The father-of-four was a guest on the latest episode of daughter Gia Giudice's (second from left) podcast Casual Chaos, where he opened up about his time on the Bravo reality TV show