While Joseph doesn't have much of a relationship with his half-siblings, he is often seen hanging out with his father