Brody (pictured 2013 with Kim and Caitlyn) has long had a rocky relationship with the Kardashian family despite appearing on some episodes of their reality show