During Wednesday's episode of their family podcast, the reality TV personality, 52, addressed the 'crazy' accusations