'I saw one of her exes somewhere at a game somewhere and I shook his hand,' remarked the Family Feud host: 'Boy, I cant even touch his a** no more'