After one of his daughter Kelly's pals quipped that he was a 'little grandfather,' he hit back: 'Don't f*****g say those words'