Brooklyn's comment comes amid an ongoing family feud which escalated when him and his wife Nicola, 30, snubbed David's 50th birthday celebrations last month