Brooklyn also missed a boys fishing trip to the Scottish Highlands with David and his younger brothers Romeo and Cruz