The duke has, however, been allowed to join King Charles, Queen Camilla and other members of the family at Balmoral