Sam and Eleanor enjoyed the event alongside Sam's mother, Lady Sarah, Princess Margaret's only daughter. She sat behind the king to watch the games