The foursome was seen enjoying some time together on the deck of his boat before Michelle and Barack departed