Their outing to the Hamptons came four years after the former actress split from her ex-husband Olivier Sarkozy (pictured together in 2017)