Wearing a pair of white-washed jeans and a black, long-sleeved shirt, the multi-millionaire sat down for a four-hour lunch with his friends