The Endeavor Talent Agency executive chairman kept his look casual as he and his pals arrived at Craig's in West Hollywood