Orlando opted for a black printed shirt and black cap for the lunch, but later went shirtless to show off his sculpted physique