But Patrick, who is now married to stunning Aussie model Pia Whitesell, didn't have a care in the world on Thursday as he enjoyed lunch in Los Angeles