For Saturday's outing, the Chrisley patriarch was clad in khaki trousers, a short-sleeved black sweater and black sneakers