They each had serious frowns on their faces as they walked through the chic town over the weekend