The White Lotus star joined the crowds to watch live music at Worthy Farm, opting for a casual olive green cap