Back in March, both Swift and Kelce whisked themselves away to the Yellowstone Club where they were seen dining with sports journalist Erin Andrews; seen in January in Kansas City