Tom briefly dated Taylor for three months back in 2016, with the star famously seen sporting an I Love TS T-shirt during one of their many loved-up outings