Ms Swift and British actor Tom Hiddleston also had a brief romance shortly after she broke up with Scottish DJ Calvin Harris