The two even walked together at the 2014 Victoria's Secret Fashion Show, though that led Taylor's most parasocial fans spread unfounded rumors that the two had had a romantic relationship, which she denied