Like Swift's boyfriend Travis Kelce, Scott also played football and has been seen attending NFL games with his daughter