Travisstood out in an eye-catching red and white hat as he cheered on his alma mater with Swift and his big brother