Indeed, the two years that have elapsed since Swift and Kelce, both 35, first got together haven't always been a bed of roses