The A-list couple's date night comes days after Swift fans convinced themselves that the pop star was pregnant thanks to an optical illusion; Swift pictured on May 31