Harry's previous romances have been doomed, including unions with the likes of Taylor Swift (the couple pictured in New York in 2012)