During their stroll through the bustling streets of the city, the pair stayed close to each other's side and appeared to be attached at the hip while holding hands