Olivia and Caspar were seen on the cosy outing together just weeks after their romance emerged following a string of secret dates