Earlier this month, a source told the Daily Mail that 'Justin wasn't thrilled about the pictures getting out' of their first date