The singer told Capital Breakfasts Chris Stark that despite the pair exchanging numbers on the night, Justin failed to respond to his message the morning after (pictured together in 2020)