The Canadian actor said that he'd love to have stayed in touch with the icon, but he's 'really hard to get a hold of you,' adding, 'There's certainly a mystique about Eminem'