The BAFTA nominee explained he turned down the chance to be associated with the Harry Potter franchise's Fantastic Beasts series because of the fame it might bring