The director admitted he could never see the cast reunite in the future due to the controversy surrounding the book's author JK Rowling