Hilary Duff's, 37, past teenage romance with rocker Joel Madden, 46, is drawing criticism from fans, who've raised concerns over their age gap; They are seen in 2005