He died this week of natural causes at home in his upstate New York hometown of Rochester, his manager informed TMZ; pictured in 2004 at the Playboy Mansion