The actor's property was originally known for being an avocado farm, but he revealed in an interview last year that droughts destroyed most of the trees as well as around 25 oaks that were more than 100 years old