Yet in recent years, Olivier has shirked the spotlight with his new lifestyle a far cry from the glitz and glamour of Hollywood he once enjoyed (pictured in 2020)