The couple raised their children in Bermuda for the first few years, and living on the island gave them a chance to shelter their kids from the spotlight