The two actors and their families share a close friendship, with Damon and Barroso regularly visiting the Hemsworths in Byron Bay, a popular tourist town on the Australian east coast