The two actors and their families share a close friendship, with Damon (far left) and his wife Luciana Barroso (second from left) regularly visiting Chris (far right) and Elsa (second from right) in Byron Bay, where they live