'Gwyneth's mother Blythe Danner also joins them in the summer to make it a big family affair,' the source added