The Friday night feast saw guests including Swift, who was seen sweeping in in a convoy of three SUVs descend on the 10-bedroom white-washed property