'Taylor has been so concerned with her mother's health struggles over the years that her focus was really only on her. Her dad was often always in great shape,' our source said. Andrea Swift (pictured) has weathered both brain and breast cancer