Ye had been complaining about not seeing his four children - daughter North, 11; son Saint, 9; daughter Chicago, 7; and son Psalm, 5 - from his six-year marriage to Kim Kardashian, which ended in 2021(pictured in 2020)