On Thursday, the HGTV star, 43, looked glum as he arrived at a Toronto airport with his wife Heather, 37