Affleck shares Violet, 19, Fin, 16, and Samuel, 13, with Garner - whom he was married to from 2005 until 2018; the former couple seen above in 2013 in West Hollywood