Appearing on Josh Horowitz' Happy Sad Confused podcast with his The Roses co-star Olivia Colman , the star was asked if there's anything about him that annoys his loved ones