'Justin's been a hard person to deal with recently because of what he's going through,' a source dished to Us Weekly earlier this week. 'It's been really hard on her'