At one point, one of the ladies rested her head on his shoulder looking visibly upset with Ne-Yo stepping in to comfort her