'I wanted to fight that guy two or three different times. Once I felt was a legitimate reason, and the other two I was just p**sed off, which, that's not right. I'm glad everybody talked me down,' he said