'No one talked about any work I was doing,' he claimed. 'They were just, like, Oh, that's the f*** stick. And that hurt so much'