I have 2 that I had for 9 years and never cleaned. One finally gave out (the handle stripped) and I bought a new one. Used it for about 600 miles and then BANG it popped the ball on the hitch end. I replaced the ball and pulled the slide bar out and it had a very heavy accumulation of black residue from the friction pad stuck to it. A wire brush cleaned it up.
I have decided that I probably created my own problem by tightening down the new bar too tight creating excess friction and thus residue on the bar. The old ones were worn enough that I was accustomed to tightening them down pretty hard. My guess is that I would be wise to replace the other as well.