Coupler gouging hitch ball

I have been noticing over the years that my hitch ball is being gouged. I always assumed it was from contaminated grease, rocks being stuck in the coupler, or a series of small animals making homes in...