I've used the same hitch pin for 16 years towing my sailboat trailer (800 lbs.), then two years ago we bought our Freedom Express. I continued to use the same hitch pin with no problems. This past summer, I noticed some rust on the pin, so it was replaced. Better safe than sorry.
Yes, the pin hole in the hitch will get hogged out. My sister-in-law had this happen. She had steel plates welded on both sides of the hitch. The holes in the plates were drilled so that the pin has a snug fit.
Just another think to keep an eye on.