It isn't something that has worn in, they come like that new. There have been a few posts in the past that address the issue. My understanding is that the receiver is 54mm and the shank is 2". Provides for a loose fit.
I fixed the issue by welding a bead across the shank at the front and back, then grinding down the welds till the fit was snug. You'll also find you'll need side welds as well.
The hole being elongated is from using a pin with a bent shank on one end. If you use a pin with a straight shank on both sides (more expensive to make I'm sure) the hole won't elongate. Learned that lesson the hard way myself. :-(