It could be your adapter plate or the adjustment on the front stop. My adapter plate is attached by four set screws and it is always been a little loose. It does clunk a little when I start and stop, but I can see and hear the difference between that noise and the front adjustment being too loose.
When the front adjustment gets loose the noise is much louder and I can feel it much more.
The front adjustment nut must be tight enough that it is difficult to rotate the hitch head.
If yours is one of the newer ones it will not have the nylon bushing, only a flat steel against the steel front of the hitch head.