I hope we are talking about the same latches. I find I have to disassemble mine every couple of years, clean and regrease the ball bearing for smooth operation. The stops are solid leading me to believe your entire lock may be rotating. I would check to see that the outer bezel isn't moving.
My trailer came with combination locks on some of the hatch doors and I would advise against them. I have to get the readers out to see the numbers and my eyes are pretty good. Plus, they have a large serrated handle perfect for Channellocks.