I'll try to elaborate with a little more info than my first post had.
The entire hitch is friction based. The heavier the tongue weight of the trailer, the more friction there is to resist sway. Most of the friction to resist sway comes from the connection of the bars to the hitch head.
In your case, the bracket that holds the L-bracket is bolted around the A-frame. When bolted together, the friction between the bracket and the A-frame is supposed to prevent the bracket from moving. The less contact between the bracket and the A-frame, the less friction there is. The bar rests on the L-bracket and when you turn there is friction between the bar and the bracket. Now making an assumption that your hitch is new or newly installed, there is a break in period. During the break in period, the steel starts to mate and parts start to wear and groove together. This break in period increases the friction between those moving parts. This is why you have a left and a right bar that you do not want to switch.
My guess is that during the break in period, the friction generated between the bar and the L-bracket has increased to a point where the resulting friction force exceeds the friction located between the A-frame and the bracket that hold the L-bracket. I believe that the fool proof way of overcoming this problem is to through bolt the bracket to the A-frame or have it welded on except that welding is more permanent. Another option would be to grease the contact point between the L-bracket and the bar (as noted above, most of the friction to prevent sway comes from the hitch head to bar connection). You could also lighten the tongue weight, but that may have negative affects on the stability of your ride.