A picture or two might be helpful here; there are quite a few different mirror designs.
I had some trouble with loose mirrors, but mine are evidently an entirely different deign. In my case, the fault seems to be wear on the shaft that the mirror head is attached to (which is held in with a sort of saddle cap arrangement, not unlike the crankshaft or connecting rod bearing caps in an engine). I believe I have solved it by interposing a shim of thin metal between the clamp caps and the shaft, though time will tell for sure; I haven't traveled far since the repair.
That's apparently not very applicable to your situation, though.