Progressive is absolutely right in this case, I think. Certainly it sounds as though the converter is working exactly as designed. It has no way of telling what the voltage at the battery is, since it doesn't have any sense lines, just what the voltage at its outputs is.
Your expected voltage drop should be computed (for comparison with actual) based on the actual current, or 20A, which would be a bit over 2%. That would work out to about a quarter volt. You're seeing four times that, so you have a poor connection or something similar somewhere.
If you could shorten the leads between the converter and the battery, perhaps by relocating the converter, that would be a big help. There's no real need for the converter to be wired into the distribution panel; it could be connected directly to the battery (with appropriate fusing) just as effectively. (Of course, if you connect it on the other side of the disconnect switch, it will alter what happens when it's set to disconnect. You would not be able to operate 12V things of the converter, as you presumably can now, but you would be able to charge the battery with the converter.)
If you have a meter with a sufficiently low enough DC voltage range, you can check along the lines to see where the voltage is dropping. Measuring between the battery negative and the converter negative terminals will show if it's in the ground path, and between the positives if it's in the positive path. Checking across the wire at various intermediate points might track down where it's happening.
