I think your battery connections are basically okay but your inverter is powering something you don't realize it's powering, like maybe the converter (if you have a separate one) or the fridge element or even the water heater element.
The battery connections should ideally be taken from diagonally opposite corners for the same thing--so 2 and 4 to the inverter, 1 and 3 to everything else, or vice-versa, or both on 1 & 3 or both on 2 & 4. It's not entirely clear to me that you've figured out if that's currently the case or not. Failing to do so would leave the batteries slightly imbalanced and eventually tend to lead to uneven wear and various difficulties associated with that, but would not cause a big loss of capacity right off.