Naio,
You are just making wild guesses. Get an ac/dc clamp meter and check out what's really going on. In all likelihood it is a low voltage/peukert event, and not so much about how well some things play with that inverter.
Every time you remove a significant load, Vbatt has a chance to rebound, and even catch up a little. You are discharging faster than you are re-charging. You need to store power in those agm's, and draw from that. So either you need two inverters, or you need to keep switching the single inverter from powering the charger to powering the 120v items.