I have a similar truck camper and set up as you (88 coachmen). I have AGM's set next to the water tank too, along with a Renogy solar panel. I left my converter connected to my batteries and so far (2 years), no ill effects.
I think it will be ok to leave your system as is, but most certainly, you will shorten the life of your agms. Installing a disconnect switch will work and it will extend the life of your batteries and you will still be able to run your camper normally. I would install the disconnect near the water tank where your batteries are, for both the black and white wires. A 15 amp 4 way switch would work just fine for a disconnect.
The only time I plug into shore power is when I need my AC or to run my fridge on electric. Hope this helps.
Does your solar charge controller have an option to charge AGMs?