replacement batteries

You need a fairly good quality digital volt meter. There is many for the $20 range. This one Walmart or this one Harbor Freight.
Check voltage level at the batteries, first with shore power disconnected then plug in shore power cord and read voltage at battery.

After the battery cooled down, I checked the water level, and it was pretty low. How can I check how many volts the converter is putting out?

I was looking at automotive/RV battery tests on Consumer Reports just yesterday. Also read a number of threads on the Ford Truck Enthusiasts Forum. Once you figure out whether or not you have a decent converter/charger in your rig, here's what I learned from my research:

The quality of the AGM batteries (such as Optima) has declined over the last few years. Evidently, they caught on (I even put one in a small trawler I had about 8 years ago - and another in a Tahoe) and the industry has had trouble keeping up with the volume. Thus, a decrease in quality of the manufactured product with a lot of premature failures.

In the wet (lead/acid) cell batteries, Interstate comes out on top in terms of longevity and reliability - including sustained CCA and reserve capacity. Funny this is just what my late father said ever since he owned a Texaco service station in the '60s.

As stated, it is critical that you figure out your inverter/charger before replacing the battery. It sounds like your battery got "cooked" and that's not a battery problem.

Rob

Biggest issue is the converter or inverter/charger how it charges the batteries. Too high of charge will boil the batteries causing sulfur smell.
Selection of batteries depends on your type of camping. One week once a year does not justify the following, switching to 6 volt true deep cycle like the golf cart batteries. Can get these for around $80 at Costco.
Minor dry camping you could get by with standard group 24 or 26 but need to be deep cycle.
My guess your converter is over charing the batteries and you are not keeping them full of water.
What is make and model of your converter?

Staying plugged in depends entirely on what converter you have. If it's the type that sits at 13.6VDC all the time then you don't want to leave it that way for ever.

A true deep cycle is best.