Those numbers sound pretty close to me.
A 100 watt probably isn't enough. That's only 7.7 amp in full sun, no clouds, no shade, clean solar panel surface, at the right angle, only at certain times per day. That will not likely do much other than add a couple days before your batteries go dead. I'm not sure how bad off your AGM batteries are now that they've been totally exhausted, I know that really shortens the life of a flooded cell battery set...but you might need to replace them.
500 Watts of solar panel would be better. 'Course that's a bit expensive. But that wattage would probably give you the kind of service you're looking for.
One thing you can do is only supply power to the refer by removing the positive leads from both battery sets while you're gone. In that case, you'd need to run a single wire up to the refer (16 ga) from the strongest battery set. You could also add a 2 X 1 battery isolator connected so both sets of batteries are supplying the refer (and ONLY the refer). Remove the +12 volt wire at the refer's 12 volt connector and substitute yours. Of course you could add a switch to toggle between the 2 sources.
So what would happen is that before you leave the lake is disconnect the battery positive cables, add your own going to the small battery isolator, run the 16 ga wire up to the refer (I'd just run it out the bottom of the battery compartment and up the wall of the RV to the outside refer vent). Connect it in place of the RVs 12 volt wire, and off you go.
And of course, do what you can to reduce the work the refer has to do. Park the RV in shade, or at least have the refer side of the RV in shade. Remember that the 12 volt will be running the relatively high current propane solenoid and circuitry, so the less often cooling is called for the better. So to reduce that run time, make sure the refer and freezer are free of frost, maybe add a battery operated fan to the inside of the refer to help cooling, don't over fill the refer or freezer, and make sure the outside vent is clear and open.
Some of those parasitic (vampire) loads are the CO & Propane detectors, the dash radio and other dash devices, the step circuitry, the transmission and engine memories in their respective ECMs, and the big one for a '04 I think would be a permanently wired inverter. Even if you eliminate those things, it always seems there's still going to be something else. And flipping off the saleman's switch doesn't always stop the vampire loads.
Have fun at the lake!
