I leave mine connected all the time but I do have a complete disconnect BLUE SEA switch (BAT1-BOTH-BAT2-ALL OFF)
I check my batteries ever so often and they very seldom gets below 12.4-5 VDC over the winter months here in Northern Neck Virginia...
Too many parasitic low drains in my trailer setup to leave the batteries connected...
I too have a small 2amp trickle charge setup as well. Here lately I have been using a small solar panel connected. It does help keep the batteries up.
My 2008 Trailer setup has the smart mode charging system in it (Progressive PD-9260C ) for my upgraded 12V battery bank. Your 2004 Model trailer most likely still has the single mode charger in it unless someone has previously upgraded it. That would be a good thing for you to upgrade and this will almost look after your batteries for you. A single voltage charger that only puts out 13.6VDC is known to boil out your battery fluids over time. If you still have the 13.6VDC single mode charger be sure to keep a close eye on your battery fluids when being left on-charge...
Roy Ken