You mention that the house battery compartment is smaller than the one for the starting battery. This may mean that your rig was designed for a single 12 volt house battery and the previous owner mistakenly installed a red cable to go to ground instead of a proper black cable. On our rig with two 6 volt golf cart type house batteries connected in series for 12 volts. There is a short red cable connecting the negative of one 6 volt battery to the positive terminal of the second 6 volt battery and a black cable connected to the negative terminal of the second battery and going to the frame for the ground connection. Make sure that you understand which cable is connected to the frame. If you connect new battery(s) with wrong polarity, serious damage to your RV or fire may occur.
Checking for proper operation of the converter/charger before installing battery is a good idea as a bad converter could damage a new battery. If you are unsure of test procedures, etc., call a mobile RV service or take rig into an RV service.