What model of converter do you have? I would guess that the converter is bad, and probably that you ought to simply replace it with a newer one rather than attempt to repair it as it sounds like a very outdated design.
As I understand it, this model of Onan generator needs 12VDC to generate 120V AC (the field is externally excited). It sounds as though your converter is an old design that switches between battery and converter power using a relay, and presumably has a separate, better regulated low-power output to charge the battery. It seemingly is not keeping up with the 12V loads imposed by the RV when the generator is also heavily loaded.
It's certainly possible that the problem may be elsewhere, too, perhaps a bad ground or bad battery connection or similar.
You are completely correct that you don't have an automatic transfer switch for the generator.