For every AC amp that the refer uses the inverter will draw about 11A DC. There is little or no charging while you drive and checking RV battery voltage with TV in high idle would be a start. It's a long run from the TV battery to the RV battery and the wiring may be inadequate for the job.
For night time you either have to much load or inadequate batteries for a res refer.
Example: With a 22A DC draw you would use 22A*5hr = 110Ah or almost 50% discharge. Adjust this for the actual draw and refer duty cycle.
Plus even when the driving charging works as it should you may not get 100% charged batteries but closer to 90% charged.
A lot of guessing in the above numbers. Do a energy audit and find out what your power usage is including refer, lights, etc.
It would not surprise me to find out that your usage exceeds the battery capability but that is speculation on my part. Solutions might include more batteries, better charging, gen or solar.