50 amp electrical issue

When voltage goes down, it's usually due to resistance in the feeder wiring to the pedestal. If so, you'd expect voltage to drop in both 120 volt legs (with same amount of load). What is the line to line voltage measuring? If only one leg is dropping, perhaps you have a poor connection at the shore power to pedestal, maybe dirty plug blades, receptacle in poor shape or bad connection inside the pedestal? But that wouldn't explain voltage rise on one leg. I would suspect a bad neutral connection somewhere. It would be interesting to see what happens if you swapped the hot-hot connections in your RV's panel - if a neutral issue, I would expect a similar rise/drop on the 120 volt legs. The pedestal (by code) should have a 20 amp recept. Maybe check that. If on a loop feeder with other 30 ampers on it, perhaps check voltage there also?

Don't know if this is of help or not.

Getting down to 104 volts is not good.