We live 4 miles off the main highway on a country maintained road not a rustic path...
You can do dirt roads but...
Be aware even if the road is graveled in dry conditions there can be extreme dust which doesn't go down well with air filters. Also be sure to keep all vents and windows closed. We hooked on the car and towed it out...never again, it filled up the engine area with dust and coated the car so well we couldn't tell what color it was. In uncertain conditions I'd not advise towing on a dirt road!
We like to head out on a day after a very light rain so the dust is settled a bit more. But too much rain and mud will be a condition a heavy RV will not like too well...had a semi go axle deep on our road and had to be pulled out by a CAT 4 miles...it weren't pretty, left ruts so deep even the ATV wouldn't go over them!
Snow, well a little snow on top of frozen ground is ok, too much on soft ground, not so good.
In winter we always tried to leave when the ground was frozen so any moisture would not create ruts.
And with gravel...the rocks can chip paint.
Best advice...we drive our most dirt, just a smattering of gravel, road very very very slowly (almost at an idle) to keep down the dust and reduce rock damage. Therefore it will take a long time to travel just a few miles.
Some country roads will have trees overhead that are not trimmed to allow safe passage so...sometimes we may to weave down the road to avoid low limbs and holes, ruts or dips in the road.
If we end up in really questionable terrain we unhook the car and drive ahead to check out the situation &/or follow with the MH both drivers chatting on the CB as to what to expect.
It works for us and since we have no choice-- we just handle it. Many folks we talk to won't even consider a dirt road. It's your choice.