I'm sorry to hear that the OHV community didn't behave better and got their privileges yanked. So many places where we ride have signs up warning us to behave or risk having the town shut down to OHV traffic. I always take these warnings seriously and keep our speeds down and remind our group to behave.
I've never ridden out of Silverton before. I've taken the narrow gauge train from Durango umpteen times since I was a kid. It's not as extreme as riding off road, but it's still a gorgeous trip. I only took my own family there once when our kids were all young. Our oldest (now a doctor) had her nose in a book the whole ride. It was her 8th birthday and she finished book four on that train ride. I think she enjoyed the day as much as the rest of us.
Losing the OHV revenue probably won't kill the town. Hopefully it's a lesson to the rest of us to watch our behavior when we're guests in similar destinations.