We have travelled Top of The World several times. The first time was back in the late 70's with tents/gear in a Chevy Blazer. The road was not much more than 2 dirt tracks and was only open about 6 weeks of the year. That is the trip we remember the best!
The road has got so much better over the years. I think if you do not drive it, you will always wonder what you missed and will regret not trying it. Do it - take your time and make several stops along the way to enjoy the scenery. Thousands of people drive it every year so it is doable - give it a try.
For a real experience, camp overnight at the gravel pull out near the border.
The border closes overnight so there is no traffic and it is so quiet and peaceful. Last time we stayed overnight there were 3 other RV's and cariboo walked around our units while we watched in awe.