Unless you are familiar and comfortable with mountain driving, I wouldn't recommend it. Once you pass Sugarlands Visitor Center @ 1,480', it's 13.3 miles constant up to Newfoundland Gap @ 5,040'. On the way down you drop to 2,800' in 8.4 miles before becoming more gradual decent for the remaining 7.4 miles to Oconaluftee Visitor Center. Then you still have to deal with 19 and it's ups/downs to Asheville.
You can figure about 1.5 hour via I40 or 3 hours via 441 because while it isn't that much further, you will be a lot slower.
I go over the mountain frequently for the last 40 years but I definitely don't go out of my way to do it. Only if I'm in GA and headed to Elkmont with a stop at Smokemount will I usually take 441.
Since you will be in Gatlinburg to start with, make sure you take a day to drive up to Clingmans Dome in the car so you can stop at the overlooks on the way up and down. Then you can decide yourself.