I would say, yes, it is crazy to pull an RV through downtown Seattle on I-5. It almost does not matter what time of day unless it is in the dead of night on a Sunday. I-405 aound Seattle is much better, and even then you can really get in a jam between Renton and Bellevue, might head up SR-18 from Federal Way to I-90 at Snoqualmie and then west on I-90 to I-405. That stretch of I-90 is actually the most underutilized until you hit Issaquah, and then it is a short 5 miles to I-405. Just the way I see it.
I second the recommendation for Fidalgo Bay RV Park in Anacortes as your home base for SJ Island explorations. Much more convenient than Burlington, and a pretty darn nice park in its own right.