GMaps gives you good route choices both ways. Notice the the west route is slightly longer, but about the same time, implying slightly faster average speeds.
The west route is mostly woods, minimal towns, and a nice stretch of open ocean views (between the trees). And a pretty stretch along Lake Crescent. BUT, there's road work along that lake, with potential delays. In May we had a half hour delay, but 4 hrs is possible
https://www.nps.gov/olym/planyourvisit/lake-crescent-highway-101-rehabilitation-2017-19.htmThe east route has divided highway (to the 108 turnoff), and some of 101 (around Shelton and Sequim), and windy waterfront driving along the Hood Canal (with parks and resorts). Overall you'll see more water this way.
If you are driving straight through I think it's a toss up. But if you intend to stop, camp, or take a side trip or two, the differences are more significant.
For heading east, check the WSDOT ferry site for info on reservations for the PT-Coupville ferry. There's also bridge painting at Deception Pass, which will limit your viewing options. It shouldn't hurt traffic.