If you are leaving Anaheim on a Sunday morning, and you want the scenic 101 rather than the dull 5 (good choice!), I would leave as early as possible -- definitely by 9 am, and 7 am is better. Take the 5 to the 134 to the 101. Traffic is generally fine till 10 am. on a weekend. On a weekday, the traffic gets bad around 6 am, and sometimes earlier. I am not kidding.
On the 101 north, the Conejo Grade from Thousand Oaks to Camarillo is pretty steep, but no big deal if you use a low gear.
Taking the 5 to the 126 and then over to Ventura is considerably longer, I think.
If the weather is very warm, you may hit some traffic north of Ventura through Santa Barbara. But usually the spring is cool and cloudy, so beach traffic should not be a problem.
If we have had any rain at all, the countryside from Santa Barbara to Paso Robles is just great at that time of year -- green rolling hills, vineyards, some ocean views here and there.