Have you compensated for the amount of traffic you will probably hit when up here? That is the beginning of peak tourist season, especially the week leading up to the fourth. I would allow for a little more of a breather between stops. We don't talk about distance in terms of miles but in terms of time. You should also seriously consider reservations as many areas book up quickly that time of year. I would also consider just staying in the US and save Canada for another day to give it the time it deserves. 7/1 is Canada Day and is a big holiday there.
You can also whale watch out of Boston, off the north shore of MA and out of Bar Harbor.