I live in Bar Harbor and I agree, you could spend easily a whole week in Bar Harbor (if you enjoy hiking, etc.). What I like about here is you can go on small hikes (great for kids). The Island Explorer bus stops at all the campgrounds and the best way to see that schedule is the Acadia app (iPhone and Android) from Chimani (it's free too). 
So give Bar Harbor a week. Trust me between getting on a boat (whale watch, Diver Ed cruise, etc.), hiking, eating good food, and maybe doing a few historic/educational things, you'll be busy for a week easily.