One day.
We drove up on Friday...did a bit of sight seeing on our own, set up the tours through the hotel for:
Saturday...US side by guided tour in the a.m., ate lunch, took a nap, guided tour of Canadian side by evening (pre-passport days when a license would do)
...chilled a bit Sunday morning before the drive home.
The only thing we didn't do that trip was the cable-car or the whirl pool jetboat. I have been back since....rode the jet boat. Not interested enough in the cable car to justify getting a passport.
OK, maybe a long weekend would be best. Bit more if you are a trinket, junk store, tourist trap kinda person.