Again, I doubt anyone is making an effort to "spy" on me, but I do see the potential issue of this technology being used - both with spying/stalking and with the noise/visual pollution. That's why I think it's good to have these discussions and realize that there will need to be laws.
One thing that hasn't been mentioned here is the potential of pedophiles to use drones to take video of little kids without being noticed. A drone flying over a playground seems innocent enough. Until you find out who is flying the drone.
I'm not saying this is a major problem now, but I do think we need to put some effort into setting boundaries BEFORE it becomes a problem.