Yep, I don't tolerate distractions from others, setting up or tearing down, too many things can (and have) gone wrong due to distractions. I politely tell them everything is under control, if you wish you can come back in a couple hours when I'm done.
I don't have much tolerance for know it all's, or motor mouths. If I need help I'll ask for it. Same in return, I just sit and watch the fun. There was one time I spoke up when a guy was inches away from backing his boat into his trailer. Other than that, it's very entertaining.