It depends. Stick to neutral stuff like folk music, old rock classics, jazz, etc. Quit playing before quiet hours. Most people will love it. It's certainly no worse than lots of laughter and kids screaming. In fact it adds to the atmosphere of the CG.
Play too early in the morning (and early is subjective), play anything religious, play after quiet hours, and yes, you will annoy people and/or just make them angry at you.
There are people, no matter what you do, who will be annoyed by you even if you are the quietest people in the park. Thankfully those people are few and far between in CGs.