DW and I love to entertain a big family crowd in our TT a few times a year. One or two nights of chaos is fun for us, but our adult kids help a lot with the cooking, cleaning and firewood hauling. And we spend most of our time outside. Some adult kids bring their own tents and I have a spare tent for them if needed.
But we also like to have quiet trips just the two of us and the dog. Much more restful.
I think it is perfectly fine to tell your brother that you really want to do this trip with just your family. And tell him you hope he understands that you need the opportunity to spend some vacation time with just your spouse and children. You could also mention that you would like to schedule another big "extended family chaos weekend" once a year when you could all get together. And maybe discuss when and where that might happen.
Others have stated this here already: just let him know you need time to bond with your own family and this trip is one you had planned to do this on. Ask him to please understand the importance of your family's time together.
Ed