Between Charlotte, Bristol and Richmond, we have camped at a racetrack for the past 16 years...never in the infield...but I don't remember any fights or too many drunks.
At Charlotte and now Richmond, we were/are on racetrack property, inside the gates, and there is security and patrols throughout the night. Bristol was across from the racetrack on private property, but it rained so much it kept most people inside their campers.
We now have a good size grassy site just inside gate 2 at Richmond, great neighbors, and within 200yds of the racetrack. Only noise we get is occasional big rig using a jake brake out on Laburnam Ave. Campground security patrols once an hour in a golf cart, and the city police come thru 3-4 times a day.