Other's opinions may vary, but I've attended the Derby twice. Once in the infield, and once in the stands. If you actually want to see the horses run, watch it on TV. There are so many people, you can't get close to the infield rail and in the grandstands, as soon as the derby horses come onto the track, everyone stands up and remains standing. Your chances of being behind a 7 foot tall, 300 pound man or a lady wearing a hat the size of a two person tent, are excellent. And you'll pay at least $60 for a general admission ticket at the gate, and that doesn't guarantee a seat, so you'll be standing for a long time. If memory serves, the actual Derby is the seventh of eight races on Derby day. And if you need to use the restroom, get in line an hour or two before you actually need one.
Finally, the last time a buddy of mine and I attended, we were early enough to get an empty spot to park the car on a side street. Some guys met us and "sold us" their promise to see that our car wasn't damaged while we were gone all day. We felt like they would be the guys doing the damage if we didn't pay, so we paid. Extortion, pure and simple.
The one problem you won't have is getting out of town after the races are over, since you'll be in a nearby RV park. The crowd arrives throughout the day, but they all leave at the same time.