When I was there at the Mardi Gras, I stayed at the St-Bernard State Park, very quiet. I took the scooter and parked at the ferry parking that goes near the French Quarter. One time I drove over the bridge and went all over downtown.
Big crowd, big party, big time music all over, parade, it was all worth the trip, I will go back for sure, maybe this year.
NO is not going worst, it is going for the better. I would not cancel, just take more precaution if you feel at risk like staying at the campground and taking a cab to go to the party area.