I remove mine because I had one stolen at a campground once, I also don't like the way they look. Mine is a non WDH so it isn't too heavy to deal with.
If someone rear ends me without the hitch in their insurance will buy me a new bumper. If someone rear ends me with the hitch in I will worry about the stress put on the hitch/frame and the unseen damage that may occur.
But to each is own, if you want to leave your hitch in thats your choice, after banging my shin on several over the years I just pay attention to where I'm walking.