We always remove ours when not towing. I believe it's illegal where I am in Canada thought not enforced. I actually thought it had to do with safety in a collision as it changes the way the forces are distributed since someone is no longer hitting your bumper that is designed to take the force. While I don't want people rear ending me, I also don't want to cause more injury to them if they should hit me, assuming I'm correct in the reasoning.
Edit: hard to find the info here but may be only if illegal if it sticks out more than 8ish inches.