Been there, done that:
Here's the problem with the 4" tube bumpers; they're nothing more than thin sheet metal. Put your bike rack on and drive down the road and have somebody follow you and watch the flex back and forth. They can really "rock and roll". It's kinda like bending a beer can back and forth which can eventually "crack" the thin bumper. It's not the strength of the bumper, but the constant flex that can do you in.
Yes, you might get away with it forever and in fact, a buddy of mine has been doing the same thing you intend to do for about 2 years. I've seen him go down the road and rack swings in and out a lot. I've been towing for over 10 years and like you, I've never seen one fall off or seen one on the side of the road either. It's just the fact we've got a couple of very expensive bikes and don't want to take a chance. On my old trailer, a welding buddy fabricated a receiver hitch welded to the frame and used the bumper only for partial support at the end of the receiver, ie, he welded the very end of the receiver to the bottom of the bumper with L-brackets. This area eventually cracked, but nothing could go anywhere because the receiver was getting 90% of it's support from the frame.
At the end of the day, you've got to do what you feel comfortable with. If you do go your intended route, keep an eye on things and check for cracks.
BTW, I have a new Yakima fold down rack (about $350) that I'm going to incorporate into the rear end of our new trailer. This thing is HEAVY, about 45-50 lbs so I'm going to incorporate some sort of receiver attached to the frame when I get around to it.