Are you using a drop receiver or extension. We had a similar situation when we first started towing 4 down. Our problem was a sloppy fit with the drop receiver. A little side to side play at the MH receiver and more where the Blue Ox draw bar went in. It translated to about 2inches of side to side play at the tow bar, made for a very scary drive. I removed the drop receiver extension after 3 miles of driving and have not had any problems since. I now tow with about a 4 inch drop from the MH to car, no problems.
Beverley and Ken