I tow with a Pathfinder, which is mechanically identical to your frontier, and I have to agree with kmbelt. I wouldn't tow anything bigger than 5000lb gross with the Pathfinder/Frontier platform.
Without knowing what WD/anti-sway hitch you have, it's hard to say if the hitch settings had anything to do with your sway, but I don't have any problems with my 5000lb, 24-foot trailer with the Pathfinder and Equal-I-zer hitch combo. I use the standard P tires as well, regular inflation. Some Frontier/Pathfinder owners who tow at the limit have recommended air bags in back to stiffen up the suspension, so you may give that a try.
Also how fast were you going? The faster you tow, the more likely to sway, especially if you don't have enough TV. I usually tow at 60-65mph on the interstate with my combo.