First off, it COULD be the truck tires. When the trailer gets off-line from the truck at all, it pushes or pulls sideways on the back of the truck. If the truck's tires are too flexible and/or underinflated, the effect is amplified and can turn into real sway. Stiffer tires will absorb or block the movement and keep the trailer in line better. It won't prevent all types of sway but it helps with certain kinds.
Second off, towing a swaying trailer is DANGEROUS, period. A swaying trailer is literally trying to move faster than the tow vehicle, and will try to pass you by regardless of how big your truck is. You would most certainly worry if the trailer was swaying.
Third off, yes, single axle travel trailers in my experience are more prone to sway, and require heavier tongue weights by percentage.