How things change! We started out with a 17' trailer (with no bathroom) pulled by our passenger car and camped all over the western US without a problem. And yes, we did it with three kids. Now we pull a 25' Cougar with Dodge 1500 hemi for just the two of us, and we are looking into buying a 2500 to make towing easier.
My suggestion would be to look into a used trailer for your first unit. Something that is suitable for the three of you and has already depreciated. Then, once you figure out whether trailer camping is really what you want to do, look into upgrading to the rig you need down the road.
Traveling with a trailer should be fun, not a white knuckle trip wondering if your tow vehicle is going to be able to get up the next hill or if the wind is going to make you swerve all over the road.