"You guys are the experts...am I still crazy? "
With those parameters of a short drive in my opinion you should be OK.
That said, you will be maxing out your Durango, but that doesn't mean it's going to self destruct the moment you hook up the trailer.
It does mean that you should be very conservative on how you drive and how you load it up. A second "shuttle" vehicle with the family in it might be a good idea if you have lots of "stuff" to take along....bikes/toys/etc.
The fact that you realize that a bigger truck is in your future is a good thing. Any longer trips would absolutely require it.
For the interim though, if you can get your family out camping in a trailer you love then I'd say go for it....but carefully :)