I went from a Excursion to a 5 seat crew cab Because we decided to get a 5'er. The5'er has been great but we still miss the passenger space of the Excursion.
Nevertheless my family fits fine in the crew cab, it is a bit snug but we manage even on long trips.
There is an issue when friends want to come, once or twice a year we may take a second car to bring extra people. However as long as my immediate family fits I have no intention of changing vehicles to accommodate others. Since your family of 5 fit in your 5 seat crew cab, I'm not sure why you need to get the X or Burb?
Now if you just want a change or would like more passenger space I understand but that is a want and there is no need to seek approval from the peanut gallery!
In the end I went from a TT to a 5'er simply because I wanted to not because there was a rational reason to make the switch.
If you are willing to give up the golf cart go for it, The X has all kinds of passenger space. Keep in mind the X makes a great TV in terms of enclosed passenger space but it lacks the versatility of a pickup bed. Especially for items you do not want inside the truck like: gas cans,grills, camping mats, wet stuff etc.