If handling ease is your goal then by all means switch to 20#. The wife and I travel quite a bit in our trailer and could easily get by with smaller tanks.
We seldom boondock and almost always heat our trailer with a ceramic heater. Your furnace will be the determining factor as it's BY FAR the biggest user of Propane. Enjoy your adventures.