The TIRE rolling diameter is the controlling dimension here, along with axle ratio. A smaller rolling diameter has the same effect as a larger axle ratio: that is, the wheels make more turns per mile, increasing the torque at the wheel. A larger rolling diameter and a smaller axle ratio decrease the number of turns per mile, and also decrease the torque.
It is probably possible to buy tires with the same rolling diameter regardless of whether you buy 16 inch wheels or 20 inch ones. The difference will be reflected in the aspect ratio of the tire dimensions, for example, an 85 series tire on a 16 inch wheel could have the same rolling diameter as a 20 series tire on a 20 inch wheel.
But there is one other important factor to consider: the tire's resistance to damage from hitting rocks/curbs/potholes. I believe you'll find that the 20 series tire will be much more easily damaged than the 85 series tire, due to the much shorter sidewalls, which afford less room for the sidewall to collapse into before damage occurs, and the larger air volume of the larger tire will cause less of a rise in pressure due to a given deflection of the tread.
Upshot: I'd stick to 16s and 85s or 75s for rough duty and leave the 20s for your pimpmobile.