The grazing fee is set according to a formula written into law. The current rate of $1.87 per AUM is the floor price. It cannot go lower. Neither political party has shown any interest in amending the formula. Outside of a small number of griping malcontents it is a non-issue in DC. The two major federal land agencies -- Forest Service and BLM -- also have legally mandated missions to promote rural economic development. Maintaining family ranches is part of that mandate. Without the public land grazing most family ranches would fail economically, with a ripple effect through the entire region. Most likely the failed ranch would be sold to developers, especially if near large cities. So in a way the grazing fee system is helping keep open land undeveloped and slowing urban sprawl. The worst managed ranch is still better wildlife habitat than the best planned subdivision.
According to the USFS figures the cost of backpacking in a Wilderness is a bit over $37 per person per day (that figure is a couple years old, probably higher now). This is obtained from dividing the total FS Wilderness management budget by the number of wilderness visitor days. I am not aware of any national forest in the country that charges $37 per person per day for wilderness permits. Most have no charge.
A study conducted by Utah State University about 20 years ago showed that while public land recreation in total was a net economic gain for rural communities, the profit was from developed and motorized recreation. Non motorized wilderness recreation was a net loss. RVers are subsidizing backpackers.