There are multiple parts to the system. One part is the satellites that orbit the earth and broadcast to the GPS receivers on the ground, on the water, in the air, etc.
Another part is the calculations that take place in GPS unit to determine a lat/long/elevation for the unit.
A third is the datum used by the GPS system (WGS84) which provides a mathematical model of the earth's surface.
Also, there are the maps which are used to display the location of the GPS unit, nearby roads, rivers, waterfalls, Walmarts, etc. etc. to the user.
Local maps in "backwoods" areas tend to be less precise, and are often older and were produced on older and local datums which don't match WGS84 (which means that a given lat/long location (say of a bridge) as measured by the GPS could be miles away from the place the bridge shows up on the non-WGS94 map.
More money and effort is spent making the maps used in the GPS in your hand or your car or RV to be accurate in busy/important areas - while out in the boondocks they may have just picked up some out of date and inaccurate map because they don't want to spend the time and money it takes to make and keep maps up to date for areas which are seldom travelled.
Note that the maps used in different GPS units may have been made by different map making companies and so may not show the same position on the MAP even though they calculate the same lat/long.
There are more factors (like the use of local datums in map making) that can affect the accuracy of your GPS - such as a large volcano eruption which changes the earth's center of gravity :).