What if the limiting factor is the size and strength of the rear end gears used in the 3.08? Perhaps they used smaller diameter, physically less-robust gears in that application. In that case, the primary danger would be breaking those gears. This is the situation with the Toyota Tacoma; the rear end gears used with the 4 cylinder engine are much lighter than those used with the V6, thus the rear end is considered by many to be the weak link in the drive train of the 4-banger Taco for those who try to tow close to its 3500 lb rating.
If this is the situation, then yes it's worth replacing with 3.55 or whatever you decide. Probably not a terribly expensive job if you get them out of a boneyard. And you might have to get them from the boneyard; I know it is getting difficult to find some replacement parts (such as seat belt assembly) for my 2000 Merc Mountaineer.