I believe the biggest concern would be your fuel mileage when not towing. All-wheel drive gets a bit worse mileage than does 4-wheel drive since you are turning both axles most of the time with all-wheel drive. Other than that, shouldn't be much difference.