Comparing similar configurations based of the 2016 Ford Trailer Towing guide.
2WD crew cab
F150 with the 3.5 Ecoboost and max trailer/payload packages & 6.5 ft bed:
GCWR 17,000, 2050 max payload, 11,900 conventional towing, 13.7 Dia brake rotors
F250 with the 6.2 gas and short bed:
GCWR 22,200, 2812 max payload, 12,500 conventional towing, 14.29" dia brake rotors
Since the OP stated that he doesn't tow that often and at the most, 400 miles from home. The F-150 isn't a slouch and would be more than adequate for his limited towing needs.