Rifle Gap is a long ways from Ouray, as is Air Force Academy and Pikes Peak. We spent four weeks in the Durango-Silverton-Ouray-Telluride area this summer. There is a lot to see and do to keep you busy for months.
As noted, many people enjoy Ridgway SP for the Ouray area. KOA is also mentioned often. Yankee Boy Basin is a great drive if you have a toad or rent a jeep. Flowers in July and August are amazing. If you want an interesting hike, the Bear Creek Trail just south of Ouray (just through the tunnel) is a National Scenic Trail. Pretty interesting after the first half mile. Not for those for whom exposure is an issue. While you're in the Ouray area, a day trip to Black Canyon of the Gunnison NP is definitely worthwhile. A great NPS CG up there if you can camp with only electricity.
Like 2gypsies, we like Haviland Lake NFS CG north of Durango. You can travel to Silverton from either Ouray or Durango. Do both, as the scenery is unique along both roads. We often spend five to seven days boondocking in the Silverton area in August. Amazing wildflowers, and Silverton changes its personality once the trains leave in the afternoon. (Do take the Durango - Silverton NGRR if you have the chance. One of the few "touristy" things we've done several times.)
As noted, Mesa Verde NP isn't that far from Durango. Plan on camping at the CG up there (a few FHU sites can be reserved). Being up on the mesa, it's definitely cooler than those CG's around Mancos, Durango, etc. Closer to Cortez, we also stay at McPhee Reservoir NFS CO. IIRC, they have some sites with FHU. Near McPhee is the Anasazi Heritage Center. Worth some time. They can also direct you to other, less visited ruins in the area. Mesa Verde is impressive, but you can get a better feel for the societal evolution in the 4-corners area by visiting some of these other places. Hovenweep just over the border in Utah is worth a visit if you have the time.
One other area to visit in the area is around Telluride. The town itself doesn't do much for us, but the nearby scenery is spectacular. Matterhorn NFS CG south of town has a few FHU sites.
For the Durango-Silverton-Telluride area, renting a jeep is really worth for a day or two or more. There's so much of the backcountry one can reach via jeep. The wildflowers with the colorful rocks make for amazing sights in July and August.