for the observatory, Davis Mountains State Park is only a few miles away. They have elec, water and dumping station. Make a reservation if you can. They're pretty popular.
at Marfa, don't pass up the chance to see the Marfa Lights...about 7 miles east of town on 90. They have a nice big rest area there and you can spend the night...dry camping...if you want..
At Big Bend, the only full hookup places are at Big Bend RV in Terlingua or the rv park down at Rio Grande Village campground...but the latter is just a small parking lot and can be very crowded with big rigs. The campground next door is shady and nicer and they have water and a dump station, too I think.
If you want to splurge, have dinner or lunch at the Gage Hotel in Marathon on your way out of Big Bend. A bit pricey, but it's a nice old hotel and the food is good.