South of Burns there are a number of interesting sights. This BLM page describes some of them, including drives
http://www.blm.gov/or/districts/burns/recreation/activities.phpFor birding there's a large shallow lake, Malheur National Wildlife Refuge.
Page Springs is a nice BLM campground at the base of Steens Mtn. The loop road up the mountain may not open till Memorial Day. There is also an historic inn nearby, Frenchglen.
On the east side of Steens Mtn is Alvord dessert and salt flat (and hot springs).
Just south of the Wildlife refuge is a lava field, and an historic round barn.
Diamond Loop scenic bywayRoads will be a mix of paved and gravel.
In terms of geology, this is the north end of the Basin & Range area that dominates Nevada.