There is another place to try. The unfortunate name is Thai Mex. It's right in Nellysford. The Thai food is exceptional. Go for the crispy duck and the drunken noodles. Ask for sticky rice with mango for dessert. Not on the menu, but they'll often make it for you. If you don't get to the area often, go to the Downtown Mall in C'ville. It's a great setting and distinctive from other cities. For lunch we like Sips and Petit Pois. For dinner, Hamiltons.
Petit Pois Restaurant
201 E Main St, Charlottesville, VA 22902
Agree with Mojoeater on Blue Mountain Brewery. If you are in that area near Veritas I highly recommend checking out Flying Fox Vineyards. I have never eaten at Devil's Backbone, but have heard good things.
Also in the Afton/Crozet area the Greenwood Grocery is a great place to stop for a packed lunch as they have hot items, gourmet sandwiches and a very nice little market. Also in Crozet both Crozet Pizza and ThreeNotchGrill are good. Closer in Ivy Duners is very good.
Than of course all the normal Charlottesville places.
Also if you go to Whitehall see if they will let you taste their Cuvee de Champs. You normally have to ask to taste it, but I think it is close competition to Octagon at Barboursville.
If you are up towards the vineyards around Barboursville I recommend Pomme in Gordonsville for lunch or dinner. Up in that area the restaurant at Barboursville Vineyards is also good.
5794 Three Notch D Rd, Crozet, VA 22932
4372 Ivy Commons, Charlottesville, VA 22903
115 S Main St, Gordonsville, VA 22942
Another excellent casual (very casual) lunch or dinner option is the Barbeque Exchange in Gordonsville. This is by far the best barbeque within 50 miles of Charlottesville and indeed the best I've had in VA. What makes this place different is that everything on the menu including side dish options is really good.
West of town in Nelson County there are some great vineyards, including King Family, Veritas and Cardinal Point. It's beautiful out there and makes a great day trip. On 151 just off 250 and very near these vineyards is Blue Mountain Brewery (http://bluemountainbrewery.com/). Great place to stop for lunch.