We've had very nice lunches and dinners at the Thornton River Grille in Sperryville, VA. This is about 5 to 10 minutes from the Inn at Little Washington.
The weblink is http://www.thorntonrivergrille.com/
The website provides directions/location and menu info. I think its right at the intersection of Highways 522 and 211 right in the small crossroads of Sperryville.
It's casual, but very nice food.
Hope you can give it a try.