Looking for Lunch close the Dept of Agriculture
Depending on where you are coming from, pick up something (ex. Teaism) and have a picnic on Mall.
You might be able to hit the Warf on Maine St. Search board for feedback.
Maybe not casual enough but the Manadrian has Sou'Wester
The Atrium might be too long a walk for your time constraint.
2009 R St NW, Washington, DC 20009
The Garden Cafe at the National Gallery of Art has a French lunch buffet with a menu devised by Michel Richard, a celebrated chef. It is impressive and quick. Allhough you can see the NGA from the Dept. of Ag, it is quite a distance to walk, so you may have to get in a cab to maximize your time. This is an area of govenment buildings, museums, and national park land, so devoid of eating options.
That area is pretty much a food wasteland. The only place I can think of that rises above the not-offensive level and fits your time constraints is the Pavillion Cafe, located next door to the National Gallery Sculpture Garden. The cafeteria in the Museum of the American Indian is somewhat further away so I don't think you'd be able to deal with the lines in this popular place, eat, and get back within your time limits. Ditto for Cafe Provencal in the west building of the National Gallery. Can you tweak your logistics? The food in the Cafe Provencal is particularly good with local super star chef Michel Richard overseeing the menu so it might be worth carving out a bit of extra time for your lunch date.