I really like the George Washington Inn, in Washington Depot. The town is VERY small so you can have a VERY short walk afterward.
If you want a longer walk, then Hopkins Inn at Lake Waramaug is a good bet, the lake is lovely to stroll around, though at places you have to share the road with cars. I'm not a huge fan of the type of cuisine at Hopkins, but it often gets favorable reviews.
For the real birthday experience, though, I'd recommend Boulders Inn, also on Lake Waramaug. Pretty fancy, but not stuffy, and the food can be really special. Not sure what lunch is like, in honesty, but dinner was fantastic.