Ondine, off Exit 6 on 84 (a Danbury exit but restaurant is on the Danbury/New Fairfield border) is a nice French fine dining restaurant. Service is gracious without being stuffy, the food is reliably good, and you certainly can linger there and enjoy the company of your friends. They have both a pre-fixe menu and an a la carte menu. If I remember correctly from our visit last month, the pre-fixe is $58.
im not sure about fine dining, but we just had a great lunch at old heidelberg. very tasty german food, great beers on tap, and a nice area outside if the weather is nice. Oktoberfest is right around the corner and they will have live music outside.
Ridgefield is one town over and there is a very nice french restaurant there whose name escapes me. The Schoolhouse Restaurant in Wilton is excellent, but maybe a tad far for some.