Party in Annapolis, Md
The Chart House also does a nice Sunday brunch, it is a chain, but I think the Annapolis location is better than many.
If neither of those work you could try Rockfish, but I think the ambiance and food at Chart House and Carrol's Creek are nicer, and Rockfish isn't a buffet.
Carrol's Creek Cafe
410 Severn Ave., Annapolis, MD 21403