H2O was very good. We didn't get a chance to try Scales and Shells but heard good things about it - fyi - they only take cash. Had appetizers (seafood donuts) at Mooring but weren't able to make reservations for dinner since they were full - I liked their menu and really wanted to try it. We had seafood appetizers at Red Parrot one day and next day had dinner there - the food was good. Wanted to go to Fluke but they weren't open when we stopped by, they're open for dinner. I had a cold lobster roll at Brick Alley Pub and it was good but I would have preferred a hot one with butter.
The Black Pearl on Bannister's Wharf is delish!
For more casual, you can join the mob scene at Clarke Cooke House.
I've heard RAVE things about the quality of the seafood at Scales and Shells. However, very simple preparations.
I personally have liked Asterisk in the past.
Mamma Luisa has amazing Italian style food -- not sure how much seafood is on the menu -- but oh so good!
How casual or fancy? Fried food, or white linen style preparations?