A fun and often overlooked place is Union St. Public House. They have a raw bar on the first floor, opposite side of the bar. I have always found the service to be fine, and like the atmosphere better than Fish Market.
If you are looking for chef's who ONLY serve fresh fish, never Sysco products...well then you have to look to the Armstrongs of Eve, Majestic and Eamonn's.
And I agree that the Chart House, albeit a chain, is one of thee nicest Chart Houses I've seen. The view is fantastic and it WILL satisfy your seafood jones. The crab soup can't be beat too.
I have always had a good experience at the Charthouse. I believe its a chain, but the food has always been to my liking. It has a nice ambiance, also - if that's important. I went to the Wharf once and it was OK - not amazing. We had a great waiter, so that always makes the experience better.
The Fish Market is a fun place - not amazing food, by any means, but very fun.