if you're in town during the market hours, you MUST try FAIDLEY's in the Lexington Market for lump crabcakes. Order the lump crabcakes, and you won't disappointed. I agree with the other reply, the IH is full of tasteless chain restaurants that deliver a view (maybe) but not much in the way of good eats. If forced to make a decision, I would reluctantly recommend Rusty Scupper or Bo Brooks (Canton).
Federal Hill has a number of very good restaurants -- away from the tourist traps.
The Inner Harbor is chain central. Steer clear of Phillips. On the far east side of the Inner Harbor is the Black Olive. Haven't been in a long while but search this board for current comments. Also Charleston has some pretty decent seafood dishes on their build-your-own-tasting menu.