Please don't go to any of the places IWantFood suggested - they are pedestrian and touristy. For the best lunch in town, go to the Ramp. Best burger, fried clams, lobster roll and mussels with a million dollar view!
I can say the same about what you suggested, majik. And I never found the ones I suggested to be so. And in tourist season, every restaurant becomes "touristy".