I like Bar Wellington (portland & wellington)... about 5-10 minute walk from where you'll be. The only time to avoid it is late on Thursday-Saturday nights, when it gets rowdy from people who can't get into the bars on King St. W.
Smokeless Joe's (John St., south of Richmond)... the menu is limited, but the food is very good. The beer selection is great. And the atmosphere is nice and cozy with a sort of secret-Toronto, you-found-where-the-locals-know feel.