Decent mid-range restaurant by Delta Chelsea
Duke of Somerset is on Bay just around the corner from the Chelsea. While I haven't yet been there myself, I've always found the food other places in the Duke chain to be reliable pub grub: http://www.thedukes.ca/
There are also the usual suspects along Elm Street, including another pub (one of the Firkins, I think?), and a couple of Italian places.
Too bad Thirty-Five Elm isn't still open, it would've been perfect.
A little late, but I thought I would pipe in. I recently had an amazing experience at the Duke of Somerset, the food and service was stellar, not to mention that they are one of the few locals in Toronto with Blanche de Chambly on tap (great microbrewery from Quebec). I have to say that I had a good time at an event there last fall/winter, but noticed a difference when we went back this Feb/March - the staff seemed a little more up beat and we really had a blast! Looking forward to the summer - big west facing patio with a parking lot across the street so no interference with the evening sun!!!
I've had good experiences at Matagali (on Elm just west of Bay). A hybrid thai/Indian place. Service tends to be slow, but if you're talking, you won't notice. Pleasant although the place is half below ground level - go down about half a dozen steps.
69 Elm St, Toronto, ON M5G1H2, CA