Casual-ish, trendy, good food?
There are very few places in Toronto where I wouldn't feel comfortable in a decent pair of jeans and a clean shirt. What's your version of "not dressed up" and where have you been that you've felt you had to get dressed up? I would say outside of the fine dining options (Canoe, Splendido, Scaramouche, etc.), you'd have to get pretty ratty to feel out of place.
88 Harbord Street, Toronto, ON M5S 1G5, CA