Just a thought, but how about some authentic dim sum? The Bright Pearl on Spadina comes complete with cart ladies. Sunday's are kind of fun, because everyone comes out. It's definitely an predominantly asian crowd, but there are many other ethnicites represented as well. They also have a location down in Harbourfront with good views of the Lake.
Hot House Cafe at the corner of Front and Church has a big brunch buffet. Not my cup of tea (I just don't like buffets in general), but certainly several large steps up from Town & Country! IIRC, they also have a live jazz trio playing during brunch.
Edited to add... here's the web site with photos of the bunch spread: