7 West serves terrible coffee and at best, mediocre desserts.
Agree with TorontoJo that DT Bistro would be a good bet.
Although there isn't much in terms of atmosphere, you could get dessert and a coffee at the Golden Wheat, Riviera Bakery or Nova Era on College Street, or at Athens Pastries on the Danforth. Quiet enough to talk, and open in the evenings.
Novecento on St. Clair W might be another option. If you like gelato, La Paloma on St Clair W, Hollywood on Bayview and Il Gelatiere Artigianale on Mt Pleasant serve gelato and coffee, and they have some tables available.
I've enjoyed the desserts I've ordered at Frank in the AGO. I haven't been there only for dessert and coffee, but I don't think it would be a problem ordering just dessert and coffee.
If anyone is looking for dessert and coffee during the gallery hours (closes around 5 most days, but open until 8 pm on Wed), the coffee shops in the AGO (one in the lower level below Frank http://www.ago.net/cafe, and an espresso bar location on the top floor, http://www.ago.net/espresso-bar ) serve nice pastries and decent coffee.
AGO members get a 10 % discount.
Another place I like for coffee, which is open in the evening is Balzac's in the Distillery. They have quite a few types of squares and cookies available.
Sweet Escape Patisserie in the Distillery also serves coffee, desserts, ice cream, and has a few tables. http://www.thesweetescapedistillery.com/
647 Mt Pleasant Rd, Toronto, ON M4S, CA
7 Charles St W, Toronto, ON M4Y1R4, CA
509 Danforth Ave, Toronto, ON M4K, CA
FRANK @ The Art Gallery of Ontario
317 Dundas Street West, Toronto, ON M5T I4G, CA
1 Bass Pro Mills Dr, Thornhill, ON L4K5W4, CA