afternoon tea in Toronto near the Fairmont?
The Royal York does a very nice afternoon tea themselves. Normally, it's a bit on the pricey side, but to mark their 100th year, they're doing a Centennial Tea all summer that is only $19.07, which includes a tour of their rooftop garden where many of the herbs used by the hotel kitchens are grown.
The Royal York itself is a nice place for afternoon tea. This summer they've got a special one on that includes a tour of their herb garden on the roof (though I'm sure that's optional if the ladies prefer just tea in the restaurant). Here is an article featuring it:
Another option is the King Edward Hotel, only a few blocks from the Royal York.