L'Espalier has wonderful food at their tea, particularly the pastries, however it is exquisitely formal. If you're looking to relax, I would second Upstairs on the Square. However, if you enjoy over the top formal, L'Espalier is lovely. (It's also a good way to get an experience there without paying what a dinner would involve...)
What about the Taj or Upstairs on the Square? Not specifically on your list, but both do a delightful job. Tea at the Taj is definitely more formal, but Tea at Upstairs on the Square is a lot of fun - especially if you sit in their Zebra Room which is really cool and funky.
I second tea at Upstairs on the Square! It's a lovely place to enjoy tea. I believe it was $20 pp when I went last year and that included your choice of tea and a variety of savory and sweet items. One of the eateries in the Boston Public Library also offered a high tea. Not sure if they still do. It was rather pricey though ~$40 pp if I remember correctly.
According to their websites, it's $28pp now. http://www.upstairsonthesquare.com/me...
Our traditional three-tiered afternoon tea:
Savory Treat, Assortment of Tea Sandwiches,
UpStairs Sweets & Pot of Tea $28
Zebra Tea (with a Kick
)Traditional Three Tiered Tea and a Glass of Bubbly $35
Grand Peppermint Tea
Chocolate Square, Fudge Brownie & Mini-Peppermint Parfait
with Pot of Wild Mint $18