I don't know what you mean by casual? I think Fleur de Sel is pretty high end and stuffy (no offense RGR). For cheap and cauel I would recommend for Thai food Spice and Cafetasia, L'Express for brunch (although it can get crowded).
roze, I wouldn't deny that Fleur de Sel is upscale. However, the $25 lunch prix-fixe is a very good deal. As far as the atmosphere goes, I would describe it as civilized as opposed to stuffy. And, as I said, no need to dress up.
If you like sandwiches, you might want to try: 1. Le Pain Quotiden (the branch in ABC carpet & home) 2. The Adore (on 13th st and university place) Both casual, relatively quiet, with good coffee and tea (for the latter) and good for lingering.