The best casual ( non-hotel) brunch in the west end of the G.T.A. is Stoney's Bread Company in Oakville. They are just south of the Q.E.W. on Kerr street. Awesome and very affordable! I have been there on several occasions and enjoyed every meal I have ever ordered.
The Breakwater Restaurant in the Waterside Inn has a great brunch and is a wonderful place for a special occasion meal, however may not be casual enough for what you have in mind. -
15 Stavebank Rd S, Mississauga, ON L5G2T2, CA
Brunch at the Waterside is pretty good, and the prices aren't outrageous either. Their burger is just okay, but it comes with this incredibly delicious tomato jam instead of ketchup. I wonder if they sell the jam?
It's not really a casual room, or a greasy spoon atmosphere, but it's comfortable, the staff are nice, and I didn't really feel out of place in jeans and a t-shirt.