My wife and I live in the area and tend to frequent Bar One at Queen and Shaw. I belive they serve brunch/breakfast as early at 9am on Sat and Sun.
Swan is also quite popular for brunch and is open both days.
Mildred Pierce is only open on Sundays and not until 10 or 11 I think.