How far are you willing to walk? Personally I think the best brunch options are in the West 80s but I used to live up there, so I'm biased.
The Neptune Room (my favorite)
Good Enough to Eat
Popover Cafe and Neptune Room will probably have the shortest waits.