Balthazar. Make a reservation. Get the waffles, french toast, bacon, and/or le panier bread basket.
I have also heard good things about Hundred Acres and Aquagrill's brunch but haven't been. Aquagrill is great for dinner, too, and you can get a zillion varieties of oysters if that interests you.
I'm not a 'brunch' person, but certainly trust Kathryn's assessment of Balthazar's brunch. We go for lunch fare usually (steak tartare, plateau de fruits de mer) and have always had a lovely time. I also like Aquagrill but, again, have only ordered 'lunch', not 'brunch', dishes.