Lunch/brunch after the memorial
Also consider Blaue Gans~they are a relatively casual Austrian restaurant with a fair amount of space and some lager tables. They have a brunch menu on Sundays (after 11)~ it also has plenty of standard brunch items for less adventurous folks. It should only be ~ 1/2 a mile from the WTC site.
BTW, I was there today and the only exit from the 9/11 Memorial is on Albany St and the West Side Highway, across from the Marriott (which has tons of signs stating there are no bathrooms -- there are also no bathrooms at the Memorial).
Right now you can NOT walk north along the West Side Highway to get to Blue Smoke, North End Grill, Shake Shack, etc. You'll have to make three lefts to compensate (it's about a 20 minute walk). The 9/11 Memorial is actually surrounded by construction on 3 sides, this is why.