Cool "literary" cafe downtown?
Third Rail Coffee. A little small but should fit.
Ost or Joe's.
Stumptown/Ace might be Downtown enough.
71 Irving isn't bad either.
None of these are of the romantic college student reading Proust while sipping espresso variety, if that's what you're looking for, but they're all lively, with good coffee. For atmosphere I would pick Ace, but it is a more adult crowd.
Less of an endorsement of their coffee....Think Coffee might fit what you're after.
441 E 12th St, New York, NY 10009
Third Rail Coffee
240 Sullivan St, New York, NY 10012