Times Square is not a hotbed of casual (and also non-chain) restaurants, but Marseille has a nice brunch, as does Blue Fin. That's if you mean <$20/entree, not <$20 all in, which is tougher. Neither is unique, as there are plenty of other seafood and French-ish restaurants in town. But again, Times Square. Not exactly an off-the-beaten-path destination.
I think you should be fine with that budget at Cafe Edison in the Edison Hotel, 228 W 47th St. (between Broadway and 8th Av.). I don't know what you mean by "unique," but Cafe Edison is honest, informal New York food - matzo ball soup, omelettes, etc. Nothing fancy, but reliable.
228 W 47th St, New York, NY 10036