I gave up junkfood for Lent one year and spent every weekend sampling breads from bread bakeries I had researched. Definitely go to Sullivan St. Bakery. There's also Bouchon, which has a great baguette, in the Time Warner Center, but both of these may be slightly out of the way for you.
Depends where you are in Midtown but:
Amy's Bread, 9th between 46th St & 47th
Sullivan St Bakery, 47th St between 10th and 11th
Le Pain Quotidien, a chain with multiple locations around Midtown
I would also check the Murray's Cheese/Meat stand in Grand Central as I believe they also carry bread from bakeries around town. Not sure about Financier but I think they have bread as well.
250 Bleecker St, New York, NY 10014