La Divina gelateria has a French Quarter outpost on St. Peter St. behind the Cathedral. Try the peppery chocolate Azteca flavor.
I can't think of any truly great pastries, but Croissant d'Or probably has the best in the quarter - 617 Ursulines St.
Chocolate - nothing is coming to mind. I don't think there are any chocolate shops in the quarter.
For super tasty popsicles in creative, delicious flavors, go to Meltdown Popsicles at 508 Dumaine. I especially love the salted caramel and anything with pineapple, but everything is good.