There are billios of reccs on the Board for brunch in Brooklyn and meals near the BBG.
The classic saturday brunch near the garden is Tom's Diner on Washington (closed Sunday, tho) . You will have to wait, dressed up diner fare, a fun place. I hear that the food at the cafe in the garden is quite good (next to the greenhouse - it looks good)
If you want a fullscale brunch experience, you may want to look through the Park Slope/Prospect Heights brunch posts. If you are really up for a walk and not just a short stroll, you should definitely go through glorious prospect park as well as the BBG - they adjoin, and if you are walking from Park Slope or Grand Army Plaza, rather than driving or taking the train direct to the garden, it is on the way. Also the Farmers Market at GAP is a good chowhound stop in the area.
If you are walking back to Plaza Street after your visit, Joyce Bakery on Vanderbilt near Park Place is a nice stop for coffee and a pastry.