Assuming you mean "the" Farmer's Market on 3rd & Fairfax, and not one of the many "real" farmers markets around town...
At the Market itself...
I'd avoid DuPar's, though it's (debatably) fine for what it is, I wouldn't choose it for a "best brunch."
There's the new Short Order, which has gotten mixed reviews in general (mix of raves and pans) but they have an actual Brunch menu and is probably your best bet in the Market for something that really resembles brunch.
There's also the French place, Monsieur Marcel, which may be your next best choice, particularly if you want more of a view of the Market itself.
Or if you want to dive in entirely, the juxtaposition of the Loteria Grill and The French Crepe Company stands offers a nice combo of choices for the group, and very decent food.
Otherwise...if you are interested in places that are near the Farmer's Market, but not necessarily within it, there are a lot of choices in the vicinity (Quality, Little Next Door, Black Cat, Tart, House, Cooks County, Eva, Playa, BLD, etc.) but depends how far from the Market you want to cast your net.
Let us know more about what you're looking for, in terms of food, vibe, and proximity, and we can hone some recs.