Depends on what you like in a brunch
Fancy/fanciful: The Veranda (Highland Ave), Chez Lou Lou (English Village). The restaurant at the Museum downtown has a monthly Sunday jazz brunch buffet. Several restaurants in town do Sunday jazz brunches (check out al.com).
Mexican: Taqueria Mi Casita, Gordo's, Taqueria La Reyna: Mi Casita (all in the Palisades/Valley Avenue area) even has breakfast tacos. All have weekend posole and menudo. Carnitas, lengua, chicharron are recommended.
Asian: Pho at Que Hong (Green Springs), Saigon Noodle House (U.S. 280) or Que Hoang (Alabama 150, near I-459).