I'll second the PV Golf Club - went a feew years back and it was great. We also like Kincaid's, on Redondo Pier. Decent food, nice surroundings and ocean view. Cheesecake Factory serves brunch too (imho that's the only meal to eat there). There's an Original Pancake House on PCH near the Bristol Farms, also.
The best kept secret in the South Bay is that Palos Verdes Country Club is open to the public for Brunch on Sunday. They will take your reservation and you do not have to be, or know a member. Excellent buffet with breakfast AND lunch foods. Nice room overlooking the golf course...Clubby atmosphere.