Tried-and-True, Casual, Institutions
What constitutes casual for you? Shorts and flip flops/no table service casual or more fine dining but no dress up needed? California is a pretty casual place but definitions are different. I'd say go to some taquerias...Cancun, San Jose, La Taqueria...all casual but maybe that's not what you're looking for.
Agree with all the above suggestions. Zuni for sure. If you can't get a reservation, they leave 1/3 of the restaurant for walk-ins, but get there early - 5:30, not later than 6. It's also one of the best places for lunch.
For dinner I would add the following to the list:
Delfina - in the Mission, great neighbrhood to explore.
Tommaso's in North Beach - for the wood-fired brick oven pizza, very casual by any standards, fun atmosphere, an SF institutuion.
Lunch : Swan's Oyster Depot ... Zuni Cafe ... L'Osteria del Forno
Dinner : Tadich Grill ... South Park Cafe ... Slanted Door
Ferry Slip, San Francisco, CA 94111
1658 Market Street, San Francisco, CA 94102
240 California St, San Francisco, CA 94111
L'Osteria del Forno
519 Columbus Ave, San Francisco, CA 94133
Swan Oyster Depot
1517 Polk St, San Francisco, CA 94109
South Park Cafe
108 South Park, San Francisco, CA 94107