I've been working downtown (U.S. Bank Tower) for 13 years, and in my opinion, you have essentially two options if you want a really good lunch: (a) spend more than $20; or (b) drive (or take the subway) someplace. Both options have their advantages, and my favorites (in each category) are as follows:
Water Grill (544 S Grand Ave., (213) 891-0900). One of the best seafood places in town.
Arnie Morton's (735 S Figueroa St., (213) 553-4566). I love their cheddar burger with sauteed mushrooms, followed generally by a strong desire to take a nap. At $13, it's also a good value.
Nick & Stef's (330 South Hope St., (213) 680-0330). Good steaks, etc. My current favorite is their grilled Scottish salmon and roasted winter squash.
Cafe Pinot (700 West 5th St., (213) 239-6500). Good poulet frites, and their churros with hot chocolate is very sweet, very rich, and very good. (Which is pretty much how I want my obituary to read.)
Ciudad (445 S. Figueroa St., (213) 486-5171). This place (like Pinot) is sometimes inconsistent, but I think they generally make a decent Cuban sandwich and I like their beef soft tacos. Good torta de tres leches too. (In fact, if you ever need a dessert to take someplace right from work, you could do much worse than ordering a whole one of these.)
Tesoro/Ciao (300 S. Grand Ave., (213) 680-0000, and 815 W. 7th St., (213) 624-2244, respectively). Two, decent, fairly reliable, mid-level Italian places owned by the same folks.
Langer's (704 S. Alvarado St., (213) 483-8050). A mere two subway stops west of Pershing Square on the redline. Before discovering Langers, I didn't think it was possible for pastrami to taste anywhere near this good. When it's time for me to go, I hope it's not a Sunday, because I want my last meal to be a number 19 on rye, hand sliced with a new pickle.
Mercado La Paloma (3655 S Grand Ave., http://www.mercadolapaloma.com/restaurants.html). Home to Taqueria Vista Hermosa (which has wonderful al pastor tacos) and Chichen Itza (good Yucatan style food, including in particular, a wonderful cochinita pibil). Also fresh juices, ice cream, etc.
Tacos Baja Ensenada (5385 Whittier Blvd., (323) 887-1980). Stunningly good fish and shrimp tacos. I think, the best in town.
Mi India Bonita (4731 E. Olympic Blvd., (323) 267-8505). Excellent albondigas, cocido de res, carnitas tortas, etc. If I ever leave LA, this may be the place I'll miss the most.
C & K Importing/Papa Cristo's (2771 W. Pico Blvd., (323) 737-2970). Good simple, flavorful Greek food. Get a side of grilled eggplant, and their Greek yogurt is phenomenal.
Philippe's the Original (1001 N. Alameda St., (213) 628-3781). Get a double dipped roast lamb, a coleslaw, and a lemonade, and you're in good shape. (Also accessible from the Union Station exit of the Redline.)
Also note that there are a ton of places in the San Gabriel Valley that are wonderful and are within easy driving distance from downtown.
And finally, the exception that proves the rule: The LA Grand Central Market (http://www.grandcentralsquare.com/ ). The carnitas burrito at Ana Maria's is huge and delicious, as are their gorditas, tortas, etc. I've heard that some of the other places in the GCM are good as well, but I can't seem to get away fro Ana Maria's. Within walking distance, and you can get a burrito big enough for two meals for $4.
Tacos Baja Ensenada
5385 Whittier Blvd, Los Angeles, CA 90022
544 South Grand, Los Angeles, CA 90071
Philippe the Original
1001 N Alameda St, Los Angeles, CA 90012
700 West Fifth St., Los Angeles, CA 90071
2771 W Pico Blvd, Los Angeles, CA 90006
704 S Alvarado St, Los Angeles, CA 90057
Grand Central Market
317 S Broadway, Los Angeles, CA 90013
445 S Figueroa St Ste 100, Los Angeles, CA 90071
Arnie Morton's the Steakhouse
735 S. Figueroa Street, Los Angeles, CA 90017
Mercado La Paloma
3655 S Grand Ave, Los Angeles, CA 90007
Nick & Stef's Steakhouse
330 S. Hope St., Los Angeles, CA 90071
Mi India Bonita
4731 E Olympic Blvd, Los Angeles, CA 90022
300 S. Grand Ave., Los Angeles, CA 90071
815 West 7th Street, Los Angeles, CA 90017
I work downtown a few blocks from Wilshire and 7th.
Roy's (on the expensive side but still a good value). To be honest I'm usually happier at Roy's than at Water Grill and Pinot. And the sushi rolls can best those at popular sushi joints.
George's Greek Cafe (7th & Fig) (only for the chicken gyro and salmon salads--mostly bad)
Charlie Kebab (7th & Fig) (excellent for the price)
Bar-B Kosher (9th around Broadway)
Tiara Cafe (9th around Broadway)
Zucca (Figueroa) (not that great overall, but a good complementary bread basket, and a nice Italian tuna salad)