Casual Lunch in Center City???
Chinatown is a good bet. There are so many good places and most offer special inexpensive lunch specials. A few that come to mind that I love and are good for sitting and talking are Ong's for excellent Vietnamese and Chinese food, CK garden for Szechuan food, Rangoon for Burmese cuisine (an especially nice dining room).