Downtown East Indian spot
I know you've said "downtown," but if you're feeling at all adventurous - or mobile - you could also try Amaya (see the discussion thread at http://www.chowhound.com/topics/422101). It's at Bayview and Eglinton, but a world apart from most of what you'd find anywhere else in the city, at least in my opinion.
Others have had different experiences - as you'll see from the discussion - but ours was most enjoyable.
Babur is good, though I haven't been in years. I am biased because I went there on a first date once (it was actually my first time eating Indian) that turned into a several year long relationship.
I also prefer Trimurti food-wise, but I think the atmosphere there is less datey. I find the tables to be very close together and things get hectic when they get swamped (with service sometimes going out the window).
The Host is very nice inside and would probably be nice for a date. You can sit in the sunroom thing at the front the restaurant. I know they get some good reviews but I went once and didn't see what the fuss was about. Our food wasn't very good at all. It was probably the first time I ever met Indian food that I didn't like. Anyway, I personally would give it another go. ETA: the service was top notch, however.
You can look into Dhaba as well. You will see mixed reviews of it on here. I have only ever been for the buffet lunch. It is fine, and a hundred times better than most of the lunch food in that neighbourhood (theatre district), which actually isn't saying much. The lunch time butter chicken is exceptionally creamy.
I am not really a fan of Kama, nor Aroma either, but they have the requisite linen tablecloths.
I don't know, for me for a date with someone newish I would let vibe, intimacy and comfort influence my decision a little more than it usually does, unless your date is also into food. If she/he is not, they will be more likely to notice that the place is cramped and noisy and not conducive to intimate conversation than the fact that their chana is particularly tasty! Just my opinion.
The Host near Bloor and Avenue has good food, an impressive decor, and a more "upscale" feel than a lot of the other Indian places downtown, if that says "romantic" to you.
A year ago I would also have recommended the Biryani House at Yonge and Wellesley, but I understand this has since closed and re-opened with a new chef, and I read some mixed reviews about it a few months back. Maybe someone else has been recently and can say whether the new incarnation is worthwhile?
I second the Host, for the excellent food. The decor has a bit of a British Empire feel -- in some corners you don't know if you should be expecting a great biriyani or beef wellington -- but it's appropriately dark, and the service is excellent.
Kama, in my experience, is decidedly second rate.
Babur on Queen is a nice airy room with good food. Just west of University. On King, there is a place called Kama below ground that is also fairly nice. I'm presuming that nice means tables have cloths and are not pressed up against each other. If you wanted a bit more funk, there are a few places on Baldwin that could cover that, and give you some before and after wanderings. Enjoy!