Completely agree on Louie's. If you can book a one-sided booth facing the water it's even better! Haven't been to Latitudes on Sunset Key (a 10-minute ferry ride from Key West) but it looks awesome too and has been featured many places as the best special occasion spot in Key West.
Louie’s Backyard. I agree completely with Zagat’s review:
Food 25 Décor 27 Service 23 Cost $57
“It doesn’t get any better” than “the sea, the sun, the food” at this “classy, romantic” “longtime winner” in Key West, voted Most Popular in the Keys for its “magnificent” ocean view, “superb” Caribbean-American cuisine and “informed” service; dinner can be a “wallet-bust” and a few cite “snooty” treatment, but to its legion of champions there’s just “no other place like it”; P.S. “make reservations well in advance.”