We took a cab but I think the warnings about the neighborhood are overstated, especially if you drive. Your car will be in plain view at all times. It's a wide open area -- a broad street with a couple of large warehouses.
After lunch we walked down Baronne to Jackson (about 8 blocks) to make our way to the Garden District. It's definitely a low income neighborhood but we -- pathetic yuppie jokes in sundresses, baseball hats and shiny loafers...total targets -- never felt threatened. Not a place I'd hang out but if you handle yourself smartly I don't think there is any problem. Especially if you are driving.