I've had the same experience. People wear everything from jeans on up to suits and nice dresses. I think that wearing jeans is underdressing a bit (and this is from someone who pretty much doesn't own anything but jeans) but you won't feel out of place whichever you wear.
And save room for dessert. Karen Barker is a goddess!
I went there a couple of months ago for dinner on a week night. There was a variety of dress, some casual (jeans, button down shirt) to business/church attire. I don't think you should feel like you HAVE to get really dressed up, unless you just want to. Have a great dinner there. BTW, I definitely recommend getting the grits souffle that has a thin layer of foie gras on top. It is heaven!