Site down

[Melanie Wong]

The site has been up and down intermittently and very slow-loading for the last two hours that I've been trying to use it.

[meshane]

The servers got hung up last night, and frankly we don't quite know why yet. It took us a couple hours to get them going again. We haven't provided an explanation yet I guess because as engineers we tend to want to have a explanation before we say anything. We have quite a bit of data on the outage that gives us a number of clues to investigate, but nothing conclusive as to what caused this yet.

What I can say is, no, this was not the product of any redesign work. None of that work is live on the site yet. We're based out of San Francisco, and when the site went down at 7:31pm PDT, most if not all of the engineering team were out and about watching the Giants clench a spot in the World Series.

As to the question regarding Amazon's problems yesterday, I don't believe it's related. Amazon's problems were specific to one hosting location in Virginia.

- Shane

[huiray]

The site is acting cranky again.

I'm getting timeouts, very slow loading of my profile page with eventual load of a cached page, loads of top-level boards with time stamps for last post different (earlier) than the time stamp for the post itself within the thread.

Here's one timeout message I got:

Gateway Timeout
The proxy server did not receive a timely response from the upstream server.

Reference #1.845efea5.1351143095.434837cc

[Melanie Wong]

Yep, timed out here on the west coast too about 5 minutes ago.

[huiray]

I think you posted after I did, right?

Interesting time stamps on our posts...

[huiray]

...and after refreshing the page...

Fascinating!!!

[meshane]

Thanks for this example on the time stamps huiray. It gives me a good idea of where to look for some of the timing issues we've seen recently.

As to the gateway timeout you reported, I'm seeing a glitch in network traffic from about 10:24pm to 10:40pm PDT. This glitch doesn't seem to be on our servers, but perhaps on some services we utilize. Everything seems to be back to normal now. This doesn't seem to be related to the problems we saw a couple nights ago.

[meshane]

This issue with the timestamps should now be fixed.

[goodhealthgourmet]

glad to see things up & running again - i was able to connect to CH just long enough to post that reply and then couldn't pull it up for the rest of the night.

[Melanie Wong]

That was my case too. I ran up a smoke signal during a brief interval that the site was live. I happened to be in an email exchange with some other chowhounds at the time and had checked with them whether they were seeing the same thing. Since some posts were getting through, it wasn't clear to me whether the problem was the site or on my end initially. So I wanted to post something for others who might be going through their own confirmation process to save them some trouble.

[Quine]

Let me guess, the fact that CH was down for hours last night (I just gave up) is "some exciting design changes that will be coming to Chowhound in the next few weeks."
IF not, wowzers can't wait to see what happens when you actually do it.

[Quine]

Just started getting some massive S l o w n e s ss , while moving around the site, from board to board.

[lcool]

not me,all OK,normal

[Engineering]

The outage should now be resolved. Our apologies for the inconvenience.

[Dave Feldman]

Good to hear. Was the outage related to Amazon's problems?

[KaimukiMan]

yes, an explanation is usually appreciated along with the apology. preferably on the front page of the site.

[goodhealthgourmet]

i was just finally able to pull it up (10:21 pm PST) after several failed attempts during the past couple of hours.