Twitter Details SXSW Traffic Preparation Measures

Update: Thomas Marban has created a new Twitter logo featuring a sad bird with a bandaid.

We’ve written about Twitter and their server/capacity issues since day 1. This week we saw downtime which could be related to heavy usage at the future of web apps conference. Next week is Twitter’s superbowl: SXSW. To that end, Twitter developer Alex Payne has provided two updates related to how they plan to handle the SXSW traffic. This news comes on the same day that Pownce has released their API 2.0.

Here are the emails (my emphasis):

Tomorrow (Saturday, March 1st) Twitter will be offline from 6PM to 8:30PM PST. We’re adding additional read-slave databases and tuning the configuration on our existing databases.

You may have noticed some downtime and intermittent availability over the last 24 hours. This was the result of some additional caching measures that we put in place which caused an unexpected memory leak, triggering cascading failures across our cluster. We’ve since taken steps to correct this issue. Performance both internal and external to the site has been stable since noon PST, and we’re watching it closely.

Earlier this week we had a different incident in which some users appeared to be signed in as other users. This occurred while were testing a more recent version of Ruby and an alternative application server on one machine in our cluster. Due to our load balancing configuration, the issue quickly spread. We disabled the site and resolved the issue within fifteen minutes. It should not have effected API clients.

Both of these changes were the result of our testing potential performance improvements in preparation for South by Southwest (SXSW), a large annual multi-disciplinary conference held every March in Austin, Texas. This time last year, our growth started to take off around SXSW, and we want to be as reliable as possible during the event this year. We will be taking more cautious steps to improve our performance and reliability over the next week.

As part of those steps, I intend to decrease the number of allowed authenticated API requests per hour from 70 to 50 from Thursday, March 6th through Wednesday, March 12th. While we are taking steps to greatly increase our capacity (and have been doing so continuously, particular since our move to our current host), the API is our foremost source of traffic, and as such is the first place we look when trying to create some breathing room for our cluster. I appreciate your understanding, and I hope that 20 fewer requests per hour don’t impact your applications too drastically for the duration of the modified rate limit.

We also intend to put some extra abuse-prevention measures in place before the event. We’ve seen a general increase in abusive traffic over the last several months, and we simply can’t afford it during a heavy-traffic event. If you’ve been scraping Twitter or consuming public API feeds unfairly, be prepared for an unpleasant surprise.

Thanks, as always, for your patience and understanding. If you have any questions, concerns, or suggestions, please do let me know.

snippit from second email

As for the proposed decreased authenticated requests per hour: I’m open to suggestions. If 50 will put too much strain on your clients and there’s some other way that we can provide you with the data you need while decreasing the overall number of requests for this six day period, please let us know.

We’re not trying to pull the wool over anyone’s eyes or shortchange you after you’ve put in hard work on your applications. We’re always open to alternatives, and we’re always listening.