Twitter Chokes Unauthenticated API Requests By IP, Sites Gasp for Air

Twitter's struggles with handling high user load have been well documented. To help the embattled site stay up between Fail Whales, they've at times reduced features to just try and keep afloat, blocking the replies tab, reducing pagination, and blocking Instant Messaging, for example.

They've also visibly tweaked the rate for authenticated API hits, first down from 70 to 20 and back up to 100. But until recently, unauthenticated API requests were unlimited, which all changed Wednesday night around 5 p.m. Pacific Time, when Twitter ratcheted them down to the same 100 per hour per IP address, effectively crushing many external services that relied on Twitter for their data. And this was done without public mention on the Twitter blog.

As you can see in this Google Groups thread, Alex Payne of Twitter told developers last Thursday:

"In our continued effort to keep things fast and prevent abuse we're planning on introducing rate-limiting by IP for unauthenticated API requests. We'll allow 100 unauthenticated requests per IP per hour, just as we currently do with authenticated requests. Please let us know if you foresee any ghastly issues with this change."

And while that sounds all nice and dandy, there were a number of developers who did see "ghastly issues", ranging from Kee Hinckley, CEO of Somewhere, Inc., Richard Cunningham of FriendBinder, and Joel Strellner of Twitturly, each of whom runs a product that depends strongly on unauthenticated API requests from the microblogging service.

But it didn't stop Twitter from making the change, even as Jodee Rich of PeopleBrowsr wrote, "this will blow us out of the water."

Since throttling down the unauthenticated API requests, services that rely on this data are a mere shadow of their former selves.


Twitturly says on status page: "Due to recent changes in Twitter's API, Twitturly is slowly dying."

Similarly, Twist's Twitter charts only show a fraction of the data they did at the beginning of the week. (See: "morning", "starbucks" and "computer".)


Twist's Twitter Search for "Morning"

Twist's Twitter Search for "Lunch vs. Dinner"
In response to the changes, FriendBinder's Cunningham said he will now be changing all API requests to Twitter to be authenticated, to work around the problem. He wrote me, in an e-mail, "There are also some parts of the API we cannot even attempt now. User authenticated requests limits are shared with other sites that the user has used - so we might not be able to get any updates for them if some other site hit the limit for them."

You can see this yourself by trying sites that aim to help you bulk follow those Twitter users who are following you. If you use a site like Twitter Karma, and then try to use Less Friends, you will get a nasty note that says you have exceeded your 100 requests per hour limit. And if you're like me, having fallen way behind on following folks, you might be revisiting one of these sites religiously every hour for the next few days until you're caught up.


Less Friends and Twitter Karma Get a Mere Fraction of the Updates
Last month, Jesse Stay wrote that developers were bailing on Twitter, thanks to outages and broken APIs. With changes like this, despite improved uptime of late, the end to Twitter's problems is nowhere in sight, and services that hoped to tap into their API this way are sore out of luck.

Note: These restrictions do not impact the four golden partners of Twitter's XMPP feature, including Zappos, Summize, FriendFeed and Twittervision.

FriendBinder Throws Hat In LifeStreaming Ring

Back in March, I bumped into rumors of FriendBinder, which, like other lifestreaming offerings including MyBlogLog, Plaxo Pulse, Profilactic, LetsProve and FriendFeed, claimed to offer a single destination to keep track of what your friends are doing on many different social networks. At the time, the author, Richard Cunningham, let me know the service was more than a year into development and would be coming soon. The service debuted in beta in the last week, and while it works, it reminds me more of Spokeo than the aforementioned apps, pulling contacts I have at different networks and displaying them in one unified stream.

Getting started with FriendBinder is relatively easy. The first step is to register which social networks you currently use, by entering your ID. Options supported at beta launch include YouTube, Twitter, Facebook, Digg, Del.icio.us, Flickr, Last.FM and the more generic News/Blog subscriptions, which asks for an RSS feed.

The second step is to add your friends into FriendBinder, by clicking the "Add All Friends" option next to each network. FriendBinder will then login to your external accounts, find your friends, and troll for updates.

When complete, you have what's called your "FriendStream", which, as anticipated, shows the latest updates in chronological order, the top item being the most recent.


As once was said about FriendFeed, if you don't make any changes to your preferences, FriendBinder is dominated by Twitter. The sheer volume of updates by friends in Twitter had me looking around to see just what other updates were contributing to the stream. I did find the occasional Digg or Del.icio.us update, but they were the exception rather than the rule.


Interestingly, one wrinkle offered by FriendBinder is the ability to rate the importance of one network's updates above another, from 1 star to 5, with 3 being the default. This is called the "interest level". I can also, by network, tag one friend's updates as more important than those from somebody else, by manually clicking the number of stars. (Frederic of The Last Podcast noted this in his review as well)


This lets you sort your best friends from lesser contacts by filtering your stream by 5 stars, 4 stars, and so on.

You can also parse the FriendBinder stream by service, showing only Del.icio.us updates, only Digg updates, etc. Given the overwhelming noise coming via Twitter, first clicking Networks and then picking a single service just might be the best way to cut through the noise.

FriendBinder does exactly what it promised to do - give one place to find all updates from friends. But that opens up more questions. Then what? The Web already has quite a few sites that serve to aggregate all friends' activities, and the ones that are gaining traction are those (read: FriendFeed) which enable a follow-on action. FriendBinder data streams are siloed, such that I won't ever interact with another FriendBinder user. I can't respond via Twitter from within the site. I can't add comments to Facebook or Flickr photos within the site. I can't post directly to the site, and wouldn't need to, considering nobody else will see it.

By registering with FriendBinder, and entering my network details, I know what I've done is set off yet another farm of servers to continue slaving away, and requesting my contacts' information, even if I never login again, just like when I registered for Iminta, Profilactic, Spokeo, Mergelab, Assetbar, Shyftr, Plaxo and any other site aiming to do the hard work for me when I'm away. As discussed last week, this strain on the infrastructure can eventually force sites to reduce features or even close, so I'm already feeling a bit guilty for making FriendBinder work on my behalf. Hopefully, FriendBinder can step up to demand, should it grow, and add new interactive features that would help it bridge the gap from being yet another lifestreaming site to one who can innovate and differentiate to make it a destination site.