Twitter Planning to Open Up the Firehose "by Thanksgiving"

By Jesse Stay of Stay N' Alive (Twitter/FriendFeed)

In an assuring statement in the Twitter API Developers Group, Alex Payne, the API lead at Twitter, shared that they have plans to have their Fire-hose of Data available to developers and users by "Thanksgiving at the Latest".  The Firehose, or Twitter's continuous XMPP stream (the protocol that powers services like Jabber and Google Talk), has been at the center of controversy for Twitter, where they opened it up briefly to developers, and had to take it down again when they realized the traffic was simply too much for their servers.

The shut-off of the firehose has set off a slew of critiques, including my own, even sending Steve Gillmor to Identi.ca briefly, and ending up in a get-together of like-minded Twitter and other developers to "bearhug" Twitter.  Twitter responded saying at times there would never be a Fire-hose again, and at others, that they were now sending users to services such as Gnip to access the data.  With Gnip closing its doors on its own XMPP stream recently though, it would appear even more frustration is brewing amongst developers.

All that controversy may just come to an end however, as Payne stated in the developers group:

For what it's worth, we've actually been cranking on the "firehose"
solution all week. We've evaluated several queueing systems, and I've just finished work on a proof-of-concept backup plan if those don't pan out. We'd really like to have a solution in place by Thanksgiving at the latest.

There's no doubt with FriendFeed opening up their Firehose of Social Media data, and Gnip cutting off it's XMPP stream that Twitter is feeling the pressure lately to open up real-time data to developers and users.  With Gnip out of the picture, Twitter now has no more resources to send people to in order to hold people off until they get their API in order.

Proof is in the pudding however, and it will definitely be interesting to watch as Twitter attempts to tackle this problem.  Maybe we'll all be getting a little early Christmas gift from Twitter after all?

Read more by Jesse Stay at Stay N' Alive.

Throttled By the Twitter API? Try Something New.

Guest Post By Rob Diana of Regular Geek (Twitter/FriendFeed)

Well, the microblogging API space sure got interesting in a hurry.

First, on Thursday, Louis Gray reported that Twitter was throttling unauthenticated API requests. This obviously would effect several applications in a very bad way. Later in the day, Dave Winer let everyone know that Identi.ca has implemented the Twitter API. And on Friday, in a surprising move, TechCrunch announced that Twitter is sending the XMPP firehose to new middle man Gnip.

So, what does this mean to you? Well, that is a good question. First, we know what the Twitter API looks like. Identi.ca replicating the API is good for interoperability as well. Yes, they copied the main Twitter API, but have yet to include the searching capabilities that Summize supplies. However, they do have RSS feeds for any search query which does suffice for basic searching. The other big players in the microblogging space, Jaiku and Pownce, also have APIs. But, what do they have to offer?

Jaiku's API contains the usual suspects, the public feed, a user's feed and a user's profile. It also allows for "presence" updates which is helpful for allowing applications like Ping.fm to post to multiple services. It also provides a method to get a user's current "presence", their last item in the "presence" stream and as well as a specific item in the "presence" stream.

Pownce's API is similar as well. There is a public "note" list, a user's note list (which can be filtered for replies, private messages and other coolness) and a user's profile. You can also retrieve a specific note, with replies included optionally, and the list of recipients for the note. For social graph fans, you can get the friends (mutual relationship) of a user, fans of the user and who the user is a fan of. For posting notes, there is the normal post method as well as separate post link, event, file and reply methods.

Interestingly, there is a method to determine the list of users a post can go to. There are some other minor goodies like feeds for the public list and a user as well as simple web post integration. Obviously, this is an API designed with developers in mind. They thought of several different ways to use the application and provided APIs accordingly. The only problem that I could see is that there is no search supported. Hopefully a third party service like Gnip will fill that void, like Summize did for Twitter.

Now that looks like a good foundation, but there are some fundamental problems. It is not obvious that Pownce and Jaiku support something like an XMPP feed, so, there may not be the ability to have the full public stream at all times. This type of thing is critical for interoperability. There is also inconsistent support for threaded messages and other post types (like the Pownce event and file posts). Why haven't we seen a real multi-microblog client? Ping.fm is doing multi-writes, but does not support multi-reads. In the instant messaging world, where the XMPP standard comes from, we do have multi-chat clients and few actually support XMPP! We are starting to see some standardization in this space as well with Identi.ca copying the Twitter API as well. If we consider the Twitter API a defacto standard and we have the XMPP standard for real time transfer, there should be little stopping developers from creating the ultimate micro-blogging client.

Now, the question is, are you willing to wait or do you want to crown someone king?

Gnip CEO's Goal: Make Twitter's Data Flow Suck Less

Data publishers and data consumers can be both friends and enemies, when it comes to the seemingly infinite demand for and growth of real-time data from Web services. Services like Digg, Flickr, Del.icio.us and Twitter are happy to see their user bases expand, and to see developer communities be built around their products. But with each new application hitting their API, and each new user, comes new demand that can put strain on their infrastructure, even if the outside application is just checking for updates which aren't there. Gnip is looking to act as a go-between for data publishers and data consumers, delivering updates from the services to the applications, and reducing the queries that can drag popular sites to a crawl.

Today, Gnip made headlines with an announcement that Twitter notifications would be sent to the service via XMPP, letting outside developers tap into Gnip instead of adding more strain to the embattled microblogging giant. And while this won't solve all of Twitter's issues, it does offer developers an alternative, taking some of the power out of Twitter's hands. The announcement did not contain any money changing hands, done quid pro quo.

After the morning's hubbub, I talked with Eric Marcoullier, CEO of Gnip, to better understand how adding Twitter to the team's growing array of partners would help users and developers, and whether this solved the growing concerns around Twitter's API limits that have seen application authors frustrated. And the answer so far, is that Gnip can solve some problems today, and is preparing to solve more issues soon. But it won't make Twitter's problems disappear.

"All these data protocols can be an exponentially scaling hassle. I like how people thought Gnip would single-handedly fix Twitter's problems, but that minimizes how big Twitter really is," Eric said. "Some developers don't just want the user stream, but the reply stream as well, and others want Track, which bangs against the Summize (now Twitter) API, to find if there is a new tweet that has a followed word. We might not ever solve that. It's a big scaling problem of reading the content, and it doesn't fix all of Twitter's problems."


Gnip's Data Flow Architecture
What Gnip is looking to do is help publishers looking to syndicate their data, and consumers, who are building businesses off user generated data, by simplifying the complex back-end work needed, and giving entrepreneurs more time to work on the front-end of their product, delivering tangible benefits.

"We're able to go to them and say, all of the effort you are doing to aggregate that data, stop now," Eric said. "Tell us what services you like, what protocol you like, and the data magically appears in real time."

While Twitter has been the most visible client so far, it's by no means the first for Gnip, which launched with two partners out of the gate, in Plaxo and MyBlogLog, where Eric was a co-founder. Since launch, Gnip has also penned partnerships with additional services, including Lijit and Iminta.


Gnip's Growing Partner Roster
Today, developers of applications are authoring products that query popular sites, like Twitter, Digg and Del.icio.us, and do so thousands of times a day, even if the overwhelming majority of the time, there are no updates. Where Gnip works well is for centralized services, like Plaxo, who can dramatically cut back on the amount of times they need to make requests. "It doesn't matter how many people are following an individual on Plaxo Pulse. They just have to ask once." Eric said.

But the way centralized services make queries to Twitter is different than the issues faced by the many apps that are struggling against the 100 API calls per hour per IP address limit discussed yesterday. For that, more work is needed.

"For the average user, 100 queries per hour is fine, as long as you're only querying the API when there is new data," Eric said. "But for, say Thwirl, where they each have their own user connections, they would have to query maybe 50 times, and that's half the load. We're looking for a simple way of creating anonymous buckets, so somebody like Twitter Karma can say 'we have 10,000 users with this collection', and we can centralize it. We're still a ways away from helping folks with distributed clients."

Gnip's initial efforts and partnerships have been completed on the first version of their product. In about a month (or two), the company expects to not just send service notifications to partners, but also, the full metadata, which will bring more rich information from its many supported services, including Del.icio.us and Disqus, Flickr and others. Maybe, at that point, you'll also see Twitter passing on reply streams as well, but that's not set in stone.

"Working with people like Twitter, we want to be sure we are serving their best interests and the developer community," Eric said. "It's a huge win for us."