Can Microblogs Just Talk To Each Other?

Guest Post By Rob Diana of Regular Geek (Twitter/FriendFeed)

Twitter has had a bad few days. After a few weeks of wonderful stability, we started to see some Fail Whales. Then people started losing followers and subscriptions. Because of this, even more people are moving to alternative micro-blogging services like Identi.ca. Of course there have been several requests for a Twitter friend importer. I think people are getting caught up in the problem and not thinking about the best solution. Previously, I mentioned that Twitter was at the crossroads. In that post I started thinking about a possible solution:

It is now late May and they are just admitting that they have infrastructure problems. Obviously, there are the redundancy and replication issues that can be solved with known techniques. The realtime API needs to be using a replicated database and not affect the main database. This is the part that is concerning me the most as it should have been obvious some time ago.

Another post appeared from Dave Winer where he mentioned that we are inching toward federation.

Right now, today I'm using an approximation to the ideal system. I try to enter my original post on FriendFeed, then I have an agent script running on one of my machines that routes it to Twitter and Identi.ca, with a pointer to the discussion thread on FriendFeed, shortened by bit.ly.

Obviously, this is not a good solution to the Twitter problem either. There are also multi-posting services like Ping.fm and Posty which allow a user to update each of their micro-blogging services with the same message. So, with one click you can update Twitter, Identi.ca, Pownce, Jaiku and Plurk. While that sounds really cool, they only provide one-way updates. Tools like Twhirl are starting to allow posting and retrieving of tweets and dents in one application. This gets a little better, but you still have multiple accounts to deal with.

Federated Microblogging

Federation is the real answer. What does federation do and why is it different from Twitter creating a distributed architecture? A distributed architecture means you have various pieces of the application sitting on different servers. Typically, several parts are replicated or redundant and there are various load balancing devices thrown in to complete the architecture. I am not going to go into tons of detail as books have been written about distributed architectures.

Federation is different from distributed in one simple sense. Federation requires a full copy of the entire system. Federation is the cooperation between various systems to act as one. Wikipedia has a good description of a federated database system.

A federated database system is a type of meta-database management system (DBMS) which transparently integrates multiple autonomous database systems into a single federated database. The constituent databases are interconnected via a computer network, and may be geographically decentralized. Since the constituent database systems remain autonomous, a federated database system is a contrastable alternative to the (sometimes daunting) task of merging together several disparate databases.

If you replace all of the database and database system terminology with microblogging and microblogging system you will understand a federated system. So, how does this help Twitter? It does not, at least not directly. By having a federated system, people can move to Identi.ca or the next clone that appears and not lose their followers. If you think about the concepts of email, you will understand how the federated system would work. My account in the federated system would be robdiana@identi.ca. A Twitter user could reply from within Twitter directly to me (robdiana@identi.ca) and I would receive it in whatever client application I had. I could also follow Twitter users, Pownce users or any other microblogging user. By enabling people to communicate across platforms would mean that the server load would be spread across various services, thus decreasing the amount of traffic on Twitter directly.

The one part of the federation that is missing is the routing between servers. This can be accomplished by following the DNS model. A local DNS server has a reference to its parent or master server. This allows new servers to be built and their location and IP address propagated to other servers. This is a very effective solution and it has worked for several years.

Because Twitter and Identi.ca follow the same API, they could act as the root servers in the system. The other option to the Twitter API is the XMPP protocol, which Twitter is using for their realtime API. The protocol choice is important because a federated system requires that all of the systems speak the same language. A federated system would also require a different client application than we have now. As I mentioned earlier, Twhirl fully supports Identi.ca and Twitter. However you are dealing with two different accounts. How cool would it be if you had one microblogging client to chat with people on any service?

Throttled By the Twitter API? Try Something New.

Guest Post By Rob Diana of Regular Geek (Twitter/FriendFeed)

Well, the microblogging API space sure got interesting in a hurry.

First, on Thursday, Louis Gray reported that Twitter was throttling unauthenticated API requests. This obviously would effect several applications in a very bad way. Later in the day, Dave Winer let everyone know that Identi.ca has implemented the Twitter API. And on Friday, in a surprising move, TechCrunch announced that Twitter is sending the XMPP firehose to new middle man Gnip.

So, what does this mean to you? Well, that is a good question. First, we know what the Twitter API looks like. Identi.ca replicating the API is good for interoperability as well. Yes, they copied the main Twitter API, but have yet to include the searching capabilities that Summize supplies. However, they do have RSS feeds for any search query which does suffice for basic searching. The other big players in the microblogging space, Jaiku and Pownce, also have APIs. But, what do they have to offer?

Jaiku's API contains the usual suspects, the public feed, a user's feed and a user's profile. It also allows for "presence" updates which is helpful for allowing applications like Ping.fm to post to multiple services. It also provides a method to get a user's current "presence", their last item in the "presence" stream and as well as a specific item in the "presence" stream.

Pownce's API is similar as well. There is a public "note" list, a user's note list (which can be filtered for replies, private messages and other coolness) and a user's profile. You can also retrieve a specific note, with replies included optionally, and the list of recipients for the note. For social graph fans, you can get the friends (mutual relationship) of a user, fans of the user and who the user is a fan of. For posting notes, there is the normal post method as well as separate post link, event, file and reply methods.

Interestingly, there is a method to determine the list of users a post can go to. There are some other minor goodies like feeds for the public list and a user as well as simple web post integration. Obviously, this is an API designed with developers in mind. They thought of several different ways to use the application and provided APIs accordingly. The only problem that I could see is that there is no search supported. Hopefully a third party service like Gnip will fill that void, like Summize did for Twitter.

Now that looks like a good foundation, but there are some fundamental problems. It is not obvious that Pownce and Jaiku support something like an XMPP feed, so, there may not be the ability to have the full public stream at all times. This type of thing is critical for interoperability. There is also inconsistent support for threaded messages and other post types (like the Pownce event and file posts). Why haven't we seen a real multi-microblog client? Ping.fm is doing multi-writes, but does not support multi-reads. In the instant messaging world, where the XMPP standard comes from, we do have multi-chat clients and few actually support XMPP! We are starting to see some standardization in this space as well with Identi.ca copying the Twitter API as well. If we consider the Twitter API a defacto standard and we have the XMPP standard for real time transfer, there should be little stopping developers from creating the ultimate micro-blogging client.

Now, the question is, are you willing to wait or do you want to crown someone king?