November 29, 2008

We've Only Just Begun to Syndicate Our Content

By Mike Fruchter of (Twitter/FriendFeed)

It wasn't too long ago that blogging and pull technology, including RSS, first became popular. If you published new Web site content, and wanted the world to know about it in real time, your delivery and distribution options were very limited.

Publishing content updates was pretty much the same as it is today. You would upload your new pages to the server and hope to see some decent search engine traffic. But you relied more on bookmark traffic, and other means of marketing, such as e-mail, to get people to your site. Important as it is to get new traffic, retention is equally as vital. Quality content, useful products, affordable prices and great customer service, are all factors in keeping people coming back to your site.

The early days of the wild, wild, Web.

In the early days, before Google, search engines took days, and often weeks, to crawl and index new content. There were a lot of hoops to jump through to get listed, and you could be waiting weeks to months for a manual review of your site for inclusion. If you didn't have the patience to wait that long, you always had the option of paying a nice fee for an express review, to get your site approved. But the days when Altavista, Lycos,Yahoo, and a few others reigned supreme were also the days the spammers dominated search results.

Therefore, if you were lucky enough to get indexed in a timely fashion, chances are some spam-related bottom feeder had already beaten you out, leaving your pages buried back in the search results. Because spam was so bad and such a problem, a lot of Webmasters adopted the " If you can't beat them join them " mentality. As a result, the search engines almost became rendered useless for a period of time, because they were filled with nothing but spam, mainly in part due to black hat SEO tactics.

E-mail was the name of the game, and it actually worked.

Newsletters are something I, and many other Webmasters, heavily used to inform our user base of new Web site updates/product offerings and so forth. This was as real time as it got back then. Composing daily and weekly e-mails got to be quite a chore, but proved to be very effective. This of course did not last long.Spammers eventually discovered, and killed e-mail marketing for the rest of us. How many of you have received or reported spam e-mail, or even what you perceived to smell like spam to Spamhaus or Spamcop? Even with these most opt-in compliant e-mail lists, you still have frequent headaches with people reporting your legitimate e-mail as spam. The spam reports are also e-mailed to your Web host, and usually to their abuse dept, which causes more unneeded headaches. For non commercial uses such as notifying small or close groups of people, e-mail is still effective and has its place. And nowadays, marketers who use commerce e-mailing must ensure their lists are opt-in/out, and that their compliant with the CAN-SPAM Act of 2003. It also takes a significant amount of more e-mail addresses to click and convert. Besides, people are a lot more reluctant to submit their e-mail addresses today than they once were. Even if they do, the chances of them actually seeing the message diminishes greatly thanks to spam filtering, and disposable e-mail addresses.

We have come a long way in a short period of time.

Today we publish and consume more content faster than ever before. WordPress has become the new FrontPage. Web sites are now blogs. e-mail and newsletters have been replaced by RSS. Micro-blogging applications such as Twitter have filled the void in between. Today when new content is created and published, it's usually done on a blog, and syndicated automatically thanks to RSS and the blogging application used. Today when you publish a blog post, it's distributed and found instantly in RSS readers within minutes of being written. Google and other search engines love blogs, because they are constantly publishing new content. Blogs that update frequently often will have more influence and higher rankings in search. Blogs and traditional Web sites get indexed in search engines, but that's where the similarities end, in terms of real time publishing and real time distribution. Blogs are indexed within minutes, but Web sites often take longer, with a lower probability for achieving higher in the search results.

So just what happens after you click the publish button on a blog post?

When you click publish, your blogging software automatically sends a ping alert to special servers maintained by Google Blog Search, Yahoo, VeriSign and others. The ping lets them know that you have recently published new content. Ping servers then alert aggregators, search engines and others to send out bots to crawl the blog for updates. The ping also alerts data miners and text miners that you have updated. Data miners are in the business of metrics, and this data is often sold to and used by corporations. Text miners are the true bottom scrapers, also commonly referred to as "splogs". Splog is short for "spam blog" and is used to describe an auto updating blog, setup to scrape feeds at regular intervals and post them. They exist for the sole purpose of either displaying ads, such as Google’s Adsense or for the purpose of creating search engine traffic, which in turn is used to promote other splogs. Splogs are automatically generated, and there is not much you can do about them nowadays other than report them to the search engines. The next step in the process, which is set in motion seconds after you press publish, is sending the new blog post to aggregators such as feed readers like Google Reader, and sites that pull RSS feeds, such as etc. The human redistribution process (sharing, bookmarking, etc) then takes over and the cycle is started all over again. Compared to the old days, all this happens within minutes.

In Closing

Publishers today do not have to worry or spend as much time with the distribution of their content as they did way back when. Time is now spent focusing on producing quality content. Gone are the days of the wild wild Web. We are now using smarter, and more effective tools and publishing methods to get the word out faster than ever before. What's next on the horizon for content syndication?

Read more by Mike Fruchter at