Battle for the planet of the APIs

Back in 2006, I gave a talk at dConstruct called The Joy Of API. It basically involved me geeking out for 45 minutes about how much fun you could have with APIs. This was the era of the mashup—taking data from different sources and scrunching them together to make something new and interesting. It was a good time to be a geek.

Anil Dash did an excellent job of describing that time period in his post The Web We Lost. It’s well worth a read—and his talk at The Berkman Istitute is well worth a listen. He described what the situation was like with APIs:

Five years ago, if you wanted to show content from one site or app on your own site or app, you could use a simple, documented format to do so, without requiring a business-development deal or contractual agreement between the sites. Thus, user experiences weren’t subject to the vagaries of the political battles between different companies, but instead were consistently based on the extensible architecture of the web itself.

Times have changed. These days, instead of seeing themselves as part of a wider web, online services see themselves as standalone entities.

So what happened?

Facebook happened.

I don’t mean that Facebook is the root of all evil. If anything, Facebook—a service that started out being based on exclusivity—has become more open over time. That’s the cause of many of its scandals; the mismatch in mental models that Facebook users have built up about how their data will be used versus Facebook’s plans to make that data more available.

No, I’m talking about Facebook as a role model; the template upon which new startups shape themselves.

In the web’s early days, AOL offered an alternative. “You don’t need that wild, chaotic lawless web”, it proclaimed. “We’ve got everything you need right here within our walled garden.”

Of course it didn’t work out for AOL. That proposition just didn’t scale, just like Yahoo’s initial model of maintaining a directory of websites just didn’t scale. The web grew so fast (and was so damn interesting) that no single company could possibly hope to compete with it. So companies stopped trying to compete with it. Instead they, quite rightly, saw themselves as being part of the web. That meant that they didn’t try to do everything. Instead, you built a service that did one thing really well—sharing photos, managing links, blogging—and if you needed to provide your users with some extra functionality, you used the best service available for that, usually through someone else’s API …just as you provided your API to them.

Then Facebook began to grow and grow. I remember the first time someone was showing me Facebook—it was Tantek of all people—I remember asking “But what is it for?” After all, Flickr was for photos, Delicious was for links, Dopplr was for travel. Facebook was for …everything …and nothing.

I just didn’t get it. It seemed crazy that a social network could grow so big just by offering …well, a big social network.

But it did grow. And grow. And grow. And suddenly the AOL business model didn’t seem so crazy anymore. It seemed ahead of its time.

Once Facebook had proven that it was possible to be the one-stop-shop for your user’s every need, that became the model to emulate. Startups stopped seeing themselves as just one part of a bigger web. Now they wanted to be the only service that their users would ever need …just like Facebook.

Seen from that perspective, the open flow of information via APIs—allowing data to flow porously between services—no longer seemed like such a good idea.

Not only have APIs been shut down—see, for example, Google’s shutdown of their Social Graph API—but even the simplest forms of representing structured data have been slashed and burned.

Twitter and Flickr used to markup their user profile pages with microformats. Your profile page would be marked up with hCard and if you had a link back to your own site, it include a rel=”me” attribute. Not any more.

Then there’s RSS.

During the Q&A of that 2006 dConstruct talk, somebody asked me about where they should start with providing an API; what’s the baseline? I pointed out that if they were already providing RSS feeds, they already had a kind of simple, read-only API.

Because there’s a standardised format—a list of items, each with a timestamp, a title, a description (maybe), and a link—once you can parse one RSS feed, you can parse them all. It’s kind of remarkable how many mashups can be created simply by using RSS. I remember at the first London Hackday, one of my favourite mashups simply took an RSS feed of the weather forecast for London and combined it with the RSS feed of upcoming ISS flypasts. The result: a Twitter bot that only tweeted when the International Space Station was overhead and the sky was clear. Brilliant!

Back then, anywhere you found a web page that listed a series of items, you’d expect to find a corresponding RSS feed: blog posts, uploaded photos, status updates, anything really.

That has changed.

Twitter used to provide an RSS feed that corresponded to my HTML timeline. Then they changed the URL of the RSS feed to make it part of the API (and therefore subject to the terms of use of the API). Then they removed RSS feeds entirely.

On the Salter Cane site, I want to display our band’s latest tweets. I used to be able to do that by just grabbing the corresponding RSS feed. Now I’d have to use the API, which is a lot more complex, involving all sorts of authentication gubbins. Even then, according to the terms of use, I wouldn’t be able to display my tweets the way I want to. Yes, how I want to display my own data on my own site is now dictated by Twitter.

Thanks to Jo Brodie I found an alternative service called Twitter RSS that gives me the RSS feed I need, ‘though it’s probably only a matter of time before that gets shuts down by Twitter.

Jo’s feelings about Twitter’s anti-RSS policy mirror my own:

I feel a pang of disappointment at the fact that it was really quite easy to use if you knew little about coding, and now it might be a bit harder to do what you easily did before.

That’s the thing. It’s not like RSS is a great format—it isn’t. But it’s just good enough and just versatile enough to enable non-programmers to make something cool. In that respect, it’s kind of like HTML.

The official line from Twitter is that RSS is “infrequently used today.” That’s the same justification that Google has given for shutting down Google Reader. It reminds of the joke about the shopkeeper responding to a request for something with “Oh, we don’t stock that—there’s no call for it. It’s funny though, you’re the fifth person to ask today.”

RSS is used a lot …but much of the usage is invisible:

RSS is plumbing. It’s used all over the place but you don’t notice it.

That’s from Brent Simmons, who penned a love letter to RSS:

If you subscribe to any podcasts, you use RSS. Flipboard and Twitter are RSS readers, even if it’s not obvious and they do other things besides.

He points out the many strengths of RSS, including its decentralisation:

It’s anti-monopolist. By design it creates a level playing field.

How foolish of us, therefore, that we ended up using Google Reader exclusively to power all our RSS consumption. We took something that was inherently decentralised and we locked it up into one provider. And now that provider is going to screw us over.

I hope we won’t make that mistake again. Because, believe me, RSS is far from dead just because Google and Twitter are threatened by it.

In a post called The True Web, Robin Sloan reiterates the strength of RSS:

It will dip and diminish, but will RSS ever go away? Nah. One of RSS’s weaknesses in its early days—its chaotic decentralized weirdness—has become, in its dotage, a surprising strength. RSS doesn’t route through a single leviathan’s servers. It lacks a kill switch.

I can understand why that power could be seen as a threat if what you are trying to do is force your users to consume their own data only the way that you see fit (and all in the name of “user experience”, I’m sure).

Returning to Anil’s description of the web we lost:

We get a generation of entrepreneurs encouraged to make more narrow-minded, web-hostile products like these because it continues to make a small number of wealthy people even more wealthy, instead of letting lots of people build innovative new opportunities for themselves on top of the web itself.

I think that the presence or absence of an RSS feed (whether I actually use it or not) is a good litmus test for how a service treats my data.

It might be that RSS is the canary in the coal mine for my data on the web.

If those services don’t trust me enough to give me an RSS feed, why should I trust them with my data?

Have you published a response to this? :


Joe Crawford

On July 1, Google Reader does indeed close. Reader has been a valuable tool for me for a long time. In 2006, I said:

Holy cow but the new Google Reader is pretty darn good. Fully worth using to read your RSS and Atom feeds.

And that same year, I wrote an extended piece that included how I used Google Reader. Time Time Time; Experts I Treasure is about how I manage my own time, and how useful GReader was to me at the time. Here’s a relevant excerpt:

One thing I’ve done is started reading RSS and Atom feeds again. I use Google Reader to organize and read blogs and anything with syndicated content. This is a great way to keep track of things, but to be “the decider” when it comes to when I look at them. I organize them into categories: ours (blogs leah and I work on or maintain, flickr feeds of ours, etc), people (blogs in a voice more or less by one person), writers.laish (blogs by people in leah’s nascent writer’s group), forsale (craigslist feeds for “for sale” items), group (blogs like metafilter or boingboing, which don’t really have a single person at the helm), jobhunt (feeds for job postings), links (link blogs like waxy’s links and larger link sites like fark), and sandiegobloggers (a holdover which i don’t beat myself up about if i don’t read, there are many). For several months I simply gave up reading blogs or keeping up with feeds, because I was so busy with work, but I ended up even more frustrated because there were people and information sources that I ended up checking anyway, and doing it ad hoc took more time than simply reading them in an RSS reader. I tried bloglines for about a minute. And I had been using NetNewsWire, which is great software, but is not nearly as mobile as I am. When Google Reader got good recently, I jumped in and have not looked back. It’s all very “Web 2.0” of me, I’m sure.

Since December, 2010 I’ve been using Reeder App to read my Google Reader feeds. It has been a tremendously worthwhile purchase at $2.99. Reeder is an excellent frontend to Google Reader. I was using Reeder originally on my Mom’s iPod Touch after she died. Reeder was great. And I’ve been glad it kept working and transferred over when Leah and I got iPhones.

I’m disappointed by Google Reader being shut down. What I’ve done now is install Tiny Tiny RSS on a domain I run and have started, tentatively using it as a replacement for Google Reader. At this point I’ll no longer be able to use Reeder App on my phone. None of the solutions out there appeal to me. Having been burned once now, it’s hard to get behind the other options. What I’m looking for now is to be able to use a service I control as a front end for Reeder App.

RSS is critical infrastructure for the web, and has been for a long time. Some of my favorite projects have been pages powered by RSS. Twitter recently killed all support for RSS or Atom feeds. It’s a horrible disservice they’ve done, but we live in an age where web giants no longer respect inter-operative technologies the way some once did. For depressing reading on this, read Battle for the planet of the APIs by Jeremy Keith.

It’s all incredibly frustrating.

I’ll leave you with a J.M. Barrie allusion in a tweet I recently tweeted:

I still believe in RSS, but we still have work to do to get back the web we lost.

Addendum July 2, 2013. The shutdown is complete:

Thank you for stopping by.

Google Reader has been discontinued. We want to thank all our loyal fans. We understand you may not agree with this decision, but we hope you’ll come to love these alternatives as much as you loved Reader.


The Google Reader team

Frequently-asked questions
  1. What will happen to my Google Reader data?

    All Google Reader subscription data (eg. lists of people that you follow, items you have starred, notes you have created, etc.) will be systematically deleted from Google servers. You can download a copy of your Google Reader data via Google Takeout until 12PM PST July 15, 2013.

  2. Will there be any way to retrieve my subscription data from Google in the future?

    No — all subscription data will be permanently, and irrevocably deleted. Google will not be able to recover any Google Reader subscription data for any user after July 15, 2013.

  3. Why was Google Reader discontinued?

    Please refer to our blog post for more information.

Goodbye, old friend.

Time & attention stole our RSS feed. Or, change your role models for a better web. June 18, 2013 (I couldn’t decide on the title. Sorry.) Jeremy Keith wrote a really great piece about how companies like Twitter and Facebook feel threatened by data interoperability and how this is setting a trend on the web: Once Facebook had proven that it was possible to be the one-stop-shop for your user’s every need, that became the model to emulate. Startups stopped seeing themselves as just one part of a bigger web. Now they wanted to be the only service that their users would ever need …just like Facebook. Jeremy is exactly right. These companies are looked up to by so many businesses today that they watch their every move and try to mimic them in hopes of mimicking their success. Is there a way to buck the trend? Yes. In a recent post by Bill Gates (on LinkedIn of all places) he writes about three things he learned from Warren Buffet. Many of us fans of Warren have heard this before but it is certainly worth re-stating here: you have to be willing to ignore the market rather than follow it, because you want to take advantage of the market’s mistakes Facebook, Twitter, and others are making the mistake of taking away read-only API access like RSS feeds or by severely limiting the usage of their APIs under a stack of hard-to-understand terms. Why? Because people, and not just geeks like Jeremy and I, will want to use their data in ways like Jeremy describes in his post and in many other ways too. So how do we do it? How do we buck the trend? We need to follow a different business model than the companies that feel threatened by something as simple as a RSS feed. Facebook and Twitter make the bulk of their money by keeping you on their sites under their well-designed user experiences. By keeping our attention. It is why Twitter cracked down so hard on third-party clients and why Facebook rarely allows its stream data outside of its own web page. They need to show ads and even more importantly they need to show ads in a way that produces real results in order to stay in business. RSS and third-party-designed experiences threatens that. Advertising as a sole business model can cause companies to do crazy things. Their customer is the advertisers (large, global brands with millions to spend) and their product is us (their users). They will do just about anything to get us to stay on their site as long as possible. We might think; “Yeah, but I love Twitter. I love Facebook.” I’m sure we do. I love Twitter. They hope we do. They work really hard and spend a lot of money to ensure that we do. However, there has never been a single ad-driven market that hasn’t seen explosive growth followed by a long, drawn out crash. Print, Radio, TV, the Internet. Advertisers are not loyal to Twitter or Facebook just like they weren’t loyal to Print or Radio. They are loyal to us. Right now you’re probably using Twitter while you watch TV! So they put #hashtags right on your TV screen. They know you better than you know you. If we want to see data interoperability increase and we want to see things like RSS begin to see growth once again we have to change our role models. We have to stop looking to Twitter and Facebook as our role models. We need to look to companies that count their users as their customer. And their customers will want to use their data however they’d like. Mimic them. Do better than them. Be more open than them. That’s when we’ll see things swing the other way. View all posts

# Monday, April 23rd, 2018 at 8:55pm

Previously on this day

12 years ago I wrote So

Such a little word.

15 years ago I wrote Hacktime

The hackclock is ticking.

15 years ago I wrote Hacknight

Hackery, Werewolf and The Doctor.

18 years ago I wrote Blooming wit

Yesterday was Bloomsday. I wasn’t in Dublin: I was on stage in Brighton with Salter Cane. Still, I couldn’t let the occasion pass unmarked.

19 years ago I wrote Browser gloom and doom

When I wrote earlier this month about Microsoft’s decision to stagnate browser development, I took a fairly pessimistic view of where we developers now stand.