Journal tags: preservation



Cool your eyes don’t change

At last November’s Build conference I gave a talk on digital preservation called All Our Yesterdays:

Our communication methods have improved over time, from stone tablets, papyrus, and vellum through to the printing press and the World Wide Web. But while the web has democratised publishing, allowing anyone to share ideas with a global audience, it doesn’t appear to be the best medium for preserving our cultural resources: websites and documents disappear down the digital memory hole every day. This presentation will look at the scale of the problem and propose methods for tackling our collective data loss.

The video is now on vimeo.

The audio has been huffduffed.

Adactio: Articles—All Our Yesterdays on Huffduffer

I’ve published a transcription over in the “articles” section.

I blogged a list of relevant links shortly after the presentation.

You can also download the slides or view them on speakerdeck but, as usual, they won’t make much sense out of context.

I hope you’ll enjoy watching or reading or listening to the talk as much as I enjoyed presenting it.

The forgotten house

The Never Forgotten House is a beautifully-written piece with a central premise that is utterly, utterly flawed. Once again the truism that “the internet never forgets” is presented as though it needed no verification.

Someday soon, the internet will fulfill its promise as a time machine. It will provide images for every space and moment so we can fact check our memories. Flickr and Facebook albums will only accumulate.

Citation needed. Badly.

Read the article. Enjoy it. But question its unquestioningness. It made me sad for exactly the opposite reasons that the author intended.

Every essential moment of a child’s life is documented if he was born in the West. With digital album after album for every birthday, every Christmas, he will never struggle to remember what his childhood home looked like.

I wish that were true.


It’s hard to believe that it’s been half a decade since The Show from Ze Frank graced our tubes with its daily updates. Five years ago to the day, he recorded the greatest three minutes of speech ever committed to video.

In the midst of his challenge to find the ugliest MySpace page ever, he received this comment:

Having an ugly Myspace contest is like having a contest to see who can eat the most cheeseburgers in 24 hours… You’re mocking people who, for the most part, have no taste or artistic training.

Ze’s response is a manifesto to the democratic transformative disruptive power of the web. It is magnificent.

In Myspace, millions of people have opted out of pre-made templates that “work” in exchange for ugly. Ugly when compared to pre-existing notions of taste is a bummer. But ugly as a representation of mass experimentation and learning is pretty damn cool.

Regardless of what you might think, the actions you take to make your Myspace page ugly are pretty sophisticated. Over time as consumer-created media engulfs the other kind, it’s possible that completely new norms develop around the notions of talent and artistic ability.

Spot on.

That’s one of the reasons why I dread the inevitable GeoCities-style shutdown of MySpace. Let’s face it, it’s only a matter of time. And when it does get shut down, we will forever lose a treasure trove of self-expression on a scale never seen before in the history of the planet. That’s so much more important than whether it’s ugly or not. As Phil wrote about the ugly and neglected fragments of Geocities:

GeoCities is an awful, ugly, decrepit mess. And this is why it will be sorely missed. It’s not only a fine example of the amateur web vernacular but much of it is an increasingly rare example of a period web vernacular. GeoCities sites show what normal, non-designer, people will create if given the tools available around the turn of the millennium.

Substitute MySpace for GeoCities and you get an idea of the loss we are facing.

Let’s not make the same mistake twice.

Digital Deathwatch

The Deatchwatch page on the Archive Team website makes for depressing reading, filled as it is with an ongoing list of sites that are going to be—or have already been—shut down. There are a number of corporations that are clearly repeat offenders: Yahoo!, AOL, Microsoft. As Aaron said last year when speaking of Museums and the Web:

Whether or not they asked to be, entire communities are now assuming that those companies will not only preserve and protect the works they’ve entrusted or the comments and other metadata they’ve contributed, but also foster their growth and provide tools for connecting the threads.

These are not mandates that most businesses take up willingly, but many now find themselves being forced to embrace them because to do otherwise would be to invite a betrayal of the trust of their users, from which they might never recover.

But occasionally there is a glimmer of hope buried in the constant avalanche of shit from these deletionist third-party custodians of our collective culture. Take Google Video, for example.

Earlier this year, Google sent out emails to Google Video users telling them the service was going to be shut down and their videos deleted as of April 29th. There was an outcry from people who rightly felt that Google were betraying their stated goal to organize the world‘s information and make it universally accessible and useful. Google backtracked:

Google Video users can rest assured that they won’t be losing any of their content and we are eliminating the April 29 deadline. We will be working to automatically migrate your Google Videos to YouTube. In the meantime, your videos hosted on Google Video will remain accessible on the web and existing links to Google Videos will remain accessible.

This gives me hope. If the BBC wish to remain true to their mission to enrich people’s lives with programmes and services that inform, educate and entertain, then they will have to abandon their plan to destroy 172 websites.

There has been a stony silence from the BBC on this issue for months now. Ian Hunter—who so proudly boasted of the planned destruction—hasn’t posted to the BBC blog since writing a follow-up “clarification” that did nothing to reassure any of us.

It could be that they’re just waiting for a nice quiet moment to carry out the demolition. Or maybe they’ve quietly decided to drop their plans. I sincerely hope that it’s the second scenario. But, just in case, I’ve begun to create my own archive of just some of the sites that are on the BBC’s death list.

By the way, if you’re interested in hearing more about the story of Archive Team, I recommend checking out these interviews and talks from Jason Scott that I’ve huffduffed.

All Our Yesterdays: the links

If you were at An Event Apart in Boston and you want to follow up on some of the things I mentioned in my talk, here are some links:

Here are some related posts of my own:

More recently, Nora Young interviewed Jason Scott on online video and digital heritage.

Full Interview: Jason Scott on online video and digital heritage | Spark | CBC Radio on Huffduffer

South by south met

South by Southwest Interactive is over for another year. Contrary to some of my expectations, it was quite wonderful.

Yes, there were plenty of social media marketing douchebags thrusting schwag and spouting pitches, but there were also shedloads of enthusiastic friendly geeks eager to hang out and share ideas.

Knowing how big the event had grown, I thought I might have trouble seeing all my friends, but I was pleasantly surprised. Instead of running around in a mad dash to see everything and meet everyone, I took things nice’n’slow and ended up meeting up with everyone anyway.

Southby is a great opportunity for me to meet up with peers that I haven’t seen in a year, but it’s an even greater opportunity to meet with new people. This year I met some of my idols in Austin, like David Baron and Fantasai from the CSS Working Group—the unsung heroes of web standards.

I also met the Jason Scott: head of Archive Team and custodian of Sockington the cat. We got together and geeked out about digital preservation with inevitable anger and vehemence when discussing the fate of Geocities or the current plan by the BBC, the technical term for which is “a dick move.”

I highly recommend that you set aside twenty minutes to listen to Jason’s talk from the Personal Digital Conference. It will entertain and energise you.

The Spendiferous Story of Archive Team on Huffduffer

The long prep

The secret to a good war movie is not in the depiction of battle, but in the depiction of the preparation for battle. Whether the fight will be for Agincourt, Rourke’s Drift, Helm’s Deep or Hoth, it’s the build-up that draws you in and makes you care about the outcome of the upcoming struggle.

That’s what 2011 has felt like for me so far. I’m about to embark on a series of presentations and workshops in far-flung locations, and I’ve spent the first seven weeks of the year donning my armour and sharpening my rhetorical sword (so to speak). I’ll be talking about HTML5, responsive design, cultural preservation and one web; subjects that are firmly connected in my mind.

It all kicks off in Belgium. I’ll be taking a train that will go under the sea to get me to Ghent, location of the Phare conference. There I’ll be giving a talk called All Our Yesterdays.

This will be non-technical talk, and I’ve been given carte blanche to get as high-falutin’ and pretentious as I like …though I don’t think it’ll be on quite the same level as my magnum opus from dConstruct 2008, The System Of The World.

Having spent the past month researching and preparing this talk, I’m looking forward to delivering it to a captive audience. I submitted the talk for consideration to South by Southwest also, but it was rejected so the presentation in Ghent will be a one-off. The SXSW rejection may have been because I didn’t whore myself out on Twitter asking for votes, or it may have been because I didn’t title the talk All Our Yesterdays: Ten Ways to Market Your Social Media App Through Digital Preservation.

Talking about the digital memory hole and the fragility of URLs is a permanently-relevant topic, but it seems particularly pertinent given the recent moves by the BBC. But I don’t want to just focus on what’s happening right now—I want to offer a long-zoom perspective on the web’s potential as a long-term storage medium.

To that end, I’ve put my money where my mouth is—$50 worth so far—and placed the following prediction on the Long Bets website:

The original URL for this prediction ( will no longer be available in eleven years.

If you have faith in the Long Now foundation’s commitment to its URLs, you can challenge my prediction. We shall then agree the terms of the bet. Then, on February 22nd 2022, the charity nominated by the winner will receive the winnings. The minimum bet is $200.

If I win, it will be a pyrrhic victory, confirming my pessimistic assessment.

If I lose, my faith in the potential longevity of URLs will be somewhat restored.

Depending on whether you see the glass as half full or half empty, this means I’m either entering a win/win or lose/lose situation.

Care to place a wager?

Voice of the Beeb hive

Ian Hunter at the BBC has written a follow-up post to his initial announcement of the plans to axe 172 websites. The post is intended to clarify and reassure. It certainly clarifies, but it is anything but reassuring.

He clarifies that, yes, these websites will be taken offline. But, he reassures us, they will be stored …offline. Not on the web. Without URLs. Basically, they’ll be put in a hole in the ground. But it’s okay; it’s a hole in the ground operated by the BBC, so that’s alright then.

The most important question in all of this is why the sites are being removed at all. As I said, the BBC’s online mothballing policy has—up till now—been superb. Well, now we have an answer. Here it is:

But there still may come a time when people interested in the site are better served by careful offline storage.

There may be a parallel universe where that sentence makes sense, but it would have to be one in which the English language is used very differently.

As an aside, the use of language in the “explanation” is quite fascinating. The post is filled with the kind of mealy-mouthed filler words intended to appease those of us who are concerned that this is a terrible mistake. For example, the phrase “we need to explore a range of options including offline storage” can be read as “the sites are going offline; live with it.”

That’s one of the most heartbreaking aspects of all of this: the way that it is being presented as a fait accompli: these sites are going to be ripped from the fabric of the network to be tossed into a single offline point of failure and there’s nothing that we—the license-payers—can do about it.

I know that there are many people within the BBC who do not share this vision. I’ve received some emails from people who worked on some of the sites scheduled for deletion and needless to say, they’re not happy. I was contacted by an archivist at the BBC, for whom this plan was unwelcome news that he first heard about here on The subsequent reaction was:

It was OK to put a videotape on a shelf, but putting web pages offline isn’t OK.

I hope that those within the BBC who disagree with the planned destruction will make their voices heard. For those of us outside the BBC, it isn’t clear how we can best voice our concerns. You could make a complaint to the BBC, though that seems to be intended more for complaints about programme content.

In the meantime, you can download all or some of the 172 sites and plop them elsewhere on the web. That’s not an ideal solution—ideally, the BBC shouldn’t be practicing a deliberate policy of link rot—but it allows us to prepare for the worst.

I hope that whoever at the BBC has responsibility for this decision will listen to reason. Failing that, I hope that we can get a genuine explanation as to why this is happening, because what’s currently being offered up simply doesn’t cut it. Perhaps the truth behind this decision lies not so much with the BBC, but with their technology partner, Siemens, who have a notorious track record for shafting the BBC, charging ludicrous amounts of money to execute the most trivial of technical changes.

If this decision is being taken for political reasons, I would hope that someone at the BBC would have the honesty to say so rather than simply churning out more mealy-mouthed blog posts devoid of any genuine explanation.


Yesterday’s account of the BBC’s decision to cull 172 websites caused quite a stir on Twitter.

Most people were as saddened as I was, although Emma described my post as being “anti-BBC.” For the record, I’m a big fan of the BBC—hence my disappointment at this decision. And, also for the record, I believe anyone should be allowed to voice their criticism of an organisational decision without being labelled “anti” said organisation …just as anyone should be allowed to criticise a politician without being labelled unpatriotic.

It didn’t take long for people to start discussing an archiving effort, which was heartening. I started to think about the best way to coordinate such an effort; probably a wiki. As well as listing handy archiving tools, it could serve as a place for people to claim which sites they want to adopt, and point to their mirrors once they’re up and running. Marko already has a head start. Let’s do this!

But something didn’t feel quite right.

I reached out to Jason Scott for advice on coordinating an effort like this. He has plenty of experience. He’s currently trying to figure out how to save the more than 500,000 videos that Yahoo is going to delete on March 15th. He’s more than willing to chat, but he had some choice words about the British public’s relationship to the BBC:

This is the case of a government-funded media group deleting. In other words, this is something for The People, and by The People I mean The Media and the British and the rest to go HEY BBC STOP

He’s right.

Yes, we can and should mirror the content of those 172 sites—lots of copies keep stuff safe—but fundamentally what we want is to keep the fabric of the web intact. Cool URIs don’t change.

The BBC has always been an excellent citizen of the web. Their own policy on handling outdated content explains the situation beautifully:

We don’t want to delete pages which users may have bookmarked or linked to in other ways.

Moving a site to a different domain will save the content but it won’t preserve the inbound connections; the hyperlinks that weave the tapestry of the web together.

Don’t get me wrong: I love the Internet Archive. I think that is doing fantastic work. But let’s face it; once a site only exists in the archive, it is effectively no longer a part of the living web. Yet, whenever a site is threatened with closure, we invoke the Internet Archive as a panacea.

So, yes, let’s make and host copies of the 172 sites scheduled for termination, but let’s not get distracted from the main goal here. What we are fighting against is .

I don’t want the BBC to take any particular action. Quite the opposite: I want them to continue with their existing policy. It will probably take more effort for them to remove the sites than to simply let them sit there. And let’s face it, it’s not like the bandwidth costs are going to be a factor for these sites.

Instead, many believe that the BBC’s decision is politically motivated: the need to be seen to “cut” top level directories, as though cutting content equated to cutting costs. I can’t comment on that. I just know how I feel about the decision:

I don’t want them to archive it. I just want them to leave it the fuck alone.

“What do we want?” “Inaction!”

“When do we want it?” “Continuously!”

Erase and rewind

In the 1960s and ’70s, it was common practice at the BBC to reuse video tapes. Old recordings were taped over with new shows. Some Doctor Who episodes have been lost forever. Jimi Hendrix’s unruly performance on Happening for Lulu would have also been lost if a music-loving engineer hadn’t sequestered the tapes away, preventing them from being over-written.

Except - a VT engineer called Bob Pratt, who really ought to get a medal, was in the habit of saving stuff he liked. Even then, the BBC policy of wiping practically everything was notorious amongst those who’d made it. Bob had the job of changing the heads on 2” VT machines. He’d be in at 0600 before everyone else and have two hours to sort the equipment before anyone else came in. Rock music was his passion, and knowing everything would soon disappear, would spend some of that time dubbing off the thing he liked onto junk tapes, which would disappear under the VT department floor.

To be fair to the BBC, the tape-wiping policy wasn’t entirely down to crazy internal politics—there were convoluted rights issues involving the actors’ union, Equity.

Those issues have since been cleared up. I’m sure the BBC has learned from the past. I’m sure they wouldn’t think of mindlessly throwing away content, when they have such an impressive archive.

And yet, when it comes to the web, the BBC is employing a slash-and-burn policy regarding online content. 172 websites are going to disappear down the memory hole.

Just to be clear, these sites aren’t going to be archived. They are going to be deleted from the web. Server space is the new magnetic tape.

This callous attitude appears to be based entirely on the fact that these sites occupy URLs in top-level directories—repeatedly referred to incorrectly as top level domains on the BBC internet blog—a space that the decision-makers at the BBC are obsessed with.

Instead of moving the sites to, say, and employing a little bit of .htaccess redirection, the BBC (and their technology partner, Siemens) would rather just delete the lot.

Martin Belam is suitably flabbergasted by the vandalism of the BBC’s online history:

I’m really not sure who benefits from deleting the Politics 97 site from the BBC’s servers in 2011. It seems astonishing that for all the BBC’s resources, it may well be my blog posts from 5 years ago that provide a more accurate picture of the BBC’s early internet days than the Corporation does itself - and that it will have done so by choice.

Many of the 172 sites scheduled for deletion are currently labelled with a banner across the top indicating that the site hasn’t been updated for a while. There’s a link to a help page with the following questions and answers:

It’ll be interesting to see how those answers will be updated to reflect change in policy. Presumably, the new answers will read something along the lines of “Fuck ‘em.”

Kiss them all goodbye. And perhaps most egregious of all, you can also kiss goodbye to WW2 People’s War:

The BBC asked the public to contribute their memories of World War Two to a website between June 2003 and January 2006. This archive of 47,000 stories and 15,000 images is the result.

I’m very saddened to see the BBC join the ranks of online services that don’t give a damn for posterity. That attitude might be understandable, if not forgivable, from a corporation like Yahoo or AOL, driven by short-term profits for shareholders, as summarised by Connor O’Brien in his superb piece on link rot:

We push our lives into the internet, expecting the web to function as a permanent and ever-expanding collective memory, only to discover the web exists only as a series of present moments, every one erasing the last. If your only photo album is Facebook, ask yourself: since when did a gratis web service ever demonstrate giving a flying fuck about holding onto the past?

I was naive enough to think that the BBC was above that kind of short-sighted approach. Looks like I was wrong.

Sad face.

Home-grown and Delicious

I’ve been using Delicious since 2005—back when it was I have over 2,000 bookmarks stored there. I moved to Magnolia for a while but we all know how that ended.

Back then I wrote:

Really, I should be keeping my links here on, maybe pinging Delicious or some other social bookmarking site as a back-up.

Recently Delicious updated its bookmarklet-conjured interface, not for the better. I thought that I could get used to the changes, but I found them getting more annoying over time. Once again, I began to toy with the idea of self-hosting my bookmarks. I even exported all my data into a big XML file.

The very next day, some of Yahoo’s shit hit the web’s fan. Delicious, it was revealed, was to be sunsetted. As someone who doesn’t randomly choose to use meteorological phenomena as verbs, I didn’t know what that meant, but it didn’t sound good.

As the twittersphere erupted in anger and indignation, I was able to share my recently-acquired knowledge:

curl https://{your username}:{your password} to get an XML file of your Delicious bookmarks.

A lot of people immediately migrated to Pinboard, which looks like an excellent service (and happens to be the work of Maciej Ceglowski, one of the best bloggers ever to put pixels to screen).

After all that, it turns out that “sunsetting” doesn’t mean “shooting in the head”, it means something more like “flogging off”, as clarified on the Delicious blog. But the damage had been done and, anyway, I had already made up my mind to bring my bookmarks in-house, so I began a fun weekend of hacking.

Setting up a new section of the site for links and importing my Delicious bookmarks was pretty straightforward. Creating a bookmarklet was pretty easy too—I already some experience of that with Huffduffer.

So now I’ll do my bookmarking right here on my own site. All’s well that ends well, right?

Well, not quite. Dom sounded a note of concern:

sigh. There goes the one thing I actually used delicious for, the social network. :(

Paul also pointed to the social aspect as the reason why he’s sticking with Delicious:

Personally, while I’ve always valued the site for its ability to store stuff, what’s always made Delicious most useful to me is its network pages in general, and mine in particular.

But it’s possible to have your Delicious cake and eat it at home. The Delicious API makes it quite easy to post links so I’ve added that into my own bookmarking code. Whenever I post a link here, it will also show up on my Delicious account. If you’re subscribed to my Delicious links, you should notice no change whatsoever.

This is exactly what Steven Pemberton was talking about when I liveblogged his XTech talk two years ago. Another Stephen, the good Mr. Hay, summed up the absurdity of the usual situation:

For a while we’ve posted our data all over the internet on all types of services. These services provide APIs so we can access the data we put into them, so that we can do things with that data. Read that again.

Now I’m hosting the canonical copies of my bookmarks, much like Tantek hosts the canonical copies of his tweets and syndicates them out to Twitter. Delicious gets to have my links as well, and I get to use Delicious as a tool for interacting with my data …only now I’m not limited to just what Delicious can offer me.

Once I had my new links section up and running, I started playing around with the Embedly API (I recently added the excellent oEmbed format to Huffduffer and I was impressed with its power). Whenever I bookmark a page with oEmbed support, I can pull content directly into my site. Take a look at the links I’ve tagged with “sci-fi” to see some examples of embedded Vimeo and Flickr content.

I definitely prefer this self-hosting-with-syndication way of doing things. I can use a service like Delicious without worrying about it going tits-up and taking all my data with it. The real challenge is going to be figuring out a way of applying that model to Twitter and Flickr. I’m curious to see which milestone I’ll hit first: 10,000 tweets or 10,000 photos. Either way, that’s a lot of my content on somebody else’s servers.

Slight return

Tantek is bringing back the blog after skipping an entire year:

I had gone from owning (most of) my content, to digital sharecropping. The past two years I watched life-changing, brilliant, and some long-lived sites get killed by owners that knew not what they had, or just gave up.

Leo Laporte is doing the same:

I feel like I’ve woken up to a bad social media dream in terms of the content I’ve put in others’ hands. It’s been lost, and apparently no one was even paying attention to it in the first place. I should have been posting it here all along.

I approve of this ongoing process of Pembertonisation.

Facing the future

There is much hand-wringing in the media about the impending death of journalism, usually blamed on the rise of the web or more specifically bloggers. I’m sympathetic to their plight, but sometimes journalists are their own worst enemy, especially when they publish badly-researched articles that fuel moral panic with little regard for facts (if you’ve ever been in a newspaper article yourself, you’ll know that you’re lucky if they manage to spell your name right).

Exhibit A: an article published in The Guardian called How I became a Foursquare cyberstalker. Actually, the article isn’t nearly as bad as the comments, which take ignorance and narrow-mindedness to a new level.

Fortunately Ben is on hand to set the record straight. He wrote Concerning Foursquare and communicating privacy. Far from being a lesser form of writing, this blog post is more accurate than the article it is referencing, helping to balance the situation with a different perspective …and a nice big dollop of facts and research. Ben is actually quite kind to The Guardian article but, in my opinion, his own piece is more interesting and thoughtful.

Exhibit B: an article by Jeffrey Rosen in The New York Times called The Web Means the End of Forgetting. That’s a bold title. It’s also completely unsupported by the contents of the article. The article contains anecdotes about people getting into trouble about something they put on the web, and—even though the consequences for that action played out in the present—he talks about the permanent memory bank of the Web and writes:

The fact that the Internet never seems to forget is threatening, at an almost existential level, our ability to control our identities.

Bollocks. Or, to use the terminology of Wikipedia, citation needed.

Scott Rosenberg provides the necessary slapdown, asking Does the Web remember too much — or too little?:

Rosen presents his premise — that information once posted to the Web is permanent and indelible — as a given. But it’s highly debatable. In the near future, we are, I’d argue, far more likely to find ourselves trying to cope with the opposite problem: the Web “forgets” far too easily.

Exactly! I get irate whenever I hear the truism that the web never forgets presented without any supporting data. It’s right up there with eskimos have fifty words for snow and people in the middle ages thought that the world was flat. These falsehoods are irritating at best. At worst, as is the case with the myth of the never-forgetting web, the lie is downright dangerous. As Rosenberg puts it:

I’m a lot less worried about the Web that never forgets than I am about the Web that can’t remember.

That’s a real problem. And yet there’s no moral panic about the very real threat that, once digitised, our culture could be in more danger of being destroyed. I guess that story doesn’t sell papers.

This problem has a number of thorns. At the most basic level, there’s the issue of . I love the fact that the web makes it so easy for people to publish anything they want. I love that anybody else can easily link to what has been published. I hope that the people doing the publishing consider the commitment they are making by putting a linkable resource on the web.

As I’ve said before, a big part of this problem lies with the DNS system:

Domain names aren’t bought, they are rented. Nobody owns domain names, except ICANN.

I’m not saying that we should ditch domain names. But there’s something fundamentally flawed about a system that thinks about domain names in time periods as short as a year or two.

Then there’s the fact that so much of our data is entrusted to third-party sites. There’s no guarantee that those third-party sites give a rat’s ass about the long-term future of our data. Quite the opposite. The callous destruction of Geocities by Yahoo is a testament to how little our hopes and dreams mean to a company concerned with the bottom line.

We can host our own data but that isn’t quite as easy as it should be. And even with the best of intentions, it’s possible to have the canonical copies wiped from the web by accident. I’m very happy to see services like Vaultpress come on the scene:

Your WordPress site or blog is your connection to the world. But hosting issues, server errors, and hackers can wipe out in seconds what took years to build. VaultPress is here to protect what’s most important to you.

The Internet Archive is also doing a great job but Brewster Kahle shouldn’t have to shoulder the entire burden. Dave Winer has written about the idea of future-safe archives:

We need one or more institutions that can manage electronic trusts over very long periods of time.

The institutions need to be long-lived and have the technical know-how to manage static archives. The organizations should need the service themselves, so they would be likely to advance the art over time. And the cost should be minimized, so that the most people could do it.

The Library of Congress has its Digital Preservation effort. Dan Gillmor reports on the recent three-day gathering of the institution’s partners:

It’s what my technology friends call a non-trivial task, for all kinds of technical, social and legal reasons. But it’s about as important for our future as anything I can imagine. We are creating vast amounts of information, and a lot of it is not just worth preserving but downright essential to save.

There’s an even longer-term problem with digital preservation. The very formats that we use to store our most treasured memories can become obsolete over time. This goes to the very heart of why standards such as HTML—the format I’m betting on—are so important.

Mark Pilgrim wrote about the problem of format obsolescence back in 2006. I found his experiences echoed more recently by Paul Glister, author of the superb Centauri Dreams, one of my favourite websites. He usually concerns himself with challenges on an even longer timescale, like the construction of a feasible means of interstellar travel but he gives a welcome long zoom perspective on digital preservation in Burying the Digital Genome, pointing to a project called PLANETS: Preservation and Long-term Access Through Networked Services.

Their plan involves the storage, not just of data, but of data formats such as JPEG and PDF: the equivalent of a Rosetta stone for our current age. A box containing format-decoding documentation has been buried in a bunker under the Swiss Alps. That’s a good start.

David Eagleman recently gave a talk for The Long Now Foundation entitled Six Easy Steps to Avert the Collapse of Civilization. Step two is Don’t lose things:

As proved by the destruction of the Alexandria Library and of the literature of Mayans and Minoans, “knowledge is hard won but easily lost.”

Long Now: Six Easy Steps to Avert the Collapse of Civilization on Huffduffer

I’m worried that we’re spending less and less time thinking about the long-term future of our data, our culture, and ultimately, our civilisation. Currently we are preoccupied with the real-time web: Twitter, Foursquare, Facebook …all services concerned with what’s happening right here, right now. The Long Now Foundation and Tau Zero Foundation offer a much-needed sense of perspective.

As with that other great challenge of our time—the alteration of our biosphere through climate change—the first step to confronting the destruction of our collective digital knowledge must be to think in terms greater than the local and the present.

The format of The Long Now

In 01992, Tim Berners-Lee wrote a document called HTML Tags.

In September 02001, I started keeping this online journal. Back then, I was storing my data in XML, using a format of my own invention. The XML was converted using PHP into (X)HTML, RSS, and potentially anything else …although the “anything else” part never really materialised.

In February 02006, I switched over to using a MySQL database to store my data as chunks of markup.

In February 02007, Ted wrote about data longevity

To me, being able to completely migrate my data — with minimal bit-rot — from system to system is the key in the never-ending and easily-lost fight to keep my data accessible over the entirety of my life.

He’s using non-binary, well-documented standards to store and structure his data: Atom, HTML and microformats.

Meanwhile, the HTML5 spec began defining error-handling for HTML documents. Ian Hickson wrote:

The original reason I got involved in this work is that I realised that the human race has written literally billions of electronic documents, but without ever actually saying how they should be processed.

I decided that for the sake of our future generations we should document exactly how to process today’s documents, so that when they look back, they can still reimplement HTML browsers and get our data back, even if they no longer have access to Microsoft Internet Explorer’s source code.

In August 2008, Ian Hickson mentioned in an interview that the timeline for HTML5 involves having two complete implementations by 02022. Many web developers were disgusted that such a seemingly far-off date was even being mentioned. My reaction was the opposite. I began to pay attention to HTML5.

HTML is starting to look like a relatively safe bet for data longevity and portability. I’m not sure the same can be said for any particular flavour of database. Sooner rather than later, I should remove the unnecessary layer of abstraction that I’m using to store my data.

This would be my third migration of content. I will take care to head Mark Pilgrim’s advice on data fidelity:

Long-term data preservation is like long-term backup: a series of short-term formats, punctuated by a series of migrations. But migrating between data formats is not like copying raw data from one medium to another.

Fidelity is not a binary thing. Data can gradually degrade with each conversion until you’re left with crap. People think this only affects the analog world, like copying cassette tapes for several generations. But I think digital preservation is actually much harder, in part because people don’t even realize that it has the same issues.

He’s also betting on HTML:

HTML is not an output format. HTML is The Format. Not The Format Of Forever, but damn if it isn’t The Format Of The Now.

I don’t think that any format could ever be The Format Of The Long Now but HTML is the closest we’ve come thus far in the history of computing to having a somewhat stable, human- and machine-readable data format with a decent chance of real longevity.

Beautiful truth

I’ve tried to articulate my feelings about data preservation, digital decay and the loss of our collective culture down the memory hole. I’ve written about Tears in the Rain, Magnoliloss and Linkrot. I’ve spoken about Open Data, The Long Web and All Our Yesterdays.

But all of my words are naught compared to a single piece of writing by Joel Johnson on Gizmodo. It’s called Raiding Eternity. From the memories stored on Flickr, past the seed bank of Svalbard, out to the Voyager golden record, it sweeps and soars in scope …but always with a single moment at its center, a single life, a single death.

Please read it. It is beautiful and it is truthful.

When old age shall this generation waste,
Thou shalt remain, in midst of other woe
Than ours, a friend to man, to whom thou say’st,
Beauty is truth, truth beauty,—that is all
Ye know on earth, and all ye need to know.

John Keats


The geeks of the UK have been enjoying a prime-time television show dedicated to the all things webby. Virtual Revoltution is a rare thing: a television programme about the web made by someone who actually understands the web (Aleks, to be precise).

Still, the four-part series does rely on the usual television documentary trope of presenting its subject matter as a series of yin and yang possibilities. The web: blessing or curse? The web: force for democracy or tool of oppression? Rhetorical questions: a necessary evil or an evil necessity?

The third episode tackles one of the most serious of society’s concerns about our brave new online world, namely the increasing amount of information available to commercial interests and the associated fear that technology is having a negative effect on privacy. Personally, I’m with Matt when he says:

If the end of privacy comes about, it’s because we misunderstand the current changes as the end of privacy, and make the mistake of encoding this misunderstanding into technology. It’s not the end of privacy because of these new visibilities, but it may be the end of privacy because it looks like the end of privacy because of these new visibilities*.

Inevitably, whenever there’s a moral panic about the web, a truism that raises its head is the assertion that The Internets Never Forget:

On the one hand, the Internet can freeze youthful folly and a small transgressions can stick with you for life. So that picture of you drunk and passed out in a skip, or that heated argument you had on a mailing list when you were twenty can come back and haunt you.

Citation needed.

We seem to have a collective fundamental attribution error when it comes to the longevity of data on the web. While we are very quick to recall the instances when a resource remains addressable for a long enough time period to cause embarrassment or shame later on, we completely ignore all the link rot and 404s that is the fate of most data on the web.

There is an inverse relationship between the age of a resource and its longevity. You are one hundred times more likely to find an embarrassing picture of you on the web uploaded in the last year than to find an embarrassing picture of you uploaded ten years ago.

If a potential boss finds a ten-year old picture of you drunk and passed out at a party, that’s certainly a cause for concern. But such an event would be extraordinary rather than commonplace. If that situation ever happened to me, I would probably feel outrage and indignation like anybody else, but I bet that I would also wonder Hmmm, where’s that picture being hosted? Sounds like a good place for off-site backups.

The majority of data uploaded to the web will disappear. But we don’t pay attention to the disappearances. We pay attention to the minority of instances when data survives.

This isn’t anything specific to the web; this is just the way we human beings operate. It doesn’t matter if the national statistics show a decrease in crime; if someone is mugged on your street, you’ll probably be worried about increased crime. It doesn’t matter how many airplanes successfully take off and land; one airplane crash in ten thousand is enough to make us very worried about dying on a plane trip. It makes sense that we’ve taken this cognitive bias with us onto the web.

As for why resources on the web tend to disappear over time, there are two possible reasons:

  1. The resource is being hosted on a third-party site or
  2. The resource is being hosted on an independent site.

The problem with the first instance is obvious. A commercial third-party responsible for hosting someone else’s hopes and dreams will pull the plug as soon as the finances stop adding up.

I’m sure you’ve seen the famous chart of Web 2.0 logos but have seen Meg Pickard’s updated version, adjusted for dead companies?

You cannot rely on a third-party service for data longevity, whether it’s Geocities, Magnolia, Pownce, or anything else.

That leaves you with The Pemberton Option: host your own data.

This is where the web excels: distributed and decentralised data linked together with hypertext. You can still ping third-party sites and allow them access to your data, but crucially, you are in control of the canonical copy (Tantek is currently doing just that, microblogging on his own site and sending copies to Twitter).

Distributed HTML, addressable by URL and available through HTTP: it’s a beautiful ballet that creates the network effects that makes the web such a wonderful creation. There’s just one problem and it lies with the URL portion of the equation.

Domain names aren’t bought, they are rented. Nobody owns domain names, except ICANN. While you get to decide the relative structure of URLs on your site, everything between the colon slash slash and the subsequent slash belongs to ICANN. Centralised. Not distributed.

Cool URIs don’t change but even with the best will in the world, there’s only so much we can do when we are tenants rather than owners of our domains.

In his book Weaving The Web, Sir Tim Berners-Lee mentions that exposing URLs in the browser interface was a throwaway decision, a feature that would probably only be of interest to power users. It’s strange to imagine what the web would be like if we used IP numbers rather than domain names—more like a phone system than a postal system.

But in the age of Google, perhaps domain names aren’t quite as important as they once were. In Japanese advertising, URLs are totally out. Instead they show search boxes with recommended search terms.

I’m not saying that we should ditch domain names. But there’s something fundamentally flawed about a system that thinks about domain names in time periods as short as a year or two. It doesn’t bode well for the long-term stability of our data on the web.

On the plus side, that embarrassing picture of you passed out at a party will inevitably disappear …along with almost everything else on the web.

Tears in the rain

When I first heard that Yahoo were planning to bulldoze Geocities, I was livid. After I blogged in anger, I was taken to task for jumping the gun. Give ‘em a chance, I was told. They may yet do something to save all that history.

They did fuck all. They told what URLs to spider and left it up to them to do the best they could with preserving internet history. Meanwhile, Jason Scott continued his crusade to save as much as he could:

This is fifteen years and decades of man-hours of work that you’re destroying, blowing away because it looks better on the bottom line.

We are losing a piece of internet history. We are losing the destinations of millions of inbound links. But most importantly we are losing people’s dreams and memories.

Geocities dies today. This is a bad day for the internet. This is a bad day for our collective culture. In my opinion, this is also a bad day for Yahoo. I, for one, will find it a lot harder to trust a company that finds this to be acceptable behaviour …despite the very cool and powerful APIs produced by the very smart and passionate developers within the same company.

I hope that my friends who work at Yahoo understand that when I pour vitriol upon their company, I am not aiming at them. Yahoo has no shortage of clever people. But clearly they are down in the trenches doing development, not in the upper echelons making the decision to butcher Geocities. It’s those people, the decision makers, that I refer to as twunts. Fuckwits. Cockbadgers. Pisstards.


As is now traditional, there will be a BarCamp in Brighton straight after dConstruct. This year it’s happening at a new venue, the Old Music Library in the middle of town—right across from the Brighton Dome, venue for dConstruct. The first batch of tickets went on sale yesterday but there’ll be more to come (if you don’t fancy playing web booking roulette, a sure-fire way of getting a ticket is to contribute to sponsoring the event).

If you’re coming to Brighton for dConstruct, I highly recommend staying for the weekend and sleeping over at BarCamp.

If you’re not coming to Brighton for dConstruct, why not? Haven’t you seen the line-up? It’s going to be fantastic.

Here’s one way to get a ticket; add something to the dConstruct time capsule:

Take a look around you. What do you see that you would like to preserve for the future? Take a picture of it, upload that picture to Flickr and tag it with dconstructcapsule.

The ticket you could win is no ordinary ticket. It’s a VIP ticket that will get you into dConstruct itself, two nights in a luxury hotel in the centre of Brighton, and a place at the speakers’ dinner the evening before the conference.

Even without the competition aspect, I think this is a pretty nifty project. People have already posted some great items:

Minidisk Player
This used to be cool. I think it still is.

Red Ring of Death
The infamous red ring of death. A symbol of recreation in the naughties and a beacon of utter despair.

Howarth S2 oboe
…though my oboe is a product of centuries of instrument making techniques and technology rather than something new, it’s certainly something (along with the skills that made it) that I believe needs preserving for the future as an example of beautiful design and craft.

time capsule banana
Clever future-people! Please clone this fruit—it’s a design classic (iconic styling, great usability), it’s nutritious, and it’s tastier than the bland efficiency-gruel you slurp down the rest of the space-week.

Now it’s your turn. What would you add to the dConstruct time capsule.

The Death and Life of Geocities

They’re trying to keep it quiet but Yahoo are planning to destroy their Geocities property. All those URLs, all that content, all those memories will be lost …like tears in the rain.

Jason Scott is mobilising but he needs help:

I can’t do this alone. I’m going to be pulling data from these twitching, blood-in-mouth websites for weeks, in the background. I could use help, even if we end up being redundant. More is better. We’re in #archiveteam on EFnet. Stop by. Bring bandwidth and disks. Help me save Geocities. Not because we love it. We hate it. But if you only save the things you love, your archive is a very poor reflection indeed.

I’m seething with anger. I hope I can tap into that anger to do something productive. This situation cannot stand. It reinforces my previously-stated opinion that Yahoo is behaving like a dribbling moronic company.

You may not care about Geocities. Keep in mind that this is the same company that owns Flickr, Upcoming, Delicious and Fire Eagle. It is no longer clear to me why I should entrust my data to silos owned by a company behaving in such an irresponsible, callous, cold-hearted way.

What would Steven Pemberton do?

Update: As numerous Yahoo employees are pointing out on Twitter, no data has been destroyed yet; no links have rotted. My toys-from-pram-throwage may yet prove to be completely unfounded. Jim invokes , seeing parallels with amazonfail, so overblown is my moral outrage. Fair point. I should give Yahoo time to prove themselves worthy guardians. As a customer of Yahoo’s other services, and as someone who cares about online history, I’ll be watching to see how Yahoo deals with this situation and I hope they deal with it well (archiving data, redirecting links).

Like I said above, I hope I can turn my anger into something productive. Clearly I’m not doing a very good job of that right now.

All Our Yesterdays

I’m back from spending a weekend in Cornwall at the inaugural Bamboo Juice conference, held in the inspiring surroundings of the Eden Project.

I opened up proceedings with a talk entitled All Our Yesterdays. I know it’s the title of a Star Trek episode, but I actually had Shakespeare in mind:

To-morrow, and to-morrow, and to-morrow,
Creeps in this petty pace from day to day,
To the last syllable of recorded time;
And all our yesterdays have lighted fools
The way to dusty death. Out, out, brief candle!

Usually my presentations follow a linear narrative but this was a rambling, self-indulgent affair. So I used a non-linear presentation tool this time; the Flash-based Prezi. You can view the presentation at

I can’t really summarise the presentation—you kinda had to be there—but there were two main points:

  1. Think about what you would put on attached to Voyager; now publish that material online.
  2. Use web standards so that we can build a .

Along the way I took in the history of writing from the Rosetta stone to the Gutenberg press via the Book of Kells, potted bios of Leibniz, Babbage and Turing, the alternative hypertext systems of Vannevar Bush and Ted Nelson, and a fairly emotional rant about the ludicrous state of affairs in the world of copyright and so-called intellectual property. There’s a bibliography of further reading tucked into the corner of the presentation:

URLs mentioned during the presentation include:

These are some of the historically important geographical locations I mentioned:

There were three video excerpts in the presentation:

My disjointed ravings on cultural preservation and space exploration would have seemed far-fetched in any other setting but after the talk, when I was wandering through the buildings of the , they seemed positively tame.

If you were at Bamboo Juice, I hope you liked the talk. If you weren’t there, sorry; you missed a beautiful day at the geodesic domes.