Tags: caching

30

sparkline

Thursday, November 2nd, 2017

The dConstruct Audio Archive works offline

The dConstruct conference is as old as Clearleft itself. We put on the first event back in 2005, the year of our founding. The last dConstruct was in 2015. It had a good run.

I’m really proud of the three years I ran the show—2012, 2013, and 2014—and I have great memories from each event. I’m inordinately pleased that the individual websites are still online after all these years. I’m equally pleased with the dConstruct audio archive that we put online in 2012. Now that the event itself is no longer running, it truly is an archive—a treasury of voices from the past.

I think that these kinds of online archives are eminently suitable for some offline design. So I’ve added a service worker script to the dConstruct archive.

Caching

To start with, there’s the no-brainer: as soon as someone hits the website, pre-cache static assets like CSS, JavaScript, the logo, and icon images. Now subsequent page loads will be quicker—those assets are taken straight from the cache.

But what about the individual pages? For something like Resilient Web Design—another site that won’t be updated—I pre-cache everything. I could do that with the dConstruct archive. All of the pages with all of the images add up to less than two megabytes; the entire site weighs less than a single page on Wired.com or The Verge.

In the end, I decided to go with a cache-as-you-go strategy. Every time a page or an image is fetched from the network, it is immediately put in a cache. The next time that page or image is requested, the file is served from that cache instead of the network.

Here’s the logic for fetch requests:

  1. First, look to see if the file is in a cache. If it is, great! Serve that.
  2. If the file isn’t in a cache, make a network request and serve the response …but put a copy of a file in the cache.
  3. The next time that file is requested, go to step one.

Save for offline

That caching strategy works great for pages, images, and other assets. But there’s one kind of file on the dConstruct archive that’s a bit different: the audio files. They can be fairly big, so I don’t want to cache those unless the user specifically requests it.

If you end up on the page for a particular talk, and your browser supports service workers, you’ll get an additional UI element in the list of options: a toggle to “save offline” (under the hood, it’s a checkbox). If you activate that option, then the audio file gets put into a cache.

Now if you lose your network connection while browsing the site, you’ll get a custom offline page with the option to listen to any audio files you saved for offline listening. You’ll also see this collection of talks on the homepage, regardless of whether you’ve got an internet connection or not.

So if you’ve got a long plane journey ahead of you, have a browse around the dConstruct archive and select some talks for your offline listening pleasure.

Or just enjoy the speediness of browsing the site.

Turning another website into a Progressive Web App.

Sunday, October 29th, 2017

The meaning of AMP

Ethan quite rightly points out some semantic sleight of hand by Google’s AMP team:

But when I hear AMP described as an open, community-led project, it strikes me as incredibly problematic, and more than a little troubling. AMP is, I think, best described as nominally open-source. It’s a corporate-led product initiative built with, and distributed on, open web technologies.

But so what, right? Tom-ay-to, tom-a-to. Well, here’s a pernicious example of where it matters: in a recent announcement of their intent to ship a new addition to HTML, the Google Chrome team cited the mood of the web development community thusly:

Web developers: Positive (AMP team indicated desire to start using the attribute)

If AMP were actually the product of working web developers, this justification would make sense. As it is, we’ve got one team at Google citing the preference of another team at Google but representing it as the will of the people.

This is just one example of AMP’s sneaky marketing where some finely-shaved semantics allows them to appear far more reasonable than they actually are.

At AMP Conf, the Google Search team were at pains to repeat over and over that AMP pages wouldn’t get any preferential treatment in search results …but they appear in a carousel above the search results. Now, if you were to ask any right-thinking person whether they think having their page appear right at the top of a list of search results would be considered preferential treatment, I think they would say hell, yes! This is the only reason why The Guardian, for instance, even have AMP versions of their content—it’s not for the performance benefits (their non-AMP pages are faster); it’s for that prime real estate in the carousel.

The same semantic nit-picking can be found in their defence of caching. See, they’ve even got me calling it caching! It’s hosting. If I click on a search result, and I am taken to page that has a URL beginning with https://www.google.com/amp/s/... then that page is being hosted on the domain google.com. That is literally what hosting means. Now, you might argue that the original version was hosted on a different domain, but the version that the user gets sent to is the Google copy. You can call it caching if you like, but you can’t tell me that Google aren’t hosting AMP pages.

That’s a particularly low blow, because it’s such a bait’n’switch. One of the reasons why AMP first appeared to be different to Facebook Instant Articles or Apple News was the promise that you could host your AMP pages yourself. That’s the very reason I first got interested in AMP. But if you actually want the benefits of AMP—appearing in the not-search-results carousel, pre-rendered performance, etc.—then your pages must be hosted by Google.

So, to summarise, here are three statements that Google’s AMP team are currently peddling as being true:

  1. AMP is a community project, not a Google project.
  2. AMP pages don’t receive preferential treatment in search results.
  3. AMP pages are hosted on your own domain.

I don’t think those statements are even truthy, much less true. In fact, if I were looking for the right term to semantically describe any one of those statements, the closest in meaning would be this:

A statement used intentionally for the purpose of deception.

That is the dictionary definition of a lie.

Update: That last part was a bit much. Sorry about that. I know it’s a bit much because The Register got all gloaty about it.

I don’t think the developers working on the AMP format are intentionally deceptive (although they are engaging in some impressive cognitive gymnastics). The AMP ecosystem, on the other hand, that’s another story—the preferential treatment of Google-hosted AMP pages in the carousel and in search results; that’s messed up.

Still, I would do well to remember that there are well-meaning people working on even the fishiest of projects.

Except for the people working at the shitrag that is The Register.

(The other strong signal that I overstepped the bounds of decency was that this post attracted the pond scum of Hacker News. That’s another place where the “well-meaning people work on even the fishiest of projects” rule definitely doesn’t apply.)

Monday, September 25th, 2017

Network Information API

It looks like this is landing in Chrome. The navigator.connection.type property will allow us to progressively enhance based on connection type:

A web application that makes use of a service worker to cache resources during installation might have different bundles of assets that it might cache: a list of crucial assets that are cached unconditionally, and a bundle of larger, optional assets that are only cached ahead of time when navigator.connection.type is 'ethernet' or 'wifi'.

There are potential security issues around fingerprinting that are addressed in this document.

Friday, September 22nd, 2017

How much storage space is my Progressive Web App using? | Dean Hume

You can use navigator.storage.estimate() to get a (vague) idea of how much space is available on a device for your service worker caches.

Thursday, May 18th, 2017

It’s not working! Should I blame caching?

Finally, the answer to one of the two hard questions in computer science.

Monday, May 1st, 2017

Offline Web Applications | Udacity

This is a free online video course recorded by Jake a couple of years back. It’s got a really good step-by-step introduction to service workers, delivered in Jake’s typically witty way. Some of the details are a bit out of date, and I must admit that I bailed when it got to IndexedDB, but I highly recommend giving this a go.

There’s also a free course on web accessibility I’m planning to check out.

Tuesday, April 18th, 2017

Progressive Web Apps - ILT  |  Web  |  Google Developers

A step-by-step guide to building progressive web apps. It covers promises, service workers, fetch, and cache, but seeing as it’s from Google, it also pushes the app-shell model.

This is a handy resource but I strongly disagree with some of the advice in the section on architectures (the same bit that gets all swoonsome for app shells):

Start by forgetting everything you know about conventional web design, and instead imagine designing a native app.

Avoid overly “web-like” design.

What a horribly limiting vision for the web! After all that talk about being progressive and responsive, we’re told to pretend we’re imitating native apps on one device type.

What’s really disgusting is the way that the Chrome team are withholding the “add to home screen” prompt from anyone who dares to make progressive web apps that are actually, y’know …webby.

Thursday, March 23rd, 2017

Retrofit Your Website as a Progressive Web App — SitePoint

Turning your existing website into a progressive web “app”—a far more appealing prospect than trying to create an entirely new app-shell architecture:

…they are an enhancement of your existing website which should take no longer than a few hours and have no negative effect on unsupported browsers.

Friday, March 17th, 2017

A Little Surprise Is Waiting For You Here — Meet The Next Smashing Magazine

An open beta of Smashing Magazine’s redesign, which looks like it could be a real poster child for progressive enhancement:

We do our best to ensure that content is accessible and enhanced progressively, with performance in mind. If JavaScript isn’t available or if the network is slow, then we deliver content via static fallbacks (for example, by linking directly to Google search), as well as a service worker that persistently stores CSS, JavaScripts, SVGs, font files and other assets in its cache.

Friday, January 27th, 2017

✨Implementing “Save For Offline” with Service Workers | Una Kravets Online✨

A great little script from Una that’s perfect for blogs and news sites—allowing users to explicitly save a page for offline reading.

Sunday, January 15th, 2017

Modernizing our Progressive Enhancement Delivery | Filament Group, Inc., Boston, MA

Scott runs through the latest improvements to the Filament Group website. There’s a lot about HTTP2, but also a dab of service workers (using a similar recipe to my site).

Wednesday, January 11th, 2017

Making Resilient Web Design work offline

I’ve written before about taking an online book offline, documenting the process behind the web version of HTML5 For Web Designers. A book is quite a static thing so it’s safe to take a fairly aggressive offline-first approach. In fact, a static unchanging book is one of the few situations that AppCache works for. Of course a service worker is better, but until AppCache is removed from browsers (and until service worker is supported across the board), I’m using both. I wouldn’t recommend that for most sites though—for most sites, use a service worker to enhance it, and avoid AppCache like the plague.

For Resilient Web Design, I took a similar approach to HTML5 For Web Designers but I knew that there was a good chance that some of the content would be getting tweaked at least for a while. So while the approach is still cache-first, I decided to keep the cache fairly fresh.

Here’s my service worker. It starts with the usual stuff: when the service worker is installed, there’s a list of static assets to cache. In this case, that list is literally everything; all the HTML, CSS, JavaScript, and images for the whole site. Again, this is a pattern that works well for a book, but wouldn’t be right for other kinds of websites.

The real heavy lifting happens with the fetch event. This is where the logic sits for what the service worker should do everytime there’s a request for a resource. I’ve documented the logic with comments:

// Look in the cache first, fall back to the network
  // CACHE
  // Did we find the file in the cache?
      // If so, fetch a fresh copy from the network in the background
      // NETWORK
          // Stash the fresh copy in the cache
  // NETWORK
  // If the file wasn't in the cache, make a network request
      // Stash a fresh copy in the cache in the background
  // OFFLINE
  // If the request is for an image, show an offline placeholder
  // If the request is for a page, show an offline message

So my order of preference is:

  1. Try the cache first,
  2. Try the network second,
  3. Fallback to a placeholder as a last resort.

Leaving aside that third part, regardless of whether the response is served straight from the cache or from the network, the cache gets a top-up. If the response is being served from the cache, there’s an additional network request made to get a fresh copy of the resource that was just served. This means that the user might be seeing a slightly stale version of a file, but they’ll get the fresher version next time round.

Again, I think this acceptable for a book where the tweaks and changes should be fairly minor, but I definitely wouldn’t want to do it on a more dynamic site where the freshness matters more.

Here’s what it usually likes like when a file is served up from the cache:

caches.match(request)
  .then( responseFromCache => {
  // Did we find the file in the cache?
  if (responseFromCache) {
      return responseFromCache;
  }

I’ve introduced an extra step where the fresher version is fetched from the network. This is where the code can look a bit confusing: the network request is happening in the background after the cached file has already been returned, but the code appears before the return statement:

caches.match(request)
  .then( responseFromCache => {
  // Did we find the file in the cache?
  if (responseFromCache) {
      // If so, fetch a fresh copy from the network in the background
      event.waitUntil(
          // NETWORK
          fetch(request)
          .then( responseFromFetch => {
              // Stash the fresh copy in the cache
              caches.open(staticCacheName)
              .then( cache => {
                  cache.put(request, responseFromFetch);
              });
          })
      );
      return responseFromCache;
  }

It’s asynchronous, see? So even though all that network code appears before the return statement, it’s pretty much guaranteed to complete after the cache response has been returned. You can verify this by putting in some console.log statements:

caches.match(request)
.then( responseFromCache => {
  if (responseFromCache) {
      event.waitUntil(
          fetch(request)
          .then( responseFromFetch => {
              console.log('Got a response from the network.');
              caches.open(staticCacheName)
              .then( cache => {
                  cache.put(request, responseFromFetch);
              });
          })
      );
      console.log('Got a response from the cache.');
      return responseFromCache;
  }

Those log statements will appear in this order:

Got a response from the cache.
Got a response from the network.

That’s the opposite order in which they appear in the code. Everything inside the event.waitUntil part is asynchronous.

Here’s the catch: this kind of asynchronous waitUntil hasn’t landed in all the browsers yet. The code I’ve written will fail.

But never fear! Jake has written a polyfill. All I need to do is include that at the start of my serviceworker.js file and I’m good to go:

// Import Jake's polyfill for async waitUntil
importScripts('/js/async-waituntil.js');

I’m also using it when a file isn’t found in the cache, and is returned from the network instead. Here’s what the usual network code looks like:

fetch(request)
  .then( responseFromFetch => {
    return responseFromFetch;
  })

I want to also store that response in the cache, but I want to do it asynchronously—I don’t care how long it takes to put the file in the cache as long as the user gets the response straight away.

Technically, I’m not putting the response in the cache; I’m putting a copy of the response in the cache (it’s a stream, so I need to clone it if I want to do more than one thing with it).

fetch(request)
  .then( responseFromFetch => {
    // Stash a fresh copy in the cache in the background
    let responseCopy = responseFromFetch.clone();
    event.waitUntil(
      caches.open(staticCacheName)
      .then( cache => {
          cache.put(request, responseCopy);
      })
    );
    return responseFromFetch;
  })

That all seems to be working well in browsers that support service workers. For legacy browsers, like Mobile Safari, there’s the much blunter caveman logic of an AppCache manifest.

Here’s the JavaScript that decides whether a browser gets the service worker or the AppCache:

if ('serviceWorker' in navigator) {
  // If service workers are supported
  navigator.serviceWorker.register('/serviceworker.js');
} else if ('applicationCache' in window) {
  // Otherwise inject an iframe to use appcache
  var iframe = document.createElement('iframe');
  iframe.setAttribute('src', '/appcache.html');
  iframe.setAttribute('style', 'width: 0; height: 0; border: 0');
  document.querySelector('footer').appendChild(iframe);
}

Either way, people are making full use of the offline nature of the book and that makes me very happy indeed.

A Tale of Four Caches · Yoav Weiss

A cute explanation of different browser caches:

  • memory cache,
  • service worker cache,
  • disk cache, and
  • push cache.

Sunday, December 4th, 2016

Service Worker, what are you? - Mariko Kosaka

This is a fun—and accurate—explanation of service workers.

There’s definitely something “alien” about a service worker—it’s kind of like a virus that gets installed on the user’s device. I’ve taken to describing it as “a man-in-the-middle attack on your own website” which makes sound a bit scarier than is necessary.

Sunday, October 16th, 2016

The Service Worker Lifecycle  |  Web  |  Google Developers

Jake goes into the details of what exactly is happening when a service worker is installed or replaced.

This is easily the most complex part of working with service workers, and I think I’m beginning to wrap my head around it, but the good news is that, for the most part, you don’t really need to know the ins and outs of this to get started (and dev tools are now making it easier to nuke from orbit if this begins to bite).

Wednesday, September 21st, 2016

Progressive Web Apps Simply Make Sense - Cloud Four

Progressive Web Apps versus native is the wrong question because every step on the path to a Progressive Web App makes sense on its own, irrespective of what a company does with their native apps.

Not all of your customers are going to have your app installed. For those who visit via the web, providing them with a better experience will make them happier and generate more revenue for your business.

It’s really that simple.

SpeedCurve | PWA Performance

Steve describes a script you can use on WebPageTest to simulate going offline so you can test how your progressive web app performs.

Monday, September 19th, 2016

An intro to progressive web apps | 8th Light

A nice introduction to progressive web apps. There’s a little bit of confusion about permissions—whether a site has been added to the home screen or not has no effect on the permissions granted to it (for things like push notifications)—but the wrap-up nails the advantages of using the web:

No more waiting to download an app, no more prompts for updating an app. From a developer perspective, it means we will be able to iterate a lot quicker. We don’t need to wait for app store approvals anymore, and we can deploy at our own leisure.

Another advantage that a progressive web app has over a native mobile app is that it is linkable, hence it is easier to share and, probably even more importantly, can be indexed by search engines. This makes discoverability of the app a lot better.

Cross-origin Service Workers: Experimenting with Foreign Fetch | Web Updates - Google Developers

This one is definitely for service worker nerds only. I’ve been trying to get my head around this idea of “foreign fetch” which allows third parties to install service workers—could be handy for sites with APIs like Huffduffer and The Session. This article does a good job of explaining the somewhat tangled process.

lyzadanger/pragmatist-service-worker: Pragmatist’s Guide to Service Worker

Lyza put together some example code for her Smashing Conference talk on service workers. If you haven’t written a service worker before, these are really nice examples of how to grok it bit by bit.