Tags: test

20

sparkline

Summer of Apollo

It’s July, 2019. You know what that means? The 50th anniversary of the Apollo 11 mission is this month.

I’ve already got serious moon fever, and if you’d like to join me, I have some recommendations…

Watch the Apollo 11 documentary in a cinema. The 70mm footage is stunning, the sound design is immersive, the music is superb, and there’s some neat data visualisation too. Watching a preview screening in the Duke of York’s last week was pure joy from start to finish.

Listen to 13 Minutes To The Moon, the terrific ongoing BBC podcast by Kevin Fong. It’s got all my favourite titans of NASA: Michael Collins, Margaret Hamilton, and Charlie Duke, amongst others. And it’s got music by Hans Zimmer.

Experience the website Apollo 11 In Real Time on the biggest monitor you can. It’s absolutely wonderful! From July 16th, you can experience the mission timeshifted by exactly 50 years, but if you don’t want to wait, you can dive in right now. It genuinely feels like being in Mission Control!

From Ideation to Iteration: Design Thinking for Work and for Life by Una Kravets

The unstoppable engine of An Event Apart in Seattle rolls onward. The second talk of the second day is from the indominatable Una Kravets. Her talk is called From Ideation to Iteration: Design Thinking for Work and for Life. Here’s the description:

Have you ever had a looming deadline and no idea where to start? Do you have a big task to face but are having trouble figuring out how to get there? Have you ever wanted to learn a technology, or build a side project but didn’t know what to build? In this talk, Una will go over an actionable approach and several techniques for applying design thinking to our work and every aspect of our lives. This includes ideating product features, blog post ideas, or even what general direction we want to move toward in our businesses. We’ll go over traditional approaches and breakout techniques that will leave you feeling more in control and ready to reach your goals.

Let’s see if I can keep up with this…

Una’s going to talk about design thinking. Una does a variety of different work outside her day job—a podcast, dev doodles, etc. Sometimes at work she’s given big, big tasks like “build a design system.” Her reaction is “whut?” How do you even start with a task like that.

Also, we make big goals sometimes. Who makes new year’s resolutions? But what does “get more fit” or “earn more income” even mean?

In this talk, Una will break things down and show how design thinking can be applied to anything.

Design thinking

A stategic, solution-based approach to solving problems.

It’s a process. It’s iterative. It’s used by IBM, Apple, GE, and it’s taught to students at a lot of different universities.

Tim Brown of Ideo points out that there’s a Venn diagram of feasability, desirability, and viability. In the middle is the point of innovation.

The steps are:

  1. Empathise — develop a deep understanding of the challenge
  2. Define — clearly articulate the problem.
  3. Ideate — brainstorm potential solutions.
  4. Prototype — design a protoype to
  5. Test — and iterate.

Una feels that the feedback part is potentially missing there. IBM uses a loop diagram to include feedback. Ideo uses these steps:

  1. Frame a question
  2. Gather inspiration
  3. Generate ideas
  4. Make ideas tangible
  5. Test to learn
  6. Share the story

Another way to think about this is how the teams interact. There’s divergence and convergence throughout. Then there’s the double diamond: design, deliver, discover.

Ideo wrote a book called Design Thinking for Libraries. It has some useful tools and diagrams. Una found this fascinating because it wasn’t specifically about products. In healthcare, GE Health used design thinking for their Adventure Series MRI scanners—it resulted in 80% less need to use sedatives. The solution might seem obvious to us in hindsight, but it wouldn’t have been obvious to medical professionals in their everyday busy lives.

Design thinking is bullshit, says Natasha Jen. She describes how it’s become an over-used term that has lost its value. Una can relate—she gets annoyed when there’s too much talking and not enough doing. Design thinking is not diagrams and sticky notes. It’s a process. It’s very much about doing something to shift perspective. It’s another tool in our toolkit, even if it has become an overused term like “synergy.”

Back in 2014, when Una was working at IBM, she thought design thinking was stupid. It seemed to be all talk, talk, talk. It felt tedious. It was 75% talking and 25% development. The balance wasn’t right.

But it’s also true that solutioning too early leads to cruft. If you end up going back to the drawing board, maybe the time could’ve been better spent doing some design thinking up front. Focus on the problem, not the solution.

Now some developers might be thinking that this is outside their area. But it can really help you in your career. It can help you choose technologies. Also, everyone on the team, regardless of role, is responsible for the product.

1. Empathise

Understand your users and the challenge. This could be a task that a user is trying to accomplish, or it could be you trying to get a raise.

Sometimes we forget who our user is. The techniques in this first step helps us solve their needs, not our needs.

You might have many users that you’re trying to help, but try focusing in on a few. You can create personas. When Una was working at Digital Ocean, the users were developers. The personas reflected this. Do the research to get to know your users.

Next, you can create an empathy map for your users. What are their goals? What are their hopes? What will they gain from your product?

Connect the empathy map to a specific context—a goal and or a scenario that the user is going through.

Bear in mind that there are many layers to your user. There are conscious rational thoughts, but also subconscious emotional thoughts. Empathy mapping helps you understand how to best communicate with your user.

Una shares a real-life scenario of hers: create a new shop-able product that increases time on site. That’s a pretty big goal. She creates a persona for a college-educated woman working in the medical field who commutes on the subway, keeps a skin-caring routing, and is getting into cake-making. Next, Una creates an empathy map for this persona. What she says, thinks, feels, and does. All of this is within the context of browsing your fashion media website casually at work.

2: Define

The problem statement should be:

  • Human-centred,
  • Specific, but not too technical (don’t solutionise too soon),
  • Narrow in scope.

How can we best create a product-highlighting web experience that Rosalyn will enjoy to increase her time on site?

You can use a tool with two columns: As-Is and to-Be. The first column is what users currently do. The second column is what you want to achieve.

3. Ideate

This is the fun part. Good old-fashioned brainstorming is good here. Go for quantity here. Get loads of ideas out.

There’s also a “worst possible ideas” game you can play at this stage. It can be a good ice-breaker.

Have a second round of brainstorming where you play the “yes, and…” game to build on the first round.

When Una was working on The Zoe Report, she found that moodboards were really useful. The iteration cycle was very fast. A moodboard allowed them to skip a lot of the back and forth between design and development. They built the website without any visual design mock-ups. They prototyped quickly, tested quickly, and shipped quickly.

Journey-mapping is the next tool you can use in this ideation phase. Map out the steps between the start and end of a user journey. Keep it simple. This is a great time to refine your product and reduce complexity.

Next, start sketching out ideas. Again, this is a great time to uncover issues and solve problems before things get too defined. But remember, when you’re showcasing your ideas in sketches, too many ideas can lead to analyis paralysis.

Oh dear. Jam. Jam. Jam. Jam. Jam. Yes, Una is using the paradox-of-choice jam example …the study who’s findings could not be reproduced.

4. Prototype

Go forth and build. A prototype can exist on a number of different axes:

  • Representation—the form it takes.
  • Precision—the detail it contains.
  • Interactivity—the extent a user can interact with it.
  • Evolution—the life stage it is at.

There are lot of prototyping tools out there. Prototypr.io helps you find the right tool for you. It breaks things down by fidelity and life cycle.

But not all prototyping has to be digital. Paper prototyping only needs pen, paper, and scissors. Some tips:

  • Use a transparency sheet for forms.
  • Use well-visible and mid-tip pens.
  • Draw up your prototype in black and white—people can get caught up in colour.

But on the web, Una recommends getting to digital as quickly as possible because interaction is such a big part of the experience. That’s why Una likes to prototype in code. But this is still a rapid prototyping phase so don’t get too caught up in the details.

5. Test

Testing with internal teams is fine during the ideation phase, but to understand how users will relate to your product, you need to test with representative people. We are not our users.

As well as the user, have a facilitator, a computer, and a scriber. As a facilitator, it’s a good idea to reduce the amount of input you give a user. Don’t hand-hold too much or you will give away your pre-existing knowledge. Encourage your user to be verbal.

Sessioncam is a tool for creating a heatmap of where people are interacting. There are also tools for tracking clicks or mouse hovers. These all feel so utterly icky to me.

The metrics you might be looking to gather could be click-through rate, time-on-site, etc. But, Una cautions, be very wary of adding all these third-party scripts to your site and slowing it down. Who’s testing the A/B tests?

On Bustle, Una wanted to measure interactions on mobile. They tested different UI elements for interactions. They ended up updating the product with a horizontal swiping component. They were able to improve the product and ship a more refined experience.

6. Review and iterate

Una feels that this step is the most important. Analyse your successes and failures, and plan to improve.

Technology changes over time so what’s feasible and viable also changes.

Design thinking on the daily

You can use design thinking in your everyday life. Maybe you want to learn JavaScript, or write blog posts, or get more fit. Una used design thinking brainstorming to break down her goals, categorise and organise them.

“Get better at JavaScript” is a goal that Una has every year.

Empathise. In this situation, the user is you. You can still create a persona of yourself. Define. Why do you want to get better at JavaScript? Is it about making better use of your time? Ideate. There are so many different ways to learn. There are books and video courses. Or you could have a project to focus on. Break. It. Down! Create actionable steps and define how you will measure progress. Match the list of things you want to learn with the list of possible side projects. Prototype. Don’t take it literally. Just build something. Test. You’re testing on yourself in this case. Review. Una does an annual review. It’s a nice therapeutic exercise and helps her move forward into the next year with actionable goals.

Another goal might be “Write a blog post.”

Empathise. Your users are your potential readers. Who do you have in mind? Make personas. Define. What’s the topic? Ideate. If you don’t know what to write about, brainstorm. What are you working on at work that you’re learning from? Select one to try. Prototype. Write. Test. Maybe show it to co-workers. Review. How did it go? Good? Bad? Refine your process for the next blog post.

Here’s a goal: “Buy a gift for someone.”

Empathise. What does this person like? What have they enjoyed receiving in the past? Define. Is the gift something they’ll enjoy for a long time? Something they can share? Ideate. Bounce ideas off friends and relatives. Prototype. In this case, this means getting the gift. Review. Did they like it?

“Get Fit.”

In this case, the review part is probably the most important part.

Marie Kondo asks “Does this spark joy?” Ask the same question of your goals.

Remember, design thinking is not just about talking, and sticky notes. It’s about getting in the right headspace for your users.

Design thinking matters—because everything we do, we do for people. Having the tools to see through the lens of those people will help you be a more well-rounded person.

Hooked and booked

At Booking.com, they do a lot of A/B testing.

At Booking.com, they’ve got a lot of dark patterns.

I think there might be a connection.

A/B testing is a great way of finding out what happens when you introduce a change. But it can’t tell you why.

The problem is that, in a data-driven environment, decisions ultimately come down to whether something works or not. But just because something works, doesn’t mean it’s a good thing.

If I were trying to convince you to buy a product, or use a service, one way I could accomplish that would be to literally put a gun to your head. It would work. Except it’s not exactly a good solution, is it? But if we were to judge by the numbers (100% of people threatened with a gun did what we wanted), it would appear to be the right solution.

When speaking about A/B testing at Booking.com, Stuart Frisby emphasised why it’s so central to their way of working:

One of the core principles of our organisation is that we want to be very customer-focused. And A/B testing is really a way for us to institutionalise that customer focus.

I’m not so sure. I think A/B testing is a way to institutionalise a focus on business goals—increasing sales, growth, conversion, and all of that. Now, ideally, those goals would align completely with the customer’s goals; happy customers should mean more sales …but more sales doesn’t necessarily mean happy customers. Using business metrics (sales, growth, conversion) as a proxy for customer satisfaction might not always work …and is clearly not the case with many of these kinds of sites. Whatever the company values might say, a company’s true focus is on whatever they’re measuring as success criteria. If that’s customer satisfaction, then the company is indeed customer-focused. But if the measurements are entirely about what works for sales and conversions, then that’s the real focus of the company.

I’m not saying A/B testing is bad—far from it! (although it can sometimes be taken to the extreme). I feel it’s best wielded in combination with usability testing with real users—seeing their faces, feeling their frustration, sharing their joy.

In short, I think that A/B testing needs to be counterbalanced. There should be some kind of mechanism for getting the answer to “why?” whenever A/B testing provides to the answer to “what?” In-person testing could be one way of providing that balance. Or it could be somebody’s job to always ask “why?” and determine if a solution is qualitatively—and not just quantitatively—good. (And if you look around at your company and don’t see anyone doing that, maybe that’s a role for you.)

If there really is a connection between having a data-driven culture of A/B testing, and a product that’s filled with dark patterns, then the disturbing conclusion is that dark patterns work …at least in the short term.

Installing Progressive Web Apps

When I was testing the dConstruct Audio Archive—which is now a Progressive Web App—I noticed some interesting changes in how Chrome on Android offers the “add to home screen” prompt.

It used to literally say “add to home screen.”

Getting the “add to home screen” prompt for https://huffduffer.com/ on Android Chrome. And there’s the “add to home screen” prompt for https://html5forwebdesigners.com/ HTTPS + manifest.json + Service Worker = “Add to Home Screen” prompt. Add to home screen.

Now it simply says “add.”

The dConstruct Audio Archive is now a Progressive Web App

I vaguely remember there being some talk of changing the labelling, but I could’ve sworn it was going to change to “install”. I’ve got to be honest, just having the word “add” doesn’t seem to provide much context. Based on the quick’n’dirty usability testing I did with some co-workers, it just made things confusing. “Add what?” “What am I adding?”

Additionally, the prompt appeared immediately on the first visit to the site. I thought there was supposed to be an added “engagement” metric in order for the prompt to appear; that the user needs to visit the site more than once.

You’d think I’d be happy that users will be presented with the home-screen prompt immediately, but based on the behaviour I saw, I’m not sure it’s a good thing. Here’s what I observed:

  1. The user types the URL archive.dconstruct.org into the address bar.
  2. The site loads.
  3. The home-screen prompt slides up from the bottom of the screen.
  4. The user immediately moves to dismiss the prompt (cue me interjecting “Don’t close that!”).

This behaviour is entirely unsurprising for three reasons:

  1. We web designers and web developers have trained users to dismiss overlays and pop-ups if they actually want to get to the content. Nobody’s going to bother to actually read the prompt if there’s a 99% chance it’s going to say “Sign up to our newsletter!” or “Take our survey!”.
  2. The prompt appears below the “line of death” so there’s no way to tell it’s a browser or OS-level dialogue rather than a JavaScript-driven pop-up from the site.
  3. Because the prompt now appears on the first visit, no trust has been established between the user and the site. If the prompt only appeared on later visits (or later navigations during the first visit) perhaps it would stand a greater chance of survival.

It’s still possible to add a Progressive Web App to the home screen, but the option to do that is hidden behind the mysterious three-dots-vertically-stacked icon (I propose we call this the shish kebab icon to distinguish it from the equally impenetrable hamburger icon).

I was chatting with Andreas from Mozilla at the View Source conference last week, and he was filling me in on how Firefox on Android does the add-to-homescreen flow. Instead of a one-time prompt, they’ve added a persistent icon above the “line of death” (the icon is a combination of a house and a plus symbol).

When a Firefox 58 user arrives on a website that is served over HTTPS and has a valid manifest, a subtle badge will appear in the address bar: when tapped, an “Add to Home screen” confirmation dialog will slide in, through which the web app can be added to the Android home screen.

This kind of badging also has issues (without the explicit text “add to home screen”, the user doesn’t know what the icon does), but I think a more persistently visible option like this works better than the a one-time prompt.

Firefox is following the lead of the badging approach pioneered by the Samsung Internet browser. It provides a plus symbol that, when pressed, reveals the options to add to home screen or simply bookmark.

What does it mean to be an App?

I don’t think Chrome for Android has any plans for this kind of badging, but they are working on letting the site authors provide their own prompts. I’m not sure this is such a good idea, given our history of abusing pop-ups and overlays.

Sadly, I feel that any solution that relies on an unrequested overlay is doomed. That’s on us. The way we’ve turned browsing the web—especially on mobile—into a frustrating chore of dismissing unwanted overlays is a classic tragedy of the commons. We blew it. Users don’t trust unrequested overlays, and I can’t blame them.

For what it’s worth, my opinion is that ambient badging is a better user experience than one-time prompts. That opinion is informed by a meagre amount of testing though. I’d love to hear from anyone who’s been doing more detailed usability testing of both approaches. I assume that Google, Mozilla, and Samsung are doing this kind of testing, and it would be really great to see the data from that (hint, hint).

But it might well be that ambient badging is just too subtle to even be noticed by the user.

On one end of the scale you’ve got the intrusiveness of an add-to-home-screen prompt, but on the other end of the scale you’ve got the discoverability problem of a subtle badge icon. I wonder if there might be a compromise solution—maybe a badge icon that pulses or glows on the first or second visit?

Of course that would also need to be thoroughly tested.

Testing

It’s tempting to think of testing with screen-readers as being like testing with browsers. With browser testing, you’re checking to see how a particular piece of software deals with the code you’re throwing at it. A screen reader is a piece of software too, so it makes sense to approach it the same way, right?

I don’t think so. I think it’s really important that if someone is going to test your site with a screen reader, it should be someone who uses a screen reader every day.

Think of it this way: you wouldn’t want a designer or developer to do usability testing by testing the design or code on themselves. That wouldn’t give you any useful data. They’re already familiar with what problems the design is supposed to be solving, and how the interface works. That’s why you need to do usability testing with someone from outside, someone who wasn’t involved in the design or development process.

It’s no different when it comes to users of assistive technology. You’re not trying to test their technology; you’re trying to test how well the thing you’re building works for the person using the technology.

In short:

Don’t think of screen-reader testing as a form of browser testing; think of it as a form of usability testing.

Brighton device lab

People of Brighton (and environs), I have a reminder for you. Did you know that there is an open device lab in the Clearleft office?

That’s right! You can simply pop in at any time and test your websites on Android, iOS, Windows Phone, Blackberry, Kindles, and more.

The address is 68 Middle Street. Ring the “Clearleft” buzzer and say you’re there to use the device lab.. There’ll always be somebody in the office. They’ll buzz you in and you can take the lift to the first floor. No need to make a prior appointment—feel free to swing by whenever you like.

There is no catch. You show up, test your sites on whatever devices you want, and maybe even stick around for a cup of tea.

Tell your friends.

I was doing a little testing this morning, helping Charlotte with a pesky bug that was cropping up on an iPad running iOS 8. To get the bottom of the issue, I needed to be able to inspect the DOM on the iPad. That turns out to be fairly straightforward (as of iOS 6):

  1. Plug the device into a USB port on your laptop using a lightning cable.
  2. Open Safari on the device and navigate to the page you want to test.
  3. Open Safari on your laptop.
  4. From the “Develop” menu in your laptop’s Safari, select the device.
  5. Use the web inspector on your laptop’s Safari to inspect elements to your heart’s content.

It’s a similar flow for Android devices:

  1. Plug the device into a USB port on your laptop.
  2. Open Chrome on the device and navigate to the page you want to test.
  3. Open Chrome on your laptop.
  4. Type chrome://inspect into the URL bar of Chrome on your laptop.
  5. Select the device.
  6. On the device, grant permission (a dialogue will have appeared by now).
  7. Use developer tools on your laptop’s Chrome to inspect elements to your heart’s content.

Using web inspector in Safari to inspect elements on a web page open on an iOS device. Using developer tools in Chrome to inspect elements on a web page open on an Android device.

Browser testing

On just about every client project that I work on, the subject of browser support comes up. Rightly so. It’s an important issue on which to get mutual understanding and agreement. But all too often, this important question is framed in a binary, true/false, go/no-go way: “Which browsers do we/don’t we support?”

Really, the first thing to get agreement on is not a list of browsers, but what we mean by the word “support”. In my mind, that word implies that a user of a particular browser should be able to accomplish the primary tasks on the website, whether that’s reading an article, booking a ticket, or buying a product. That doesn’t mean that the task must be experienced in pixel-perfect fidelity to an ideal visual design.

But to others, that’s exactly what “support” means. Personally, I’d call that optimisation. As Brad puts it:

There is a difference between support and optimization.

So to put it in glib terms, I support every browser …but I optimise for none.

Alright, fine. But I still need to get to some mutual understanding with a client about which browsers will get the optimised experience and which browsers will simply be supported.

Personally, I like the Filament Group’s approach of discussing this in terms of features rather than browsers. It makes sense to me to say the browsers that support geolocation will get the geolocation features, or the browsers that support offline caching will get the offline caching features. There’s no need to produce a list of what those browsers are for each feature, and in any case, the list would be constantly changing and updating with each new browser release.

But—and this is a big but—nine times out of ten, when the issue of browser support comes up, it isn’t about functionality; it’s about branding. What clients generally want to know is which browsers will get the ideal visual design. Obviously the newer versions of Chrome and Firefox are going to get all the lovely layouts, rounded corners, gradients, transparencies, and animations …but what about older versions of Internet Explorer? Even if users of IE8 and IE7 can accomplish their tasks, will the “degraded” visual presentation hurt their experience?

My hypothesis is that it won’t. Users of older versions of Internet Explorer aren’t doing a side-by-side comparison of the same website opened up in the latest Chrome nightly. Considering what their daily usage must be like—unable to use Facebook, unable to use Google services—I suspect that they are happy just to be able to complete their task, regardless of the site’s visual fidelity.

There’s another viewpoint—one that I’ve heard expressed by clients—that even users of older browsers should still get the ideal, pixel-perfect visual design. The hypothesis here is that, by allowing someone to experience anything less than the perfect presentation, the client’s brand will be damaged in the mind of that person.

Like I said, this is something that comes up on most client projects, and this is the point at which we’d have to come to an agreement about which hypothesis we’re going to go with. Of course I’m going to argue in favour of the first hypothesis, but I’ve come to realise that arguing in favour of either hypothesis is the wrong approach. We shouldn’t be debating this …we should be testing it.

We have two competing hypotheses about a group of users. Instead of trying to read their minds, why not test with that group of users to find out which hypothesis is correct? No matter what the results of the test, they will be valuable either way.

Think about the amount of work that’s going to go in to optimising for older browser versions—it’s going to take quite a bit of time and money. It makes sense to ensure that this time and money isn’t being spent on little more than a hunch that pixel-perfection is important to those users. On the other hand, if the test reveals that actually those users really will have a lesser opinion of a brand unless they get pixel-perfect parity with newer browsers, then you’ll know that the time and money spent making that happen isn’t wasted.

Josh wrote recently that 1 hour of research saves 10 hours of development time:

Or, in longer terms if more people appreciated how one day of user research can save weeks of coding I think they would do it more. It is remarkable what you decide to not build after talking to a few people closely.

When it comes to decisions around browser support/optimisation, I think that even a little bit of up-front research and testing could potentially save a lot of time, money, and heartache. I’m not sure exactly what form the testing should take, but I’m interested in figuring it out.

Anniversary

A funny thing happened when I was in Berlin two weekends ago. I was walking down the street that my AirBnB apartment was on when I heard someone say “Jeremy Keith?” It turned out it was Andre Jay Meissner, one of the founders of the excellent Open Device Lab website. We had emailed but never met before. Small world!

The Twitter account for the open device lab in Nuremburg pointed out that it’s been one year since I wrote a blog post about the open device lab I set up:

Much as I’d love to take credit for the idea of an open device lab, it simply isn’t true. Jason and Lyza had been working on setting up the open device lab in Portland for quite a while when I flung open the doors of the Clearleft test lab. But I will take credit for the “Ah, fuck it!” attitude that I introduced to the idea of sharing test devices with the community. Partly because I had seen how long it was taking the Portland device lab to get off the ground while they did everything by the book, I decided to just wait for the worst to happen instead of planning for it:

There are potential pitfalls to opening up a testing suite like this. What about the insurance? What about theft? What about breakage? But the thing about potential pitfalls is that they’re just that: potential. I’m treating all of them as YAGNI issues. I’ll address any problems if and when they occur rather than planning for worst-case scenarios.

It proved to be a great policy. So far, nothing has gone wrong. And it also served as an example to other people thinking about opening up device labs at their companies: “don’t sweat it; I didn’t!”

But as far as anniversaries go, the one-year birthday of the Clearleft device lab is not the most significant event of April 30th. Today is the twentieth anniversary of the publication of one of the most important documents in technological history: the document that officially put the World Wide Web into the public domain.

Open device labs are a small, small part of working on the web but I like to think that they represent the same kind of spirit of openness and sharing that Tim Berners-Lee and his colleagues demonstrated at CERN:

I really, really like the way that communal device labs have taken off. It’s like a physical manifestation of the sharing and openness that has imbued the practice of web design and development right from the start. View source, mailing lists, blog posts, Stack Overflow, and Github are made of bits; device labs are made of atoms. But they are all open for you to use and contribute to.

At UX London I had dinner with a Swiss entrepreneur who was showing off his proprietary native app on his phone and proudly declaring that he had been granted a patent. He seemed like a nice chap, but his attitude kind of made my skin crawl. It seemed so antithetical to the spirit of sharing and openness that I’m used to from the web.

James Gleick once described the web as the patent that never was:

Tim Berners-Lee invented the Web and the Web browser — that is, the world as we now know it — pretty much single-handedly, starting in about 1989, when he was working as a software engineer at CERN, the particle-physics laboratory in Geneva. He didn’t patent it, or any part of it. On the contrary, he has labored tirelessly to keep cyberspace open and nonproprietary.

We are all reaping the benefits of Sir Tim’s kindness and generosity.

Promo

The lovely and talented Paul and Kelly from Maxine Denver were in the Clearleft office to do some video work last week. After finishing a piece I was in, I suggested they keep “rolling” (to use an anachronistic term) so I could do a little tongue-in-cheek piece about the Clearleft device lab, a la Winnebago Man.

Here’s the result.

It reminded me of something, but I couldn’t figure out what. Then I remembered.

Open device labs

It’s been just nine months since I threw open the doors to the device lab in the Clearleft office. The response in just the first week was fantastic — people started donating their devices to the communal pool, doubling, then tripling the amount of phones and tablets.

The idea of having a communal device lab wasn’t new; Jason had been talking about setting up a lab in Portland but the paperwork involved was bogging it down. So when I set up the Brighton lab, I deliberately took an “ah, fuck it!” attitude …and that was new:

There are potential pitfalls to opening up a testing suite like this. What about the insurance? What about theft? What about breakage? But the thing about potential pitfalls is that they’re just that: potential. I’m treating all of them as YAGNI issues. I’ll address any problems if and when they occur rather than planning for worst-case scenarios.

So far, so good.

Since then I’ve been vocally encouraging others to set up communal devices labs wherever they may be—linking and tweeting whenever anybody so much as mentioned the possibility of getting a device lab up and running. Then the Lab Up! site was established to help people do just that.

Now there’s a brand new site that’s not just for people setting up device labs, but also for people looking a device lab to use: OpenDeviceLab.com.

  • Help people to locate the right Open Device Lab for the job,
  • explain and promote the Open Device Lab movement,
  • attract Contributors and Sponsors to help and donate to ODLs.

It’s an excellent resource. Head on over there and find out where your nearest device lab is located. And if you can’t find one, think about setting one up.

I really, really like the way that communal device labs have taken off. It’s like a physical manifestation of the sharing and openness that has imbued the practice of web design and development right from the start. View source, mailing lists, blog posts, Stack Overflow, and Github are made of bits; device labs are made of atoms. But they are all open for you to use and contribute to.

The test

There was once a time when the first thing you would do when you went to visit a newly-launched website was to run its markup through a validator.

Later on that was replaced by the action of bumping up the font size by a few notches—what Dan called the Dig Dug test.

Thanks to Ethan, we all started to make our browser windows smaller and bigger as soon as we visited a newly-launched site.

Now when I go to a brand new site I find myself opening up the “Network” tab in my browser’s developer tools to count the HTTP requests and measure the page weight.

Just like old times.

RoboHornet’s nest

Paul Irish recently announced that the RoboHornet browser benchmarking tool is being open-sourced. This is great news!

RoboHornet is designed to avoid the selective dick-measuring that characterises so many benchmarking results touted by browser vendors in their marketing spiel. Instead, the criteria that RoboHornet tests against are decided by developers like you and me. It’s like Stack Overflow for browser performance.

Sadly, Roger Capriotti from Microsoft used the announcement as an opportunity to engage in even more swaggering selective dick-measuring. Bizarrely, he seems to have completely misunderstood how RoboHornet works. Repeatedly mischaracterising it as “micro-framework”, he takes it to task as a tool that “only focuses on specific aspects of browser performance” …completely glossing over the fact that those “specific aspects” are chosen by us, the developers who build the websites that the browsers are supposed to render.

Instead, he chooses “a real-world scenario” …imitating the scrolling text effect seen in the 1999 movie The Matrix, concluding:

This is a great example of why we have consistently said real-world performance matters when evaluating a browser.

WAT?

But, y’know, the risible example and complete misrepresentation of RoboHornet isn’t what bothers me about the post. It’s the tone. I’ve had it with this sort of sniping, mean-spirited, playground politics. This does not move the web forward. This does not make a more beautiful web.

On the plus side, crap like this makes you appreciate the professionalism of the people working on Firefox, Chrome and Opera (Apple, of course, dodges the issue entirely by having absolutely zero developer relations when it comes to their browser).

Don’t get me wrong: there are very, very good people working on Internet Explorer at Microsoft. But they’re not the ones writing petulant blog posts. I feel bad for them. If Roger Capriotti—whose job title is “Director, Internet Explorer Marketing”—is supposed to be speaking for them, he is letting them down badly.

Open device labs

It seems like there’s been a lot of activity lately around the idea of open device labs and I’ve been doing my best to keep track of it all.

There are communal device labs springing up all over the place: Berlin, Cape Town, Helsinki, Amsterdam and soon Washington DC. Here’s a list of most of them.

If you’re running an open device lab, or hoping to set one up, be sure to register at Lab Up, a site that aims to pool resources and hopefully get some device manufacturers to distribute their wares. There’s also a mailing list for open device labs. Sign up if you have or want a communal device lab. Sharing is caring.

Finally, there’s an article on Smashing Magazine that goes into great detail on every aspect of setting up and running a communal collection of devices. If you’ve been thinking about starting an open device lab in your area, now is the time. Do it.

Laboratory conditions

Whenever I mention the communal Brighton device lab on Twitter, or post pictures of the devices, I always get questions about it. So here’s a quick rundown of how things are set up.

Most of the devices are plugged in to two 10-port powered USB hubs made by Trust.

Powered USB hub

I tend to clump the devices by operating system: a bunch of Android devices here, a bunch of Windows Phones there. That might just be my mild OCD; there isn’t really any particular advantage to doing that.

I’ve also rearranged the home screens to be consistent. All the Android devices have these icons, in this order:

Adobe Shadow is a very handy way of avoiding manual refreshes on Android and iOS devices. It has a lot of moving parts, but it’s worth setting it up. As well as installing the Adobe Shadow app on each device, you’ll need to download and run the desktop app on the computer you’re “driving” with and you’ll need to install a Google Chrome plug-in. It’s worth taking the time to do it though: you’d spent a lot more time manually refreshing each and every device.

Running Adobe Shadow on eleven devices

More often than not, you’ll want to test websites before they launch so you’ll need some way of looking at locally-hosted sites. Localtunnel and showoff.io are just two ways of doing that—I’m sure there are many more. Personally, I use showoff.io for $5 a month and it works well for me.

Testing

Finally, there’s the question I get asked more than any other…

The stands are from The Iron Mill just outside Ballymena in County Antrim. The iPhone stands are nice and sturdy, they work for other models just fine, and delivery is free within the UK and Northern Ireland.

And now, here’s Remy conducting mad experiments with JSbin at the device lab:

Device labs

Following the example of the communal Clearleft test lab in Brighton that I keep harping on about, a number of other locations are also sprouting shared device labs.

The London Device Lab has just been set up by Shaun Dunne at Mozilla’s Space.

The Malmö Open Device Lab should prove very handy for inhabitants of southern Sweden (and Copenhagen too—just a short train ride away).

Here’s the Exeter device library.

Cole Henley is looking for devices that could be contributed to a communal device lab in Frome in Somerset.

Want a communal testing lab for your town? Do it. It doesn’t matter how small it is initially; it will grow once your fellow developers come out of the woodwork and start contributing their devices.

I guarantee you that every web developer out there is thinking the same thing: “I don’t have enough devices to test on.”

Start a device lab. Get the word out. Tell me about it and I’ll shout it from the rooftops.

Share and enjoy.

Update: Here’s a new device lab in The Netherlands.

Device expansion

Remy popped ‘round to the Clearleft office the other day with some more devices for the communal testing lab …although the Matrix-era Nokia is probably not all that useful.

Yay! More phones for the communal testing lab from @rem Getting more phone chargers and power strips for the communal device lab

The list of devices is starting to look pretty good. Of course there’s no such thing as “enough” devices, but this is at least beginning to look like a nice cross-section. I’ve been buying phone chargers and power strips in enough quantities that I’m starting to get bulk discounts.

If you’re in the neighbourhood, come on ‘round and try out your sites on these devices.

Your local mobile device lab

Last week I described how I go about getting hold of mobile devices to test with at Clearleft. I finished by throwing open the door to any other web devs in Brighton who want to test on our devices:

We’ve always had an open-door policy here, so if you want to pop around, use our WiFi, and test on our devices, you’re more than welcome.

There was a great response from my fellow Brightonians offering to add some of their devices to the collection:

Aegir popped ‘round yesterday and dropped off an iPhone 3GS and a very cute Nokia N9.

Getting some testing devices from @aegirthor

This morning I arrived in to the office to find that Aron had dropped off a Samsung Galaxy Tab.

This is great! Just by opening up the testing suite to everyone, it has now expanded beyond what I had been able to gather together by myself.

Josh has put together a page on the Clearleft site that we’ll keep updated with a list of the devices we’ve got.

We’ve ordered some nice stands for the phones so that we can clear away some of the cable clutter. We don’t want the experience of testing to be any more unpleasant than it needs to be. To that end, I’ve installed Adobe Shadow on the iOS and Android devices.

If you want to test a site that isn’t live yet, I find that services like showoff.io and localtunnel are handy for previewing local sites. Do bear in mind that these devices are intended for testing websites: if you want to install apps or update software, sorry; that’s not what the test lab is for.

If you’re based in Brighton, pop ‘round whenever you want to use our devices.

If you’re not based in Brighton, maybe you could kick-start a similar sharing scheme with a local company or co-working space. Belfast, Bristol, Manchester, Birmingham …you’ve got loads of smart geeks and I bet they’ve got plenty of devices they could be sharing.

There are potential pitfalls to opening up a testing suite like this. What about the insurance? What about theft? What about breakage? But the thing about potential pitfalls is that they’re just that: potential. I’m treating all of them as issues. I’ll address any problems if and when they occur rather than planning for worst-case scenarios.

“What’s the worst that could happen?” is really only valuable as a rhetorical question.

Left to our own devices

In the final part of his series on responsive design in .net magazine, Paul talks about testing:

The key thing to remember here is that any testing is better than no testing. Build up a test suite as large as you can afford, and find opportunities to test on other devices whenever you can.

I agree wholeheartedly and I think that any kind of testing with real devices should be encouraged. I find it disheartening when somebody blogs or tweets about testing on a particular device, only to be rebuffed with sentiments like “it isn’t enough.” It’s true—it isn’t enough …but it’s never enough. And the fact that a developer is doing any testing at all is a good thing.

There are some good resources out there to help you get started in putting together a collection of test devices.

Actually, you should probably just read everything on Stephanie’s site: it’s all thoughtful and useful. In her post on the value of 1000 Androids she wrote:

The dirty little secret is that the more you test—the more accurately you will determine when it’s ok not to.

That’s crucial. I think there’s a common misunderstanding that testing on many different devices means developing for many different devices. Nothing could be further from the truth. Just as in the world of the desktop, we shouldn’t be developing for any specific devices or browsers. That’s why we develop with standards.

In fact I’ve found that one of the greatest benefits of testing on as many different platforms as possible is that it stops me from straying down the path of device-specific development. When I come across a problem in my testing, my reaction isn’t to think “how can I fix this problem on this particular device?”, which would probably involve throwing more code at it. Instead I think “how can I avoid this problem?” The particular device may have highlighted the issue, but there’s almost always a more fundamental problem to be tackled …and it’s very rarely tackled by throwing more code at it.

I wish I had more devices to test on …even if it might increase the risk of me getting reported to the police. At the same time, I realise that it’s a pretty crumby and expensive situation for developers, particularly freelancers. As Paul wrote:

Not only does it present a potential barrier to entry for anyone wanting to build responsive websites, but it encourages the purchase of yet more devices and gadgets.

I’ve been keeping my costs down by shopping in dodgy second-hand electronics shops that sell devices that have fallen off the back of a lorry. Most of them sell phones with a classification of A, B, or C. If something is labelled A, it is mint condition. If something is labelled C, there might be something wrong with it or you might not get all the cables or the box. But for my purposes, that’s absolutely fine. If anything, I’m usually hunting for outdated devices with older versions of operating systems.

So far I’ve got:

  • An HTC thingy running Windows Phone 7.5,
  • Another HTC thingy running Android 2.3.4,
  • A little Samsung running Android 2.2,
  • A Blackberry 9800 (Torch, right?),
  • A Blackberry Playbook,
  • A Kindle,
  • An iPod running iOS 3.1.3,
  • An iPad running iOS 5.something,
  • And I’ve got my own iPhone 4 and Kindle Touch.

Devices

I don’t know about you, but my eyes glaze over when it comes to phone models and operating system numbers, especially when they just end up sounding like condoms. Luckily we’ve got people like PPK to help us figure out the smartphone browser landscape.

You’ll notice plenty of glaring omissions in my paltry list—there’s nary a Nokia device to be found—but I aim to plug those gaps as soon as I can.

In the meantime I’ve been setting up a desk at the Clearleft office for these devices so that they can stay charged up and within reach.

Devices

We’ve always had an open-door policy here, so if you want to pop around, use our WiFi, and test on our devices, you’re more than welcome. Give me some advance warning on Twitter and I can put the kettle on for a cup of tea. We’re at 28 Kensington Street, Suite 2.

Think of it as a quick’n’dirty, much smaller-scale version of Mobile Portland’s Device Lab.

HTML5 test results

As promised, I’ve gathered the data from one of the exercises I administered at the dConstruct HTML5 and CSS3 workshop and totted the results up in a table.

There were 30 people in the workshop but I only managed to retrieve 22 results—I don’t know what happened to the missing eight sheets of answers. This is a smaller sampling than I was hoping for and I realise that it’s too small to be considered scientifically accurate but I think it’s still interesting to see the responses of 22 smart, savvy web developers.

Across the top (in the table header) are the possible answers; nine new elements in the HTML5 spec. Each row shows the answers given for each element as workshop attendees attempted to match up the names of the elements with the nine definitions provided from the spec. The most common answer in each case has been highlighted.

HTML5 Workshop Results
article aside details figure footer header hgroup nav section
article 5 0 2 1 1 1 0 1 9
aside 1 11 1 3 2 0 0 0 2
details 1 6 8 1 4 0 0 0 0
figure 3 2 2 14 1 0 0 0 0
footer 0 0 10 0 10 1 0 0 1
header 0 0 0 2 0 16 0 4 0
hgroup 0 0 0 0 0 1 21 0 0
nav 0 0 1 0 0 0 0 19 2
section 12 0 0 0 1 1 1 0 7

Ideally, the table would show a nice uniform graph: a solid diagonal line from the top left left to the bottom right. The diagonal line is there but it isn’t exactly uniform.

For the record, here are the element names together with their correct definitions. I’ve included a little sparkline with each one to show the distribution of answers. A unanimous result would show one clear spike. Wobbly uneven sparklines indicate a corresponding uncertainty in the answers.

article

…a section of a page that consists of a composition that forms an independent part of a document, page, or site.

aside

…represents a section of a page that consists of content that is tangentially related to the content around it, and which could be considered separate from that content.

details

…additional information or controls which the user can obtain on demand.

figure

…some flow content, optionally with a caption, that is self-contained and is typically referenced as a single unit from the main flow of the document.

footer

…typically contains information about its section such as who wrote it, links to related documents, copyright data, and the like.

header

…a group of introductory or navigational aids.

hgroup

…used to group a set of h1–h6 elements when the heading has multiple levels, such as subheadings, alternative titles, or taglines.

nav

…a section of a page that links to other pages or to parts within the page: a section with navigation links.

section

…a thematic grouping of content, typically with a heading, possibly with a footer.

There are some interesting results here; some of them surprising, some of them expected.

The most easily identified element is hgroup. That’s also the only element name that isn’t taken from an existing English word. I don’t think that’s a coincidence. When the name of an element is created specifically to describe what that element does, it’s no surprise that the name and the definition are tightly coupled.

I was expecting to see more confusion between aside and figure. Both elements serve a similar purpose and in my opinion, the names could be swapped around without changing the definitions in the spec. But the data doesn’t support my hypothesis.

The confusion around footer surprised me. I would have expected to see more confusion between footer and aside than between footer and details.

The most glaring problem lies with section and article. This doesn’t surprise me. At this point, section and article are practically indistinguishable. For a while there, article had some more optional attributes—cite and pubdate—but those have now been removed. Now section and article have the same content model and appear to cover much the same kind of content:

…a thematic grouping of content, typically with a heading, possibly with a footer.

…a section of a page that consists of a composition that forms an independent part of a document, page, or site.

Witness the recent article on HTML5 Doctor wherein Bruce attempts to clear up the confusion and points out where the HTML5 Doctor site has been getting it wrong. If it’s tough for those smart guys to figure out when to use section, article or div, it doesn’t bode well for the rest of us.

Choosing the right element would be easier if there were some clear rules like “You can only use sections within articles” or “You can only use articles within a section” but as it is, you can nest sections in articles and articles in a section. With that much rope, authors are likely to hang themselves, overdosing on the semantic freedom.

I don’t think there needs to be a section element and an article element. I don’t have a particularly strong opinion about which one should stay and which one should go but a little trimming is definitely in order.

Now, if you’ll excuse me, I’m off to present this data to the list.

Silverbacking

It’s been a busy week for Clearleft. I wasn’t in the office for the start of the week though; I was in up north delivering some Ajax training to the good people at the Library of the University of Liverpool. Alas, due to construction work, I didn’t have the chance to peruse the world-famous science fiction collection. I’ll just have to return to Merseyside sometime when the builders are gone.

I made it back to Brighton in time to press the proverbial button and launch the website of Silverback, the project that’s been keeping a portion of Clearleft very busy for a while now.

It’s been fascinating to watch Silverback take shape from the spark of an idea from Andy to the conflagration that is desktop software development. It’s been a learning experience for everyone involved. If you want to delve into all the details, be sure to read Garrett’s in-depth look at Silverback.

I didn’t have that much to do with the development. In fact, all I did was mark up and style the website (oh, and integrate the PayPal stuff …joy). Still, I’ve found myself caught up in the excitement of an honest-to-goodness product launch. We’ve all been tracking the feedback on Twitter and on blogs. On the whole, it seems like people really, really like it. But what’s far more important than whether people do or don’t buy this piece of software is the fact that people are talking about usability testing.

Silverback is all about usability testing — Rich has summed up exactly what Silverback does nicely. It’s a Mac app that we built to scratch our own itch. We wanted a way to be able to run usability tests quickly and cheaply.

Usability testing is one of those things that always seems to be amongst the first to get cut from projects, usually because of cost or time concerns. Maybe Silverback can help tip the balance back in favour of doing at least some usability testing even if it’s really quick’n’dirty.

I’m constantly amazed by just how far a little user-testing goes. The analysis of the results needn’t be time-consuming either. Having a handful of people try out your wireframes can lead to forehead-slapping revelations of obvious issues.

So I’m really happy that, if nothing else, Silverback will encourage more people to think about doing some quick usability tests. I guarantee that after just one round, the benefits will be so self-evident that usability testing will become indispensable.

There’s one other forthcoming release that I’m hoping will spur on the growth of usability testing. It’s not another piece of software. A little birdie tells me that Steve Krug—author of the classic Don’t Make Me Think—is writing a new book on… yup, quick and easy usability testing.

The rewards of usability testing are within reach for the price of one book and one piece of ~$50 dollar software.