Tags: testing




It’s tempting to think of testing with screen-readers as being like testing with browsers. With browser testing, you’re checking to see how a particular piece of software deals with the code you’re throwing at it. A screen reader is a piece of software too, so it makes sense to approach it the same way, right?

I don’t think so. I think it’s really important that if someone is going to test your site with a screen reader, it should be someone who uses a screen reader every day.

Think of it this way: you wouldn’t want a designer or developer to do usability testing by testing the design or code on themselves. That wouldn’t give you any useful data. They’re already familiar with what problems the design is supposed to be solving, and how the interface works. That’s why you need to do usability testing with someone from outside, someone who wasn’t involved in the design or development process.

It’s no different when it comes to users of assistive technology. You’re not trying to test their technology; you’re trying to test how well the thing you’re building works for the person using the technology.

In short:

Don’t think of screen-reader testing as a form of browser testing; think of it as a form of usability testing.

Brighton device lab

People of Brighton (and environs), I have a reminder for you. Did you know that there is an open device lab in the Clearleft office?

That’s right! You can simply pop in at any time and test your websites on Android, iOS, Windows Phone, Blackberry, Kindles, and more.

The address is 68 Middle Street. Ring the “Clearleft” buzzer and say you’re there to use the device lab.. There’ll always be somebody in the office. They’ll buzz you in and you can take the lift to the first floor. No need to make a prior appointment—feel free to swing by whenever you like.

There is no catch. You show up, test your sites on whatever devices you want, and maybe even stick around for a cup of tea.

Tell your friends.

I was doing a little testing this morning, helping Charlotte with a pesky bug that was cropping up on an iPad running iOS 8. To get the bottom of the issue, I needed to be able to inspect the DOM on the iPad. That turns out to be fairly straightforward (as of iOS 6):

  1. Plug the device into a USB port on your laptop using a lightning cable.
  2. Open Safari on the device and navigate to the page you want to test.
  3. Open Safari on your laptop.
  4. From the “Develop” menu in your laptop’s Safari, select the device.
  5. Use the web inspector on your laptop’s Safari to inspect elements to your heart’s content.

It’s a similar flow for Android devices:

  1. Plug the device into a USB port on your laptop.
  2. Open Chrome on the device and navigate to the page you want to test.
  3. Open Chrome on your laptop.
  4. Type chrome://inspect into the URL bar of Chrome on your laptop.
  5. Select the device.
  6. On the device, grant permission (a dialogue will have appeared by now).
  7. Use developer tools on your laptop’s Chrome to inspect elements to your heart’s content.

Using web inspector in Safari to inspect elements on a web page open on an iOS device. Using developer tools in Chrome to inspect elements on a web page open on an Android device.

Browser testing

On just about every client project that I work on, the subject of browser support comes up. Rightly so. It’s an important issue on which to get mutual understanding and agreement. But all too often, this important question is framed in a binary, true/false, go/no-go way: “Which browsers do we/don’t we support?”

Really, the first thing to get agreement on is not a list of browsers, but what we mean by the word “support”. In my mind, that word implies that a user of a particular browser should be able to accomplish the primary tasks on the website, whether that’s reading an article, booking a ticket, or buying a product. That doesn’t mean that the task must be experienced in pixel-perfect fidelity to an ideal visual design.

But to others, that’s exactly what “support” means. Personally, I’d call that optimisation. As Brad puts it:

There is a difference between support and optimization.

So to put it in glib terms, I support every browser …but I optimise for none.

Alright, fine. But I still need to get to some mutual understanding with a client about which browsers will get the optimised experience and which browsers will simply be supported.

Personally, I like the Filament Group’s approach of discussing this in terms of features rather than browsers. It makes sense to me to say the browsers that support geolocation will get the geolocation features, or the browsers that support offline caching will get the offline caching features. There’s no need to produce a list of what those browsers are for each feature, and in any case, the list would be constantly changing and updating with each new browser release.

But—and this is a big but—nine times out of ten, when the issue of browser support comes up, it isn’t about functionality; it’s about branding. What clients generally want to know is which browsers will get the ideal visual design. Obviously the newer versions of Chrome and Firefox are going to get all the lovely layouts, rounded corners, gradients, transparencies, and animations …but what about older versions of Internet Explorer? Even if users of IE8 and IE7 can accomplish their tasks, will the “degraded” visual presentation hurt their experience?

My hypothesis is that it won’t. Users of older versions of Internet Explorer aren’t doing a side-by-side comparison of the same website opened up in the latest Chrome nightly. Considering what their daily usage must be like—unable to use Facebook, unable to use Google services—I suspect that they are happy just to be able to complete their task, regardless of the site’s visual fidelity.

There’s another viewpoint—one that I’ve heard expressed by clients—that even users of older browsers should still get the ideal, pixel-perfect visual design. The hypothesis here is that, by allowing someone to experience anything less than the perfect presentation, the client’s brand will be damaged in the mind of that person.

Like I said, this is something that comes up on most client projects, and this is the point at which we’d have to come to an agreement about which hypothesis we’re going to go with. Of course I’m going to argue in favour of the first hypothesis, but I’ve come to realise that arguing in favour of either hypothesis is the wrong approach. We shouldn’t be debating this …we should be testing it.

We have two competing hypotheses about a group of users. Instead of trying to read their minds, why not test with that group of users to find out which hypothesis is correct? No matter what the results of the test, they will be valuable either way.

Think about the amount of work that’s going to go in to optimising for older browser versions—it’s going to take quite a bit of time and money. It makes sense to ensure that this time and money isn’t being spent on little more than a hunch that pixel-perfection is important to those users. On the other hand, if the test reveals that actually those users really will have a lesser opinion of a brand unless they get pixel-perfect parity with newer browsers, then you’ll know that the time and money spent making that happen isn’t wasted.

Josh wrote recently that 1 hour of research saves 10 hours of development time:

Or, in longer terms if more people appreciated how one day of user research can save weeks of coding I think they would do it more. It is remarkable what you decide to not build after talking to a few people closely.

When it comes to decisions around browser support/optimisation, I think that even a little bit of up-front research and testing could potentially save a lot of time, money, and heartache. I’m not sure exactly what form the testing should take, but I’m interested in figuring it out.


A funny thing happened when I was in Berlin two weekends ago. I was walking down the street that my AirBnB apartment was on when I heard someone say “Jeremy Keith?” It turned out it was Andre Jay Meissner, one of the founders of the excellent Open Device Lab website. We had emailed but never met before. Small world!

The Twitter account for the open device lab in Nuremburg pointed out that it’s been one year since I wrote a blog post about the open device lab I set up:

Much as I’d love to take credit for the idea of an open device lab, it simply isn’t true. Jason and Lyza had been working on setting up the open device lab in Portland for quite a while when I flung open the doors of the Clearleft test lab. But I will take credit for the “Ah, fuck it!” attitude that I introduced to the idea of sharing test devices with the community. Partly because I had seen how long it was taking the Portland device lab to get off the ground while they did everything by the book, I decided to just wait for the worst to happen instead of planning for it:

There are potential pitfalls to opening up a testing suite like this. What about the insurance? What about theft? What about breakage? But the thing about potential pitfalls is that they’re just that: potential. I’m treating all of them as YAGNI issues. I’ll address any problems if and when they occur rather than planning for worst-case scenarios.

It proved to be a great policy. So far, nothing has gone wrong. And it also served as an example to other people thinking about opening up device labs at their companies: “don’t sweat it; I didn’t!”

But as far as anniversaries go, the one-year birthday of the Clearleft device lab is not the most significant event of April 30th. Today is the twentieth anniversary of the publication of one of the most important documents in technological history: the document that officially put the World Wide Web into the public domain.

Open device labs are a small, small part of working on the web but I like to think that they represent the same kind of spirit of openness and sharing that Tim Berners-Lee and his colleagues demonstrated at CERN:

I really, really like the way that communal device labs have taken off. It’s like a physical manifestation of the sharing and openness that has imbued the practice of web design and development right from the start. View source, mailing lists, blog posts, Stack Overflow, and Github are made of bits; device labs are made of atoms. But they are all open for you to use and contribute to.

At UX London I had dinner with a Swiss entrepreneur who was showing off his proprietary native app on his phone and proudly declaring that he had been granted a patent. He seemed like a nice chap, but his attitude kind of made my skin crawl. It seemed so antithetical to the spirit of sharing and openness that I’m used to from the web.

James Gleick once described the web as the patent that never was:

Tim Berners-Lee invented the Web and the Web browser — that is, the world as we now know it — pretty much single-handedly, starting in about 1989, when he was working as a software engineer at CERN, the particle-physics laboratory in Geneva. He didn’t patent it, or any part of it. On the contrary, he has labored tirelessly to keep cyberspace open and nonproprietary.

We are all reaping the benefits of Sir Tim’s kindness and generosity.


The lovely and talented Paul and Kelly from Maxine Denver were in the Clearleft office to do some video work last week. After finishing a piece I was in, I suggested they keep “rolling” (to use an anachronistic term) so I could do a little tongue-in-cheek piece about the Clearleft device lab, a la Winnebago Man.

Here’s the result.

It reminded me of something, but I couldn’t figure out what. Then I remembered.

Open device labs

It’s been just nine months since I threw open the doors to the device lab in the Clearleft office. The response in just the first week was fantastic — people started donating their devices to the communal pool, doubling, then tripling the amount of phones and tablets.

The idea of having a communal device lab wasn’t new; Jason had been talking about setting up a lab in Portland but the paperwork involved was bogging it down. So when I set up the Brighton lab, I deliberately took an “ah, fuck it!” attitude …and that was new:

There are potential pitfalls to opening up a testing suite like this. What about the insurance? What about theft? What about breakage? But the thing about potential pitfalls is that they’re just that: potential. I’m treating all of them as YAGNI issues. I’ll address any problems if and when they occur rather than planning for worst-case scenarios.

So far, so good.

Since then I’ve been vocally encouraging others to set up communal devices labs wherever they may be—linking and tweeting whenever anybody so much as mentioned the possibility of getting a device lab up and running. Then the Lab Up! site was established to help people do just that.

Now there’s a brand new site that’s not just for people setting up device labs, but also for people looking a device lab to use: OpenDeviceLab.com.

  • Help people to locate the right Open Device Lab for the job,
  • explain and promote the Open Device Lab movement,
  • attract Contributors and Sponsors to help and donate to ODLs.

It’s an excellent resource. Head on over there and find out where your nearest device lab is located. And if you can’t find one, think about setting one up.

I really, really like the way that communal device labs have taken off. It’s like a physical manifestation of the sharing and openness that has imbued the practice of web design and development right from the start. View source, mailing lists, blog posts, Stack Overflow, and Github are made of bits; device labs are made of atoms. But they are all open for you to use and contribute to.

The test

There was once a time when the first thing you would do when you went to visit a newly-launched website was to run its markup through a validator.

Later on that was replaced by the action of bumping up the font size by a few notches—what Dan called the Dig Dug test.

Thanks to Ethan, we all started to make our browser windows smaller and bigger as soon as we visited a newly-launched site.

Now when I go to a brand new site I find myself opening up the “Network” tab in my browser’s developer tools to count the HTTP requests and measure the page weight.

Just like old times.

RoboHornet’s nest

Paul Irish recently announced that the RoboHornet browser benchmarking tool is being open-sourced. This is great news!

RoboHornet is designed to avoid the selective dick-measuring that characterises so many benchmarking results touted by browser vendors in their marketing spiel. Instead, the criteria that RoboHornet tests against are decided by developers like you and me. It’s like Stack Overflow for browser performance.

Sadly, Roger Capriotti from Microsoft used the announcement as an opportunity to engage in even more swaggering selective dick-measuring. Bizarrely, he seems to have completely misunderstood how RoboHornet works. Repeatedly mischaracterising it as “micro-framework”, he takes it to task as a tool that “only focuses on specific aspects of browser performance” …completely glossing over the fact that those “specific aspects” are chosen by us, the developers who build the websites that the browsers are supposed to render.

Instead, he chooses “a real-world scenario” …imitating the scrolling text effect seen in the 1999 movie The Matrix, concluding:

This is a great example of why we have consistently said real-world performance matters when evaluating a browser.


But, y’know, the risible example and complete misrepresentation of RoboHornet isn’t what bothers me about the post. It’s the tone. I’ve had it with this sort of sniping, mean-spirited, playground politics. This does not move the web forward. This does not make a more beautiful web.

On the plus side, crap like this makes you appreciate the professionalism of the people working on Firefox, Chrome and Opera (Apple, of course, dodges the issue entirely by having absolutely zero developer relations when it comes to their browser).

Don’t get me wrong: there are very, very good people working on Internet Explorer at Microsoft. But they’re not the ones writing petulant blog posts. I feel bad for them. If Roger Capriotti—whose job title is “Director, Internet Explorer Marketing”—is supposed to be speaking for them, he is letting them down badly.

Open device labs

It seems like there’s been a lot of activity lately around the idea of open device labs and I’ve been doing my best to keep track of it all.

There are communal device labs springing up all over the place: Berlin, Cape Town, Helsinki, Amsterdam and soon Washington DC. Here’s a list of most of them.

If you’re running an open device lab, or hoping to set one up, be sure to register at Lab Up, a site that aims to pool resources and hopefully get some device manufacturers to distribute their wares. There’s also a mailing list for open device labs. Sign up if you have or want a communal device lab. Sharing is caring.

Finally, there’s an article on Smashing Magazine that goes into great detail on every aspect of setting up and running a communal collection of devices. If you’ve been thinking about starting an open device lab in your area, now is the time. Do it.

Laboratory conditions

Whenever I mention the communal Brighton device lab on Twitter, or post pictures of the devices, I always get questions about it. So here’s a quick rundown of how things are set up.

Most of the devices are plugged in to two 10-port powered USB hubs made by Trust.

Powered USB hub

I tend to clump the devices by operating system: a bunch of Android devices here, a bunch of Windows Phones there. That might just be my mild OCD; there isn’t really any particular advantage to doing that.

I’ve also rearranged the home screens to be consistent. All the Android devices have these icons, in this order:

Adobe Shadow is a very handy way of avoiding manual refreshes on Android and iOS devices. It has a lot of moving parts, but it’s worth setting it up. As well as installing the Adobe Shadow app on each device, you’ll need to download and run the desktop app on the computer you’re “driving” with and you’ll need to install a Google Chrome plug-in. It’s worth taking the time to do it though: you’d spent a lot more time manually refreshing each and every device.

Running Adobe Shadow on eleven devices

More often than not, you’ll want to test websites before they launch so you’ll need some way of looking at locally-hosted sites. Localtunnel and showoff.io are just two ways of doing that—I’m sure there are many more. Personally, I use showoff.io for $5 a month and it works well for me.


Finally, there’s the question I get asked more than any other…

The stands are from The Iron Mill just outside Ballymena in County Antrim. The iPhone stands are nice and sturdy, they work for other models just fine, and delivery is free within the UK and Northern Ireland.

And now, here’s Remy conducting mad experiments with JSbin at the device lab:

Device labs

Following the example of the communal Clearleft test lab in Brighton that I keep harping on about, a number of other locations are also sprouting shared device labs.

The London Device Lab has just been set up by Shaun Dunne at Mozilla’s Space.

The Malmö Open Device Lab should prove very handy for inhabitants of southern Sweden (and Copenhagen too—just a short train ride away).

Here’s the Exeter device library.

Cole Henley is looking for devices that could be contributed to a communal device lab in Frome in Somerset.

Want a communal testing lab for your town? Do it. It doesn’t matter how small it is initially; it will grow once your fellow developers come out of the woodwork and start contributing their devices.

I guarantee you that every web developer out there is thinking the same thing: “I don’t have enough devices to test on.”

Start a device lab. Get the word out. Tell me about it and I’ll shout it from the rooftops.

Share and enjoy.

Update: Here’s a new device lab in The Netherlands.

Device expansion

Remy popped ‘round to the Clearleft office the other day with some more devices for the communal testing lab …although the Matrix-era Nokia is probably not all that useful.

Yay! More phones for the communal testing lab from @rem Getting more phone chargers and power strips for the communal device lab

The list of devices is starting to look pretty good. Of course there’s no such thing as “enough” devices, but this is at least beginning to look like a nice cross-section. I’ve been buying phone chargers and power strips in enough quantities that I’m starting to get bulk discounts.

If you’re in the neighbourhood, come on ‘round and try out your sites on these devices.

Your local mobile device lab

Last week I described how I go about getting hold of mobile devices to test with at Clearleft. I finished by throwing open the door to any other web devs in Brighton who want to test on our devices:

We’ve always had an open-door policy here, so if you want to pop around, use our WiFi, and test on our devices, you’re more than welcome.

There was a great response from my fellow Brightonians offering to add some of their devices to the collection:

Aegir popped ‘round yesterday and dropped off an iPhone 3GS and a very cute Nokia N9.

Getting some testing devices from @aegirthor

This morning I arrived in to the office to find that Aron had dropped off a Samsung Galaxy Tab.

This is great! Just by opening up the testing suite to everyone, it has now expanded beyond what I had been able to gather together by myself.

Josh has put together a page on the Clearleft site that we’ll keep updated with a list of the devices we’ve got.

We’ve ordered some nice stands for the phones so that we can clear away some of the cable clutter. We don’t want the experience of testing to be any more unpleasant than it needs to be. To that end, I’ve installed Adobe Shadow on the iOS and Android devices.

If you want to test a site that isn’t live yet, I find that services like showoff.io and localtunnel are handy for previewing local sites. Do bear in mind that these devices are intended for testing websites: if you want to install apps or update software, sorry; that’s not what the test lab is for.

If you’re based in Brighton, pop ‘round whenever you want to use our devices.

If you’re not based in Brighton, maybe you could kick-start a similar sharing scheme with a local company or co-working space. Belfast, Bristol, Manchester, Birmingham …you’ve got loads of smart geeks and I bet they’ve got plenty of devices they could be sharing.

There are potential pitfalls to opening up a testing suite like this. What about the insurance? What about theft? What about breakage? But the thing about potential pitfalls is that they’re just that: potential. I’m treating all of them as issues. I’ll address any problems if and when they occur rather than planning for worst-case scenarios.

“What’s the worst that could happen?” is really only valuable as a rhetorical question.

Left to our own devices

In the final part of his series on responsive design in .net magazine, Paul talks about testing:

The key thing to remember here is that any testing is better than no testing. Build up a test suite as large as you can afford, and find opportunities to test on other devices whenever you can.

I agree wholeheartedly and I think that any kind of testing with real devices should be encouraged. I find it disheartening when somebody blogs or tweets about testing on a particular device, only to be rebuffed with sentiments like “it isn’t enough.” It’s true—it isn’t enough …but it’s never enough. And the fact that a developer is doing any testing at all is a good thing.

There are some good resources out there to help you get started in putting together a collection of test devices.

Actually, you should probably just read everything on Stephanie’s site: it’s all thoughtful and useful. In her post on the value of 1000 Androids she wrote:

The dirty little secret is that the more you test—the more accurately you will determine when it’s ok not to.

That’s crucial. I think there’s a common misunderstanding that testing on many different devices means developing for many different devices. Nothing could be further from the truth. Just as in the world of the desktop, we shouldn’t be developing for any specific devices or browsers. That’s why we develop with standards.

In fact I’ve found that one of the greatest benefits of testing on as many different platforms as possible is that it stops me from straying down the path of device-specific development. When I come across a problem in my testing, my reaction isn’t to think “how can I fix this problem on this particular device?”, which would probably involve throwing more code at it. Instead I think “how can I avoid this problem?” The particular device may have highlighted the issue, but there’s almost always a more fundamental problem to be tackled …and it’s very rarely tackled by throwing more code at it.

I wish I had more devices to test on …even if it might increase the risk of me getting reported to the police. At the same time, I realise that it’s a pretty crumby and expensive situation for developers, particularly freelancers. As Paul wrote:

Not only does it present a potential barrier to entry for anyone wanting to build responsive websites, but it encourages the purchase of yet more devices and gadgets.

I’ve been keeping my costs down by shopping in dodgy second-hand electronics shops that sell devices that have fallen off the back of a lorry. Most of them sell phones with a classification of A, B, or C. If something is labelled A, it is mint condition. If something is labelled C, there might be something wrong with it or you might not get all the cables or the box. But for my purposes, that’s absolutely fine. If anything, I’m usually hunting for outdated devices with older versions of operating systems.

So far I’ve got:

  • An HTC thingy running Windows Phone 7.5,
  • Another HTC thingy running Android 2.3.4,
  • A little Samsung running Android 2.2,
  • A Blackberry 9800 (Torch, right?),
  • A Blackberry Playbook,
  • A Kindle,
  • An iPod running iOS 3.1.3,
  • An iPad running iOS 5.something,
  • And I’ve got my own iPhone 4 and Kindle Touch.


I don’t know about you, but my eyes glaze over when it comes to phone models and operating system numbers, especially when they just end up sounding like condoms. Luckily we’ve got people like PPK to help us figure out the smartphone browser landscape.

You’ll notice plenty of glaring omissions in my paltry list—there’s nary a Nokia device to be found—but I aim to plug those gaps as soon as I can.

In the meantime I’ve been setting up a desk at the Clearleft office for these devices so that they can stay charged up and within reach.


We’ve always had an open-door policy here, so if you want to pop around, use our WiFi, and test on our devices, you’re more than welcome. Give me some advance warning on Twitter and I can put the kettle on for a cup of tea. We’re at 28 Kensington Street, Suite 2.

Think of it as a quick’n’dirty, much smaller-scale version of Mobile Portland’s Device Lab.


It’s been a busy week for Clearleft. I wasn’t in the office for the start of the week though; I was in up north delivering some Ajax training to the good people at the Library of the University of Liverpool. Alas, due to construction work, I didn’t have the chance to peruse the world-famous science fiction collection. I’ll just have to return to Merseyside sometime when the builders are gone.

I made it back to Brighton in time to press the proverbial button and launch the website of Silverback, the project that’s been keeping a portion of Clearleft very busy for a while now.

It’s been fascinating to watch Silverback take shape from the spark of an idea from Andy to the conflagration that is desktop software development. It’s been a learning experience for everyone involved. If you want to delve into all the details, be sure to read Garrett’s in-depth look at Silverback.

I didn’t have that much to do with the development. In fact, all I did was mark up and style the website (oh, and integrate the PayPal stuff …joy). Still, I’ve found myself caught up in the excitement of an honest-to-goodness product launch. We’ve all been tracking the feedback on Twitter and on blogs. On the whole, it seems like people really, really like it. But what’s far more important than whether people do or don’t buy this piece of software is the fact that people are talking about usability testing.

Silverback is all about usability testing — Rich has summed up exactly what Silverback does nicely. It’s a Mac app that we built to scratch our own itch. We wanted a way to be able to run usability tests quickly and cheaply.

Usability testing is one of those things that always seems to be amongst the first to get cut from projects, usually because of cost or time concerns. Maybe Silverback can help tip the balance back in favour of doing at least some usability testing even if it’s really quick’n’dirty.

I’m constantly amazed by just how far a little user-testing goes. The analysis of the results needn’t be time-consuming either. Having a handful of people try out your wireframes can lead to forehead-slapping revelations of obvious issues.

So I’m really happy that, if nothing else, Silverback will encourage more people to think about doing some quick usability tests. I guarantee that after just one round, the benefits will be so self-evident that usability testing will become indispensable.

There’s one other forthcoming release that I’m hoping will spur on the growth of usability testing. It’s not another piece of software. A little birdie tells me that Steve Krug—author of the classic Don’t Make Me Think—is writing a new book on… yup, quick and easy usability testing.

The rewards of usability testing are within reach for the price of one book and one piece of ~$50 dollar software.