Tags: testing

128

sparkline

Sunday, December 3rd, 2017

Saturday, November 18th, 2017

Hooked and booked

At Booking.com, they do a lot of A/B testing.

At Booking.com, they’ve got a lot of dark patterns.

I think there might be a connection.

A/B testing is a great way of finding out what happens when you introduce a change. But it can’t tell you why.

The problem is that, in a data-driven environment, decisions ultimately come down to whether something works or not. But just because something works, doesn’t mean it’s a good thing.

If I were trying to convince you to buy a product, or use a service, one way I could accomplish that would be to literally put a gun to your head. It would work. Except it’s not exactly a good solution, is it? But if we were to judge by the numbers (100% of people threatened with a gun did what we wanted), it would appear to be the right solution.

When speaking about A/B testing at Booking.com, Stuart Frisby emphasised why it’s so central to their way of working:

One of the core principles of our organisation is that we want to be very customer-focused. And A/B testing is really a way for us to institutionalise that customer focus.

I’m not so sure. I think A/B testing is a way to institutionalise a focus on business goals—increasing sales, growth, conversion, and all of that. Now, ideally, those goals would align completely with the customer’s goals; happy customers should mean more sales …but more sales doesn’t necessarily mean happy customers. Using business metrics (sales, growth, conversion) as a proxy for customer satisfaction might not always work …and is clearly not the case with many of these kinds of sites. Whatever the company values might say, a company’s true focus is on whatever they’re measuring as success criteria. If that’s customer satisfaction, then the company is indeed customer-focused. But if the measurements are entirely about what works for sales and conversions, then that’s the real focus of the company.

I’m not saying A/B testing is bad—far from it! (although it can sometimes be taken to the extreme). I feel it’s best wielded in combination with usability testing with real users—seeing their faces, feeling their frustration, sharing their joy.

In short, I think that A/B testing needs to be counterbalanced. There should be some kind of mechanism for getting the answer to “why?” whenever A/B testing provides to the answer to “what?” In-person testing could be one way of providing that balance. Or it could be somebody’s job to always ask “why?” and determine if a solution is qualitatively—and not just quantitatively—good. (And if you look around at your company and don’t see anyone doing that, maybe that’s a role for you.)

If there really is a connection between having a data-driven culture of A/B testing, and a product that’s filled with dark patterns, then the disturbing conclusion is that dark patterns work …at least in the short term.

Monday, November 6th, 2017

Installing Progressive Web Apps

When I was testing the dConstruct Audio Archive—which is now a Progressive Web App—I noticed some interesting changes in how Chrome on Android offers the “add to home screen” prompt.

It used to literally say “add to home screen.”

Getting the “add to home screen” prompt for https://huffduffer.com/ on Android Chrome. And there’s the “add to home screen” prompt for https://html5forwebdesigners.com/ HTTPS + manifest.json + Service Worker = “Add to Home Screen” prompt. Add to home screen.

Now it simply says “add.”

The dConstruct Audio Archive is now a Progressive Web App

I vaguely remember there being some talk of changing the labelling, but I could’ve sworn it was going to change to “install”. I’ve got to be honest, just having the word “add” doesn’t seem to provide much context. Based on the quick’n’dirty usability testing I did with some co-workers, it just made things confusing. “Add what?” “What am I adding?”

Additionally, the prompt appeared immediately on the first visit to the site. I thought there was supposed to be an added “engagement” metric in order for the prompt to appear; that the user needs to visit the site more than once.

You’d think I’d be happy that users will be presented with the home-screen prompt immediately, but based on the behaviour I saw, I’m not sure it’s a good thing. Here’s what I observed:

  1. The user types the URL archive.dconstruct.org into the address bar.
  2. The site loads.
  3. The home-screen prompt slides up from the bottom of the screen.
  4. The user immediately moves to dismiss the prompt (cue me interjecting “Don’t close that!”).

This behaviour is entirely unsurprising for three reasons:

  1. We web designers and web developers have trained users to dismiss overlays and pop-ups if they actually want to get to the content. Nobody’s going to bother to actually read the prompt if there’s a 99% chance it’s going to say “Sign up to our newsletter!” or “Take our survey!”.
  2. The prompt appears below the “line of death” so there’s no way to tell it’s a browser or OS-level dialogue rather than a JavaScript-driven pop-up from the site.
  3. Because the prompt now appears on the first visit, no trust has been established between the user and the site. If the prompt only appeared on later visits (or later navigations during the first visit) perhaps it would stand a greater chance of survival.

It’s still possible to add a Progressive Web App to the home screen, but the option to do that is hidden behind the mysterious three-dots-vertically-stacked icon (I propose we call this the shish kebab icon to distinguish it from the equally impenetrable hamburger icon).

I was chatting with Andreas from Mozilla at the View Source conference last week, and he was filling me in on how Firefox on Android does the add-to-homescreen flow. Instead of a one-time prompt, they’ve added a persistent icon above the “line of death” (the icon is a combination of a house and a plus symbol).

When a Firefox 58 user arrives on a website that is served over HTTPS and has a valid manifest, a subtle badge will appear in the address bar: when tapped, an “Add to Home screen” confirmation dialog will slide in, through which the web app can be added to the Android home screen.

This kind of badging also has issues (without the explicit text “add to home screen”, the user doesn’t know what the icon does), but I think a more persistently visible option like this works better than the a one-time prompt.

Firefox is following the lead of the badging approach pioneered by the Samsung Internet browser. It provides a plus symbol that, when pressed, reveals the options to add to home screen or simply bookmark.

What does it mean to be an App?

I don’t think Chrome for Android has any plans for this kind of badging, but they are working on letting the site authors provide their own prompts. I’m not sure this is such a good idea, given our history of abusing pop-ups and overlays.

Sadly, I feel that any solution that relies on an unrequested overlay is doomed. That’s on us. The way we’ve turned browsing the web—especially on mobile—into a frustrating chore of dismissing unwanted overlays is a classic tragedy of the commons. We blew it. Users don’t trust unrequested overlays, and I can’t blame them.

For what it’s worth, my opinion is that ambient badging is a better user experience than one-time prompts. That opinion is informed by a meagre amount of testing though. I’d love to hear from anyone who’s been doing more detailed usability testing of both approaches. I assume that Google, Mozilla, and Samsung are doing this kind of testing, and it would be really great to see the data from that (hint, hint).

But it might well be that ambient badging is just too subtle to even be noticed by the user.

On one end of the scale you’ve got the intrusiveness of an add-to-home-screen prompt, but on the other end of the scale you’ve got the discoverability problem of a subtle badge icon. I wonder if there might be a compromise solution—maybe a badge icon that pulses or glows on the first or second visit?

Of course that would also need to be thoroughly tested.

Wednesday, September 27th, 2017

Canonical test podcasts (Joe Clark)

Are you the creator, programmer, or quality-tester of a podcasting application? This page provides a range of podcasts that exemplify a range of atypical use case from merely uncommon to exceedingly fringe. If your app can handle all these, you’re doing well.

The Coming Software Apocalypse - The Atlantic

The title is pure clickbait, and the moral panic early in this article repeats the Toyota myth, but then it settles down into a fascinating examination of abstractions in programming. On the one hand, there’s the problem of the not enough abstraction: having to write in code is such a computer-centric way of building things. On the other hand, our world is filled with dangerously abstracted systems:

When your tires are flat, you look at your tires, they are flat. When your software is broken, you look at your software, you see nothing.

So that’s a big problem.

Bret Victor, John Resig and Margaret Hamilton are featured. Doug Engelbart and J.C.R. Licklider aren’t mentioned but their spirits loom large.

Thursday, September 21st, 2017

Chrome to force .dev domains to HTTPS via preloaded HSTS

Well, I guess it’s time to change all my locally-hosted sites from .dev domains to .test. Thanks, Google.

Thursday, July 27th, 2017

Testing the accessibility of pattern libraries

Riffing on Rachel’s talk at Patterns Day:

At the Patterns Day conference last month, Rachel Andrew mentioned something interesting about patterns. She said that working with reusable interface components, where each one has its own page, made her realise that those work quite well as isolated test cases. I feel this also goes for some accessibility tests: there is a number of criteria where isolation aids testing.

Hidde specifically singles out these patterns:

  • Collapsible (“Show/hide”)
  • Form field
  • Video player

Sunday, July 23rd, 2017

What I’ve learned about motor impairment

James gives—if you’ll pardon the pun— hands-on advice on making sites that consider motor impairment:

  • Don’t assume keyboard access is all you need
    • Auto complete/Autofill
    • Show me my password
  • Allow for fine motor control issues
    • Don’t autoplay videos
    • Avoid hover-only controls
    • Infinite scrolling considerations
  • Be mindful of touch
    • Avoid small hit targets
    • Provide alternate controls for touch gestures

Far from being a niche concern, visitors with some form of motor impairment likely make up a significant percentage of your users. I would encourage you to test your website or application with your less dominant hand. Is it still easy to use?

Tuesday, June 6th, 2017

Left to our own devices. — Ethan Marcotte

Your website’s only as strong as the weakest device you’ve tested it on.

Wednesday, May 31st, 2017

Emmet Re:view — fast and easy way to test responsive design in multiple viewports

It’s no substitute for testing with real devices, but the “device wall” view in this Chrome plug-in is a nifty way of getting an overview of a site’s responsiveness at a glance.

Sunday, May 14th, 2017

ngrok - secure introspectable tunnels to localhost

This looks like a useful tool, not just for testing locally-hosted sites (say, at a device lab), but also for making locally-hosted sites run on HTTPS so you can test service workers.

Sunday, March 12th, 2017

Design Ethics in Practice – The Interconnected

Excellent and practical advice for before, during, and after research sessions and usability tests.

Tuesday, March 7th, 2017

bitsofcode | Linting HTML using CSS

Smart use of attribute selectors in CSS to catch mistakes in HTML.

Thursday, January 19th, 2017

Improving accessibility in Co-op wills – Digital blogs

Some interesting insights from usability and accessibility testing at the Co-op.

We used ‘nesting’ to reduce the amount of information on the page when the user first reaches it. When the user chooses an option, we ask for any other details at that point rather than having all the questions on the page at once.

Monday, November 21st, 2016

FormLinter—Detect common issues that hurt conversions

A little tool for testing common form issues.

  • Did we remember to give every input a label? (No, placeholders are not an adequate replacement)?
  • Do our labels’ for attributes match our inputs’ ids?
  • Did we take advantage of the url, email, and password input types, or did we forget and just use text?
  • Are our required fields marked as such?

Sunday, November 6th, 2016

bitsofcode | Tools for Developing Accessible Websites

Ire rounds up a bunch of tools you can use to test accessibility, from dev tools to Tenon.

Tuesday, October 25th, 2016

Web Bloat Score Calculator

Here’s an interesting metric for measuring performance: take the overall page weight of a URL and divide it by the file size of the screenshot of that URL.

Monday, September 19th, 2016

What, Exactly, Makes Something A Progressive Web App? | Infrequently Noted

Alex runs through the features that a progressive web app must have, should have, and would be nice to have.

In general, installability criteria are tightening. Today’s Good-To-Haves may become part of tomorrow’s baseline. The opposite is unlikely because at least one major browser has made a strong commitment to tightening up the rules for installability.

Right now, this is in the nice-to-have category:

Mobile-friendly, not mobile-only.

Personally, I’d put that in the must-have category, and not just for progressive web apps.

Anyway, read on for some advice on testing and tooling when it comes to evaluating progressive web apps.

Sunday, June 12th, 2016

Research with blind users on mobile devices | Accessibility

Some interesting outcomes from testing gov.uk with blind users of touchscreen devices:

Rather than reading out the hierarchy of the page, some of the users navigated by moving their finger around to ‘discover’ content.

This was really interesting - traditionally good structure for screen readers is about order and hierarchy. But for these users, the physical placement on the screen was also really important (just as it is for sighted users).

Thursday, May 12th, 2016

The Sonos Pattern Library — zdfs

There’s a lot I disagree with here. I don’t think this pattern library process is very elegant or scalable, and it certainly wouldn’t work for me.

But I’m still linking to it. Why? Because I think it’s absolutely wonderful that people share their processes like this. It doesn’t matter one whit whether or not it would work for me.

Frontend development may have gotten a lot more complicated, but the simple premise of sharing what you’ve learned hasn’t.

I couldn’t agree more!