Tags: testing



Tuesday, July 7th, 2020

We need more inclusive web performance metrics | Filament Group, Inc.

Good point. When we talk about perceived performance, the perception in question is almost always visual. We should think more inclusively than that.

Thursday, June 25th, 2020

Accessible to some - Manuel Matuzović

A score of 100 in Lighthouse or 0 errors in axe doesn’t mean that you’re done, it means that you’re ready to start manual testing and testing with real users, if possible.

Tuesday, June 23rd, 2020

Introduction to Screen Readers Using Voiceover | Gymnasium

This is a great short introduction to using VoiceOver with Safari by the one and only Ethan Marcotte.

Thursday, April 30th, 2020

Prioritizing users in a crisis: Building the California COVID-19 response site

This is a great case study of the excellent California COVID-19 response site. Accessibility and performance are the watchwords here.

Want to know their secret weapon?

A $20 device running Android 9, with no contract commitment has been one of the most useful and effective tools in our effort to be accessible.

Leaner, faster sites benefit everybody, but making sure your applications run smoothly on low-end hardware makes a massive difference for those users.

Tuesday, March 10th, 2020

Lighthouse bookmarklet

I use Firefox. You should too. It’s fast, secure, and more privacy-focused than the leading browser from the big G.

When it comes to web development, the CSS developer tooling in Firefox is second-to-none. But when it comes to JavaScript and network-related debugging (like service workers), Chrome’s tools are currently better than Firefox’s (for now). For example, Chrome has a tab in its developer tools that lets you run Lighthouse on the currently open tab.

Yesterday, I got the Calibre newsletter, which always has handy performance-related links from Karolina. She pointed to a Lighthouse extension for Firefox. “Excellent!”, I thought, and I immediately installed it. But I had some qualms about installing a plug-in from Google into a browser from Mozilla, particularly as the plug-in page says:

This is not a Recommended Extension. Make sure you trust it before installing

Well, I gave it a go. It turns out that all it actually does is redirect to the online version of Lighthouse. “Hang on”, I thought. “This could just be a bookmarklet!”

So I immediately uninstalled the browser extension and made this bookmarklet:


Drag that up to your desktop browser’s bookmarks toolbar. Press it whenever you’re on a site that you want to test.

Tuesday, March 3rd, 2020

HTML: The Inaccessible Parts - daverupert.com

Well, this is a grim collection from Dave:

There are some cases where even using plain ol’ HTML causes accessibility problems. I get frustrated and want to quit web development whenever I read about these types of issues. Because if browsers can’t get this right, what hope is there for the rest of us.

It’s worth clicking through each link he lists—the situation is often much more nuanced than simply “Don’t use X.”

Friday, January 10th, 2020

Listen To Me And Not Google: HeydonWorks

We have to stop confusing the excesses of capitalism with the hallmarks of quality. Sometimes Google aren’t better, they’re just more pervasive.

cough AMP cough

Thursday, November 7th, 2019

What I’ve learned about accessibility in SPAs

Nolan writes up what he learned making accessibiity improvements to a single page app. The two big takeways involve letting the browser do the work for you:

Here’s the best piece of accessibility advice for newbies: if something is a button, make it a <button>. If something is an input, make it an <input>. Don’t try to reinvent everything from scratch using <div>s and <span>s.

And then there are all the issues that crop up when you take over the task of handling navigations:

  • You need to manage focus yourself.
  • You need to manage scroll position yourself.

For classic server-rendered pages, most browser engines give you this functionality for free. You don’t have to code anything. But in an SPA, since you’re overriding the normal navigation behavior, you have to handle the focus yourself.

Friday, November 1st, 2019

Location, Privilege and Performant Websites

Testing on a <$100 Android device on a 3G network should be an integral part of testing your website. Not everyone is on a brand-new device or upgrades often, especially with the price point of a high-end phones these days.

When we design and build our websites with the outliers in mind, whether it’s for performance or even user experience, we build an experience that can be easy for all to access and use — and that’s what the web is about, access and information for all.

Wednesday, October 16th, 2019

Beyond automatic accessibility testing: 6 things I check on every website I build - Manuel Matuzović

Six steps that everyone can do to catch accessibility gotchas:

  1. Check image descriptions
  2. Disable all styles
  3. Validate HTML
  4. Check the document outline
  5. Grayscale mode
  6. Use the keyboard

Thursday, September 19th, 2019

An HTML attribute potentially worth $4.4M to Chipotle - Cloud Four

When I liveblogged Jason’s talk at An Event Apart in Chicago, I included this bit of reporting:

Jason proceeds to relate a long and involved story about buying burritos online from Chipotle.

Well, here is that story. It’s a good one, with some practical takeaways (if you’ll pardon the pun):

  1. Use HTML5 input features
  2. Support autofill
  3. Make autofill part of your test plans

Friday, August 23rd, 2019

Is client side A/B testing always a bad idea in your experience? · Issue #53 · csswizardry/ama

Harry enumerates the reasons why client-side A/B testing is terrible:

  • It typically blocks rendering.
  • Providers are almost always off-site.
  • It happens on every page load.
  • No user-benefitting reuse.
  • They likely skip any governance process.

While your engineers are subject to linting, code-reviews, tests, auditors, and more, your marketing team have free rein of the front-end.

Note that the problem here is not A/B testing per se, it’s client-side A/B testing. For some reason, we seem to have collectively decided that A/B testing—like analytics—is something we should offload to the JavaScript parser in the user’s browser.

Thursday, August 22nd, 2019

Mobile E-Commerce UX: Deemphasize ‘Install App’ Ads or Avoid Them Entirely

The test results are in:

During our testing “Install App” banners were the direct and sole cause of several abandonments of some of the world’s largest e-commerce websites.

Read on for details…

Saturday, August 3rd, 2019

LukeW | “Mad Libs” Style Form Increased Conversion by 25-40%

I was chatting with Rachel at work the other day about conversational forms, and I mentioned that I kicked that whole thing off with the mad libs style form on Huffduffer. Here’s the research that Luke later did on whether this style of form could increase conversion.

Saturday, July 27th, 2019

How to test vanilla JS performance | Go Make Things

Did you know about console.time() and console.timeEnd()? I did not.

Tuesday, July 16th, 2019

Initial thoughts on standardizing form controls | Greg Whitworth

Greg has done a lot of research into developer frustrations with customising form controls.

My current thinking in this space, and I know some folks will find this controversial, but I think we should completely standardize in-page form controls with no limitations on their styling capabilities. What do I mean by in-page controls? I am referring to any form control or component that is rendered within the content process. This standardization would include the sub-parts and their related states and how these are exposed (probably through CSS psuedo classes or HTML attributes). This will enable the shadow-dom to be encapsulated while providing web developers with a consistent experience to adjust to match their brand and needs of their site/application.

Monday, May 27th, 2019

Plain Text vs. HTML Emails: Which Is Better? [New Data]

Spoiler: it’s plain text. Every time.

Nothing boosts opens and clicks as well as an old school, plain-text email.

I feel vindicated.

People say they prefer HTML emails ..but they actually prefer plain-text.

This seems like a plausable explanation:

Think about how you email colleagues and friends: Do you usually add images or use well-designed templates? Probably not, and neither does your audience. They’re used to using email to communicate in a personal way, so emails from companies that look more personal will resonate more.

Now get off my lawn, you pesky HTML-email lovin’ kids.

Wednesday, April 10th, 2019

Improving accessibility with accessibility acceptance criteria — Paul Hayes

Wouldn’t it be great if every component in your design system had accessibility acceptance criteria? Paul has some good advice for putting those together:

  • Start with accessibility needs
  • Don’t be too generic
  • Don’t define the solution
  • Iterate criteria

Wednesday, March 13th, 2019

Accessibility Insights

A handy accessibility tool. The browser plug-in is only for Chrome right now (I use Firefox as my main browser) but it’s pretty nifty—the tool for visualising tabbing is really useful.

Tuesday, March 5th, 2019

From Ideation to Iteration: Design Thinking for Work and for Life by Una Kravets

The unstoppable engine of An Event Apart in Seattle rolls onward. The second talk of the second day is from the indominatable Una Kravets. Her talk is called From Ideation to Iteration: Design Thinking for Work and for Life. Here’s the description:

Have you ever had a looming deadline and no idea where to start? Do you have a big task to face but are having trouble figuring out how to get there? Have you ever wanted to learn a technology, or build a side project but didn’t know what to build? In this talk, Una will go over an actionable approach and several techniques for applying design thinking to our work and every aspect of our lives. This includes ideating product features, blog post ideas, or even what general direction we want to move toward in our businesses. We’ll go over traditional approaches and breakout techniques that will leave you feeling more in control and ready to reach your goals.

Let’s see if I can keep up with this…

Una’s going to talk about design thinking. Una does a variety of different work outside her day job—a podcast, dev doodles, etc. Sometimes at work she’s given big, big tasks like “build a design system.” Her reaction is “whut?” How do you even start with a task like that.

Also, we make big goals sometimes. Who makes new year’s resolutions? But what does “get more fit” or “earn more income” even mean?

In this talk, Una will break things down and show how design thinking can be applied to anything.

Design thinking

A stategic, solution-based approach to solving problems.

It’s a process. It’s iterative. It’s used by IBM, Apple, GE, and it’s taught to students at a lot of different universities.

Tim Brown of Ideo points out that there’s a Venn diagram of feasability, desirability, and viability. In the middle is the point of innovation.

The steps are:

  1. Empathise — develop a deep understanding of the challenge
  2. Define — clearly articulate the problem.
  3. Ideate — brainstorm potential solutions.
  4. Prototype — design a protoype to
  5. Test — and iterate.

Una feels that the feedback part is potentially missing there. IBM uses a loop diagram to include feedback. Ideo uses these steps:

  1. Frame a question
  2. Gather inspiration
  3. Generate ideas
  4. Make ideas tangible
  5. Test to learn
  6. Share the story

Another way to think about this is how the teams interact. There’s divergence and convergence throughout. Then there’s the double diamond: design, deliver, discover.

Ideo wrote a book called Design Thinking for Libraries. It has some useful tools and diagrams. Una found this fascinating because it wasn’t specifically about products. In healthcare, GE Health used design thinking for their Adventure Series MRI scanners—it resulted in 80% less need to use sedatives. The solution might seem obvious to us in hindsight, but it wouldn’t have been obvious to medical professionals in their everyday busy lives.

Design thinking is bullshit, says Natasha Jen. She describes how it’s become an over-used term that has lost its value. Una can relate—she gets annoyed when there’s too much talking and not enough doing. Design thinking is not diagrams and sticky notes. It’s a process. It’s very much about doing something to shift perspective. It’s another tool in our toolkit, even if it has become an overused term like “synergy.”

Back in 2014, when Una was working at IBM, she thought design thinking was stupid. It seemed to be all talk, talk, talk. It felt tedious. It was 75% talking and 25% development. The balance wasn’t right.

But it’s also true that solutioning too early leads to cruft. If you end up going back to the drawing board, maybe the time could’ve been better spent doing some design thinking up front. Focus on the problem, not the solution.

Now some developers might be thinking that this is outside their area. But it can really help you in your career. It can help you choose technologies. Also, everyone on the team, regardless of role, is responsible for the product.

1. Empathise

Understand your users and the challenge. This could be a task that a user is trying to accomplish, or it could be you trying to get a raise.

Sometimes we forget who our user is. The techniques in this first step helps us solve their needs, not our needs.

You might have many users that you’re trying to help, but try focusing in on a few. You can create personas. When Una was working at Digital Ocean, the users were developers. The personas reflected this. Do the research to get to know your users.

Next, you can create an empathy map for your users. What are their goals? What are their hopes? What will they gain from your product?

Connect the empathy map to a specific context—a goal and or a scenario that the user is going through.

Bear in mind that there are many layers to your user. There are conscious rational thoughts, but also subconscious emotional thoughts. Empathy mapping helps you understand how to best communicate with your user.

Una shares a real-life scenario of hers: create a new shop-able product that increases time on site. That’s a pretty big goal. She creates a persona for a college-educated woman working in the medical field who commutes on the subway, keeps a skin-caring routing, and is getting into cake-making. Next, Una creates an empathy map for this persona. What she says, thinks, feels, and does. All of this is within the context of browsing your fashion media website casually at work.

2: Define

The problem statement should be:

  • Human-centred,
  • Specific, but not too technical (don’t solutionise too soon),
  • Narrow in scope.

How can we best create a product-highlighting web experience that Rosalyn will enjoy to increase her time on site?

You can use a tool with two columns: As-Is and to-Be. The first column is what users currently do. The second column is what you want to achieve.

3. Ideate

This is the fun part. Good old-fashioned brainstorming is good here. Go for quantity here. Get loads of ideas out.

There’s also a “worst possible ideas” game you can play at this stage. It can be a good ice-breaker.

Have a second round of brainstorming where you play the “yes, and…” game to build on the first round.

When Una was working on The Zoe Report, she found that moodboards were really useful. The iteration cycle was very fast. A moodboard allowed them to skip a lot of the back and forth between design and development. They built the website without any visual design mock-ups. They prototyped quickly, tested quickly, and shipped quickly.

Journey-mapping is the next tool you can use in this ideation phase. Map out the steps between the start and end of a user journey. Keep it simple. This is a great time to refine your product and reduce complexity.

Next, start sketching out ideas. Again, this is a great time to uncover issues and solve problems before things get too defined. But remember, when you’re showcasing your ideas in sketches, too many ideas can lead to analyis paralysis.

Oh dear. Jam. Jam. Jam. Jam. Jam. Yes, Una is using the paradox-of-choice jam example …the study who’s findings could not be reproduced.

4. Prototype

Go forth and build. A prototype can exist on a number of different axes:

  • Representation—the form it takes.
  • Precision—the detail it contains.
  • Interactivity—the extent a user can interact with it.
  • Evolution—the life stage it is at.

There are lot of prototyping tools out there. Prototypr.io helps you find the right tool for you. It breaks things down by fidelity and life cycle.

But not all prototyping has to be digital. Paper prototyping only needs pen, paper, and scissors. Some tips:

  • Use a transparency sheet for forms.
  • Use well-visible and mid-tip pens.
  • Draw up your prototype in black and white—people can get caught up in colour.

But on the web, Una recommends getting to digital as quickly as possible because interaction is such a big part of the experience. That’s why Una likes to prototype in code. But this is still a rapid prototyping phase so don’t get too caught up in the details.

5. Test

Testing with internal teams is fine during the ideation phase, but to understand how users will relate to your product, you need to test with representative people. We are not our users.

As well as the user, have a facilitator, a computer, and a scriber. As a facilitator, it’s a good idea to reduce the amount of input you give a user. Don’t hand-hold too much or you will give away your pre-existing knowledge. Encourage your user to be verbal.

Sessioncam is a tool for creating a heatmap of where people are interacting. There are also tools for tracking clicks or mouse hovers. These all feel so utterly icky to me.

The metrics you might be looking to gather could be click-through rate, time-on-site, etc. But, Una cautions, be very wary of adding all these third-party scripts to your site and slowing it down. Who’s testing the A/B tests?

On Bustle, Una wanted to measure interactions on mobile. They tested different UI elements for interactions. They ended up updating the product with a horizontal swiping component. They were able to improve the product and ship a more refined experience.

6. Review and iterate

Una feels that this step is the most important. Analyse your successes and failures, and plan to improve.

Technology changes over time so what’s feasible and viable also changes.

Design thinking on the daily

You can use design thinking in your everyday life. Maybe you want to learn JavaScript, or write blog posts, or get more fit. Una used design thinking brainstorming to break down her goals, categorise and organise them.

“Get better at JavaScript” is a goal that Una has every year.

Empathise. In this situation, the user is you. You can still create a persona of yourself. Define. Why do you want to get better at JavaScript? Is it about making better use of your time? Ideate. There are so many different ways to learn. There are books and video courses. Or you could have a project to focus on. Break. It. Down! Create actionable steps and define how you will measure progress. Match the list of things you want to learn with the list of possible side projects. Prototype. Don’t take it literally. Just build something. Test. You’re testing on yourself in this case. Review. Una does an annual review. It’s a nice therapeutic exercise and helps her move forward into the next year with actionable goals.

Another goal might be “Write a blog post.”

Empathise. Your users are your potential readers. Who do you have in mind? Make personas. Define. What’s the topic? Ideate. If you don’t know what to write about, brainstorm. What are you working on at work that you’re learning from? Select one to try. Prototype. Write. Test. Maybe show it to co-workers. Review. How did it go? Good? Bad? Refine your process for the next blog post.

Here’s a goal: “Buy a gift for someone.”

Empathise. What does this person like? What have they enjoyed receiving in the past? Define. Is the gift something they’ll enjoy for a long time? Something they can share? Ideate. Bounce ideas off friends and relatives. Prototype. In this case, this means getting the gift. Review. Did they like it?

“Get Fit.”

In this case, the review part is probably the most important part.

Marie Kondo asks “Does this spark joy?” Ask the same question of your goals.

Remember, design thinking is not just about talking, and sticky notes. It’s about getting in the right headspace for your users.

Design thinking matters—because everything we do, we do for people. Having the tools to see through the lens of those people will help you be a more well-rounded person.