Journal tags: research

7

sparkline

Making Research Count by Cyd Harrell

The brilliant Cyd Harrell is opening up day two of An Event Apart in Chicago. I’m going to attempt to liveblog her talk on making research count…

Research gets done …and then sits in a report, gathering dust.

Research matters. But how do we make it count? We need allies. Maybe we need more money. Perhaps we need more participation from people not on the product team.

If you’re doing real research on a schedule, sharing it on a regular basis, making people’s eyes light up …then you’ve won!

Research counts when it answers questions that people care about. But you probably don’t want to directly ask “Hey, what questions do you want answered?”

Research can explain oddities in analytics weird feedback from customers, unexpected uses of products, and strange hunches (not just your own).

Curious people with power are the most useful ones to influence. Not just hierarchical power. Engineers often have a lot of power. So ask, “Who is the most curious engineer, and how can I drag them out on a research session with me?”

At 18F, Cyd found that a lot of the nodes of power were in the mid level of the organisation who had been there a while—they know a lot of people up and down the chain. If you can get one of those people excited about research, they can spread it.

Open up your practice. Demystify it. Put as much effort into communicating as into practicing. Create opportunities for people to ask questions and learn.

You can think about communities of practice in the obvious way: people who do similar things to us, and other people who make design decisions. But really, everyone in the organisation is affected by design decisions.

Cyd likes to do office hours. People can come by and ask questions. You could open a Slack channel. You can run brown bag lunches to train people in basic user research techniques. In more conventional organisations, a newsletter is a surprisingly effective tool for sharing the latest findings from research. And use your walls to show work in progress.

Research counts when people can see it for themselves—not just when it’s reported from afar. Ask yourself: who in your organisation is disconnected from their user? It’s difficult for people to maintain their motivation in that position.

When someone has been in the field with you, the data doesn’t have to be explained.

Whoever’s curious. Whoever’s disconnected. Invite them along. Show them what you’re doing.

Think about the qualities of a good invitation (for a party, say). Make the rules clear. Make sure they want to come back. Design the experience of observing research. Make sure everyone has tools. Give everyone a responsibility. Be like Willy Wonka—he gave clear rules to the invitied guests. And sure, things didn’t go great when people broke the rules, but at the end, everyone still went home with the truckload of chocolate they were promised.

People who get to ask a question buy in to the results. Those people feel a sense of ownership for the research.

Research counts when methods fit the question. Think about what the right question is and how you might go about answering it.

You can mix your methods. Interviews. Diary studies. Card sorting. Shadowing. You can ground the user research in competitor analysis.

Back in 2008, Cyd was contacted by a company who wanted to know: how do people really use phones in their cars? Cyd’s team would ride along with people, interviewing them, observing them, taking pictures and video.

Later at the federal government, Cyd was asked: what are the best practices for government digital transformation? How to answer that? It’s so broad! Interviews? Who knows what?

They refined the question: what makes modern digital practices stick within a government entity? They looked at what worked when companies were going online, so see if there was anything that government could learn from. Then they created a set of really focused interview questions. What does digital transformation mean? How do you know when you’re done? What are the biggest obstacles to this work? How do you make changes last?

They used atechnique called cluster recruiting to figure out who else to talk to (by asking participants who else they should be talking to).

There is no one research method that will always work for you. Cutting the right corners at the right time lets you be fast and cheap. Cyd’s bare-bones research kit costs about $20: a notebook, a pen, a consent form, and the price of a cup of coffee. She also created a quick score sheet for when she’s not in a position to have research transcribed.

Always label your assumptions before beginning your research. Maybe you’re assuming that something is a frustrating experience that needs fixing, but it might emerge that it doesn’t need fixing—great! You’ve just saved a whole lotta money.

Research counts when researchers tell the story well. Synthesis works best as a conversational practice. It’s hard to do by yourself. You start telling stories when you come back from the field (sometimes it starts when you’re still out in the field, talking about the most interesting observations).

Miller’s Law is a great conceptual framework:

To understand what another person is saying, you must assume that it is true and try to imagine what it could be true of.

You’re probably familiar with the “five whys”. What about the “five ways”? If people talk about something five different ways, it’s virtually certain that one of them will be an apt metaphor. So ask “Can you say that in a different way?” five time.

Spend as much time on communicating outcomes as you did on executing the work.

After research, play back how many people you spoke to, the most valuable insight you gained, the themes that are emerging. Describe the question you wanted to answer, what answers you got, and what you’re going to do next. If you’re in an organisation that values memos, write a memo. Or you could make a video. Or you could write directly into backlog tickets. And don’t forget the wall work! GDS have wonderfully full walls in their research department.

In the end, the best tool for research is an illuminating story.

Cyd was doing research at the Bakersfield courthouse. The hypothesis was that a lot of people weren’t engaging with technology in the court system. She approached a man named Manuel who was positively quaking. He was going through a custody battle. He said, “I don’t know technology but it doesn’t scare me. I’m shaking because this paperwork just gets to me—it’s terrifying.” He said who would gladly pay for someone to help him with the paperwork. Cyd wrote a report on this story. Months later, they heard people in the organisation asking questions like “How would this help Manuel?”

Sometimes you do have to fight (nicely).

People will push back on the time spent on research—they’ll say it doesn’t fit the sprint plan. You can have a three day research plan. Day 1: write scripts. Day 2: go to the users and talk to them. Day 3: play it back. People on a project spend more time than that in Slack.

People will say you can’t talk to the customers. In that situation, you could talk to people who are in the same sector as your company’s customers.

People will question the return on investment for research. Do it cheaply and show the very low costs. Then people stop talking about the money and start talking about the results.

People will claim that qualitative user research is not statistically significant. That’s true. But research is something else. It answers different question.

People will question whether a senior person needs to be involved. It is not fair to ask the intern to do all the work involved in research.

People will say you can’t always do research. But Cyd firmly believes that there’s always room for some research.

  • Make allies in customer research.
  • Find the most curious engineer on the team, go to lunch with them, and feed them the most interesting research insights.
  • Record a pain point and a send a video to executives.
  • If there’s really no budget, maybe you can get away with not paying incentives, but perhaps you can provide some other swag instead.

One of the best things you can do is be there, non-judgementally, making friends. It takes time, but it works. Research is like a dandelion in flight. Once it’s out and about, taking root, the more that research counts.

Workshops

There’s a veritable smörgåsbord of great workshops on the horizon…

Clearleft presents a workshop with Jan Chipchase on field research in London on May 29th, and again on May 30th. The first day is sold out, but there are still tickets available for the second workshop (tickets are £654). If you’ve read Jan’s beautiful Field Study Handbook, then you’ll know what a great opportunity it is to spend a day in his company. But don’t dilly-dally—that second day is likely to sell out too.

This event is for product teams, designers, researchers, insights teams, in agencies, in-house, local and central government. People who are curious about human interaction, and their place in the world.

I’m really excited that Sarah and Val are finally bringing their web animation workshop to Brighton (I’ve been not-so-subtly suggesting that they do this for a while now). It’s a two day workshop on July 9th and 10th. There are still some tickets available, but probably not for much longer (tickets are £639). The workshop is happening at 68 Middle Street, the home of Clearleft.

This workshop will get you up and running with web animation in less time than it would take to read all the tutorials you have bookmarked. Over two days, you’ll go from beginner or novice web animator to having expert level knowledge of the current web animation landscape. You’ll get an in-depth look at animating with CSS, JavaScript, and SVG through hands-on exercises and learn the most efficient workflows for each.

A bit before that, though, there’s a one-off workshop on responsive web typography from Rich on Thursday, June 29th, also at 68 Middle Street. You can expect the same kind of brilliance that he demonstrated in his insta-classic Web Typography book, but delivered by the man himself.

You will learn how to combine centuries-old craft with cutting edge technology, including variable fonts, to design and develop for screens of all shapes and sizes, and provide the best reading experiences for your modern readers.

Whether you’re a designer or a developer, just starting out or seasoned pro, there will be plenty in this workshop to get your teeth stuck into.

Tickets are just £435, and best of all, that includes a ticket to the Ampersand conference the next day (standalone conference tickets are £235 so the workshop/conference combo is a real bargain). This year’s Ampersand is shaping up to be an unmissable event (isn’t it always?), so the workshop is like an added bonus.

See you there!

Links from a talk

In two weeks time, I’ll be in Seattle for An Event Apart. I’ll be giving a brand new talk. The title is The Way Of The Web (although perhaps a more accurate title would be The Layers Of The Web).

Here’s the description:

Do you ever get overwhelmed by the ever-changing nature of web design and development? Exhausting, isn’t it? How are you supposed to know which technologies and tools you should invest your time in? Will they stick around or will you just have to relearn everything in another few months? Join Jeremy as he takes a tour of the past, present, and future of working on the web. From the building blocks of HTML, CSS, and JavaScript through to frameworks and libraries right up to the latest and greatest Progressive Web Apps, this talk will examine our collective assumptions with a critical eye. By learning from the past, we can make sensible design decisions today to build the web of tomorrow.

There’s a direct evolution line from my previous talks—Resilence and Evaluating Technology—to this new one. (Spoiler: everything I talk about is in some way related to progressive enhancement …even if I never use the words “progressive" or “enhancement" in the talks.)

I’ve been preparing this new talk for months. It started with a mind map—an A3 sheet of paper with disconnected thoughts, like something from the scene in the crime movie where they enter the lair of the serial killer and find a crazy wall.

Then I set it aside and began procrastinating. But it was the good kind of procrastinating, right? I mean, I had made a start and all those thoughts were now bubbling around in my head.

Eventually I forced myself to put things in some sort of order and started creating slides. That’s the beginning of the horrible process bouncing between thinking “this is pretty good!” and “this is absolute crap!” To be honest, I never actually know if a talk is any good until I give it in front of an audience (practice runs at work are great for getting feedback but they’re not the same as doing the talk for real).

Anyway, I think the talk is ready to roll. If you see me giving this talk and you’re interested in diving deeper into the topics raised, I’ve gathered together some of sources I used.

Further Reading

Related posts on adactio.com

Progressive Web Apps

Books

Films

Balance

This year’s Render conference just wrapped up in Oxford. It was a well-run, well-curated event, right up my alley: two days of a single track of design and development talks (see also: An Event Apart and Smashing Conference for other events in this mold that get it right).

One of my favourite talks was from Frances Ng. She gave a thoroughly entertaining account of her journey from aerospace engineer to front-end engineer, filled with ideas about how to get started, and keep from getting overwhelmed in the world of the web.

She recommended taking the time to occasionally dive deep into a foundational topic, pointing to another talk as a perfect example; Ana Balica gave a great presentation all about HTTP. The second half of the talk was about HTTP 2 and was filled with practical advice, but the first part was a thoroughly geeky history of the Hypertext Transfer Protocol, which I really loved.

While I’m mentoring Amber, we’ve been trying to find a good balance between those deep dives into the foundational topics and the hands-on day-to-day skills needed for web development. So far, I think we’ve found a good balance.

When Amber is ‘round at the Clearleft office, we sit down together and work on the practical aspects of HTML, CSS, and (soon) JavaScript. Last week, for example, we had a really great day diving into CSS selectors and specificity—I watched Amber’s knowledge skyrocket over the course of the day.

But between those visits—which happen every one or two weeks—I’ve been giving Amber homework of sorts. That’s where the foundational building blocks come in. Here are the questions I’ve asked so far:

  • What is the difference between the internet and the web?
  • What is the difference between GET and POST?
  • What are cookies?

The first question is a way of understanding the primacy of URLs on the web. Amber wrote about her research. The second question was getting at an understanding of HTTP. Amber wrote about that too. The third and current question is about state on the web. I’m looking forward to reading a write-up of that soon.

We’re still figuring out this whole mentorship thing but I think this balance of research and exercises is working out well.

Research on evaluating technology

I’ve spent the past few months preparing a new talk for An Event Apart San Francisco (and hopefully some more AEAs after that). As always happens, I spent the whole time vacillating between thinking “this is good!” and thinking “this is awful!” I’m still bouncing between those poles. I won’t really know whether the talk is up to snuff until I actually give it to a live audience.

Over the past few years, my presentations have built upon one another. Two years ago, my talk was called Enhance! and it set the groundwork for using a layered approach to web design and development. My 2016 talk, Resilience, follows on with a process and examples for that approach (I also set myself the challenge of delivering a talk about progressive enhancement without ever using the phrase “progressive enhancement”).

My new talk goes a bit meta, but in my mind, it’s very much building on the previous talks. The talk is all about evaluating technology. I haven’t settled on a final title, but I was thinking about something obtuse, like …Evaluating Technology.

Here’s my hastily scribbled description:

We work with technology every day. And every day it seems like there’s more and more technology to understand: graphic design tools, build tools, frameworks and libraries, not to mention new HTML, CSS and JavaScript features landing in browsers. How should we best choose which technologies to invest our time in? When we decide to weigh up the technology choices that confront us, what are the best criteria for doing that? This talk will help you evaluate tools and technologies in a way that best benefits the people who use the websites that we are designing and developing. Let’s take a look at some of the hottest new web technologies like service workers and web components. Together we will dig beneath the hype to find out whether they will really change life on the web for the better.

As ever, I’ll begin and end with a long-zoom pretentious arc of history, but I’ll dive into practical stuff in the middle. That’s become a bit of a cliché for my presentations, but the formula works as a sort of microcosm of a good conference—a mixture of the inspirational and the practical, trying to keep a good balance of both.

For this new talk, the practical focus will be on some web technologies that are riding high on the hype cycle right now: service workers, web components, progressive web apps. I’ll use them as a lens for applying broader questions about how we make decisions about the technologies we embrace, and the technologies we reject.

Technology. Now there’s a big subject. It’s literally the entirety of human history. I had to be careful not to go down too many rabbit holes. I’m still not sure if I’ve succeeded, but I’ve already had to ruthlessly cull some darlings.

One of the nice things that the An Event Apart crew started doing was to provide link lists for each talk to attendees. That gives me an opportunity to touch briefly on a topic in the talk itself, but allow any interested attendees to dive deeper at their leisure.

For this talk on evaluating technology, I’ve put together this list of hyperlinks for further reading, watching, listening, and researching…

People

Papers

Presentations

Books

Wibbly-wobbly timey-wimey stuff

I met up with Remy a few months back to try to help him finalise the line-up for this year’s Full Frontal conference. Remy puts a lot of thought into crafting a really solid line-up. He was in a good position too: the conference was already sold out so he didn’t have to worry about having a big-name speaker to put bums on seats—he could concentrate entirely on finding just the right speaker for the final talk.

He described the kind of “big picture” talk he was looking for, and I started naming some names and giving him some ideas of people to contact.

Imagine my surprise then, when—while we were both in New York for Brooklyn Beta—I received a lengthy email from Remy (pecked out on his phone), saying that he had decided who wanted to do the closing talk at Full Frontal. He wanted me to do it.

Now, this was just a couple of weeks ago so my first thought was “No way! I don’t have enough time to prepare a talk.” It takes me quite a while to prepare a new presentation.

But then he described—in quite some detail—what he wanted me to talk about …and it’s exactly the kind of stuff that I really enjoy geeking out about: long-term thinking, digital preservation, and all that jazz. So I said yes.

That’s why I’ve spent the last couple of weeks quietly freaking out, attempting to marshall my thoughts and squeeze them into Keynote. The title of my talk is Time. Pretentious? Moi?

I’m trying to pack a lot into this presentation. I’ve already had to kill some of my darlings and drop some of the more esoteric stuff, but damn it, it’s hard to still squeeze everything in.

I’ve been immersed in research and link-making, reading and huffduffing all things time-related. In the course of my hypertravels, I discovered that there’s an entire event devoted to “the origins, evolution, and future of public time.” It’s called Time For Everyone and it’s taking place in California …at exactly the same time as Full Frontal.

Here’s the funny thing: the description for the event is exactly the same as the description I gave Remy for my talk:

This thing all things devours:
Birds, beasts, trees, flowers;
Gnaws iron, bites steel;
Grinds hard stones to meal;
Slays king, ruins town,
And beats high mountain down.

If you’re coming along to Full Frontal next Friday, I hope you’ll be in a receptive mood. I also hope that Remy won’t mind that what I’m going to present isn’t exactly what he asked for …but I think it’s interesting stuff.

I just wish I had more time.

Testing Huffduffer’s sign-up

Ever since I launched Huffduffer, one of the features that really caught people’s attention was the sign up form.

I have to admit, I didn’t really think it was that revolutionary an idea. All I was trying to do was make the sign-up process a little friendlier and if web standards have taught us anything, it’s that there’s nothing inherent in the presentation of any element, much less forms. So I made the form more conversational and less blocky and rigid.

Well, it turns out that people love it. I’ve received bucketloads of Twitter messages and emails from people telling me how much they enjoyed the sign-up process.

But amongst all the positive comments I saw about the sign-up form when Huffduffer launched, I saw some armchair UX practitioners wondering about the usability of this somewhat unorthodox approach to forms. Fair point. Without user testing, how can I really know if the mad-libs approach is really working?

Now, it happens that Luke W. likes the Huffduffer sign-up form, as evidenced by a recent chat he had with Jared.

SpoolCast: Moving Beyond Static Forms with Luke Wroblewski on Huffduffer

If anyone knows anything about the usability of web forms, it’s Luke. He literally wrote the book on it.

Not content with simply expressing a liking for the Huffduffer-style of human-friendly form presentation, he decided to put it to the test with Vast.com:

After seeing the Huffduffer form in action, I was curious how it would perform against a traditional form. Would people be more inclined to complete it because of the narrative format? Or would the unfamiliar presentation format confuse people? Thanks to Ron Kurti and the team at Vast.com, I now have some early answers.

Ron and his team ran some A/B testing online that compared a traditional Web form layout with a narrative “Mad Libs” format. In Vast.com’s testing, Mad Libs style forms increased conversion across the board by 25-40%.

That seems to be a statistically-significant result, even accounting for Cennydd’s reality-check on A/B testing.

It’ll be interesting to see if this is the start of a trend. If nothing else, it’s a way of getting designers to think about the presentation of common human-computer interactions, such as signing up to a new website.