I moderated this panel in London last week, all about the growing field of research ops—I genuinely love moderating panels. Here, Richard recounts some of the thought nuggets I prised from the mind casings of the panelists.
Monday, October 14th, 2019
Tuesday, August 27th, 2019
Making Research Count by Cyd Harrell
Research gets done …and then sits in a report, gathering dust.
Research matters. But how do we make it count? We need allies. Maybe we need more money. Perhaps we need more participation from people not on the product team.
If you’re doing real research on a schedule, sharing it on a regular basis, making people’s eyes light up …then you’ve won!
Research counts when it answers questions that people care about. But you probably don’t want to directly ask “Hey, what questions do you want answered?”
Research can explain oddities in analytics weird feedback from customers, unexpected uses of products, and strange hunches (not just your own).
Curious people with power are the most useful ones to influence. Not just hierarchical power. Engineers often have a lot of power. So ask, “Who is the most curious engineer, and how can I drag them out on a research session with me?”
At 18F, Cyd found that a lot of the nodes of power were in the mid level of the organisation who had been there a while—they know a lot of people up and down the chain. If you can get one of those people excited about research, they can spread it.
Open up your practice. Demystify it. Put as much effort into communicating as into practicing. Create opportunities for people to ask questions and learn.
You can think about communities of practice in the obvious way: people who do similar things to us, and other people who make design decisions. But really, everyone in the organisation is affected by design decisions.
Cyd likes to do office hours. People can come by and ask questions. You could open a Slack channel. You can run brown bag lunches to train people in basic user research techniques. In more conventional organisations, a newsletter is a surprisingly effective tool for sharing the latest findings from research. And use your walls to show work in progress.
Research counts when people can see it for themselves—not just when it’s reported from afar. Ask yourself: who in your organisation is disconnected from their user? It’s difficult for people to maintain their motivation in that position.
When someone has been in the field with you, the data doesn’t have to be explained.
Whoever’s curious. Whoever’s disconnected. Invite them along. Show them what you’re doing.
Think about the qualities of a good invitation (for a party, say). Make the rules clear. Make sure they want to come back. Design the experience of observing research. Make sure everyone has tools. Give everyone a responsibility. Be like Willy Wonka—he gave clear rules to the invitied guests. And sure, things didn’t go great when people broke the rules, but at the end, everyone still went home with the truckload of chocolate they were promised.
People who get to ask a question buy in to the results. Those people feel a sense of ownership for the research.
Research counts when methods fit the question. Think about what the right question is and how you might go about answering it.
You can mix your methods. Interviews. Diary studies. Card sorting. Shadowing. You can ground the user research in competitor analysis.
Back in 2008, Cyd was contacted by a company who wanted to know: how do people really use phones in their cars? Cyd’s team would ride along with people, interviewing them, observing them, taking pictures and video.
Later at the federal government, Cyd was asked: what are the best practices for government digital transformation? How to answer that? It’s so broad! Interviews? Who knows what?
They refined the question: what makes modern digital practices stick within a government entity? They looked at what worked when companies were going online, so see if there was anything that government could learn from. Then they created a set of really focused interview questions. What does digital transformation mean? How do you know when you’re done? What are the biggest obstacles to this work? How do you make changes last?
They used atechnique called cluster recruiting to figure out who else to talk to (by asking participants who else they should be talking to).
There is no one research method that will always work for you. Cutting the right corners at the right time lets you be fast and cheap. Cyd’s bare-bones research kit costs about $20: a notebook, a pen, a consent form, and the price of a cup of coffee. She also created a quick score sheet for when she’s not in a position to have research transcribed.
Always label your assumptions before beginning your research. Maybe you’re assuming that something is a frustrating experience that needs fixing, but it might emerge that it doesn’t need fixing—great! You’ve just saved a whole lotta money.
Research counts when researchers tell the story well. Synthesis works best as a conversational practice. It’s hard to do by yourself. You start telling stories when you come back from the field (sometimes it starts when you’re still out in the field, talking about the most interesting observations).
Miller’s Law is a great conceptual framework:
To understand what another person is saying, you must assume that it is true and try to imagine what it could be true of.
You’re probably familiar with the “five whys”. What about the “five ways”? If people talk about something five different ways, it’s virtually certain that one of them will be an apt metaphor. So ask “Can you say that in a different way?” five time.
Spend as much time on communicating outcomes as you did on executing the work.
After research, play back how many people you spoke to, the most valuable insight you gained, the themes that are emerging. Describe the question you wanted to answer, what answers you got, and what you’re going to do next. If you’re in an organisation that values memos, write a memo. Or you could make a video. Or you could write directly into backlog tickets. And don’t forget the wall work! GDS have wonderfully full walls in their research department.
In the end, the best tool for research is an illuminating story.
Cyd was doing research at the Bakersfield courthouse. The hypothesis was that a lot of people weren’t engaging with technology in the court system. She approached a man named Manuel who was positively quaking. He was going through a custody battle. He said, “I don’t know technology but it doesn’t scare me. I’m shaking because this paperwork just gets to me—it’s terrifying.” He said who would gladly pay for someone to help him with the paperwork. Cyd wrote a report on this story. Months later, they heard people in the organisation asking questions like “How would this help Manuel?”
Sometimes you do have to fight (nicely).
People will push back on the time spent on research—they’ll say it doesn’t fit the sprint plan. You can have a three day research plan. Day 1: write scripts. Day 2: go to the users and talk to them. Day 3: play it back. People on a project spend more time than that in Slack.
People will say you can’t talk to the customers. In that situation, you could talk to people who are in the same sector as your company’s customers.
People will question the return on investment for research. Do it cheaply and show the very low costs. Then people stop talking about the money and start talking about the results.
People will claim that qualitative user research is not statistically significant. That’s true. But research is something else. It answers different question.
People will question whether a senior person needs to be involved. It is not fair to ask the intern to do all the work involved in research.
People will say you can’t always do research. But Cyd firmly believes that there’s always room for some research.
- Make allies in customer research.
- Find the most curious engineer on the team, go to lunch with them, and feed them the most interesting research insights.
- Record a pain point and a send a video to executives.
- If there’s really no budget, maybe you can get away with not paying incentives, but perhaps you can provide some other swag instead.
One of the best things you can do is be there, non-judgementally, making friends. It takes time, but it works. Research is like a dandelion in flight. Once it’s out and about, taking root, the more that research counts.
Thursday, August 1st, 2019
This’ll be good—the inside story of the marvelous Zooniverse project as told by Chris Lintott. I’m looking forward to getting my hands on a copy of this book when it comes out in a couple of months.
Wednesday, May 15th, 2019
I completely agree with every single one of Terence’s recommendations here. The difference is that, in my case, they’re just hot takes, whereas he has actually joined the AMP Advisory Committee, joined their meetings, and listened to the concerns of actual publishers.
- AMP isn’t loved by publishers
- AMP is not accessible
- No user research
- AMP spreads fake news
- Signed Exchanges are not the answer
There’s also a very worrying anti-competitive move by Google Search in only showing AMP results to users of Google Chrome.
I’ve been emailing with Paul from the AMP team and I’ve told him that I honestly think that AMP’s goal should be to make itself redundant …the opposite of the direction it’s going in.
As I said in the meeting - if it were up to me, I’d go “Well, AMP was an interesting experiment. Now it is time to shut it down and take the lessons learned back through a proper standards process.”
I suspect that is unlikely to happen. Google shows no sign of dropping AMP. Mind you, I thought that about Google+ and Inbox, so who knows!
Monday, April 8th, 2019
200 discarded objects from a dump in San Francisco, meticulously catalogued, researched, and documented by Jenny Odell. The result is something more revealing than most pre-planned time capsule projects …although this project may be somewhat short-lived as it’s hosted on Tumblr.
Sunday, June 17th, 2018
This looks like a really good (and free!) online book all about design ops.
Monday, June 4th, 2018
I really like the way that this pattern library includes research insights to provide justification for design decisions.
Tuesday, May 15th, 2018
Monday, April 23rd, 2018
There’s a veritable smörgåsbord of great workshops on the horizon…
Clearleft presents a workshop with Jan Chipchase on field research in London on May 29th, and again on May 30th. The first day is sold out, but there are still tickets available for the second workshop (tickets are £654). If you’ve read Jan’s beautiful Field Study Handbook, then you’ll know what a great opportunity it is to spend a day in his company. But don’t dilly-dally—that second day is likely to sell out too.
This event is for product teams, designers, researchers, insights teams, in agencies, in-house, local and central government. People who are curious about human interaction, and their place in the world.
I’m really excited that Sarah and Val are finally bringing their web animation workshop to Brighton (I’ve been not-so-subtly suggesting that they do this for a while now). It’s a two day workshop on July 9th and 10th. There are still some tickets available, but probably not for much longer (tickets are £639). The workshop is happening at 68 Middle Street, the home of Clearleft.
A bit before that, though, there’s a one-off workshop on responsive web typography from Rich on Thursday, June 29th, also at 68 Middle Street. You can expect the same kind of brilliance that he demonstrated in his insta-classic Web Typography book, but delivered by the man himself.
You will learn how to combine centuries-old craft with cutting edge technology, including variable fonts, to design and develop for screens of all shapes and sizes, and provide the best reading experiences for your modern readers.
Whether you’re a designer or a developer, just starting out or seasoned pro, there will be plenty in this workshop to get your teeth stuck into.
Tickets are just £435, and best of all, that includes a ticket to the Ampersand conference the next day (standalone conference tickets are £235 so the workshop/conference combo is a real bargain). This year’s Ampersand is shaping up to be an unmissable event (isn’t it always?), so the workshop is like an added bonus.
See you there!
Wednesday, March 21st, 2018
Links from a talk
In two weeks time, I’ll be in Seattle for An Event Apart. I’ll be giving a brand new talk. The title is The Way Of The Web (although perhaps a more accurate title would be The Layers Of The Web).
Here’s the description:
There’s a direct evolution line from my previous talks—Resilence and Evaluating Technology—to this new one. (Spoiler: everything I talk about is in some way related to progressive enhancement …even if I never use the words “progressive" or “enhancement" in the talks.)
I’ve been preparing this new talk for months. It started with a mind map—an A3 sheet of paper with disconnected thoughts, like something from the scene in the crime movie where they enter the lair of the serial killer and find a crazy wall.
Then I set it aside and began procrastinating. But it was the good kind of procrastinating, right? I mean, I had made a start and all those thoughts were now bubbling around in my head.
Eventually I forced myself to put things in some sort of order and started creating slides. That’s the beginning of the horrible process bouncing between thinking “this is pretty good!” and “this is absolute crap!” To be honest, I never actually know if a talk is any good until I give it in front of an audience (practice runs at work are great for getting feedback but they’re not the same as doing the talk for real).
Anyway, I think the talk is ready to roll. If you see me giving this talk and you’re interested in diving deeper into the topics raised, I’ve gathered together some of sources I used.
- Pace Layering: How Complex Systems Learn and Keep Learning by Stewart Brand
- The Human Computer’s Dreams Of The Future (PDF) by Ida Rhodes
- 3D Glasses On Reality by Kim Stanley Robinson
- The Rule of Least Power by Tim Berners-Lee and Noah Mendelsohn
- Everything Easy Is Hard Again by Frank Chimero
- Over-engineering is under-engineering by Baldur Bjarnason
- The Burden of Precision by Daniel Eden
- The work I like by Ethan Marcotte
- A Sound Of Thunder (PDF) by Ray Bradbury
Related posts on adactio.com
- 02018-03-06 Minimal viable service worker
- 02017-12-23 Ubiquity and consistency
- 02017-11-02 The dConstruct Audio Archive works offline
- 02017-03-15 Progressive Web App questions
- 02017-01-11 Making Resilient Web Design work offline
- 02016-10-18 Choice
- 02015-11-15 Home Screen
Progressive Web Apps
- Make It So by Nathan Shedroff and Christopher Noessel
- How Buildings Learn by Stewart Brand
- Time Travel by James Gleick
Tuesday, August 22nd, 2017
If research on biases has told us anything, it is that humans make better decisions when we learn to recognize and correct for bias.
Sunday, April 2nd, 2017
This year’s Render conference just wrapped up in Oxford. It was a well-run, well-curated event, right up my alley: two days of a single track of design and development talks (see also: An Event Apart and Smashing Conference for other events in this mold that get it right).
One of my favourite talks was from Frances Ng. She gave a thoroughly entertaining account of her journey from aerospace engineer to front-end engineer, filled with ideas about how to get started, and keep from getting overwhelmed in the world of the web.
She recommended taking the time to occasionally dive deep into a foundational topic, pointing to another talk as a perfect example; Ana Balica gave a great presentation all about HTTP. The second half of the talk was about HTTP 2 and was filled with practical advice, but the first part was a thoroughly geeky history of the Hypertext Transfer Protocol, which I really loved.
While I’m mentoring Amber, we’ve been trying to find a good balance between those deep dives into the foundational topics and the hands-on day-to-day skills needed for web development. So far, I think we’ve found a good balance.
But between those visits—which happen every one or two weeks—I’ve been giving Amber homework of sorts. That’s where the foundational building blocks come in. Here are the questions I’ve asked so far:
- What is the difference between the internet and the web?
- What is the difference between GET and POST?
- What are cookies?
The first question is a way of understanding the primacy of URLs on the web. Amber wrote about her research. The second question was getting at an understanding of HTTP. Amber wrote about that too. The third and current question is about state on the web. I’m looking forward to reading a write-up of that soon.
We’re still figuring out this whole mentorship thing but I think this balance of research and exercises is working out well.
Sunday, March 12th, 2017
Excellent and practical advice for before, during, and after research sessions and usability tests.
Tuesday, January 31st, 2017
The ‘Credit Card Number’ Field Must Allow and Auto-Format Spaces (80% Don’t) - Articles - Baymard Institute
A deep dive into formatting credit card numbers with spaces in online forms.
Monday, November 14th, 2016
This is a thorough write-up of an interesting case where SVG looks like the right tool for the job, but further research leads to some sad-making conclusions.
I love SVG. It’s elegant, scalable and works everywhere. It’s perfect for mobile… as long as it doesn’t move. There is no way to animate it smoothly on Android.
Monday, October 24th, 2016
Research on evaluating technology
I’ve spent the past few months preparing a new talk for An Event Apart San Francisco (and hopefully some more AEAs after that). As always happens, I spent the whole time vacillating between thinking “this is good!” and thinking “this is awful!” I’m still bouncing between those poles. I won’t really know whether the talk is up to snuff until I actually give it to a live audience.
Over the past few years, my presentations have built upon one another. Two years ago, my talk was called Enhance! and it set the groundwork for using a layered approach to web design and development. My 2016 talk, Resilience, follows on with a process and examples for that approach (I also set myself the challenge of delivering a talk about progressive enhancement without ever using the phrase “progressive enhancement”).
My new talk goes a bit meta, but in my mind, it’s very much building on the previous talks. The talk is all about evaluating technology. I haven’t settled on a final title, but I was thinking about something obtuse, like …Evaluating Technology.
Here’s my hastily scribbled description:
As ever, I’ll begin and end with a long-zoom pretentious arc of history, but I’ll dive into practical stuff in the middle. That’s become a bit of a cliché for my presentations, but the formula works as a sort of microcosm of a good conference—a mixture of the inspirational and the practical, trying to keep a good balance of both.
For this new talk, the practical focus will be on some web technologies that are riding high on the hype cycle right now: service workers, web components, progressive web apps. I’ll use them as a lens for applying broader questions about how we make decisions about the technologies we embrace, and the technologies we reject.
Technology. Now there’s a big subject. It’s literally the entirety of human history. I had to be careful not to go down too many rabbit holes. I’m still not sure if I’ve succeeded, but I’ve already had to ruthlessly cull some darlings.
One of the nice things that the An Event Apart crew started doing was to provide link lists for each talk to attendees. That gives me an opportunity to touch briefly on a topic in the talk itself, but allow any interested attendees to dive deeper at their leisure.
For this talk on evaluating technology, I’ve put together this list of hyperlinks for further reading, watching, listening, and researching…
- Design Principles
- The Extensible Web Manifesto
- Developer Fallacies
Progressive Web Apps
- Home Screen
- Regressive Web Apps
- The Progressive Web App Dev Summit
- The Imitation Game
- Progressive Web Apps: Escaping Tabs Without Losing Our Soul by Alex Russell
- The Building Blocks of Progressive Web Apps by Ada Rose Edwards
- Progressive Web Apps: The Long Game by Remy Sharp
- What, Exactly, Makes Something A Progressive Web App? by Alex Russell
- Reports and Working Notes on DNA by Rosalind Franklin
- I, Pencil by Leonard E. Read
- HTML Design Principles edited by Anne van Kesteren and Maciej Stachowiak
- Sketch of The Analytical Engine Invented by Charles Babbage by L. F. Menabrea with notes upon the memoir by the translator Ada Augusta, Countess of Lovelace
- The Coming Technological Singularity: How to Survive in the Post-Human Era by Vernor Vinge
- The Real World of Technology by Ursula M. Franklin @ the CBC Massey Lectures, 1989
- The Triumph Of Technology by Lord Sir Alec Broers @ the BBC Reith Lectures, 2005
- How Technology Evolves by Kevin Kelly @ TED, 2005
- When Ideas Have Sex by Matt Ridley @ TED, 2010
- How I Built A Toaster—From Scratch by Thomas Thwaites @ TED, 2010
- Admiral Shovel and the Toilet Roll by James Burke @ dConstruct, 2012
- Unexpected Item In The Bagging Area by Dan Williams @ dConstruct, 2013
- Hypertext As An Agent Of Change by Mandy Brown @ dConstruct 2014
- The Humane Representation Of Thought by Bret Victor @ the UIST and SPLASH conferences, 2014
- Our Comrade The Electron by Maciej Cegłowski @ Webstock, 2014
- Step Off This Hurtling Machine by Alex Feyerke @ JSConf.au, 2014
- The Moral Economy of Tech by Maciej Cegłowski @ the Society For The Advancement Of Socio-Economics, 2016
- The Real World Of Technology by Ursula M. Franklin
- The Rational Optimist by Matt Ridley
- What Technology Wants by Kevin Kelly
- The Inevitable by Kevin Kelly
- Where Good Ideas Come From: The Natural History of Innovation by Steven Johnson
- How We Got To Now: Six Innovations That Made The Modern World by Steven Johnson
- 101 Unuseless Japanese Inventions: The Art of Chindogu by Kenji Kawakami
- The Toaster Project (Or A Heroic Attempt to Build a Simple Electric Appliance from Scratch) by Thomas Thwaites
- Connections by James Burke
Tuesday, August 2nd, 2016
The security research that went into improving the spec for the Battery Status API. This is why it’s so important that the web holds itself to high standard.
Even most unlikely mechanisms bring unexpected consequences from privacy point of views. That’s why it is necessary to analyze new features, standards, designs, architectures - and products with a privacy angle. This careful process will yield results, decrease the number of issues, abuses and unwelcome surprizes.
Thursday, February 11th, 2016
An examination of how sites like The Session are meshing with older ideas of traditional Irish music:
There is a very interesting tension at play here – one that speaks directly to the design of new technologies. On the one hand, Irish musicians appear to be enthusiastically adopting digital media to establish a common repertoire of tunes, while on the other the actual performance of these tunes in a live session is governed by a strong etiquette that emphasizes the importance of playing by ear.
There’s an accompanying paper called Supporting Traditional Music-Making: Designing for Situated Discretion (PDF).
Friday, January 22nd, 2016
A fascinating slice of ethnographic research in Myanmar by Craig. There’s no mention of the web, which is certainly alarming, but then again, that’s not the focus of the research.
Interestingly, while Facebook is all omnipresent and dominant, nobody is using it the way that Facebook wants: all the accounts are basically “fake”.
What I found fascinating are the ways that people have found to bypass app stores. They’re basically being treated as damage and routed ‘round. So while native apps are universal, app stores would appear to be a first world problem.
Now if there were only some kind of universally accessible distribution channel that didn’t require any kind of installation step …hmmm.