Tags: search

Back when the web was young, it wasn’t yet clear what the rules were. Like, could you really just link to something without asking permission?

Then came some legal rulings to establish that, yes, on the web you can just link to anything without checking if it’s okay first.

What about search engines and directories? Technically they’re rifling through all the stuff we publish and reposting snippets of it. Is that okay?

Again, through some legal precedents—but mostly common agreement—everyone decided that on balance it was fine. After all, those snippets they publish are helping your site get traffic.

In short order, search came to rule the web. And Google came to rule search.

The mutually beneficial arrangement persisted uneasily. Despite Google’s search results pages getting worse and worse in recent years, the company’s huge market share of search means you generally want to be in their good books.

Google’s business model relies on us publishing web pages so that they can put ads around the search results linking to that content, and we rely on Google to send people to our websites by responding smartly to search queries.

That has now changed. Instead of responding to search queries by linking to the web pages we’ve made, Google is instead generating dodgy summaries rife with hallucina… lies (a psychic hotline, basically).

Google still benefits from us publishing web pages. We no longer benefit from Google slurping up those web pages.

With AI, tech has broken the web’s social contract:

Google has steadily been manoeuvring their search engine results to more and more replace the pages in the results.

As Chris puts it:

Me, I just think it’s fuckin’ rude.

Google is a portal to the web. Google is an amazing tool for finding relevant websites to go to. That was useful when it was made, and it’s nothing but grown in usefulness. Google should be encouraging and fighting for the open web. But now they’re like, actually we’re just going to suck up your website, put it in a blender with all other websites, and spit out word smoothies for people instead of sending them to your website. Instead.

Ben proposes an update to robots.txt that would allow us to specify licensing information:

Robots.txt needs an update for the 2020s. Instead of just saying what content can be indexed, it should also grant rights.

Like crawl my site only to provide search results not train your LLM.

It’s a solid proposal. But Google has absolutely no incentive to implement it. They hold all the power.

Or do they?

There is still the nuclear option in robots.txt:

User-agent: Googlebot
Disallow: /

That’s what Vasilis is doing:

I have been looking for ways to not allow companies to use my stuff without asking, and so far I coulnd’t find any. But since this policy change I realised that there is a simple one: block google’s bots from visiting your website.

The general consensus is that this is nuts. “If you don’t appear in Google’s results, you might as well not be on the web!” is the common cry.

I’m not so sure. At least when it comes to personal websites, search isn’t how people get to your site. They get to your site from RSS, newsletters, links shared on social media or on Slack.

And isn’t it an uncomfortable feeling to think that there’s a third party service that you absolutely must appease? It’s the same kind of justification used by people who are still on Twitter even though it’s now a right-wing transphobic cesspit. “If I’m not on Twitter, I might as well not be on the web!”

The situation with Google reminds me of what Robin said about Twitter:

The speed with which Twitter recedes in your mind will shock you. Like a demon from a folktale, the kind that only gains power when you invite it into your home, the platform melts like mist when that invitation is rescinded.

We can rescind our invitation to Google.

I was talking to someone recently about a forgotten battle in the history of the early web. It was a battle between search engines and directories.

These days, when the history of the web is told, a whole bunch of services get lumped into the category of “competitors who lost to Google search”: Altavista, Lycos, Ask Jeeves, Yahoo.

But Yahoo wasn’t a search engine, at least not in the same way that Google was. Yahoo was a directory with a search interface on top. You could find what you were looking for by typing or you could zero in on what you were looking for by drilling down through a directory structure.

Yahoo wasn’t the only directory. DMOZ was an open-source competitor. You can still experience it at DMOZlive.com:

The official DMOZ.com site was closed by AOL on February 17th 2017. DMOZ Live is committed to continuing to make the DMOZ Internet Directory available on the Internet.

Search engines put their money on computation, or to use today’s parlance, algorithms (or if you’re really shameless, AI). Directories put their money on humans. Good ol’ information architecture.

It turned out that computation scaled faster than humans. Search won out over directories.

Now an entire generation has been raised in the aftermath of this battle. Monica Chin wrote about how this generation views the world of information:

Catherine Garland, an astrophysicist, started seeing the problem in 2017. She was teaching an engineering course, and her students were using simulation software to model turbines for jet engines. She’d laid out the assignment clearly, but student after student was calling her over for help. They were all getting the same error message: The program couldn’t find their files.

Garland thought it would be an easy fix. She asked each student where they’d saved their project. Could they be on the desktop? Perhaps in the shared drive? But over and over, she was met with confusion. “What are you talking about?” multiple students inquired. Not only did they not know where their files were saved — they didn’t understand the question.

Gradually, Garland came to the same realization that many of her fellow educators have reached in the past four years: the concept of file folders and directories, essential to previous generations’ understanding of computers, is gibberish to many modern students.

Dr. Saavik Ford confirms:

We are finding a persistent issue with getting (undergrad, new to research) students to understand that a file/directory structure exists, and how it works. After a debrief meeting today we realized it’s at least partly generational.

We live in a world ordered only by search:

While some are quite adept at using labels, tags, and folders to manage their emails, others will claim that there’s no need to do because you can easily search for whatever you happen to need. Save it all and search for what you want to find. This is, roughly speaking, the hot mess approach to information management. And it appears to arise both because search makes it a good-enough approach to take and because the scale of information we’re trying to manage makes it feel impossible to do otherwise. Who’s got the time or patience?

There are still hold-outs. You can prise files from Scott Jenson’s cold dead hands.

More recently, Linus Lee points out what we’ve lost by giving up on directory structures:

Humans are much better at choosing between a few options than conjuring an answer from scratch. We’re also much better at incrementally approaching the right answer by pointing towards the right direction than nailing the right search term from the beginning. When it’s possible to take a “type in a query” kind of interface and make it more incrementally explorable, I think it’s almost always going to produce a more intuitive and powerful interface.

Directory structures still make sense to me (because I’m old) but I don’t have a problem with search. I do have a problem with systems that try to force me to search when I want to drill down into folders.

I have no idea what Google Drive and Dropbox are doing but I don’t like it. They make me feel like the opposite of a power user. Trying to find a file using their interfaces makes me feel like I’m trying to get a printer to work. Randomly press things until something happens.

Anyway. Enough fist-shaking from me. I’m going to ponder Linus’s closing words. Maybe defaulting to a search interface is a cop-out:

Text search boxes are easy to design and easy to add to apps. But I think their ease on developers may be leading us to ignore potential interface ideas that could let us discover better ideas, faster.

I’ve added five more faces to the UX London line-up.

Irina Rusakova will be giving a talk on day one, the day that focuses on research. Her talk on designing with the autistic community is one I’m really looking forward to.

Also on day one, my friend and former Clearleftie Cennydd Bowles will be giving a workshop called “What could go wrong?” He literally wrote the book on ethical design.

Day two is all about creation. My co-worker Chris How will be speaking. “Nepotism!” you cry! But no, Chris is speaking because I had the chance to his talk—called “Unexpectedly obvious”—and I thought “that’s perfect for UX London!”:

Let him take you on a journey through time and across the globe sharing stories of designs that solve problems in elegant if unusual ways.

Also on day two, you’ve got two additional workshops. Lou Downe will be running a workshop on designing good services, and Giles Turnbull will be running a workshop called “Writing for people who hate writing.”

I love that title! Usually when I contact speakers I don’t necessarily have a specific talk or workshop in mind, but I knew that I wanted that particular workshop from Giles.

When I wrote to Giles to ask come and speak, I began by telling how much I enjoy his blog—I’m a long-time suscriber to his RSS feed. He responded and said that he also reads my blog—we’re blog buddies! (That’s a terrible term but there should be a word for people who “know” each other only through reading each other’s websites.)

Anyway, that’s another little treasure trove of speakers added to the UX London roster:

• Irina Rusakova
• Cennydd Bowles
• Chris How
• Lou Downe
• Giles Turnbull

That’s nineteen speakers already and we’re not done yet—expect further speaker announcements soon. But don’t wait on those announcements before getting your ticket. Get yours now!

It wasn’t that long ago that I told you about some of the speakers that have been added to the line-up for UX London in June: Steph Troeth, Heldiney Pereira, Lauren Pope, Laura Yarrow, and Inayaili León. Well, now I’ve got another five speakers to tell you about!

Aleks Melnikova will be giving a workshop on day one, June 28th—that’s the day with a focus on research.

Stephanie Marsh—who literally wrote the book on user research—will also be giving a workshop that day.

Before those workshops though, you’ll get to hear a talk from the one and only Kat Zhou, the creator of Design Ethically. By the way, you can hear Kat talking about deceptive design in a BBC radio documentary.

Day two has a focus on content design so who better to deliver a workshop than Sarah Winters, author of the Content Design book.

Finally, on day three—with its focus on design systems—I’m thrilled to announce that Adekunle Oduye will be giving a talk. He too is an author. He co-wrote the Design Engineering Handbook. I also had the pleasure of talking to Adekunle for an episode of the Clearleft podcast on design engineering.

So that’s another five excellent speakers added to the line-up:

• Aleks Melnikova
• Stephanie Marsh
• Kat Zhou
• Sarah Winters
• Adekunle Oduye

That’s a total of fifteen speakers so far with more on the way. And I’ll be updating the site with more in-depth descriptions of the talks and workshops soon.

If you haven’t yet got your ticket for UX London, grab one now. You can buy tickets for individual days, or to get the full experience and the most value, get a ticket for all three days.

If you’ve signed up to the UX London newsletter then this won’t be news to you, but more speakers have been added to the line-up.

Steph Troeth will be giving a workshop on day one. That’s the day with a strong focus on research, and when it comes to design research, Steph is unbeatable. You can hear some of her words of wisdom in an episode of the Clearleft podcast all about design research.

Heldiney Pereira will be speaking on day two. That’s the day with a focus on content design. Heldiney previously spoke at our Content By Design event and it was terrific—his perspective on content design as a product designer is invaluable.

Lauren Pope will also be on day two. She’ll be giving a workshop. She recently launched a really useful content audit toolkit and she’ll be bringing that expertise to her UX London workshop.

Day three is going to have a focus on design systems (and associated disciplines like design engineering and design ops). Both Laura Yarrow and Inayaili León will be giving talks on that day. You can expect some exciting war stories from the design system trenches of HM Land Registry and GitHub.

I’ve got some more speakers confirmed but I’m going to be a tease and make you wait a little longer for those names. But check out the line-up so far! This going to be such an excellent event (I know I’m biased, but really, look at that line-up!).

June 28th to 30th. Tobacco Dock, London. Get your ticket if you haven’t already.