Tags: google

Back when the web was young, it wasn’t yet clear what the rules were. Like, could you really just link to something without asking permission?

Then came some legal rulings to establish that, yes, on the web you can just link to anything without checking if it’s okay first.

What about search engines and directories? Technically they’re rifling through all the stuff we publish and reposting snippets of it. Is that okay?

Again, through some legal precedents—but mostly common agreement—everyone decided that on balance it was fine. After all, those snippets they publish are helping your site get traffic.

In short order, search came to rule the web. And Google came to rule search.

The mutually beneficial arrangement persisted uneasily. Despite Google’s search results pages getting worse and worse in recent years, the company’s huge market share of search means you generally want to be in their good books.

Google’s business model relies on us publishing web pages so that they can put ads around the search results linking to that content, and we rely on Google to send people to our websites by responding smartly to search queries.

That has now changed. Instead of responding to search queries by linking to the web pages we’ve made, Google is instead generating dodgy summaries rife with hallucina… lies (a psychic hotline, basically).

Google still benefits from us publishing web pages. We no longer benefit from Google slurping up those web pages.

With AI, tech has broken the web’s social contract:

Google has steadily been manoeuvring their search engine results to more and more replace the pages in the results.

As Chris puts it:

Me, I just think it’s fuckin’ rude.

Google is a portal to the web. Google is an amazing tool for finding relevant websites to go to. That was useful when it was made, and it’s nothing but grown in usefulness. Google should be encouraging and fighting for the open web. But now they’re like, actually we’re just going to suck up your website, put it in a blender with all other websites, and spit out word smoothies for people instead of sending them to your website. Instead.

Ben proposes an update to robots.txt that would allow us to specify licensing information:

Robots.txt needs an update for the 2020s. Instead of just saying what content can be indexed, it should also grant rights.

Like crawl my site only to provide search results not train your LLM.

It’s a solid proposal. But Google has absolutely no incentive to implement it. They hold all the power.

Or do they?

There is still the nuclear option in robots.txt:

User-agent: Googlebot
Disallow: /

That’s what Vasilis is doing:

I have been looking for ways to not allow companies to use my stuff without asking, and so far I coulnd’t find any. But since this policy change I realised that there is a simple one: block google’s bots from visiting your website.

The general consensus is that this is nuts. “If you don’t appear in Google’s results, you might as well not be on the web!” is the common cry.

I’m not so sure. At least when it comes to personal websites, search isn’t how people get to your site. They get to your site from RSS, newsletters, links shared on social media or on Slack.

And isn’t it an uncomfortable feeling to think that there’s a third party service that you absolutely must appease? It’s the same kind of justification used by people who are still on Twitter even though it’s now a right-wing transphobic cesspit. “If I’m not on Twitter, I might as well not be on the web!”

The situation with Google reminds me of what Robin said about Twitter:

The speed with which Twitter recedes in your mind will shock you. Like a demon from a folktale, the kind that only gains power when you invite it into your home, the platform melts like mist when that invitation is rescinded.

We can rescind our invitation to Google.

When the web.dev team at Google contacted Clearleft about writing a course on responsive design, our eyes lit up.

This was clearly a good fit. For one thing, Clearleft has been pioneering responsive design from day one—we helped launch some of the first responsive sites in the UK. But there was another reason why this partnership sounded good: we had the same approach to writing and sharing.

Ever since Clearleft was founded in 2005 we’ve taken on board the motto of the World Wide Web itself: let’s share what we know. As well as doing the work, we enjoy sharing how the work gets done. Whether it’s case studies, blog posts, podcasts, or conference talks, we’re always thinking about ways to contribute to the web community.

Many other great resources have contributed to our collective knowledge: A List Apart, CSS Tricks, Smashing Magazine, Mozilla Developer Network, and more recently web.dev, which has become an excellent resource for front-end developers. But they wanted to make sure that designers were also included. Una described her plan for a fifteen-part course on modern responsive design aimed at web designers.

It was ambitious. The plan included some cutting edge technology that’s just shipping in browsers now. It sounded daunting and exciting in equal measure. Mostly it sounded like far too good an opportunity for Clearleft to pass up so we jumped on it.

With my fellow Clearlefties otherwise engaged in client work, it fell to me to tap out the actual words. Fortunately I’ve had plenty of experience with my own website of moving my fingers up and down on a keyboard in attempt to get concepts out of my head and onto the screen. I familiarised myself with the house style and got to work.

I had lots of help from the Chrome developer relations team at Google. Project management (thanks, Terry!), technical editing (thanks, Adam!), and copy editing (thanks, Rachel!) were provided to me.

Working with Rachel again was a real treat—she wrote the second edition of my book, HTML5 For Web Designers. Every time she suggested a change to something I had written, I found myself slapping my forehead and saying “Of course! That’s so much better!” It felt great to have someone else be a content buddy for me.

We had a weekly video call to check in and make sure everything was on track. There was also an epic spreadsheet to track the flow of each module as they progressed from outline to first draft to second draft.

Those were just the stages when the words were in a Google doc. After that, the content moved to Github and there was a whole other process to shepherd it towards going live.

Take note of the license in that repo: Creative Commons Attribution 3.0. That means that you—or anyone—is free to use and reuse all the material (as long as you include a credit). I think I might republish the fifteen articles on my site at some point.

If you’d like to peruse the outcome of this collaboration between Clearleft and Google, head on over to web.dev and learn responsive design. Then feel free to share it!

1. Introduction
2. Media queries
3. Internationalization
4. Macro layouts
5. Micro layouts
6. Typography
7. Responsive images
8. The picture element
9. Icons
10. Theming
11. Accessibility
12. Interaction
13. User interface patterns
14. Media features
15. Screen configurations