Link tags: robots.txt



Robots.txt - Jim Nielsen’s Blog

I realized why I hadn’t yet added any rules to my robots.txt: I have zero faith in it.

Documentation for GPTBot - OpenAI API

Now that the horse has bolted—and ransacked the web—you can shut the barn door:

To disallow GPTBot to access your site you can add the GPTBot to your site’s robots.txt:

User-agent: GPTBot
Disallow: /

User-agent: zombies Disallow: /brains