robots.txt can block AI training while keeping search exposure for OpenAI bots, but Google-Extended ties training and citation into one switch
A robots.txt file decides whether AI crawlers may use your content, and providers differ in how finely you can control them. OpenAI documents independent tokens so you can allow search and block training, while Google-Extended bundles training and citation into a single lever.