robots.txt
A root-level file that communicates crawl permissions to compliant bots.
What is robots.txt?
Robots.txt can allow or disallow paths for named user agents. It is a crawl directive, not an access-control mechanism, and different AI crawlers may use different tokens. Blocking retrieval can reduce the chance that current site content is discovered.
In practiceWhat does this look like in a real AI answer?
A site allows GPTBot on public documentation while disallowing private utility paths.
Why it matters for AI visibility
AI answers can shape awareness and buying decisions before a person visits a website. Understanding robots.txt helps teams connect what a model says with the prompts, sources, and technical signals that influenced the result. The useful next step is to observe the evidence, compare it over time, and act on the specific gap.
How to measure or use it
- Define the scope. Choose the relevant market, prompts, models, locations, and time window.
- Keep the evidence. Save the complete answer, its citations, and the page or claim being evaluated.
- Compare patterns. Look for repeated movement across prompts and models instead of reacting to one response.
- Take a specific action. Improve the owned page, correct a claim, earn third-party inclusion, or create better evidence.