A file at /robots.txt that tells search engines and AI crawlers which paths to crawl and which to skip.
Reviewed by Ali Yasin Jatoi · Updated
robots.txt blocks crawling but does not block indexing on its own. Use meta robots noindex for that. A modern robots.txt also lists rules for GPTBot, ClaudeBot, Google Extended, PerplexityBot, and CCBot.
Related terms
XML Sitemap
A list of every canonical URL on your site, submitted to Google Search Console for indexing.
Google Search Console
Google's free tool that shows which keywords drive clicks to your site, plus indexing and Core Web Vitals data.
AEO (Answer Engine Optimization)
Optimizing pages so AI engines like ChatGPT, Perplexity, and Google AI Overviews cite you in answers.
We have shipped hundreds of fixes for exactly this kind of issue. Book a 20 minute call and we will tell you straight whether it is a quick fix or a bigger root cause.
Once you know what the term means, here is how it gets handled.
Every service maps to a specific glossary term. Pick the one that hurts.
The playbook behind every fix. Written from 150 plus site scale.
Learn once, protect forever. Month to month, no lock in.
