A file at /robots.txt that tells search engines and AI crawlers which paths to crawl and which to skip.
Last updated ยท Reviewed by Ali Hassan
robots.txt blocks crawling but does not block indexing on its own. Use meta robots noindex for that. A modern robots.txt also lists rules for GPTBot, ClaudeBot, Google Extended, PerplexityBot, and CCBot.
Related terms
XML Sitemap
A list of every canonical URL on your site, submitted to Google Search Console for indexing.
Google Search Console
Google's free tool that shows which keywords drive clicks to your site, plus indexing and Core Web Vitals data.
AEO (Answer Engine Optimization)
Optimizing pages so AI engines like ChatGPT, Perplexity, and Google AI Overviews cite you in answers.
We have shipped hundreds of fixes for exactly this kind of issue. Book a 20 minute call and we will tell you straight whether it is a quick fix or a bigger root cause.