robots.txt
A file that tells crawlers which paths they may or may not crawl.
What it means
A text file at yourdomain.com/robots.txt with Allow/Disallow rules for crawler access.
Why it matters for indexing
A Disallow on the wrong path blocks crawling entirely — the page can't be indexed even if it's perfect.
How to check it
Open /robots.txt directly, or use the robots.txt report / URL Inspection to confirm a URL isn't blocked.
How FastIndexing helps
Our pre-flight check flags robots.txt blocks so you fix access before submitting.
Frequently asked questions
What is robots.txt used for?
It tells crawlers which paths they may or may not request — managing crawl access and budget. It lives at yourdomain.com/robots.txt.
Does robots.txt prevent indexing?
No — it only controls crawling. A URL blocked in robots.txt can still be indexed (without its content) if it's linked elsewhere. To keep a page out of the index, use noindex and let it stay crawlable.
Where is the robots.txt file?
Always at the site root: https://yourdomain.com/robots.txt. There's one per host — subdomains have their own.