The doorman file every site needs
robots.txt is the first thing a search-engine crawler requests from your domain — a plain-text doorman standing at yoursite.com/robots.txt, telling bots which corridors are open and which are off-limits. A missing file is survivable (everything gets crawled); a broken one is a catastrophe, and a single mistyped line has deleted entire businesses from Google. This generator produces syntactically correct rules from plain choices, so the doorman says exactly what you mean.
How to build your robots.txt
- Pick a policy: allow everything (right for most sites), block everything (staging/dev sites), or custom rules.
- In custom mode, list paths to disallow — one per line, like
/wp-admin/or/private/— and any exceptions to re-allow. - Add your sitemap URL (strongly recommended) and an optional crawl-delay.
- Download
robots.txtand upload it to your site's root directory.
Reading the rules you just made
User-agent: * addresses all bots; Disallow: /private/ asks them to skip that path and everything under it; Allow: carves exceptions out of broader blocks (the classic WordPress pair: disallow /wp-admin/ but allow /wp-admin/admin-ajax.php, which front-end features need). Paths match by prefix — /private blocks /private-files/ too, so trailing slashes matter. The Sitemap: line is pure profit: it hands crawlers a map of everything you do want indexed.
The two mistakes that end careers
First: Disallow: / — one character of path, and the entire site is asked to vanish from search. It belongs only on staging servers, and forgetting to remove it at launch is a legendary post-mortem. Second, subtler: robots.txt is not a security tool. Disallowed paths aren't hidden — the file is public, and listing /secret-admin/ in it is literally publishing a map to it. Use it to manage crawling; use authentication to manage secrets. Also know: a disallowed URL can still appear in results (link-only, no snippet) if other sites link it — true de-indexing needs a noindex meta tag instead.
Deployment checklist
- Root only: the file works at
/robots.txt, never in a subfolder. - Verify live: visit yoursite.com/robots.txt after upload — what you see is what bots get.
- Sitemap line included? If you run our ToolNest plugin, that's
https://yoursite.com/sitemap.xml. - Crawl-delay: only if bots genuinely strain your server; Google ignores it, Bing and others respect it.
Thirty seconds of choices, one file in the root, and your crawl budget goes where it earns rankings instead of wandering the back corridors.
Quick reference
| Property | Detail |
|---|---|
| Policies | Allow all / block all / custom paths |
| Custom rules | Disallow + Allow exceptions, prefix-matched |
| Extras | Sitemap URL line, optional crawl-delay |
| Output | Standards-compliant robots.txt download |
| Install location | Site root: yoursite.com/robots.txt |
| Security note | Public file — never list secret paths |
| Cost | Free |
Frequently asked questions
Will Disallow remove a page from Google?
Not reliably — it stops crawling, but the URL can still be indexed from external links (shown without a snippet). To truly remove a page, allow crawling and add a noindex meta tag, or use Search Console's removal tool.
What should a normal WordPress site put here?
Typically: allow everything, disallow /wp-admin/, allow /wp-admin/admin-ajax.php, plus the Sitemap line. WordPress serves a similar virtual default; a real file in the root gives you explicit control.
Is robots.txt case-sensitive?
Paths are — /Private/ and /private/ are different rules. Directive names (User-agent, Disallow) are not. When in doubt, match the exact casing of your real URLs.
Can I block AI crawlers with this?
The generator targets all bots with User-agent: *. Blocking specific crawlers (GPTBot, CCBot etc.) requires named User-agent blocks — generate the base file here, then add those stanzas by hand if you want them.