Free Custom Robots.txt generator For Blogger (Blogspot)
Robots.txt Generator for Blogger
Build a Blogger-ready robots.txt in seconds. Live preview, plain-English explanations, AdSense-safe rules and a path tester. Free, and everything runs in your browser.
Preset
Crawl policy
Sitemaps
Advanced: crawlers, crawl-delay, custom rules
Path tester · is a URL blocked?
What each line does
What is a robots.txt file? (beginner basics)
A robots.txt file is a plain text file at the root of your website. It tells search engine crawlers which URLs they may crawl and which to skip, and it can point them to your sitemap.
A rulebook for crawlers
Bots like Googlebot and Bingbot read it first, then follow the rules written for them before crawling your pages.
One file per host
On Blogger it lives at yourblog.blogspot.com/robots.txt. A custom domain has its own file, and every subdomain needs a separate one.
A request, not a lock
Good bots obey it, bad bots can ignore it. Never use it to hide private data, and remember it does not delete pages already indexed.
Robots.txt syntax cheat sheet
Every robots.txt file is built from a few simple directives. Learn these and you can read any file.
| Directive | What it does | Example |
|---|---|---|
| User-agent | Names the crawler the next rules apply to. * means all crawlers. | User-agent: Googlebot |
| Disallow | Blocks a path from being crawled. An empty value blocks nothing. | Disallow: /search |
| Allow | Permits a path, often to make an exception inside a blocked folder. | Allow: / |
| Sitemap | Gives the full URL of your XML sitemap. It can appear anywhere in the file. | Sitemap: https://site.com/sitemap.xml |
| Crawl-delay | Asks bots to wait between requests. Google ignores it. | Crawl-delay: 10 |
| * wildcard | Matches any run of characters inside a path. | Disallow: /*?m=1 |
| $ end marker | Matches the end of a URL exactly. | Disallow: /*.pdf$ |
| # comment | Notes for humans. Crawlers skip the line. | # block search pages |
Rules are case-sensitive and match from the start of the path. When several rules match a URL, the most specific (longest) one wins, and Google prefers Allow if two rules tie.
Robots.txt examples you can copy
Copy-ready files for the most common needs: Blogger, disallow all, allow all, WordPress, folders and AI bots.
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: https://yourblog.blogspot.com/sitemap.xml
User-agent: * Disallow: /
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/wp-sitemap.xml
User-agent: * Disallow: /private/ Disallow: /*.pdf$
User-agent: GPTBot User-agent: CCBot User-agent: ClaudeBot User-agent: Google-Extended Disallow: /
How to add a custom robots.txt in Blogger
Blogger needs no file upload. You paste the text into a setting. Menu names can change slightly, so look for the same wording.
Pick a preset above
Choose Blogger SEO for most blogs, set your blog URL, then check the preview on the right.
Copy the file
Press Copy. You can also press .txt if you want a downloaded copy for your records.
Open Blogger Settings
Sign in to Blogger, pick your blog, open Settings and scroll to Crawlers and indexing.
Enable custom robots.txt
Turn on Enable custom robots.txt, then paste your rules into the Custom robots.txt box and save.
Verify it is live
Open yourblog.blogspot.com/robots.txt in a browser. It should match what you pasted. Then check it in Google Search Console.
Robots.txt vs noindex vs canonical: which should you use?
Blogger also has "custom robots header tags" under the same settings. These common values work well for most blogs.
allLet search engines index and follow your homepage.
noindexKeeps thin label, search and date pages out of results while links still pass through.
allYour real posts and pages stay fully indexable.
| Tool | Stops crawling? | Stops indexing? | Best for |
|---|---|---|---|
| robots.txt | Yes | No. The URL can still appear without a description. | Saving crawl budget, blocking whole sections. |
| noindex tag | No | Yes, once the page is crawled. | Keeping thin or duplicate pages out of results. |
| Canonical tag | No | No, it consolidates signals. | Handling URL variants of the same page. |
Do not block a page in robots.txt and also noindex it. Google cannot see the noindex on a blocked page. Pick one method per URL group.
How to test and validate your robots.txt
Always check the file after saving. A single wrong line can hide your whole blog.
Open it in a browser
Visit yourblog.blogspot.com/robots.txt and confirm the text matches what you saved.
Search Console report
Open Settings in Search Console, then the robots.txt report to see the fetched file and any problems.
Inspect a URL
URL Inspection shows whether a page is blocked by robots.txt or allowed to be crawled.
Use the path tester
Type any path in the tester above and pick a crawler to see which rule decides the result.
| Common error | Fix |
|---|---|
| Disallow: / on a live blog | This blocks the whole site. Remove it or use the Blogger SEO preset. |
| Missing leading slash | Paths must start with /, for example /search. |
| Blocking CSS or JS | Google needs them to render pages. Leave them crawlable. |
| Wrong sitemap URL | Use the full https:// URL of a sitemap that exists. |
| noindex inside robots.txt | Google stopped supporting it in 2019. Use a meta or header tag. |
| Wrong letter case | Paths are case-sensitive. /Search is not /search. |
| File too large | Google reads only the first 500 KiB, so keep the file short. |
Pro tips for robots.txt in 2026
Go beyond the basics with these habits used by experienced SEOs.
Protect crawl budget
Block endless duplicate URLs such as search results and parameter variants so crawlers spend time on real posts.
Know the match order
The most specific rule wins. Use Allow to carve out an exception inside a blocked folder.
Decide your AI policy
Blocking training bots keeps content out of model training. Some AI search bots affect visibility in answers, so choose deliberately.
Do not use it for secrets
The file is public. Listing a private path actually advertises it. Protect private areas with a login instead.
Keep sitemaps current
Include every sitemap URL and update the file after a domain change, redesign or migration.
Re-test after changes
Re-check the robots.txt report after edits. Google caches the file, so changes can take a day to apply.
Robots.txt generator FAQ
Short answers to the questions people search most.
What is a robots.txt file?
A robots.txt file is a plain text file at the root of your site that tells search engine crawlers which URLs they may or may not crawl. It is a request that well-behaved bots follow, not a security lock.
How do I add a custom robots.txt in Blogger?
In Blogger open Settings, scroll to Crawlers and indexing, turn on Enable custom robots.txt, paste your rules and save. Then open yourblog.blogspot.com/robots.txt to confirm.
What is the best robots.txt for Blogger?
A safe setup lets the AdSense crawler Mediapartners-Google in, blocks /search for all crawlers, allows everything else and lists your sitemap. This generator's Blogger SEO preset builds exactly that.
How do I disallow all crawlers in robots.txt?
Use User-agent: * followed by Disallow: /. This blocks every compliant crawler from the whole site, so use it only for staging or private sites, never on a live blog you want to rank.
How do I test or validate my robots.txt?
Open yourblog.blogspot.com/robots.txt in a browser, check the robots.txt report in Google Search Console, inspect a URL to see if it is blocked, and use the path tester in this generator.
Does robots.txt remove a page from Google?
No. Disallow stops crawling, not indexing. A blocked URL can still appear in results without a description. To keep a page out of results, leave it crawlable and use a noindex tag.
Does Google respect Crawl-delay?
No. Googlebot ignores Crawl-delay. Some other crawlers honor it, so the generator lets you add it with a clear warning.
What is the robots.txt for WordPress?
A common WordPress file uses User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php and a Sitemap line such as https://example.com/wp-sitemap.xml.
Should I block AI crawlers in robots.txt?
It is your choice. Blocking AI training bots such as GPTBot, CCBot and ClaudeBot keeps your content out of training sets but does not affect normal Google search. Compliance is voluntary, so some bots may ignore it.
Ready to build your robots.txt?
Pick a preset, check the preview and paste it into Blogger in under a minute.
Back to the generator