robots.txt // blogger control deck

Tell the crawlers
exactly where to go.

Compile a Blogger‑ready robots.txt, read what every line actually does, and test any path against your rules — with a one‑click shield against duplicate content and AI scrapers.

preset

crawl policy · main group

extra shields

this policy applies to

sitemaps

custom rules · main group

robots.txt · 0 chars
Blogger: Settings → Search preferences → Custom robots.txt → paste & save.

path tester · does a crawler reach it?

what each line does

The duplicate‑content trap hiding in every Blogger blog

On Blogger, your real posts live at clean URLs like /2026/07/my-post.html, but the very same words also appear on every label page and every search result — all of which sit under /search. To a crawler that looks like dozens of copies of one page, and it burns crawl budget deciding which copy to keep. Disallowing /search closes that leak. It is safe to do on Blogger specifically, because no genuine content is ever published at a /search path, so you block nothing you actually want indexed.

Parameters are the second leak

Append ?m=1 to any Blogger URL and you get its mobile variant; add ?updated-max or ?max-results and you get paginated slices. Same content, new URL, more duplicates. The Block parameter duplicates switch adds wildcard rules that catch these variants wherever they appear, while leaving your clean canonical URLs fully open.

Read the file, don't just trust it

A robots.txt line is easy to write and easy to get wrong. The explainer panel translates every directive you generate into a plain sentence, and the preview and the explainer are linked — hover a line in either (on a pointer device) and its twin lights up, so you always know which rule produces which effect before you paste it live.

// on your phone

Flip a switch and the file rebuilds instantly. The bar pinned to the bottom keeps the live verdict and the copy / download buttons within thumb‑reach the whole time you're tuning — it steps aside the moment you scroll down to read this.

The AI‑scrapers nobody told you about

Long before your content shows up in a chatbot's answer, a fleet of bots — GPTBot, CCBot, Google‑Extended, ClaudeBot, Bytespider and friends — crawls it for training. The Block AI training bots switch adds a dedicated group that turns them away at the door, and because it is a separate group it does not touch your normal search indexing. The path tester lets you confirm this: test as GPTBot and it is blocked; test as Googlebot and your pages stay open.

The crawl‑delay myth

People add Crawl-delay hoping to slow Google down. Googlebot ignores it completely. The generator still lets you set it — some smaller crawlers do respect it — but it labels the line honestly so you never build a false sense of control. For Google, the real throttle is the crawl budget you protect by blocking the duplicate URLs above.

Putting it on Blogger

Copy the generated text with the copy button. In Blogger open Settings, scroll to Search preferences, find Custom robots.txt, switch it on, paste, and save. The output is formatted exactly for that box — comments, grouped user‑agents, rules and sitemap lines in the right order. Come back here whenever you add a section you want to hide; the file rebuilds the instant you flip a switch.