How to Set robots.txt and Sitemap on BlogSpot

How to set robots.txt and sitemap on BlogSpot is mostly about using Blogger’s built-in tools, then submitting the sitemap in Google Search Console. You do not upload robots.txt over FTP. You also do not hand-build a sitemap file for a normal BlogSpot blog. Blogger already publishes both paths for you.

I use robots.txt to keep search pages out of the index, keep AdSense crawlers allowed, and point crawlers at sitemap.xml. After that I verify the property in Search Console and submit the sitemap. For titles, mobile, and on-page SEO that work with this crawl setup, read how to make a BlogSpot blog SEO and mobile friendly.

Default Blogger robots.txt

Open your live robots.txt in a browser:

https://yourblog.blogspot.com/robots.txt

Blogger generates a basic file. A typical pattern looks like this (example for gojist):

User-agent: Mediapartners-Google

Disallow:

User-agent: *

Disallow: /search

Allow: /

Sitemap: https://gojist.blogspot.com/sitemap.xml

Mediapartners-Google is the AdSense crawler line. Disallow: /search keeps blog search result URLs from cluttering the index. Allow: / opens the rest of the blog. The Sitemap line tells crawlers where the XML map lives.

Turn on custom robots.txt in Blogger

1. Sign in to Blogger and select the blog.

2. Open Settings.

3. Under Crawlers and indexing, enable Custom robots.txt.

4. Paste your rules in the box and save.

Keep the Mediapartners-Google allow rule if you use AdSense. Keep Disallow: /search unless you have a clear reason not to. Point Sitemap to your real sitemap URL, matching http or https and the domain you actually use in Search Console.

After saving, reload /robots.txt on the live site and confirm the text matches what you entered. If you use a custom domain, check robots.txt on that domain too.

Custom robots header tags (brief)

Separate from robots.txt, Blogger can add per-post robots meta tags. In Settings under Crawlers and indexing, enable custom robots tags. Then in a post or page editor, open Options and set tags such as noindex, nofollow, noarchive, or nosnippet when you truly need them.

Use those tags sparingly. noindex on a money post by mistake will hide it from Google. For most public how-to posts, leave the default allow-index behavior alone and control crawl noise with robots.txt instead.

BlogSpot sitemap.xml (you do not invent the file)

Blogger builds the sitemap automatically. Open:

https://yourblog.blogspot.com/sitemap.xml

That URL is what you submit. There is no need to generate an XML file by hand for a standard BlogSpot site. If you use a custom domain, submit the sitemap on the same property URL you verified.

Submit the sitemap in Google Search Console

1. Sign in to Google Search Console.

2. Add your BlogSpot (or custom domain) property if it is not there yet.

3. Verify ownership. HTML tag, Google Analytics, or other methods Search Console lists all work when set up correctly.

4. Open Sitemaps.

5. Enter sitemap.xml (or the full sitemap path Search Console accepts for that property) and Submit.

6. Check status later. “Success” means Google could fetch it. Indexing still depends on quality and crawl demand.

Also list the same sitemap URL in your custom robots.txt so other crawlers see it. Keep both Search Console and robots.txt pointing at one consistent domain version.

Common mistakes

Blocking the whole site with a blanket Disallow: /. Leaving Mediapartners-Google blocked when you run AdSense. noindex on posts you want ranked. Submitting sitemap.xml on a property that does not match the live domain. Mixing www and non-www or http and https without picking one primary URL. Assuming sitemap submit equals instant ranking. It only helps discovery.

If ads and crawl setup both matter to you, pair this guide with how to set up AdSense on Blogger (code, ads.txt, and payments). Clear crawl rules plus a submitted sitemap give Google a clean map of what you want indexed.

How the pieces fit together

Think of robots.txt as the gate list for crawlers, custom robots tags as per-page overrides, and sitemap.xml as the map you hand to Search Console. Most BlogSpot blogs only need a careful robots.txt and one sitemap submit. Per-post noindex is for drafts you published by mistake, thank-you pages, or thin tag archives you do not want in search.

When you change custom robots.txt, give crawlers time. The robots.txt report in Search Console settings, or a simple browser load of /robots.txt, is enough for a quick sanity check. If the live file still shows the old default, hard-refresh or try an incognito window. Caching can show the old version for a short while.

If you moved from blogspot.com to a custom domain, re-check three URLs on the new host: /robots.txt, /sitemap.xml, and a sample post. Then verify the custom domain property in Search Console and submit sitemap.xml there. Leaving the old blogspot.com property alone without redirects can split signals. Prefer one primary domain and stick to it in the Sitemap line.

Do not paste random robots.txt samples from unrelated CMS docs. WordPress disallow rules for /wp-admin/ do not apply on BlogSpot. Copying them can confuse you during troubleshooting even if Blogger ignores some lines. Stay with Blogger-shaped rules: allow AdSense’s crawler, disallow /search, allow the rest, and declare the sitemap.

After crawl setup is stable, keep publishing. A perfect robots.txt will not rank empty blogs. Use it to stop noise, then spend your time on posts people finish reading.

Save robots.txt changes, confirm the live files, submit the sitemap once, then focus on publishing useful posts. That is the durable BlogSpot crawl setup.