Sitemaps help search engines find your pages; robots.txt tells crawlers where they may go. Misunderstanding either can quietly hide important content. Here is how both work.
Two small files influence how search engines explore your website. An XML sitemap is a list of the pages you want search engines to know about. A robots.txt file gives crawlers rules about which parts of the site they may request. Both are simple, and both are frequently misconfigured in ways that are hard to notice.
XML sitemaps
What a sitemap does
A sitemap helps search engines discover URLs, especially on new sites, large sites, or pages with few internal links. It is a hint, not a command: listing a page does not guarantee it will be indexed. Google's sitemaps overview explains when a sitemap helps most.
What to include
- Canonical URLs of pages you want indexed
- Only pages that return a 200 status, not redirects or errors
- An accurate
lastmoddate that changes when the content meaningfully changes
What to leave out
- Pages with a noindex tag
- Duplicate versions such as tracking-parameter URLs
- Thin pages such as empty tag archives or internal search results
Size limits
A single sitemap file can contain up to 50,000 URLs or 50 MB uncompressed. Larger sites use a sitemap index file that points to several sitemaps, often split by content type, such as articles, categories and authors.
Priority and changefreq
Google has said it ignores the priority and changefreq values. An accurate lastmod is far more useful.
robots.txt
What robots.txt does
The robots.txt file sits at the root of your domain, for example example.com/robots.txt. It tells crawlers which paths they should not request. It is commonly used to stop crawlers wasting time on admin areas, internal search results or endless filter combinations. Google's introduction to robots.txt covers the syntax.
A simple example
User-agent: *
Disallow: /admin/
Disallow: /search
Sitemap: https://www.example.com/sitemap.xmlThe mistakes that hurt
1. Using robots.txt to remove pages from search
Blocking a URL in robots.txt stops crawling, not indexing. If other pages link to it, the URL can still appear in results, just without a description. To keep a page out of search, allow crawling and add a noindex meta tag, or protect it with a password.
2. Blocking a page and also adding noindex
If robots.txt blocks a page, crawlers never see its noindex tag. The two instructions cancel each other out.
3. Blocking CSS and JavaScript
Search engines render pages much like a browser. Blocking the files needed to render the page can prevent them understanding your layout and content.
4. Leaving a staging rule in place
Disallow: / blocks the whole site. It is a common leftover from a development environment and can quietly remove a site from search after a launch.
5. A stale or bloated sitemap
Sitemaps listing deleted pages, redirects or noindexed URLs send mixed signals and waste crawl attention.
How to test both
- Submit your sitemap in Google Search Console under Sitemaps and check for errors.
- Review the robots.txt report in Search Console to see the version Google fetched and any problems.
- Use URL Inspection on key pages to confirm they are crawlable and indexable.
- After any site migration or redesign, recheck both files on launch day.
Sitemaps help discovery, but internal links do much of the work; see our internal linking strategy. For a page-by-page check, use the blog SEO audit checklist.
0 Comments
No comments yet — be the first to share what you think.