Sitemap & Robots

Difficulty: Intermediate

Sitemaps and robots.txt are the primary mechanisms for communicating with search engine crawlers about your site's structure and crawling preferences. While meta tags control behavior on a per-page basis, sitemaps and robots.txt operate at the site level - telling crawlers which pages exist, how important they are, and which areas should or should not be crawled.

A sitemap.xml file is an XML document that lists the URLs on your site along with optional metadata about each URL: when it was last modified (lastmod), how frequently it changes (changefreq), and its relative importance within the site (priority). Sitemaps help search engines discover pages that might not be found through normal crawling - such as pages behind JavaScript navigation, dynamically generated URLs, or newly created content. The sitemap should be submitted to search engines through Google Search Console and Bing Webmaster Tools. Large sites can use sitemap index files that reference multiple sitemaps, with each individual sitemap limited to 50,000 URLs or 50MB.

The robots.txt file sits at the root of your domain (example.com/robots.txt) and provides directives to web crawlers about which parts of the site they should or should not access. The User-agent directive specifies which crawler the rules apply to (use * for all), Disallow blocks crawling of specific paths, and Allow overrides a Disallow for specific sub-paths. The Sitemap directive points crawlers to your sitemap.xml location. Robots.txt is a suggestion, not a security mechanism - well-behaved crawlers respect it, but malicious bots will ignore it. Never rely on robots.txt to hide sensitive content.

Canonical URLs solve the duplicate content problem. When the same or very similar content is accessible at multiple URLs (with/without www, HTTP vs HTTPS, with query parameters, paginated pages), search engines may not know which version to index and rank. The <link rel="canonical" href="..."> tag in the page head tells search engines which URL is the authoritative version. This consolidates ranking signals to a single URL instead of splitting them across duplicates.

The hreflang attribute handles international and multilingual content. When your site has the same content in multiple languages or regional variants, hreflang tags tell search engines which version to show to users based on their language and location. Each page includes link tags pointing to all its alternate versions, including itself. The x-default value specifies the fallback page for users whose language or region does not match any specific variant. Incorrect hreflang implementation is one of the most common international SEO mistakes.

Code examples

Sitemap.xml Structure

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-03-10</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/blog</loc>
    <lastmod>2026-03-09</lastmod>
    <changefreq>daily</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://example.com/blog/css-grid-guide</loc>
    <lastmod>2026-03-01</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.6</priority>
  </url>
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2026-01-15</lastmod>
    <changefreq>yearly</changefreq>
    <priority>0.4</priority>
  </url>
</urlset>

<!-- Sitemap Index (for large sites with multiple sitemaps) -->
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-pages.xml</loc>
    <lastmod>2026-03-10</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-blog.xml</loc>
    <lastmod>2026-03-09</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products.xml</loc>
    <lastmod>2026-03-08</lastmod>
  </sitemap>
</sitemapindex>

Each <url> entry lists a page's location, last modification date, expected change frequency, and priority (0.0 to 1.0). Priority is relative within your site, not compared to other sites. For sites with over 50,000 URLs, use a sitemap index that references multiple sitemaps.

Robots.txt Configuration

# robots.txt - placed at site root: example.com/robots.txt

# Rules for all crawlers
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /search?*
Disallow: /user/*/settings
Allow: /api/public/

# Block specific crawlers from heavy pages
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

# Slower crawl rate for specific bot
User-agent: Bingbot
Crawl-delay: 5

# Reference sitemap location
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-blog.xml

Robots.txt uses User-agent to target specific crawlers, Disallow to block paths, and Allow to make exceptions. Wildcard (*) matches all crawlers. The Sitemap directive tells crawlers where to find your sitemaps. Crawl-delay requests a slower crawl rate. Multiple Sitemap entries can be listed.

Canonical URLs

<!-- Canonical tag: Tell search engines the preferred URL -->
<head>
  <link rel="canonical" href="https://example.com/blog/css-grid-guide" />
</head>

<!-- Problem: Same content at multiple URLs -->
<!-- https://example.com/blog/css-grid-guide -->
<!-- https://example.com/blog/css-grid-guide?ref=twitter -->
<!-- https://www.example.com/blog/css-grid-guide -->
<!-- http://example.com/blog/css-grid-guide -->
<!-- All should point to the canonical version above -->

<!-- Canonical on paginated content -->
<!-- Page 1 (canonical to itself) -->
<link rel="canonical" href="https://example.com/blog?page=1" />

<!-- Page 2 (canonical to itself, NOT page 1) -->
<link rel="canonical" href="https://example.com/blog?page=2" />

<!-- Cross-domain canonical (syndicated content) -->
<!-- On partner-site.com that republishes your content -->
<link rel="canonical" href="https://example.com/blog/original-article" />

The canonical tag consolidates ranking signals to one URL when content exists at multiple addresses. Each page should have a self-referencing canonical (even if there are no duplicates). For paginated content, each page is canonical to itself. Cross-domain canonicals tell search engines where the original content lives.

Hreflang for Multilingual Sites

<!-- English version (US) -->
<head>
  <link rel="alternate" hreflang="en-US" href="https://example.com/en/about" />
  <link rel="alternate" hreflang="en-GB" href="https://example.com/en-gb/about" />
  <link rel="alternate" hreflang="es" href="https://example.com/es/about" />
  <link rel="alternate" hreflang="fr" href="https://example.com/fr/about" />
  <link rel="alternate" hreflang="ja" href="https://example.com/ja/about" />
  <link rel="alternate" hreflang="x-default" href="https://example.com/about" />
</head>

<!-- The same set of hreflang tags must appear on ALL language versions -->
<!-- Spanish version must include all alternates too -->
<!-- On https://example.com/es/about: -->
<head>
  <link rel="alternate" hreflang="en-US" href="https://example.com/en/about" />
  <link rel="alternate" hreflang="en-GB" href="https://example.com/en-gb/about" />
  <link rel="alternate" hreflang="es" href="https://example.com/es/about" />
  <link rel="alternate" hreflang="fr" href="https://example.com/fr/about" />
  <link rel="alternate" hreflang="ja" href="https://example.com/ja/about" />
  <link rel="alternate" hreflang="x-default" href="https://example.com/about" />
</head>

Every language/region version of a page must include hreflang tags pointing to ALL versions, including itself. The x-default value specifies the fallback for unmatched languages. Hreflang can use language only (es) or language-region (en-US). All versions must cross-reference each other symmetrically.

Key points

Robots.txt is a crawl directive, not a security measure. Blocked pages can still appear in search results (showing the URL without a snippet) if other pages link to them. Use noindex to fully remove pages from search results.
Every page should have a self-referencing canonical tag, even if no duplicate versions exist. This prevents issues from query parameters, session IDs, or tracking codes that crawlers might discover.
Hreflang must be bidirectional - if page A points to page B as an alternate, page B must also point back to page A. Missing return links cause hreflang to be ignored.

Concepts covered

sitemap.xml, robots.txt, Canonical URLs, hreflang, Crawl Control