Case study · Developer Tools

Robots & Sitemap Analyzer

Analyze robots.txt and XML sitemaps into structured crawlability data across one or multiple websites.

Overview

Analyzes robots.txt files and XML sitemaps for one or many websites. It discovers sitemap files, evaluates robots rules for a selected user agent, inspects sitemap coverage, follows sitemap indexes, and reports common technical issues without crawling website pages.

Problem

Auditing robots.txt rules and sitemap structures across multiple websites requires inspecting several files, nested sitemap indexes, URL entries, and crawl restrictions.

Solution

Automates robots.txt and sitemap analysis and produces normalized crawlability data for each website, including discovered sitemaps, evaluated rules, URL statistics, and detected issues.

How it works

- Analyze multiple websites in one run - Evaluate robots.txt rules for a selected user agent - Discover declared and common sitemap locations - Follow sitemap indexes and inspect child sitemaps - Detect missing files and invalid XML - Detect duplicate URLs and invalid dates - Detect cross-domain sitemap entries - Configurable sitemap and URL limits - Proxy support