Fetch robots.txt, discover sitemaps, and extract URLs. Supports sitemap index files and regular sitemaps.