A free Chrome extension that audits the crawl layer of any site —
sitemap, robots.txt, llms.txt, and 13 AI bots — entirely in your
browser, never on a server.
Parses robots.txt for Sitemap: directives and probes
12 common paths. Follows <sitemapindex> files to count real
total URLs. Generates one if none exists.
Score your AI readiness
One 0–100 number weighing sitemap hygiene, llms.txt presence, AI search
bot access, and explicit configuration. Specific fixes to reach 100, not vague
advice.
See the bot matrix
All 13 AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended,
Bytespider, CCBot, and seven more — parsed against your robots.txt and
grouped by vendor.
See it in action
One click. One scan. One audit.
From "what do crawlers see on this page?" to a full audit, a score, and a download.
Open the popup. Click Scan.Read the score. See what to fix.Expand the matrix. Check every bot.
Privacy by design
Nothing leaves your browser to anyone.
The extension fetches robots.txt, sitemaps, and
llms.txt directly from the site you're auditing.
Same kind of request your browser makes when it loads the page normally.
No backend. No analytics. No transit through Sukat servers.
Your URL list, your audit results, your sitemap exports — all stay inside the tab.
Every public release of Sukat Crawler, dated and versioned. Updates ship through the Chrome Web Store — if auto-updates are on, you're already on the latest.
v1.0.0CurrentInitial release
First public release on the Chrome Web Store. One-click audits of sitemap, robots.txt, llms.txt, and 13 AI bots — without anything leaving the browser.
Added
Sitemap detection. Parses robots.txt for Sitemap: directives and probes 12 common paths (/sitemap.xml, /wp-sitemap.xml, Yoast variants). Follows <sitemapindex> files to count real total URLs across all sub-sitemaps.
Sitemap generation. Same-origin BFS crawl that produces a valid sitemap.xml for download when none exists. Configurable page/depth limits, includes-query-strings toggle.
URL validator. HEAD requests against every URL to surface 404s, redirects, and server errors. Concurrency-limited, cancellable. Validation results carry through to CSV / JSON / TXT exports.
llms.txt detection. Checks /llms.txt and /llms-full.txt for the emerging AI-readable site-map convention.
AI Crawler Matrix. Parses robots.txt against 13 AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalAgent, Bytespider, Applebot-Extended, CCBot) and shows an allow/block matrix grouped by vendor.
AI Readiness score. A single 0–100 number weighing sitemap hygiene, llms.txt presence, AI search bot access, and explicit AI configuration. Surfaces the specific fixes that would raise the score.
robots.txt viewer. Collapsible view of the raw file, with copy and open-in-tab buttons.
Export URLs. CSV, JSON, or TXT, with validation results included if the validator has run.
Lastmod uniformity warning. Flags the common CMS bug where every URL shares the same <lastmod> date.
Side panel mode. A toolbar button opens Crawler in Chrome's side panel so it stays open while you click around the audited site.
Want a heads-up when Crawler updates? Drop a line and Bernard will tag you on the next release post.
Common questions
Does it cost anything?
No. Free forever, no account required, no premium tier, no ads. Same model as the rest of Sukat.
Will it slow my browser down?
No — the extension does nothing unless you click it. It doesn't run on every page you visit, doesn't track you, doesn't sit in the background fetching things. Open the popup, click Scan, and only then does it audit the current site.
What sites does it work on?
Any http(s):// site. WordPress, Shopify, custom React apps, static blogs, headless CMS deployments — anywhere a sitemap could live, Crawler can find it (or generate one if it doesn't exist).
What's the difference between Sukat, Inspector, and Crawler?
Sukat (this site) compresses images to an exact KB/MB target.
Inspector audits images on a page and hands the heavy ones to Sukat.
Crawler audits the crawl layer of a site — sitemap, robots.txt, llms.txt, AI bot access. Three tools, one privacy model: everything runs in your browser.
Is my data really safe?
Yes. The extension has no backend — there is literally no server we control that could receive your data. We don't use Google Analytics, advertising scripts, or any tracking in the extension itself. The only requests leaving your browser are the ones fetching robots.txt, sitemaps, and llms.txt from the site origin — exactly what your browser does when it loads the page normally. Full disclosure here.
Does it cover all 13 AI bots?
Yes — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Meta-ExternalAgent, Applebot-Extended, Bytespider, and CCBot. Grouped by vendor in the matrix with allow / block / default badges so you can see at a glance which AI search products can read your content.
Firefox / Safari / Edge support?
Chrome and Chromium-based browsers (Edge, Brave, Arc, Vivaldi) work today. Firefox support is on the roadmap. Safari requires significant rework due to Apple's extension model and isn't planned yet.
Privacy policy
The full privacy policy
Last updated June 10, 2026. This covers the Sukat Crawler extension; the Sukat website has its own website privacy policy.
The short version
Sukat Crawler runs entirely in your browser. The only network requests it makes go to the site you choose to audit — to read its public robots.txt, sitemap, and llms.txt. There is no backend, no account, and no analytics or tracking in the extension. Your URL list and audit results never leave the tab.
What it connects to
The only network requests Sukat Crawler makes go to the site you choose to audit. To build a report it reads that site's publicly available files — robots.txt, the XML sitemap and any nested sitemaps, and llms.txt — and checks the response status of the URLs listed in the sitemap. Those requests go straight from your browser to the target site. None of them pass through Sukat.
What it never does
No reading the pages you browse. It does not read page content or inject scripts into the sites you visit.
No cross-site tracking. It does not watch your activity across sites or build a profile of you.
No analytics, telemetry, or advertising code of any kind inside the extension.
No personal data collected, transmitted, or sold — there is none to collect.
What's stored on your device
Sukat Crawler keeps only your settings — crawl depth, maximum pages, and the query-string toggle — in Chrome's local storage on your own machine. When you export a report as CSV, JSON, or TXT, the file is generated locally and saved through your browser's normal download. This data:
stays on your device,
is never synced to an account or sent off your device, and
is cleared when you clear the extension's storage or remove it.
Because nothing lives on a server, there is nothing for Sukat to retain.
Permissions, and why each is needed
On install, Chrome warns that Sukat Crawler can "read and change your data on all websites." That broad wording is required because you can point the audit at any site — but in practice the extension only reads the public crawl files and URL statuses of the specific site you scan. It never reads or changes page content. Each permission maps to one job:
Host access
Reads the public crawl files and checks the URLs of the site you're auditing. Used only on sites you choose to scan.
downloads
Saves the reports you export to your device.
storage
Remembers your settings between sessions.
sidePanel
Shows results in Chrome's side panel, alongside the page.
activeTab
Detects which site is open when you start a scan.
About this website
This page (sukatapp.com) uses standard, privacy-respecting analytics to count visits and see which pages are useful. That applies to the website only — the Sukat Crawler extension itself sends nothing. See the site's website privacy policy.
Contact
Questions about privacy or the extension? Reach the developer, Bernard Brillo, via the Sukat contact page.
