Crawler access
HTTP availability, redirects, robots.txt, AI crawler rules, and page directives.
AI agent readiness & technical GEO audit
Can AI agents access and understand your website?
Audit crawler access, technical discoverability, content clarity, structured data, and other signals that help AI search systems find and interpret your public pages.
No sign-up required · Evidence-backed results · No ranking guarantees
robots.txtMeta robotsX-Robots-Tag7 crawlers
Understand this check
Scope, output, and practical use
What is it?
A deterministic test of whether known AI and search crawler user agents can access the submitted URL.
What does it check?
Reads robots.txt, page robots metadata, response headers, sitemap declarations, and path-specific rules.
Why is it needed?
A crawler cannot retrieve content that the site intentionally or accidentally blocks.
What does the output mean?
Allowed means the tested rules permit retrieval; blocked or partial points to the exact matching evidence. It does not guarantee indexing or citation.
Example use case
Use it after changing robots.txt to verify that OAI-SearchBot is allowed while private paths remain blocked.
No scan selected
Submit a public domain or page URL to inspect crawler access controls.
What we check
Technical evidence you can verify
Discoverability
Sitemaps, canonicals, indexability, internal links, and bounded URL samples.
Content clarity
Raw main content, headings, direct answers, meaningful links, and semantic structure.
Structured data
Valid JSON-LD, entity properties, value formats, and visible-content consistency.
Trust and identity
Brand names, descriptions, entity home, identifiers, and essential trust pages.
How it works
From URL to prioritized fixes
- Enter a public URL.The scanner validates and safely fetches a bounded set of resources.
- We run focused checks.Versioned deterministic rules preserve evidence for each finding.
- Review and verify fixes.Copy remediation, download the report, publish changes, and rescan.
What this audit does not promise
Technical readiness can improve eligibility for crawling and interpretation. It does not guarantee that any AI service will mention, recommend, rank, or cite your website.
Read the scoring methodologyCommon questions
Short, honest answers
Does a high score guarantee an AI citation?
No. Scores summarize observed technical or editorial signals only.
Does GEO Toolkit change my website?
No. It performs read-only public requests and gives reviewable guidance.
Do I need an llms.txt file?
No. It is experimental and optional; core crawlability, content, and structured data come first.
Can I scan private pages?
No. Do not submit authenticated or sensitive URLs. Private and reserved network targets are blocked.