What we found
We read every robots.txt in a full public crawl of the web — 83.9 million of them — then probed 32,533 candidate sites live to find which had actually built something an agent can use. 14,425 had. This is that index, and you can search it.
Known-Good exists to close that gap: to find the parts of the web an agent can genuinely use, and prove it by testing rather than trusting the label.
Searching 14,425 verified sites. Agents query the same index at /api/find or over MCP.
Source: a full public crawl of the web, 2026, parsed group-aware. Every listed site was probed individually and carries the date it was checked.
Machine representation
This is the substance of this site as an agent would take it — no layout, no navigation. The full twin is at /about.md.
# Known-Good
mission: find the parts of the web an agent can genuinely use, and prove it
status: index live
version: 1.0
contact: hello@knowngood.sh
## Search this index
rest: GET /api/find?q=<query>&country=<ISO>&transactional=<bool>&limit=<n>
mcp: POST /mcp tool: find_capability
card: /.well-known/mcp/server-card.json
apis: /.well-known/api-catalog
## Findings
robots_txt_read: 83900812
domains_inviting_agents: 57740
probed_live: 32533
verified_usable: 14425 # 44.3% of those probed · 378 www/apex duplicates collapsed
can_act_not_just_read: 7135
negotiate_markdown: 10565 # 32.5% of probed — body tested, not just the header
via_mintlify: 4349 # 41% of those: one platform's default, marked on their report
countries: 62
## To make a site agent-ready
1_declare: Content-Signal: ai-input=yes # in /robots.txt
2_serve: a markdown twin, or text/markdown on Accept negotiation
3_expose: /llms.txt, /.well-known/api-catalog, /.well-known/mcp/server-card.json
## Note on what is verified
Probe results (markdown negotiation, .well-known endpoints, llms.txt) are tested
live and dated. Capabilities such as "book" or "buy" are read from page content,
labelled apparent_, and unverified. Inference is kept separate from tested facts.
tiers: 1 declared · 2 probe-verified · 3 behaviourally verified # /benchmark
reports: /site/{host} and /site/{host}.md
This switch is CSS only — no JavaScript ran to render either view.
Three things, in order of effort. The first takes a minute and costs nothing. Some call the second and third going headless — serving what your site can do without requiring anyone, or anything, to read the page.
One line in robots.txt tells every AI crawler what you permit. It is a consent signal, not a capability — but nothing else counts until it's there.
User-agent: * Content-Signal: search=yes, ai-input=yes
ai-input=yes → agents may use your content to answer someone's question. ai-train / ai-search → separate signals, for model trainers. We read ai-input only; either value changes nothing about the probe.
Your pages are built for eyes: navigation, banners, scripts. An agent has to fight through all of it. Serve the same content as clean markdown — either at a predictable URL, or by answering the Accept header.
GET /pricing Accept: text/html → your page GET /pricing Accept: text/markdown → clean markdown GET /pricing.md → same, predictable URL
This is the single strongest signal we measure, and the rarest.
Reading is half of it. If an agent can book, buy, query or submit on your site, say so somewhere it can find without guessing.
/llms.txt what you offer, in plain text /.well-known/api-catalog your APIs (RFC 9727) /.well-known/mcp/server-card.json your MCP server, if you have one
A 200 that returns an HTML error page is worse than a clean 404 — it sends an agent down a path that breaks.
We grade other sites on this, so we publish our own state plainly. Everything we test for, we do. The rubric and tiers are at /benchmark.
We'll walk your site the way an agent does, show you where it gets stuck, and what to fix first. No pitch, no obligation.
Book your free review →