# Known-Good

> Finding the parts of the web an agent can genuinely use — and proving it by testing,
> rather than trusting the label.

- mission: find the parts of the web an agent can genuinely use, and prove it
- status: index live
- version: 1.0
- contact: hello@knowngood.sh

## What we found

We read every robots.txt in a full public crawl of the web — 83.9 million of them — then
probed 32,533 candidate sites live to find which had actually built something an agent can
use. 14,425 had. This is that index, and it is searchable.

Source: a full public crawl of the web, 2026, parsed group-aware. Every listed site was
probed individually and carries the date it was checked.

- robots_txt_read: 83,900,812
- domains_inviting_agents: 57,740
- probed_live: 32,533
- verified_usable: 14,425  (44.3% of those probed; 378 www/apex duplicate hosts collapsed to one report each, 26 Aug)
- can_act_not_just_read: 7,135
- countries: 62
- cdn_default_share: 0.86 (declarations that are one CDN's default, not a decision)
- negotiate_markdown_strict: 10,565 of 32,533 probed (32.5%) — body is markdown and differs from HTML, not just the header
- negotiate_via_mintlify: 4,349 (41% of those) — one platform's default; marked "via mintlify" on their report

## Searching the index

- REST: `GET https://knowngood.sh/api/find?q=<query>&country=<ISO>&transactional=<bool>&has_mcp=<bool>&limit=<n>`
- MCP servers, tested live: https://knowngood.sh/mcp-servers · platform defaults attributed at `/platform/{name}` · not listed? https://knowngood.sh/submit
- MCP: `POST https://knowngood.sh/mcp` — tool `find_capability`
- Server card: https://knowngood.sh/.well-known/mcp/server-card.json
- API catalog: https://knowngood.sh/.well-known/api-catalog

Results separate what was tested from what was inferred. Probe results — markdown
negotiation, `.well-known` endpoints, `llms.txt` — are live-tested and dated. Capabilities
such as "book" or "buy" are read from page content, labelled `apparent_`, and are
unverified. Inference is kept separate from tested facts.

Every listing sits on a three-tier ladder (https://knowngood.sh/benchmark):

- tier 1 — declared: the site says agents are welcome in robots.txt. A permission, not a capability.
- tier 2 — probe-verified: we fetched the site and tested each signal ourselves, dated. Every listing.
- tier 3 — behaviourally verified: an agent completed the action end to end. Opening to a founding cohort.

Each site has a dated report at `https://knowngood.sh/site/{host}` (and `.md`).

## Making a site agent-ready

Three things, in order of effort. The first takes a minute and costs nothing. Some call the
second and third going *headless* — serving what your site can do without requiring anyone,
or anything, to read the page.

### 1. Say agents are welcome

One line in `robots.txt` tells every AI crawler what you permit. It is a consent signal, not
a capability — but nothing else counts until it's there.

```
User-agent: *
Content-Signal: search=yes, ai-input=yes
```

`ai-input=yes` means agents may use your content to answer someone's question.
`ai-train` and `ai-search` are separate signals, between you and model trainers — we read `ai-input` only, and either value changes nothing about the probe.

### 2. Offer a machine-readable version

Your pages are built for eyes: navigation, banners, scripts. An agent has to fight through
all of it. Serve the same content as clean markdown — either at a predictable URL, or by
answering the `Accept` header.

```
GET /pricing     Accept: text/html      -> your page
GET /pricing     Accept: text/markdown  -> clean markdown
GET /pricing.md                         -> same, predictable URL
```

This is the single strongest signal we measure, and the rarest.

### 3. Publish endpoints an agent can act on

Reading is half of it. If an agent can book, buy, query or submit on your site, say so
somewhere it can find without guessing.

```
/llms.txt                            what you offer, in plain text
/.well-known/api-catalog             your APIs (RFC 9727)
/.well-known/mcp/server-card.json    your MCP server, if you have one
```

A 200 that returns an HTML error page is worse than a clean 404 — it sends an agent down a
path that breaks.

## What's live here today

We grade other sites on this, so we publish our own state plainly. Everything we test
for, we now do.

- Content negotiation — `Accept: text/markdown` returns markdown from any page URL
- `/about.md` — markdown twin of this page
- `/api/find` — the search API
- `/.well-known/mcp/server-card.json` — MCP server card
- `/.well-known/api-catalog` — RFC 9727
- `/llms.txt` — curated index of this site
- `/robots.txt` — content signals: `ai-input=yes`, `ai-train=no`
- `/bot` and `/bot/ips.json` — crawler transparency, opt-out, published IPs

## Own a listed site?

Your report at `https://knowngood.sh/site/{host}` names each check that did not pass.
Behavioural verification (tier 3) is opening to a founding cohort: `https://knowngood.sh/verify/{host}`.
Questions: hello@knowngood.sh