# KnownGood-Verifier — crawler information

> What the KnownGood-Verifier crawler does, what it fetches, how to control or
> block it, and how to confirm a request is really from us.

You're likely reading this because you saw `KnownGood-Verifier` in your server
access logs. This page explains what it does and how to control it.

## Block or limit us — immediately

We check `robots.txt` on every visit and honour it at once. To block entirely:

```
User-agent: KnownGood-Verifier
Disallow: /
```

To stay listed but slow us down:

```
User-agent: KnownGood-Verifier
Crawl-delay: 30
```

We treat the Content Signals directive `ai-input=no` as a hard opt-out: a site
declaring it is never listed and never re-probed, regardless of anything else.

## Every "no" we honour, in order

1. `robots.txt` — `Disallow` for `KnownGood-Verifier` or `*` on the paths we fetch.
   Binding (RFC 9309).
2. `ai-input=no` — the Content Signals directive in `robots.txt`. A convention, widely
   deployed; we treat it as binding.
3. `Content-Usage: search=n` — as an HTTP response header or a `robots.txt` line. The
   IETF AI Preferences draft (draft-ietf-aipref-attach); not yet an RFC, honoured now.
4. `X-Robots-Tag: noai` or `<meta name="robots" content="noai">` — informal, no
   standard behind it; honoured as a courtesy because a site that wrote it meant it.

## How a site comes to be checked

We check a site when it declares agents welcome in `robots.txt`, **or** publishes
an agent endpoint — an MCP server card, an RFC 9727 API catalog, an `llms.txt` —
**or** appears in a public MCP registry, **or** an owner asks at
https://knowngood.sh/submit. Publishing an endpoint for agents is an invitation to
agents; we treat it as one. If that is not what you meant, the two lines above
stop us, and one email does too.

## What it does

KnownGood-Verifier checks whether an AI agent can read and use your site. It is a
verifier, not a scraper — it reads a handful of standard files to assess readiness,
and does not harvest content, prices, or customer data.

- User-agent: `KnownGood-Verifier/0.3 (+https://knowngood.sh/bot)`
- Fetches: `/robots.txt`, your homepage, a few `/.well-known/` paths, `/llms.txt`, and at most one contact page.
- Rate: at most one concurrent request, with pauses. A full check is a few requests, not a crawl.
- Frequency: an initial check, then periodic re-checks. Not continuous.
- JavaScript: we read what the server returns; we do not render or interact on a live check.
- Personal data: none collected.

## Verify it's really us

Confirm a request's source IP falls within our published range at `/bot/ips.json`.
Our crawlers resolve under `*.knowngood.sh`. Our name from an IP outside that range
is an impostor — block it, and optionally report it to us.

## Our commitments

- We honour `robots.txt` for our user-agent, every visit.
- We treat `ai-input=no` as a hard opt-out.
- We identify ourselves honestly and never disguise our user-agent.
- We rate-limit ourselves to negligible load.
- We collect no personal data.
- One email stops us.

## Contact

Reach a human at bot@knowngood.sh for questions, complaints, or to stop us entirely.

## Why agents are in your logs

The web has a second reader now: AI agents sent by real customers, reading sites and
completing tasks. Most of the web is invisible or unusable to them. Known-Good measures
whether yours is.

Book a free 30-minute agent-readiness review over video: we'll screen-share your site,
show you where an agent gets stuck, and what to fix first. No pitch, no obligation.
Book at https://cal.com/knowngood.sh/30min