| Internet-Draft | AI Terms | November 2026 |
| Ovalles | Expires 22 May 2027 | [Page] |
This document defines AI Terms: a JSON document published at the well-known URI /.well-known/ai-terms in which a website states, for each of four uses of its content by automated AI systems (search, answer, training and agent), whether the use is allowed, allowed for a price, or not allowed. Each use corresponds to one category of the IETF AI Preferences vocabulary, and the document defines how a site's terms are written, conservatively, as AI Preferences statements (the Content-Usage header field and robots.txt rule) for clients that read only those. AI Terms extends the AI Preferences work with what it leaves out: a price and a free allowance, a payment binding (HTTP 402, with x402 as the first), a purpose declaration by the client (the AI-Purpose header field), a description of a paid page that the payment depends on, and a receipt that records the use.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 22 May 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
The Robots Exclusion Protocol [RFC9309] lets a site say whether a named crawler may fetch a path. The IETF AI Preferences working group [AIPREF] defines a vocabulary for how content may be used by automated processing, including AI training [I-D.ietf-aipref-vocab], and ways to attach those preferences to content [I-D.ietf-aipref-attach]. Its charter excludes technical enforcement, client authentication and auditing.¶
Sites increasingly want to permit a use in exchange for payment rather than only allow or refuse it, and want a record of what was paid for. AI Terms builds on the AI Preferences vocabulary to add that. It is an extension of that work, not an alternative to it: every AI Terms use corresponds to one AI Preferences category (Section 2.1), a site's terms can always be written as AI Preferences statements that grant no more than the terms (Section 5), and a site can keep stating its preferences the AI Preferences way. AI Terms adds five things:¶
A price per use. A use can be allow, paid (with a price and unit) or deny, with an optional free allowance for verified clients.¶
A payment binding. For paid uses, the terms name a payment protocol and endpoint. This document uses HTTP 402 with the x402 protocol [X402] as the first binding.¶
Purpose declaration by the client. A request header field, AI-Purpose, in which the client states its intended use and so accepts that use's terms.¶
Description before payment. A quote for a paid page can describe the page, and a payment answering such a quote is settled only if the page delivered is the page described (Section 8.1).¶
Receipts. A paid response carries proof of payment and the declared purpose, giving both parties a record of licensed use.¶
Individual drafts cover neighbouring ground: [I-D.car-ai-txt-wellknown] declares usage policy and links to a licensing or pricing page, and [I-D.reilly-aipref-compliance] defines verifiable compliance records while excluding licensing, compensation and payment. Section 10 describes how AI Terms relates to them and to Really Simple Licensing [RSL].¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
This document defines four uses. Each access by an automated AI system is made for one of them, and the client declares which (Section 6).¶
Indexing content in order to show links to it, or excerpts of it, in AI-powered search results.¶
Using content, fetched in real time or taken from an index, as input to a generative AI model that answers a question asked by a person. A summary of the content shown in AI search results is an answer use.¶
Collecting content to train or fine-tune machine-learning models.¶
An autonomous software agent reading content or invoking endpoints to complete a task.¶
Each use corresponds to one category of the AI Preferences vocabulary [I-D.ietf-aipref-vocab]. The correspondence is not exact, and the notes say where the two differ; Section 5.1 writes terms as preferences so that the preferences never grant more than the terms. (Sections 3 and 4 of [I-D.ietf-aipref-vocab] do not yet have working group consensus; this mapping follows its -08 text and will follow its changes.)¶
| AI Terms use | AI Preferences category | Notes |
|---|---|---|
training
|
AI Training (train-ai) |
AI Training covers generative models only; training also covers training models used for other tasks. |
answer
|
AI Use (ai-use) |
Includes summaries of the content shown in AI search results. |
agent
|
AI Use (ai-use) |
An agent that does not give the content to a generative AI model makes no AI Use; its terms still apply. |
search
|
Search (search) |
Search excludes generating summaries, which AI Terms counts as answer. Search covers every search application, not only AI-powered ones, and allows training models used only within the search application. |
The training use also corresponds to the cross-vendor robots.txt policy token __usage_ai-training proposed in [I-D.kelley-aipref-training-preference-signal], which expresses the AI Training preference per path; Section 5.3 writes it from a site's terms. AI Terms uses that token rather than defining one of its own.¶
No use spans two categories. AI Use is split into answer and agent because sites commonly want to treat a person's live question differently from an autonomous agent's task. If the AI Preferences vocabulary gains categories for these, a later revision will map them. Extensions to that vocabulary are made by standards-track RFCs (Section 4.4 of [I-D.ietf-aipref-vocab]), which this document is not; it only asks whether the split would be useful there.¶
A site publishes its terms as a JSON [RFC8259] object at the well-known URI [RFC8615] /.well-known/ai-terms, served with media type application/json and UTF-8 encoding. The document MAY instead be hosted elsewhere and referenced as described in Section 4; such a reference MUST use HTTPS.¶
The version of this specification, "0.1" for this document.¶
The origin the terms apply to, serialized as in [RFC6454], for example https://example.com.¶
Date of the last change, YYYY-MM-DD.¶
One member for each of the four uses (search, answer, training, agent), each a Use Terms object (Section 3.3). A document that lacks one is invalid; a reader that meets one anyway applies default to the missing use.¶
allow, paid or deny. It applies to a request that declares no use (Section 6), to a request whose client fails the site's identity checks (Section 7), and to a use for which no valid terms are found (Section 3.5). When default is paid, the price is the one the 402 response states (Section 8); since the terms then state no price to compare with, a client SHOULD NOT pay such a quote without a price ceiling of its own.¶
Overrides for parts of the site (Section 3.5).¶
verify, an array of identity methods the site applies, and unverified, what applies to a client that fails them (Section 7).¶
"AI-Purpose" in this version.¶
default is paid, or a paths entry lists a use whose access is paid and carries no payment of its own):protocol, version, endpoint and protocol-specific members (Section 8).¶
Whether quotes for paid pages describe the page and whether delivery is checked against the description (Section 8.1).¶
A description of the receipt a paid access returns.¶
true if the site currently measures automated access without charging and has not yet chosen terms. No AI Preferences statement is derived from such a document (Section 5.1).¶
licensing, a URL for bulk or custom licensing.¶
Unknown members MUST be ignored. A document is valid when it meets the requirements of this section. A JSON Schema [JSON-SCHEMA] for them is published as an aid [TOOLS]; it cannot express every rule (for example that a paid use needs a way to pay). A client SHOULD treat a document that is not valid, or whose ai_terms major version it does not implement, as absent; the one exception is a missing use, to which it applies default.¶
allow, paid or deny.¶
paid):amount (a decimal string greater than zero), currency (an [ISO4217] code; USD in this version) and per: page (each page delivered) or request (each request answered).¶
required, requested or none: whether the client is to cite and link the source when content is shown to a person.¶
paid or deny):A free allowance: requests (an integer from 1 to 1,000,000,000) and per (hour or day). See Section 3.4.¶
The terms state that content is not to be used for a use whose access is deny, and is used for a use whose access is paid only after payment, beyond any free allowance (Section 3.4). A use whose value is not an object with a valid access is treated as not listed.¶
A use whose access is paid or deny MAY carry a free member, for example { "requests": 100, "per": "day" }. The site answers up to requests requests for that use from each client whose identity it has verified (Section 7), in each UTC hour or UTC day, with the content, without payment; after that, the use's access applies for the rest of the window. Only requests answered with the content count toward the allowance. A client whose identity is not verified gets no allowance. Attribution applies to requests made within the allowance. On a use whose access is allow, free has no meaning and MUST be ignored.¶
The allowance is a pacing signal: a client that reads steadily can stay within it, and only a burst meets the price or the refusal. A site counts requests per verified client identity, per use and per window. A site that counts approximately, for example on several servers that each keep their own count, MAY grant more than the stated allowance and SHOULD NOT grant less.¶
Each entry of paths has match (a path pattern starting with /), uses (Use Terms for one or more uses) and optionally payment (the payment for the prices it sets).¶
In match, * matches any sequence of zero or more characters other than /, and every other character matches itself. A pattern matches a request when it matches the whole path component of the request target, without the query, after both are normalized as in Section 6.2.2 of [RFC3986]: hexadecimal digits in percent-encodings are compared in upper case, percent-encoded unreserved characters are decoded, characters outside the set allowed in a path (for example non-ASCII characters) are percent-encoded as UTF-8, and dot-segments are removed from the request path (Section 5.2.4 of [RFC3986]). An encoded slash (%2F) is not a segment separator. Patterns MUST NOT contain dot-segments.¶
The terms for a use on a path are found for each use separately: among the entries that match the path and list that use with a valid Use Terms object, the one with the most characters in the normalized match other than * applies; if two have the same number, the first in the array applies. If no matching entry lists the use, the top-level uses member for that use applies, and if it is missing or invalid, default applies. Because * does not cross /, an entry applies at one depth; a subtree at several depths needs one entry per depth.¶
An implementation MUST match patterns in time linear in the length of the pattern and the path (for example segment by segment, since * never crosses /). It MAY ignore patterns longer than 512 characters, after normalization, or containing more than 16 * characters.¶
{
"ai_terms": "0.1",
"site": "https://example.com",
"updated": "2026-11-18",
"uses": {
"search": { "access": "allow", "attribution": "required" },
"answer": { "access": "allow", "attribution": "required" },
"training": { "access": "paid",
"price": { "amount": "0.002", "currency": "USD",
"per": "page" },
"free": { "requests": 100, "per": "day" } },
"agent": { "access": "paid",
"price": { "amount": "0.002", "currency": "USD",
"per": "page" } }
},
"default": "paid",
"identity": { "verify": ["published-ip-ranges"] },
"purpose_header": "AI-Purpose",
"payment": { "protocol": "x402", "version": 2,
"endpoint": "https://pay.example/g/abc123def456",
"network": "eip155:8453", "asset": "USDC" },
"preview": { "shows": ["title", "description", "excerpt", "words"],
"delivery": "per-quote",
"described_quote":
"https://pay.example/g/abc123def456{+path}" }
}
¶
The AI Preferences statement these terms imply for every path (Section 5.1) is train-ai=n: training is paid, so n; answer is allowed but agent is paid, so nothing is stated for AI Use; and Search is not stated, because a Search preference would also allow training for search, which these terms price.¶
In addition to the well-known URI, a server MAY reference terms from any response with a Link header field [RFC8288] using the relation type ai-terms. The target MUST be an https URI and MAY be on another origin, for example a document that a rightsholder or a registry publishes for the site's content:¶
Link: <https://example.com/.well-known/ai-terms>; rel="ai-terms"¶
A server MUST include this link in every HTTP 402 (Payment Required) response [RFC9110] that it sends because of its AI Terms, so that a client refused for payment can find the terms it is being asked to pay under. A server SHOULD include it in other responses to automated clients, in particular successful ones, so that a client or archive that stores response header fields keeps the pointer with the content.¶
A document's site names the origin whose content it covers, wherever the document is hosted. A client MUST ignore a document whose site is not the origin of the content it is deciding about: the origin of the well-known URI it fetched, of the robots.txt that named it, or of the response whose Link field referenced it. A document hosted on another origin (by a rightsholder, a registry or a provider that generates terms) thus applies only when the site's own responses or its robots.txt reference it, and only to that site's content.¶
A document referenced by a Link field applies to the target resource of the request the response answers (its path and query), whatever the response's status. For that resource it replaces the site-wide document entirely: its paths are matched against the request path, and its default, identity and free members apply; a free allowance it states is counted separately. Only Link header fields of a final response count, not those of a 1xx response or link elements in content. A link with an anchor parameter is ignored, and if a response carries more than one ai-terms link, the client ignores them all. If the linked document cannot be retrieved or is not valid, the client applies the site-wide document; a site that depends on stricter linked terms enforces them at the server (Section 11). Every redirect followed while retrieving a linked document MUST use https, and a linked document is cached by its URI.¶
An AI-Terms record in the origin's robots.txt names a site-wide document, as the well-known URI does; if the well-known URI yields a valid document, that document is used and the record is ignored. Responses from a payment endpoint or a described_quote URI on another origin never establish terms: such a settling party may link the terms as a pointer, and the client checks the linked document against the terms it obtained for the site.¶
A header field travels only with the response. Content copied to another host loses the link, and this document defines no way to name a work independently of the origin that serves it (see Appendix C).¶
A site SHOULD keep the terms document and /robots.txt free to fetch for every client, whatever its terms for other paths: they are how a client learns the rules and the price, and a client that is refused /robots.txt may treat the site as having no rules at all (Section 2.3.1.3 of [RFC9309]).¶
A site MAY add a line AI-Terms: <URL> to its robots.txt. Section 2.2.4 of [RFC9309] allows records beyond the protocol's own; crawlers that do not interpret this one are unaffected by it.¶
Most automated clients do not read AI Terms. Some are starting to read AI Preferences: at least one operator documents that, since 2026-10-30, it has begun recognizing a "no AI training" preference based on the AI Preferences drafts [ANTHROPIC-CRAWL]. This section defines how a site's AI Terms are written as AI Preferences statements and as robots.txt rules. The statements are a conservative approximation: they never grant more than the terms, and they cannot express what the vocabulary has no category for, such as a price or training models that are not generative.¶
The AI Preferences statement implied by a site's terms for a path is derived from the access that applies to each use on that path (Section 3.5):¶
train-ai):y if training is allow; n if it is paid or deny.¶
ai-use):y if answer and agent are both allow; n if neither is allow; otherwise no preference is stated.¶
search):y if search and training are both allow; otherwise no preference is stated. A Search preference of y also allows training models used only within the search application, and overrides the other categories (Section 4.3 of [I-D.ietf-aipref-vocab]), so it is not stated where training is priced or refused. AI Terms governs AI-powered search only, and the Search category covers every search application, so a price or a refusal for AI-powered search is not a reason to state search=n.¶
A paid use is written as n because an AI Preferences statement carries no price and no identity: a client that reads only the preference cannot pay, and must not make the use for free. A client that pays under the AI Terms is licensed by its payment (Section 8), not by the preference. A free allowance (Section 3.4) does not change the derived value: it applies only to verified clients, and a preference applies to everyone.¶
The statement is serialized as in Section 6 of [I-D.ietf-aipref-vocab], a Dictionary [RFC9651] with the categories in the order above, for example train-ai=n, ai-use=y. Because train-ai is always stated, the statement is never empty. No statement is derived from a document whose report_only is true: its operator has not chosen terms.¶
The derivation uses the terms for each use, not default. A preference describes uses, and a client that acts on one is making a particular use, which is what declaring it would state (Section 6); default governs clients that do not say what they are doing. A site whose default is more restrictive than its terms for a use, and which wants clients that read only preferences to meet that default, writes the more restrictive terms for that use.¶
A server that publishes AI Terms SHOULD send the Content-Usage header field [I-D.ietf-aipref-attach] with the statement derived for the request path on successful responses to automated clients. The field describes the content of the message, and a 402 response carries a price, not the page, so a 402 response does not carry it for the page; the Link field (Section 4) points to the terms instead. A server that already sends its own Content-Usage field for a response combines it with the derived statement as Section 5.1 of [I-D.ietf-aipref-vocab] combines preferences: for each category, n in either gives n.¶
For a response that carries an ai-terms link, the statement is derived from the linked document (Section 4). robots.txt rules (Section 5.3) are derived from the site-wide document only; where the two differ, Section 5.4 makes the more restrictive apply.¶
A successful response to a paid request carries the same statement as an unpaid one. The payment licenses the client that paid; the receipt (Section 9), not the preference, records that. An archive that stores the paid response together with its header fields keeps the site's preference for everyone else.¶
A site MAY add rules derived from its terms to its robots.txt:¶
Content-Usage rules ([I-D.ietf-aipref-attach]): the statement derived from the top-level terms, and a rule for each paths entry with the entry's pattern as the rule's path, its characters that robots.txt reads specially ($, # and others outside the path set) percent-encoded. A rule's value is the derived statement where that is no looser than the terms on every path the rule reaches, and otherwise the most restrictive value the terms give any such path.¶
A group for the product tokens whose operators document them as collecting content for AI training or as controlling its use for AI training (for example GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended), with Disallow for the paths where training is paid or deny. Search crawlers and fetchers acting for a person are not in this group; their uses are not training.¶
A group for the policy token __usage_ai-training ([I-D.kelley-aipref-training-preference-signal]), with Disallow for the paths where training is paid or deny and Allow where it is allow, including Allow: / when training is allowed everywhere, since a missing group states no preference. The group holds only these rules: it states a preference, not crawl rules, so a site's own Disallow rules and Content-Usage rules do not belong in it, and it is kept apart from groups that name crawlers. No group is written for a document whose report_only is true.¶
A robots.txt path pattern cannot express every AI Terms pattern exactly: in robots.txt, * also matches /, a pattern matches a prefix of the path and query, and the longest pattern wins rather than the one with the most literal characters. A site or generator that writes these rules MUST NOT write a rule that, for any path it applies to under [RFC9309] and [I-D.ietf-aipref-attach], states a preference less restrictive than the terms (where n is more restrictive than no statement, and no statement more restrictive than y), or allows a training-only crawler a path where training is not allow, or states a preference in favour of training through the policy token on such a path. Where a rule cannot be exact, it errs toward the more restrictive value. Because readers differ on which of two rules of equal length applies, a generator MUST assume that the less restrictive one does. It MUST NOT copy into the training group a site's own Allow rule that reaches a path where training is not allow. The Content-Usage header field (Section 5.2) is computed per path and is exact.¶
Because a crawler named in a group of its own reads only that group (Section 2.2.1 of [RFC9309]), the site's own rules and the Content-Usage rules need to appear in every group of the file to reach every crawler. No preference is implied for a disallowed path ([I-D.ietf-aipref-attach]), so a group that disallows every path needs no Content-Usage rules.¶
The Disallow rules stop training-only crawlers before they see a 402. That is the intent for crawlers that cannot pay: they are kept from taking for free what the terms sell. A crawler that can pay finds the price in the AI Terms document, which the AI-Terms record points to; a site that expects such crawlers can leave their product tokens out of the group. A site that prefers every crawler to meet the 402 can omit these rules and keep the Content-Usage rules alone.¶
An example for the terms in Section 3.6, where training is paid site-wide, with a site's own Disallow: /admin kept in both groups (some training tokens left out for brevity):¶
User-agent: * Disallow: /admin Content-Usage: train-ai=n User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Google-Extended User-agent: Applebot-Extended Disallow: /admin Disallow: / User-agent: __usage_ai-training Disallow: / AI-Terms: https://example.com/.well-known/ai-terms¶
A client that reads a site's AI Terms and also its AI Preferences statements combines them as follows. Only statements attached by the site's origin are considered here: its robots.txt and the Content-Usage field of its own responses. A statement attached by anyone else, including one embedded in the content, is not overridden by a payment under the AI Terms, and is combined as [I-D.ietf-aipref-vocab] says.¶
The access that applies under the AI Terms is the result, with three qualifications. First, an n from any statement attached by the site's origin, for the category of a use the AI Terms allow, denies that use, as Section 5.1 of [I-D.ietf-aipref-vocab] requires. A robots.txt rule derived from terms can be stricter than the terms (Section 5.3), so a site that wants such uses honoured uses patterns robots.txt can express exactly. Second, an AI Preferences y never makes a paid or deny use allowed. Third, an n for the category of a use the AI Terms make paid is the derived statement: a payment made under the AI Terms is offered by the site as a specific arrangement in the sense of Section 5.2 of [I-D.ietf-aipref-vocab], which can take precedence over it, so it does not stop a client that pays from making the use. Likewise, content a client receives within a free allowance (Section 3.4) may be used for that use. A site that does not want a use made by any client, paying or not, states deny with no free allowance.¶
In the other direction, a site that publishes only AI Preferences statements has no AI Terms: y and n do not become allow and deny terms, and no price is implied.¶
AI-Purpose is a request header field. Its value is a Token [RFC9651] naming one of the uses defined in this document:¶
AI-Purpose = "search" / "answer" / "training" / "agent"¶
Senders MUST send one of these tokens, in lower case and with no parameters. A recipient ignores any other value, treating the request as if it carried no AI-Purpose; it MAY accept a value that differs only in case.¶
An automated AI client SHOULD send AI-Purpose with each request to a site that publishes AI Terms. The field states the use the client intends, and the terms for that use apply to the request. A server MAY record the declaration and MAY echo it in the response. A request without the field gets the default terms.¶
A client's claimed identity (for example a user-agent product token) is not by itself evidence of who operates it. The identity.verify member lists the methods a site applies:¶
The request's source address is within the address ranges the claimed operator publishes for that client.¶
The request carries an HTTP message signature verifiable with a key the claimed operator publishes, as defined in [I-D.ietf-webbotauth-httpsig-protocol].¶
A client fails a check when it claims an identity whose operator publishes the means to check it (address ranges or keys) and the check does not confirm it. The identity.unverified member says what applies to a client that fails every listed check:¶
The default terms apply to it. This is the value when the member is absent or has any other value.¶
The terms for the use its claimed identity is documented for (for example training for a product token its operator documents as a training crawler) apply to it, or default when no use is documented; it earns no free allowance.¶
In both cases a failed check never earns a free allowance. A client that claims no identity the site can check is not verified (it earns no free allowance) but has not failed, and the terms for its declared use apply. When identity is absent, the site verifies no client: none fails a check, and none earns a free allowance.¶
payment.endpoint is an absolute https URI with no query or fragment. The content at path P (with its query) is paid for at the endpoint with any trailing / removed, followed by P; a site that answers 402 itself MAY give its own origin. A settling party MUST ensure that P cannot reach a path outside the site's: it normalizes P as in Section 3.5 and then removes dot-segments (Section 5.2.4 of [RFC3986]), and it does not decode an encoded slash or backslash into a separator.¶
When the applicable access is paid, a server answers a request that carries no valid payment with status 402 and the payment requirements of the protocol named in payment.protocol. For the x402 binding [X402], the requirements are carried in the PAYMENT-REQUIRED response header field; the client retries with a signed authorization in the PAYMENT-SIGNATURE request header field, and the content is returned only after the payment is verified. With the exact scheme on EVM networks, settlement transfers the stated amount from the client's wallet to the payee named in the payment requirements. The x402 binding settles in the asset named in payment.asset, a stablecoin denominated in the price's currency; a client compares a requirement's amount with the price by converting the asset's atomic units with its decimals, counting one unit of the asset as one unit of the currency, and SHOULD refuse a requirement that asks for more than the price. A client MUST check that a requirement's network and asset identify the asset the terms name, by network and contract address rather than by symbol, and MUST take the asset's decimals from its own knowledge of that asset, not from the quote.¶
A client SHOULD pay only at an endpoint named by the site's well-known document, or by a linked document (Section 4) whose endpoint the well-known document also names, in its own payment member or in a paths entry. A linked document that names any other endpoint, or a higher price than the well-known document for the same use, makes a claim that the site's operator has not confirmed in its site-wide terms; a client that pays under it does so only within its own limits. Terms that the site neither publishes nor references say nothing about whether it authorized anyone to sell its content (Section 11).¶
Other payment protocols can be bound by defining their payment members in a future revision.¶
A payment of this kind cannot be taken back, so a client needs to know what it is buying before it pays, and a site should not be paid for something else. This section defines how a quote describes a page and how the delivery is held to that description. It applies to quotes made by a settling party that is not the site: when a site settles its own payments, it holds every key the check depends on and could describe one page and deliver another, so such a site MUST NOT send delivery set to must-match. A client cannot itself tell whether the settling party is independent of the site; it relies on the terms it obtained from the site's origin naming that party (Section 8).¶
A quote (the 402 response) for a paid page SHOULD describe the page. For the x402 binding [X402V2], the description is carried as the extension named ai-terms in the quote's extensions object. As that protocol requires of every extension, its value has two members: info, the description, and schema, a JSON Schema for it. info is an object with these members:¶
The page's title.¶
The page's own summary, when it has one.¶
The opening of the page's main text; null when the site's snippet controls forbid showing it.¶
The length of the page's main text in words.¶
must-match when the quote is held to this description, as defined below; not-checked otherwise.¶
Unknown members MUST be ignored. A site's snippet controls (nosnippet, max-snippet) apply to the text shown.¶
A quote that describes a page commits to it. The commitment is an integrity-protected value, bound to the request URI and limited in time, that travels with the payment, so that the delivery is checked against what that client was shown and not against a later reading of the page. For the x402 binding, the settling party places the commitment in the extra member of each payment requirement it offers; a version 2 client includes the requirement it accepted, extra included, in its payment payload, and the settling party reads the commitment from there. The name and format of the commitment are private to the settling party, and a client treats extra as opaque.¶
A payment answering a described quote MUST NOT be settled unless the main text delivered is at least half as long as the main text described and the settling party's comparison accepts it. The comparison SHOULD reject content that shares less than half of the described main text; how it measures shared text is an implementation matter, and Appendix A gives the method and measurements of one implementation. The main text is the page's main region when it marks one (for HTML, a single main or article element), otherwise its visible text. A page that is empty, much shorter, or mostly different text is not a delivery, whatever its HTTP status.¶
A description taken after the payment request is received MUST NOT be used to judge that payment's delivery: a site that serves the same wrong content to every request would then match itself.¶
A settling party that describes pages MUST NOT settle a payment for a page when the payment carries no valid commitment, unless the quote that payment answers was not held to a description (see below). It answers such a payment with a fresh described quote instead.¶
The description in a quote is not itself authenticated: a site that builds its own quotes could claim must-match for a page the settling party never described. A client SHOULD therefore return the quote's ai-terms extension with its payment, as the x402 version 2 specification provides for extensions, and a settling party MUST NOT settle a payment that returns a description with delivery set to must-match unless the payment also carries a valid commitment. A client that does not return the extension is not protected against such a claim.¶
The request made to describe a page and the request made to deliver it SHOULD be indistinguishable to the site, so that a site cannot show one page to be described and another to be paid for.¶
A quote is held to a description only if it carries one, with delivery set to must-match. Any other quote is not checked: it is paid when the page is returned successfully, or answered with a fresh described quote. Such a quote SHOULD carry an ai-terms extension whose info has delivery set to not-checked and, when a described quote can be had elsewhere, a described_quote member saying where, in the form defined for the terms document below. A client that wants the check MUST treat a quote with no ai-terms extension, or one whose delivery is not must-match, as not checked, and asks for a described quote instead.¶
The preview member of the terms document declares which of these a site does:¶
What a description contains: any of title, description, excerpt, words.¶
must-match: every quote describes the page and is held to it. per-quote: each quote says which applies; quotes obtained from described_quote describe the page and are held to it, and any other quote is paid when the page is returned successfully. not-checked: pages are not described and deliveries are not compared.¶
per-quote):Where a client asks for a described quote: a URI Template [RFC6570] with one variable, path, the path and query of the page starting with a slash, for example https://pay.example/g/abc123def456{+path}.¶
This mechanism does not judge whether a page is good, accurate or worth its price. It holds the site to the page it showed.¶
A successful paid response SHOULD carry:¶
the payment protocol's proof of settlement (for x402, the PAYMENT-RESPONSE header field identifying the settled transaction), and¶
the AI-Purpose header field echoing the declared use.¶
Together with the request URI and time, this forms a record of licensed use. The site can show which automated client accessed which content, for which use, and what was paid; the client's operator can show that it licensed the content it used. A receipt shows that a payment was made for a request; it does not show that the content was later used only for the declared use. A future revision may define a signed receipt object independent of the payment protocol; the compliance records and transparency receipts of [I-D.reilly-aipref-compliance] are a candidate format to reuse rather than reinvent.¶
Unchanged. A disallowed path remains disallowed; AI Terms governs the use and price of content that is fetched. Section 5.3 defines robots.txt rules derived from AI Terms.¶
AI Terms is an extension of this work. Its uses refine the vocabulary's categories (Section 2.1), its terms are written as AI Preferences statements and as the robots.txt training policy token of [I-D.kelley-aipref-training-preference-signal] (Section 5), and Section 5.4 says how a client combines the two. It adds what the working group's charter leaves out: a price and payment binding for uses a site permits for a fee, client-side purpose declaration, a check that a paid page is the page described, and receipts.¶
Declares usage policy per category and per agent, with a Training-Fee member that links to a pricing page. AI Terms adds a machine-actionable price and payment binding, client purpose declaration and receipts. If preferred, the Use Terms, payment and purpose members of this document could be defined as an extension of the ai.json format instead of a separate well-known URI; the author is open to that.¶
[I-D.reilly-aipref-compliance] records what a processor observed and decided, and excludes payment. AI Terms receipts record paid uses; the two can coexist, and a later revision may reuse its record format.¶
RSL 1.0 expresses licensing terms as an XML document discovered through a License: line in robots.txt, a Link header field with rel="license", or markup. It defines payment types including per-crawl and per-use, a license server protocol based on OAuth 2.0, and can advertise x402 as an accepted payment method. RSL is the more complete licensing framework, has publisher backing, and can also state amounts and answer with 402. AI Terms is deliberately smaller: a lightweight x402 profile that works alongside RSL, with one JSON document of per-use access, no license server, a purpose declaration by the client and a receipt on each paid response. The two can be published side by side, and a site with an RSL license can point to it from contact.licensing. Where both are present and disagree, a client should follow the more restrictive, counting a refusal as more restrictive than a requirement for a license, and either as more restrictive than a price. A mapping between RSL payment types and AI Terms uses is future work, and the author would welcome doing it with the RSL community.¶
Terms are statements, not enforcement. Sites that require them to hold SHOULD enforce them at the server or edge, for example by answering 402 for paid uses and verifying identity before granting an allowance. A site whose default is allow gives every use to a client that declares none; a site that prices a use and wants that price paid sets default to paid or deny.¶
Terms referenced from another origin MUST be fetched over HTTPS. A document's terms apply to the origin named in site, and only when retrieved for that origin (Section 4). Anyone can publish a document naming any site and a payment endpoint of their own, and a gateway can answer 402 for content it does not own. A client SHOULD therefore pay only at an endpoint named by terms obtained as Section 4 describes (Section 8), and a settling party that sells content on a site's behalf SHOULD verify that the site's operator controls the site, for example by finding its terms, or another token it issued, served from the site's origin. Whoever can set header fields on a response can link terms for it, including a payment endpoint. A host that lets several parties set header fields under one origin SHOULD remove ai-terms links that its tenants set, unless it intends their terms, including their payment endpoint, to govern their content, as Section 4 provides; clients limit the exposure by paying only at endpoints the site-wide terms name (Section 8).¶
A malicious or compromised terms document, or a quote, can state any price. Clients MUST apply their own spending limits per request and per period and MUST NOT pay without such limits. A client SHOULD compare the price in a 402 with the price in the terms and refuse a quote that asks for more.¶
A client can misstate its identity or purpose. Identity verification limits the former. This protocol cannot detect a client that declares a cheaper use than the one it makes (for example search to get content it trains on); only a paid access leaves a receipt, and independent detection of copied content is the only check. A free allowance is counted per verified identity (Section 3.4), so claiming a well-known client's name does not earn its allowance. Published address ranges change; a site SHOULD refresh them regularly, since a stale list can refuse a genuine client or accept one whose address was reassigned.¶
A robots.txt rule or Content-Usage value that is looser than the terms would grant uses the site did not grant. Section 5.3 therefore requires derived rules to be no less restrictive than the terms on any path they reach. Values copied from a remote source (for example a gateway's quote) into response header fields MUST be checked against the syntax of the field before being sent, so that a compromised source cannot inject other header fields.¶
A terms document is parsed by clients and its patterns are matched on every request. Clients SHOULD bound the size of the document they accept (for example 64 KiB) and the number of paths entries they consider. Pattern matching MUST NOT backtrack (Section 3.5): a pattern made of many * characters separated by single letters makes naive regular-expression matching take exponential time.¶
The delivery check of Section 8.1 compares text, not meaning. A site can describe and deliver the same poor page and be paid for it, and a delivered page may differ from the described one by up to about half of its main text. The check does not cover a quote that carries no description. A description is only as good as the settling party: the protection in Section 8.1 against a forged description depends on the client returning the quote's extension with its payment. A client that needs the check MUST look for the description in the quote it pays (Section 8.1), not only at the terms document. The commitment MUST be integrity-protected with a key the site cannot obtain from the quote, bound to the request URI and short-lived; otherwise a site or a third party could substitute a description after the fact.¶
A description discloses part of a paid page without payment. Sites decide how much through preview.shows and their snippet controls.¶
Clients SHOULD cache terms for a bounded time and respect HTTP caching header fields. A 402 response and a paid response MUST carry Cache-Control: private or no-store, so that a shared cache does not serve a price or a paid page to another client. A response served within a free allowance or after an identity check SHOULD also be marked private, and a response whose status depends on AI-Purpose SHOULD carry Vary: AI-Purpose. A shared cache in front of the server answers every client alike, so a site that needs per-client terms enforces them in front of such a cache. A settling party MUST NOT settle the same payment twice (with the exact scheme on EVM networks, authorizations carry a nonce for this) and SHOULD accept a payment only for the request URI it was quoted for.¶
The AI-Purpose header field reveals the client's intended use to the server; this is its purpose.¶
Payment links a payer to the content it bought. With x402 on a public blockchain, the transfer itself (payer and payee addresses, amount and time) is public, while the URI bought is known to the site and the settling party; the site's records connect the two. Operators that consider this sensitive should use payment accounts dedicated to automated access.¶
Verifying identity by address and counting allowances require the site to process client addresses and identities. Sites should keep only what counting needs, for no longer than the window it needs.¶
IANA is requested to register the following in the "Well-Known URIs" registry [RFC8615]:¶
IANA is requested to register the following in the "Link Relation Types" registry [RFC8288]:¶
IANA is requested to register the following in the "Hypertext Transfer Protocol (HTTP) Field Name Registry" [RFC9110]:¶
Note to the RFC Editor: please remove this section before publication.¶
This section records the status of known implementations of the protocol defined by this specification at the time of posting of this Internet-Draft, and is based on a proposal described in [RFC7942]. The description of implementations in this section is intended to assist the IETF in its decision processes in progressing drafts to RFCs. Please note that the listing of any individual implementation here does not imply endorsement by the IETF. Furthermore, no effort has been spent to verify the information presented here that was supplied by IETF contributors. This is not intended as, and must not be construed to be, a catalog of available implementations or their features. Readers are advised to note that other implementations may exist.¶
AIPayGen (the author's).¶
Server side: generates the AI Terms document for each site from its settings, serves a site's own document at /.well-known/ai-terms, sends Link: rel="ai-terms" on every 402 it sends, verifies client identity against published IP ranges and, for agents that sign their requests, by Web Bot Auth ([I-D.ietf-webbotauth-httpsig-protocol]: an Ed25519 HTTP message signature covering @authority and signature-agent, with created and expires, checked against the key in the agent's published key directory), settles with x402 and echoes AI-Purpose on paid responses. It derives AI Preferences statements as in Section 5: the gateway sends Content-Usage on paid responses and serves robots.txt rules for each site; the author's own site sends Link on every 200 and 402 response, Content-Usage on every 200 response, and writes its robots.txt from its own terms. The robots.txt rules include the __usage_ai-training policy token group of [I-D.kelley-aipref-training-preference-signal], as Section 5.3 describes, in the rules served for each site and in the author's own robots.txt. The robots.txt generator computes, for each rule, the strictest value the terms give any path the rule reaches, by exhaustive search over the product of the patterns' automata, so that no rule is looser than the terms. A site that only measures traffic (report_only) gets no derived preferences until its operator chooses terms. For pages bought through a quote the gateway made, the 402 describes the page (title, description, a 200-character excerpt, word count) and the payment is settled only if the delivered page is at least half the described length and shares enough of its main text, measured as an estimated Jaccard similarity of at least 0.38 between bottom-k MinHash sketches (k = 64) of the four-word shingles of the first 1,500 words of each main text. In the author's trials with one article, deliveries keeping half or less of the described text were refused in every trial; acceptance began at about 52 percent and was complete at 60 percent. The description is sent as the ai-terms extension and, for older clients, a second time under a vendor key. The commitment expires after 15 minutes. A payment for a page that carries neither a valid commitment nor a valid not-checked marker is answered with a described quote and not settled, and so is one that carries the marker but returns a description claiming must-match. The quote that a site's own install fetches from the gateway and answers with is not held to a description: it says so in an ai-terms extension with delivery set to not-checked, and in a marker in each requirement's extra; installs that rebuild the quote themselves pass on only the marker. Responses that refuse or fail a payment carry a vendor header field saying whether anything was charged: nothing, or unknown when a settlement call broke off. The free allowance is applied by the Cloudflare Workers, Express, Next.js, Netlify, Vercel and Django integrations, counted per process, and by the WordPress plugin since its version 2.4.0; the gateway publishes the allowance in the terms it generates but does not itself count or grant it. Client side: an agent tool that, before paying for a page, reads the document a Link field in the page's response points to (any https host whose name is not private; redirects are not followed) or, when the response links none, the site's well-known document; a link it cannot read counts as no terms. It refuses a use the document denies or prices above the caller's limit, and sends AI-Purpose: agent. It pays only the requirements in the page's own 402 response, within the caller's limit, and never pays an endpoint that terms name, so terms can only make it refuse. It applies a document only when its site is the page's origin (Section 4), whether the document was linked, found at the well-known URI, or linked from the gateway's quote for one of its routes.¶
Running in production on one site and offered to others; early.¶
This document except: signed-request identity (Section 7) as a basis for the free allowance or for terms (Web Bot Auth signatures are verified, and a public endpoint shows any client the verdict for its own request, but they are used only to label and count clients; the site integrations grant allowances by published IP ranges, and generated documents list only published-ip-ranges); choosing terms by AI-Purpose (Section 6), for which crawler class is used instead (see Description); the free allowance (Section 3.4) in the gateway and in the Nginx, Apache and Caddy integrations; and the free allowance on a use whose access is deny, which no integration applies yet. Content-Usage is sent by the gateway (so also through the Caddy integration, which relays it) and by the Cloudflare Workers, Express, Next.js, Netlify, Vercel and Django integrations (on pages they serve to AI crawlers), not by the Nginx, Apache and WordPress integrations, which send only the Link, on their 402s. The gateway publishes preview.delivery as per-quote. The gateway's own responses link the route's document, whose site is the merchant's origin, as a payment-endpoint pointer; the author's own site's document does not describe the gateway's paths. The documents the site integrations link name the route's upstream origin, so they apply only to pages served from exactly that origin (not, for example, its www variant). The integrations choose the terms for a request by the crawler's class, from its user agent and published addresses, not from AI-Purpose, which the gateway only records and echoes. A client that claims a crawler's name from an address its operator does not publish is answered as that crawler without a free allowance, and the documents the implementation generates say so (identity.unverified is treated-as-claimed); in a site's "training" mode, search and on-demand crawlers are therefore let in free by name. The WordPress plugin's own 402s carry the Link (since 2.5.1), but 402s it relays from the gateway for a paid retry do not yet. A new route that sells a site's pages must name the site's origin, so that the endpoint rule of Section 8 holds exactly. A route for an API, or a page route created before that rule, can have a base path; its document still names the whole origin and the endpoint rule does not hold for it (clients buy at the route's own URL), and no robots.txt rules are generated for it.¶
The site integrations (WordPress, Cloudflare Workers, Nginx, Apache, Caddy, Next.js, Netlify, Vercel, Express, Django) are open source [TOOLS]: GPL-2.0-or-later for the WordPress plugin, MIT for the others. The gateway is not.¶
The author.¶
The author.¶
A single-file reader using only the Python standard library: fetches a document, resolves the terms for a use and path (Section 3.5), derives the AI Preferences statement for a path (Section 5.1), checks a price against a client limit, and validates a document.¶
New; written from this document's text.¶
Document parsing, Section 3.3, Section 3.5 (matching segment by segment, without backtracking) and Section 5.1. No payment, no robots.txt generation, and no delivery check.¶
A JSON Schema for the document and a set of test vectors are published [TOOLS]: 26 validity cases, 24 path-resolution cases for Section 3.5, 8 AI Preferences derivation cases for Section 5.1, and 5 robots.txt cases (23 checks of what an RFC 9309 reader concludes from generated rules) for Section 5.3. Both implementations above pass every case they cover; the robots.txt cases apply only to the gateway, which generates robots.txt. The two implementations share an author, so they are not independent; reports from others are welcome and will be listed.¶
Note to the RFC Editor: please remove this section before publication.¶
These are measurements from one site, the author's own [REFIMPL], read once from its public counter (the path /ai-crawlers/live.json on that site) at 2026-11-18T22:16:48Z. They describe one small site over three days and are not a survey. The site's pages are free to every AI use; only its marketplace listings (third-party APIs, servers and agents) are priced, at USD 0.001 a page, and AI crawlers that request a listing get a 402 with x402 payment requirements and a Link to the site's AI Terms.¶
| Measure | Value |
|---|---|
| Counting began | 2026-11-16 |
| Requests from AI crawlers (by user agent, and by published address where the operator publishes one) | 6,731 |
| Of those, answered with a 402 quote | 3,231 |
| Payments by crawlers | 0 |
| Requests identified as fake (a crawler's name from an address its operator does not publish) or disguised (a browser's name from an address an AI operator publishes) | 90 |
| Distinct crawler labels | 25 |
The crawlers with the most requests were GoogleOther (3,442), Amazonbot (800), GPTBot (717), meta-externalagent (646), ClaudeBot (468) and ElevenlabsCrawler (466). Anthropic's and Mistral's crawlers are checked by address only from 2026-11-17, when their address lists were first used; earlier requests were counted by name.¶
With the caution that one small site allows: crawlers that collect in bulk appear to treat a 402 as a failed fetch and move on. None paid. This is why Section 5 writes the terms into the signals such crawlers already read: robots.txt rules, which they follow, and the AI Preferences statement, which at least one operator has begun to recognize for training [ANTHROPIC-CRAWL]. The counter also records which address-verified crawlers fetch robots.txt, /llms.txt and /.well-known/ai-terms; at the time of the reading it held no such reads, so this document reports none.¶
Note to the RFC Editor: please remove this section before publication.¶
Questions the author would like the working group's and readers' views on:¶
The document covers an origin (site). Terms that travel with a work across hosts, for embedded metadata or a registry lookup, would need to name the work instead, for example by a URI or an identifier. An optional member doing so would change what a document covers: a reader of this version would ignore it and apply the terms to the whole of site when it finds the document at the well-known URI or through robots.txt (a linked document is already limited to one resource). It is therefore left out of this version, and adding it would come with a new major version of ai_terms. It would preferably be aligned with any asset-identification mechanism the AI Preferences work adopts rather than defined separately.¶
* never crosses /, so terms for a whole subtree need one paths entry per depth. A form such as ** would change the matching of this version.¶
A site cannot yet say in its site-wide document which hosts may publish terms, or name payment endpoints, for its content; Section 8 limits payment to endpoints that document names. A list of delegated hosts would let a rightsholder's or a registry's document be confirmed rather than only linked.¶
Section 5.1 writes a paid use as n. Whether that reading is acceptable, against Section 5.1 of [I-D.ietf-aipref-vocab] and Section 5.2 of [I-D.ietf-aipref-vocab], is asked of the working group.¶
payment.asset is a symbol; clients check the asset by network and contract address themselves (Section 8). A later version could carry the address, for example as a CAIP-19 identifier.¶
Uses are mapped normatively onto the AI Preferences vocabulary (Section 2.1); a summary in AI search results is an answer use.¶
New section: writing AI Terms as AI Preferences statements: the derivation, the Content-Usage header field, robots.txt rules (including the __usage_ai-training policy token) that are never looser than the terms, and how a client reads both (Section 5).¶
Paths: each use is resolved separately; ties go to the first entry; * may match nothing; paths and patterns are compared after normalization, without the query; an invalid use value counts as not listed; matching must not backtrack (Section 3.5). The -02 text was ambiguous on these points; the new text matches the published implementations and test vectors, but a reader that followed -02 another way may resolve some documents differently.¶
Clarifications of format 0.1 (the document format is unchanged): payment.endpoint is an absolute https URI, and content at path P is paid for at the endpoint followed by P (Section 8); every 402 sent because of the terms carries Link: rel="ai-terms" (Section 4); a document applies only to the origin named in its site, and only when retrieved as Section 4 describes; what default applies to, and what a paid default costs; report_only derives no preferences.¶
A site that settles its own payments does not claim must-match (Section 8.1).¶
AI-Purpose is a Structured Field Token; unknown values are ignored (Section 6).¶
identity.unverified says whether a client that fails identity checks gets default or the terms of the identity it claims (Section 7).¶
Security and Privacy Considerations expanded: origin of the terms and of the payment endpoint, derived signals, header injection, resource use, caching and replay, undetectable purpose misdeclaration, public settlement data.¶
Implementation Status: AI Preferences derivation, Content-Usage, robots.txt generation, Link on every 402, and the WordPress plugin's free allowance (since 2.4.0).¶
Discovery: a Link target may be on another origin (a rightsholder's or a registry's); a linked document governs the resource the response answers and replaces the site-wide document for it; rules for several links, anchor, unreachable targets and redirects; a client pays only at endpoints the site-wide document names unless it accepts an unconfirmed claim within its limits (Section 4, Section 8).¶
New appendices: Field Observations (Appendix B) and Open Issues (Appendix C).¶
Thanks to the site owners and developers whose public discussion of AI crawler access shaped the uses, the free allowance, the delivery check and the receipt design.¶