Network Working Group S. McKenna Internet-Draft Independent Intended status: Informational 17 November 2026 Expires: 21 May 2027 Attested Conduct for Anonymous Web Data Collection: Use Cases, Requirements, and a Tiered Model draft-mckenna-webbotauth-collection-attestation-02 Abstract Anonymous Bot Authentication lets a web site learn that an automated client holds a credential from an Anchor whose policies it accepts, without learning which client it is. This document describes web data collection use cases that need that property, and what they require of the evidence an Anchor evaluates, of what a site can verify about conduct it cannot attribute, and of what a buyer of collected data needs to see. It proposes a tiered model. Collectors register signed statements of their own conduct in a SCITT Transparency Service; the site, its security intermediary, or a network intermediary may corroborate them; and an Anchor acting as a certifying body may appraise them. Coarse, unlinkable attributes about the data's eventual recipient let a site restrict use by category of recipient without learning who the recipient is. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 21 May 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. McKenna Expires 21 May 2027 [Page 1] Internet-Draft Attested Collection November 2026 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.1. Financial Research . . . . . . . . . . . . . . . . . . . 4 3.2. Price Comparison and Consumer Protection . . . . . . . . 4 3.3. Archiving and Research . . . . . . . . . . . . . . . . . 4 4. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 4 5. A Tiered Model . . . . . . . . . . . . . . . . . . . . . . . 5 5.1. Tier 0: Self-Attested, Logged Conduct . . . . . . . . . . 6 5.1.1. Collector-Governed Rates . . . . . . . . . . . . . . 7 5.2. Tier 1: Corroborated Conduct . . . . . . . . . . . . . . 8 5.2.1. Intermediary Witnesses . . . . . . . . . . . . . . . 8 5.3. Tier 2: Certified Conduct . . . . . . . . . . . . . . . . 9 5.3.1. Key Lifecycle . . . . . . . . . . . . . . . . . . . . 9 6. Recipient Categories . . . . . . . . . . . . . . . . . . . . 10 7. Relationship to Other Work . . . . . . . . . . . . . . . . . 11 8. Open Questions . . . . . . . . . . . . . . . . . . . . . . . 11 9. Security Considerations . . . . . . . . . . . . . . . . . . . 12 10. Privacy Considerations . . . . . . . . . . . . . . . . . . . 12 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 12.1. Normative References . . . . . . . . . . . . . . . . . . 13 12.2. Informative References . . . . . . . . . . . . . . . . . 13 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 14 1. Introduction A large share of automated web traffic is data collection: crawlers, monitors, and research tools that retrieve public pages for indexing, archiving, price and availability tracking, financial research, academic study, and consumer protection. Sites have good reasons to manage this traffic, and the IETF is developing two ways to do so: identified bots that sign their requests [I-D.ietf-webbotauth-httpsig-protocol], and Anonymous Bot Authentication (ABA) [I-D.rescorla-anonymous-webbotauth], in which a bot presents an anonymous credential issued under the policies of an McKenna Expires 21 May 2027 [Page 2] Internet-Draft Attested Collection November 2026 Anchor. Many legitimate collectors cannot use the identified path. A financial firm that reveals which retailers it monitors reveals its research. A consumer protection monitor that is identified can be served different content or blocked. ABA addresses this, and its authors note the tension directly: abuse controls that depend on knowing the identity of the abusive party do not work with anonymous authentication. ABA specifies how a credential is issued and presented. It leaves open three questions this document takes up: what evidence an Anchor relies on when it decides that a collector complies with its policies, how parties who do not trust the collector can check that evidence, and what a site or a data buyer can learn about conduct it cannot attribute. This document defines no new protocol. It shows how existing pieces (the SCITT architecture [RFC9943], the RATS roles [RFC9334], and ABA with MoLE [I-D.jms-mole-architecture]) fit this use case, and where gaps remain. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 2. Terminology Collector: An automated client that retrieves web content in order to produce data for its operator or for a recipient. Recipient: The party that ultimately receives the collected data, which may be the collector's operator or a downstream buyer. Site: The web site whose content is collected, together with any security intermediary (for example a bot management or WAF provider) acting for it. Anchor: McKenna Expires 21 May 2027 [Page 3] Internet-Draft Attested Collection November 2026 As in [I-D.rescorla-anonymous-webbotauth]: a party that evaluates whether a collector complies with its policies and issues endorsements. In RATS terms [RFC9334], an Anchor appraising evidence about a collector acts as a Verifier. Conduct Statement: A signed statement by a collector describing its own behavior toward a site over a stated interval. Transparency Service: As defined in the SCITT architecture [RFC9943]: an append-only, verifiable log that registers signed statements and returns receipts. 3. Use Cases 3.1. Financial Research Investment firms buy data on prices, inventory, job postings, and product listings. Their compliance teams increasingly ask for evidence of how the data was collected: whether robots exclusion [RFC9309] and other stated preferences were honored, whether request rates were reasonable, and whether the data was collected under the terms the vendor claims. The firms also cannot let sites learn which companies they watch. 3.2. Price Comparison and Consumer Protection [I-D.rescorla-anonymous-webbotauth] lists shopping sites blocking price comparison bots, and employment or housing sites blocking bots that monitor for discrimination, among the harms of identity-only schemes. These collectors have to show good conduct because they cannot show who they are. 3.3. Archiving and Research Archives and academic projects collect at scale for public-interest purposes. They are usually willing to be identified, and a verifiable record of conduct helps them when a site questions their behavior. 4. Requirements R1 (Unlinkability): A site MUST NOT be able to link a collector's requests to a specific collector or recipient beyond what the credential scheme in use allows. McKenna Expires 21 May 2027 [Page 4] Internet-Draft Attested Collection November 2026 R2 (Verifiable conduct): The behavior that matters to sites (request rate, backoff when a site slows or signals overload, adherence to robots exclusion and other published preferences) SHOULD be recorded in a form that a party other than the collector can check. R3 (Non-equivocation): A collector MUST NOT be able to present one record of its conduct to a site or Anchor and a different record to a recipient. R4 (Progressive assurance): The model SHOULD work without any certifying body, and SHOULD let stronger assurance be added later without changing what collectors record. R5 (Minimal attributes): Attributes disclosed to sites MUST be coarse. As participants noted when Web Bot Auth discussed ABA, fine-grained attributes become fingerprints that defeat anonymity. R6 (Recipient restrictions): A site SHOULD be able to express that certain categories of recipient may not receive data collected from it, and a collector SHOULD be able to prove membership in a permitted category without revealing the recipient. R7 (Accountability without identity): Misconduct SHOULD have consequences, chiefly loss of endorsement at the Anchor, without requiring that the site learn who misbehaved. R8 (Collector-governed rates): A well-behaved collector SHOULD set and enforce its own request rate for each site, and record the basis for it. Sites keep full control of their access policy. The aim is to give them a reason to rely on attested self-governance instead of limits per client, which can be used to discriminate among collectors and are not tied to the site's actual load. A site MAY publish capacity hints that apply equally to all automated clients. As with TCP congestion control, collectors need not coordinate: if each backs off when a site slows, their combined load falls as the site's load rises. 5. A Tiered Model The model has three tiers. Each adds a party and a stronger claim. A collector at any tier records the same Conduct Statements. McKenna Expires 21 May 2027 [Page 5] Internet-Draft Attested Collection November 2026 5.1. Tier 0: Self-Attested, Logged Conduct The collector produces Conduct Statements automatically from its own operation. A statement covers one site over one interval and contains: * the interval; * a keyed reference to the site, so that the log does not disclose which sites the collector visits; * request counts and the maximum and mean request rate; * the rate ceiling and per-interval budget the collector set for itself, and the inputs it used to set them (see Section 5.1.1); * evidence of backoff: response-time and error-rate observations, how each slowdown was classified, and the collector's corresponding rate changes; * a digest of the robots exclusion file and of any other preference signals in force during the interval, and whether they were followed; * a digest of the extraction configuration and software version used. The collector signs each statement and registers it with a Transparency Service [RFC9943]. The receipt proves that the statement existed at that time, and the collector cannot later present a different one (R3). Keyed site references are not enough on their own. A long-lived signing key, or software and configuration digests repeated in every statement, would let anyone reading the log link all of one collector's statements and see its whole footprint. Collectors SHOULD therefore sign with keys that are not reused across sites or epochs, for example keys bound to the credential or derived per site and epoch, and SHOULD salt or coarsen digests in the log, disclosing exact values only to a verifier that needs them. For the same reason, the key behind a keyed site reference SHOULD change per epoch, so that statements about one site in different epochs carry different references. Tier 0 proves what the collector said and when, not that it is true. It is cheap to adopt, it creates a consistent record that later tiers can check, and it lets a recipient check a vendor's claims against a log the vendor cannot rewrite. McKenna Expires 21 May 2027 [Page 6] Internet-Draft Attested Collection November 2026 5.1.1. Collector-Governed Rates Under R8 the collector, not the site, sets its rate: 1. It estimates the site's normal traffic and capacity from available signals: third-party traffic estimates, observed response times, error rates and page weight, time-of-day patterns, and any capacity hint the site publishes for all automated clients. 2. It sets its rate ceiling to a small fraction of that estimate, and keeps its own request budget per site and interval. 3. It monitors response times continuously against the site's baseline. 4. It separates site slowdowns from slowdowns in its own network path, such as a proxy vendor, by comparing measurements across several independent egress paths, and where available consults the site's status page or a third-party uptime monitor. A slowdown seen on most paths is treated as the site's; one confined to a single path or vendor is treated as the path's. 5. It separates overload signals from path-specific refusals. An HTTP 429 or 503, a Retry-After header, or rising errors seen across several paths indicate that the site is under load. A refusal confined to one path, such as a 429 on the first request from an address the site or its bot-management service has blocked, while other paths and ordinary browsers reach the site normally, reflects the reputation of that path rather than the site's load. 6. It reduces its rate on a site slowdown or overload signal, honors any Retry-After, and recovers gradually once the site returns to baseline. On a path-specific slowdown or refusal it may move requests to another path. Rerouting changes the path, never the rate: the collector's total rate to the site stays within its ceiling. The Conduct Statement records each of these decisions and the measurements behind them, including each reroute and the evidence that the refusal was confined to one path. A verifier can then judge whether the collector backed off and whether its ceiling and classifications had a reasonable basis. Estimation methods are left to implementations; recording their inputs and outputs is what makes them checkable. McKenna Expires 21 May 2027 [Page 7] Internet-Draft Attested Collection November 2026 5.2. Tier 1: Corroborated Conduct A site, or a security intermediary acting for it, observes requests that present a given credential. ABA lets the site apply budgets and rate limits per credential; under R8 these are a backstop, not the main control. In this tier the site also signs a statement of what it observed for that credential over an interval (request volume and rate, and any limits it applied), and either returns it to the collector or registers it in a Transparency Service. Matching the site's observations against the collector's Conduct Statements removes the main weakness of Tier 0: the collector is no longer the only source of evidence about its own behavior. Figures are compared within a stated tolerance, since the two parties' clocks and interval boundaries will not align exactly. How a site statement is bound to a collector's statements without creating a linkable identifier is an open question (see Section 8). 5.2.1. Intermediary Witnesses Corroboration need not come from the site. A network intermediary that carries the collector's traffic, such as a proxy network, VPN or egress gateway, scraping-infrastructure provider, or cloud network service, sees that traffic directly, and the collector cannot alter what it sees. Such an intermediary can sign a witness statement for a keyed site reference and interval. What it can attest to depends on what it can see. A proxy that only tunnels encrypted connections sees the destination host, timing, and volume: it can attest to connection and request rates and how they changed, but not to response statuses or headers. A service that terminates TLS for the collector, such as an unblocking API or a hosted browser, can also attest to the statuses, response times, and Retry-After values it relayed, and to how the collector's rate changed afterward. When a collector uses several paths or intermediaries, each records which of its paths received refusals or slowdowns, so a verifier can confirm that a reroute followed a refusal confined to one path and did not raise the collector's total rate to the site. Witness statements use the collector's keyed site references, so they reveal nothing about which sites were visited beyond what the intermediary already knows, and they SHOULD use signing keys that do not link the collector across sites. They can be registered in a Transparency Service, matched against the collector's Conduct Statements, combined across independent intermediaries so that none is trusted alone, and shown to Anchors, auditors, or data buyers. They add evidence from a party other than the collector without any McKenna Expires 21 May 2027 [Page 8] Internet-Draft Attested Collection November 2026 change at the site, but they supplement site corroboration rather than replace it. Each site and each Anchor decides whether to accept witness statements, and from which intermediaries. 5.3. Tier 2: Certified Conduct An Anchor, acting as a certifying body, appraises the collector's Conduct Statements and any corroborating statements against its policies, in the sense of a RATS Verifier [RFC9334]. It may also verify the identity of the collector's operator and of the recipient, and keeps both to itself. On a favorable appraisal it issues an endorsement under [I-D.rescorla-anonymous-webbotauth]. Because the Anchor sees identity and the site sees only the credential, identity checks can be strict without weakening R1. An Anchor that endorses a collector whose logged conduct contradicts its policies can be held to account, because the evidence it relied on is in a log it does not control. That accountability is strongest when the appraisal itself can be re- checked. An Anchor SHOULD record each appraisal result with references to the statements it relied on and to the version of its policy it applied, so that a party that does not trust the Anchor can re-run the mechanical parts: whether the collector's ceiling was within policy, whether its rate and the rate the site observed stayed under that ceiling, and whether it backed off after each overload signal. A verdict record [I-D.wright-verdict-record] is one way to do this. Judgements, such as whether a capacity estimate was reasonable, cannot be re-run, and the record shows what they rested on. Withdrawing an endorsement is a new record that supersedes the old one and names no one. A mechanical check MUST follow the distinction in Section 5.1.1: a reroute after a refusal confined to one path is not a failure to back off, provided the collector's total rate to the site stayed at or below its ceiling. 5.3.1. Key Lifecycle Conduct Statements are signed with keys that do not link a collector across sites or epochs, so the log alone does not show which statements belong to which collector. Only the Anchor needs to make that link. The collector generates all of its own keys; the Anchor issues endorsements, not signing keys. McKenna Expires 21 May 2027 [Page 9] Internet-Draft Attested Collection November 2026 1. Bootstrap. Before it holds a credential, the collector derives each signing key from a secret it holds, the keyed site reference, and the epoch, and logs its Conduct Statements at Tier 0. 2. Appraisal. When it applies for an endorsement, the collector proves to the Anchor that the key for each submitted statement was derived from its secret. The Anchor can then link those statements, but learns no site behind a keyed reference. 3. Issuance. If the appraisal is favorable, the Anchor issues an endorsement, from which the collector obtains credentials. 4. Operation. The collector now derives its signing keys from the credential secret, still per site and epoch. Each statement is tied to a credential period, but a reader of the log still cannot link one statement to another. The same derivation can produce the per-interval pseudonyms considered in Section 8. 5. Renewal. Before reissuing, the Anchor appraises the statements logged in the previous period, using the same proof as in step 2. A collector that loses its secret can no longer claim its earlier statements, and one that shares it lets someone else claim them. Anchors SHOULD require proof of possession of the secret at appraisal, and SHOULD NOT accept statements logged after a reported compromise of the secret. 6. Recipient Categories Some sites object less to collection itself than to who receives the data: for example, direct competitors, or automated pricing systems in the same product category. Competition law in several jurisdictions is also concerned with the exchange of commercially sensitive information among competitors, including through intermediaries and algorithms. This document takes no position on any law. Recipient categories are not a general tool for sites to choose who may read public data; used that way, they would bring back the discrimination R8 is meant to avoid. They SHOULD be limited to restrictions with a legal basis, such as competitors and pricing systems in the same product category, and SHOULD NOT be used to exclude journalists, researchers, archives, or consumer protection monitors. McKenna Expires 21 May 2027 [Page 10] Internet-Draft Attested Collection November 2026 A Tier 2 Anchor can verify a recipient's category and include it as a coarse attribute of the endorsement, for example "financial research" or "not a seller in category X". A site can then state, alongside its other preferences, which categories may not receive its data, and the collector proves membership in a permitted category without revealing the recipient. Attributes MUST be few and broad (R5). 7. Relationship to Other Work * [I-D.rescorla-anonymous-webbotauth] and [I-D.jms-mole-architecture] define how endorsements and credentials are issued and presented. This document describes evidence an Anchor can rely on, and does not change those mechanisms. * [I-D.ietf-webbotauth-httpsig-protocol] covers identified collectors. Conduct Statements apply to them as well. * [RFC9943] provides the transparency log. Conduct Statements are SCITT Signed Statements. * [RFC9334] provides the Verifier and Relying Party roles used in Tier 2. * PACE (draft-meunier-webbotauth-capacity-management, an editor's copy), proposed for the webbotauth working group, has sites moderate anonymous quota per tenant using Web Bot Auth and MoLE. Quota governs access; Conduct Statements record how a collector used it. The two differ on who sets the rate (see Section 5.1.1). * [I-D.gilda-audit-agent-audit-record] defines per-call audit records for agent actions. Conduct Statements are aggregate and collection-oriented, and need an unlinkable mode that per-call records with identified principals do not provide. * [I-D.illyes-webbotauth-cbcp] and [I-D.farzdusa-webbot-datacollection] describe the conduct that Conduct Statements would record. 8. Open Questions 1. How can a site's corroborating statement be matched to a collector's Conduct Statements without creating a linkable identifier? Candidate approaches include per-interval pseudonyms derived from the credential (see Section 5.3.1), and corroboration only in aggregate per Anchor. A pseudonym that both statements carry keeps appraisals checkable per collector and per interval; aggregate-only corroboration does not. McKenna Expires 21 May 2027 [Page 11] Internet-Draft Attested Collection November 2026 2. Should Transparency Services for Conduct Statements be public, permissioned, or operated by Anchors? 3. Which conduct fields are useful enough to standardize, and which should stay implementation-specific? 4. What minimum set of recipient categories is useful without becoming a fingerprint? 5. What should happen to a collector's endorsement when its logged conduct and a site's corroborating statements disagree? 9. Security Considerations Tier 0 statements are self-serving: a collector can log favorable statements that are false. The tiers let reliance grow only as independent evidence is added. Logs can leak metadata. Site references MUST be keyed so that the log does not reveal which sites a collector visits, and registration timing can itself correlate collectors with sites. Batching and delayed registration reduce this. An intermediary that signs witness statements is usually paid by the collector, and some intermediaries also sell help in getting past site blocks. A witness may therefore have an interest in attesting that a refusal was confined to one path, or in understating rates. A witness statement is only as credible as its signer, so sites choose which witnesses to accept, and the site's own corroboration, where available, carries more weight. An Anchor and a collector can collude. Logged evidence makes collusion detectable after the fact but does not prevent it. Sites choose which Anchors to accept. Attributes and timing patterns can fingerprint collectors despite anonymous credentials (R5). 10. Privacy Considerations The model is intended to protect the identity of collectors and recipients from sites, and the identity of sites from public logs. An Anchor at Tier 2 holds identity information and becomes a sensitive party: a record that a given recipient buys data collected by a given operator is exactly what the model hides from sites, and held in one place it is a target for breach, subpoena, or sale. Anchors SHOULD retain only what an appraisal needs and delete it after a stated period, SHOULD verify a recipient's category without McKenna Expires 21 May 2027 [Page 12] Internet-Draft Attested Collection November 2026 retaining the recipient's identity where the category can be established that way, and SHOULD publish their retention and disclosure policies. 11. IANA Considerations This document has no IANA actions. 12. References 12.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC9309] Koster, M., Illyes, G., Zeller, H., and L. Sassman, "Robots Exclusion Protocol", RFC 9309, DOI 10.17487/RFC9309, September 2022, . 12.2. Informative References [I-D.farzdusa-webbot-datacollection] Farzaneh, Levy, J., and S. McKenna, "Best Practices for Responsible Web Data Collection", Work in Progress, Internet-Draft, draft-farzdusa-webbot-datacollection-01, 23 September 2026, . [I-D.gilda-audit-agent-audit-record] Gilda, S., "An Audit Record Format for AI Agent Authorization Decisions", Work in Progress, Internet- Draft, draft-gilda-audit-agent-audit-record-03, 29 October 2026, . [I-D.ietf-webbotauth-httpsig-protocol] Meunier, T. and S. Major, "HTTP Message Signatures for automated traffic", Work in Progress, Internet-Draft, draft-ietf-webbotauth-httpsig-protocol-01, 29 October 2026, . McKenna Expires 21 May 2027 [Page 13] Internet-Draft Attested Collection November 2026 [I-D.illyes-webbotauth-cbcp] Illyes, G., Kühlewind, M., and K. Aj, "Crawler best practices", Work in Progress, Internet-Draft, draft- illyes-webbotauth-cbcp-01, 22 October 2026, . [I-D.jms-mole-architecture] Schlesinger, S., Jackson, D., and T. Meunier, "Moderation of unLinkable Endorsements (MOLE) Architecture", Work in Progress, Internet-Draft, draft-jms-mole-architecture-01, 2 November 2026, . [I-D.rescorla-anonymous-webbotauth] Rescorla, E. and R. Barnes, "Anonymous Bot Authentication: Authorization and Rate Limiting for Web Agents", Work in Progress, Internet-Draft, draft-rescorla-anonymous- webbotauth-01, 19 July 2026, . [I-D.wright-verdict-record] Wright, D., "Verdict Records: Results on Claims that Cite Receipts as Evidence", Work in Progress, Internet-Draft, draft-wright-verdict-record-02, 2 November 2026, . [RFC9334] Birkholz, H., Thaler, D., Richardson, M., Smith, N., and W. Pan, "Remote ATtestation procedureS (RATS) Architecture", RFC 9334, DOI 10.17487/RFC9334, January 2023, . [RFC9943] Birkholz, H., Delignat-Lavaud, A., Fournet, C., Deshpande, Y., and S. Lasker, "An Architecture for Trustworthy and Transparent Digital Supply Chains", RFC 9943, DOI 10.17487/RFC9943, June 2026, . Author's Address Sarah McKenna Independent Email: mckenna.sarah+ietf@gmail.com McKenna Expires 21 May 2027 [Page 14]