Bot Service Index (BSI): A Global Discovery Infrastructure for Autonomous Agent Services

1.1.1. 1.1.1. Motivation: A Concrete Origin

The Bot Service Index was not conceived in the abstract. It emerged from a concrete practical failure.¶

A buying bot was built for a private use case: monitoring online shops for a specific product and purchasing it automatically the moment it became available. This is a straightforward task for an autonomous agent — exactly the kind of task agents are well-suited for.¶

The bot failed, not because the task was technically complex, but because the internet's infrastructure is actively hostile to it:¶

HTML-only product pages. Product availability, price, and purchase state were encoded in HTML rendered for a human eye. No machine-readable API existed. The bot had to parse HTML — fragile, maintenance-intensive, and broken by every redesign.¶

Cloudflare Bot Management and equivalent shields. The majority of commercial web services now sit behind bot mitigation infrastructure. Cloudflare Bot Management, and equivalent products from Akamai, Imperva, and others, are deployed specifically to detect and block non-human request patterns. Repeated automated requests — even at modest frequency — trigger rate limiting, CAPTCHA challenges, or silent blocking. A buying bot that polls a product page to detect availability is treated identically to a malicious scraper or a DDoS participant.¶

CAPTCHA payment barriers. Even when product pages were reachable, payment flows required solving CAPTCHAs that explicitly excluded non-human actors. The purchasing step — the final, necessary action — was deliberately made inaccessible to the bot.¶

Proxy network pollution. To work around rate limits and bot detection, the bot required a rotating proxy network — different IP addresses on each request to disguise its automated origin. This is not a solution: it pollutes internet traffic with avoidable requests, raises the cost of operation, and contributes directly to the adversarial dynamic between bots and infrastructure operators. Every proxy request is a wasted roundtrip that a machine-readable API endpoint would have made unnecessary.¶

Polling as the only state-change mechanism. Because the bot had no way to subscribe to product availability events, it had to poll the product page continuously. This is architecturally wasteful: the bot consumes server resources and network bandwidth to repeatedly ask a question whose answer has not changed. If the service had provided a notification registration endpoint — a webhook, a server-sent event stream, or a WebSocket channel — the bot could subscribe once and receive a push notification when the product became available. No polling. No proxy network. No CAPTCHA exposure.¶

These are not edge cases. They are the standard experience for any autonomous agent attempting to consume a commercial internet service today. The buying bot illustrates why the Bot Service Index is necessary: not as an academic exercise, but as the infrastructure layer that makes autonomous agents functional participants in the commercial internet.¶

1.1.2. 1.2. The Discovery Problem

When an autonomous agent must fulfill a task that requires an external service, it faces a fundamental discovery problem: how does it find services that can fulfill its requirement?¶

Current approaches are inadequate:¶

• Hardcoded URLs: brittle, require human maintenance, do not adapt to new or changed services.¶

• LLM training data: stale, non-authoritative, not machine-verifiable.¶

• Human-curated lists: do not scale, not machine-navigable, lack structured metadata.¶

• Web search: returns HTML documents designed for humans, not structured service descriptions for agents.¶

What is needed is a machine-native equivalent of a search engine: a global, always-current, structured index of services that autonomous agents can query by capability, trust level, liveness, and other machine-relevant criteria.¶

1.1.3. 1.3. Lessons from Prior Art

The BSI is not the first attempt at a global service registry. Prior efforts must be understood explicitly so that their failure modes are not repeated.¶

UDDI (Universal Description, Discovery and Integration)
UDDI was a SOAP-era standard for a global service registry with the same conceptual goal as BSI, published as an OASIS Committee Draft in October 2004 (editors: Clement, Hately, von Riegen, Rogers). It failed for three reasons: (1) extreme complexity of the XML-based data model; (2) no automatic verification — all data was self-asserted with no crawling or validation; (3) no adoption incentive — there was no commercial model to sustain registration or discovery. BSI addresses all three directly: a simple JSON manifest, automated spider verification, and a commercial tier model.¶

robots.txt (Robots Exclusion Protocol)
Machine-readable, but concerned with exclusion — telling crawlers what not to access — not with discovery of capabilities. Per-domain only. Not a registry.¶

MCP (Model Context Protocol)
Defines tool and capability descriptions for LLM-based agents. Excellent for consumption once a server URL is known. Does not address the discovery problem: there is no index of MCP servers. BSI is complementary to MCP — it can index MCP servers as one supported spec type.¶

Well-Known URIs (RFC 8615)
Per-domain machine-readable metadata at /.well-known/. Useful for per-service metadata but requires the consumer to already know the domain. No cross-service search or global index.¶

DNS
DNS resolves names to addresses but carries no capability semantics. It is an architectural analogy for BSI's federation model, not a comparable system.¶

1.1.4. 1.4. Related IETF and W3C Work

As of April 2026, the number of Internet-Drafts working in adjacent areas of agent/bot infrastructure has grown significantly. None addresses the same problem as BSI. This section documents each and states the relationship explicitly.¶

draft-pioli-agent-discovery (ARDP — Agent Registration and Discovery Protocol)
Proposes a federated agent registration and discovery protocol. Agents self-register capabilities (MCP, A2A, HTTP, gRPC) with federated resolvers. Deliberately decentralised — no global registry mandate, no central query URL. Relationship to BSI: complementary. ARDP addresses agent-to-agent capability advertisement within a federation. BSI addresses global, cross-organisation service discovery from a neutral central index. The BSM spec.type vocabulary could reference ARDP-registered agents as a future extension.¶

draft-narajala-courtney-ansv2 (ANS v2 — Agent Name Service v2)
Full title: "Agent Name Service v2 (ANS): A Domain-Anchored Trust Layer for Autonomous AI Agent Identity" (Courtney, Narajala, Huang, Habler, Sheriff; last revised April 2026). Anchors autonomous agent identities to DNS domain names, with Registration Authority verification via ACME, dual certificates, and an append-only Transparency Log compliant with IETF SCITT standards. Defines three verification tiers: Bronze (PKI), Silver (PKI + DANE), and Gold (PKI + DANE + Transparency Log). Focused on agent identity and trust anchoring, not service capability discovery — there is no global index, no capability search, and no liveness model. Relationship to BSI: complementary. ANS v2 could serve as an identity and trust anchoring layer for service operators registered in the BSI, with BSI Organisation levels potentially mapping to ANS verification tiers in a future alignment. (Supersedes the expired draft-narajala-ans-00.)¶

draft-vandemeent-ains-discovery (AINS — AInternet Name Service)
Agent discovery and trust resolution via signed, append-only replication logs. Supports multi-registry federation. No central authority. No commercial sustainability model. Relationship to BSI: different philosophy. AINS prioritises decentralisation and cryptographic verifiability. BSI prioritises a single authoritative global index with a governed trust model. The two approaches represent a genuine design tension that this document's Open Questions section (Section 12, item 8) invites community input on.¶

draft-aiendpoint-ai-discovery (AI Discovery Endpoint)
Defines /.well-known/ai as a per-host machine-readable capability document, analogous to robots.txt for agent consumers. Per-domain only; not a global index. Relationship to BSI: directly complementary. The BSI Spider SHOULD read /.well-known/ai when present on a registered service's domain as an additional source of capability metadata. Service Owners publishing /.well-known/ai documents reduce the Spider's verification burden.¶

draft-batum-aidre (AIDRE — AI Discovery and Retrieval Endpoint)
Defines /.well-known/ai-discovery as a per-origin discovery document exposing collections, metadata, and optional vector-native query interfaces. Decentralised by design — each origin publishes its own endpoint; no central authority aggregates them. Relationship to BSI: complementary. AIDRE and the AI Discovery Endpoint draft (above) both address per-origin machine-readable capability publication. The BSI Spider SHOULD treat /.well-known/ai-discovery as an additional metadata source when present on a registered service's domain. BSI provides the global aggregation and trust verification layer that per-origin endpoints cannot provide alone.¶

draft-cui-ai-agent-discovery-invocation (AI Agent Discovery and Invocation Protocol)
(Cui, Chao, Du — Tsinghua University / Zhongguancun Laboratory, February 2026.) Specifies a metadata format for agent capabilities and a registry-based discovery and invocation mechanism. Explicitly permits multiple coexisting registries; no global authority is defined. No trust model, no commercial sustainability layer, no liveness verification. Relationship to BSI: partially overlapping problem space, different architectural philosophy. Where this draft describes how agents register with and query a registry, BSI specifies what a globally authoritative, commercially sustainable, trust-verified registry looks like and how it is governed. The two are not mutually exclusive; a BSI-registered service could also self-register with local registries described by this protocol.¶

draft-am-layered-ai-discovery-architecture (A Layered Approach to AI Discovery)
(Moussa, Akhavain — Huawei Canada, March 2026.) Proposes a two-layer architecture separating the discovery transport mechanism from the metadata format for the object being discovered. Delegates security and trust to other groups. No global registry, no funding model. Relationship to BSI: architectural framing only. BSI's HATEOAS model is consistent with this layered framing — the Index API provides the transport layer, the BSM provides the discovery object format.¶

draft-hood-agtp-discovery (AGTP Agent Discovery and Name Service)
(Hood — Nomotic, March 2026.) A governance-focused, zone-constrained discovery broker tied to a specific agent protocol (AGTP). Returns ranked agent manifests within trust-gated zones. Multiple independent ANS instances per organisation; optional federation between trusted peers. Relationship to BSI: different scope. AGTP addresses intra-organisation, permission-aware agent orchestration. BSI addresses cross-organisation, open, global service discovery.¶

draft-mozley-aidiscovery (AI Agent Discovery Problem Statement)
(Mozley, Williams — Infoblox; Sarikaya; Schott — Deutsche Telekom, April 2026.) A problem statement that explicitly argues against centralised discovery directories, citing organisational autonomy requirements. Proposes distributed discovery via well-known entry points per organisation. Relationship to BSI: this draft articulates the strongest counter-position to BSI's architecture. The argument from autonomy is valid for intra- organisation agent discovery but does not address the cross-organisation case: an autonomous agent that needs to discover payment, logistics, or data services provided by unknown third parties cannot rely on a decentralised model without a trust anchor. BSI is designed precisely for that cross-boundary, zero-prior-relationship discovery scenario.¶

draft-mozleywilliams-dnsop-dnsaid (DNS for AI Discovery)
(Mozley, Williams — Infoblox; Sarikaya; Schott — Deutsche Telekom, March 2026.) Proposes DNS-AID: using SVCB records under a leaf zone convention (e.g. _agents.example.com) to publish agent service endpoints. Leverages existing DNS infrastructure with DNSSEC for integrity. Supports four discovery models: direct, index-based, multi-domain, and registry-based. Relationship to BSI: complementary at the infrastructure layer. DNS provides routing and namespace anchoring; BSI provides capability search, trust metadata, and liveness verification. A consuming agent could use DNS-AID to resolve a known domain's agent endpoints and use BSI to discover unknown providers by capability.¶

draft-meunier-webbotauth-registry (Registry and Signature Agent card for Web bot auth)
Authored by Maxime Guerreiro (Cloudflare), Ulas Kirazci (Amazon), and Thibault Meunier (Cloudflare). Defines a JSON-based "Signature Agent Card" format for entities originating or forwarding signed HTTP requests to advertise metadata about themselves, and establishes an IANA registry for Signature Agent Card parameters. Focused on bot authentication — how a bot proves who it is to a service — not on service discovery. Related to the active webbotauth IETF Working Group (chaired by David Schinazi and Rifaat Shekh-Yusef, Area Director Mike Bishop), which is currently the most active formal IETF effort in the bot/agent infrastructure space. Relationship to BSI: orthogonal but complementary. webbotauth addresses bot consumer identity; BSI addresses service provider discovery. A future version of BSI may reference webbotauth identity cards as a mechanism for consuming agents to authenticate to the Index API.¶

W3C AI Agent Protocol Community Group
Proposed May 2025, targeting agent interoperability protocols. 216 participants as of April 2026. Pre-specification as of this writing. Relationship to BSI: BSI will monitor this group's outputs and align the BSM capability taxonomy with any vocabulary standardised by the W3C CG where applicable.¶

Positioning
The agent/bot infrastructure space has grown rapidly in early 2026, with over a dozen active Internet-Drafts addressing adjacent problems. The dominant architectural tendency across these drafts is decentralised and federated — distributed registries, DNS-anchored discovery, per-origin well-known endpoints, and zone-based governance.¶

BSI takes a deliberate counter-position for a specific use case: global, cross-organisation, zero-prior-relationship service discovery. This use case cannot be solved by decentralised architectures alone, because decentralisation requires that the discovering agent already knows where to look. BSI provides the answer to "where to look" — a single stable entry point, commercially operated by a neutral foundation, with verified trust metadata and structured capability search.¶

BSI remains the first and only proposed global, bot-native, commercially sustainable service discovery index with a structured multi-dimensional trust model, where the autonomous agent is the primary consumer. The combination of: (1) a single globally queryable central index, (2) capability-based search, (3) a three-dimensional verifiable trust model, and (4) a commercial sustainability layer — does not appear in any other current draft or standard.¶

A.1.1. 14.1. Normative References

• RFC 2119 — Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.¶

• RFC 8615 — Nottingham, M., "Well-Known Uniform Resource Identifiers (URIs)", RFC 8615, May 2019.¶

• RFC 8446 — Rescorla, E., "The Transport Layer Security (TLS) Protocol Version 1.3", RFC 8446, August 2018.¶

• RFC 9110 — Fielding, R., Nottingham, M., Reschke, J. (Eds.), "HTTP Semantics", RFC 9110, June 2022.¶

A.1.2. 14.2. Informative References

• OpenAPI Specification 3.1 — OpenAPI Initiative, https://spec.openapis.org/oas/v3.1.0¶

• Model Context Protocol — Anthropic, https://modelcontextprotocol.io¶

• AsyncAPI Specification 3.0 — AsyncAPI Initiative, https://www.asyncapi.com/docs/reference/specification/v3.0.0¶

• RFC 8949 — Bormann, C., Hoffman, P., "Concise Binary Object Representation (CBOR)", RFC 8949, December 2020.¶

• Robots Exclusion Protocol — Koster, M., 1994. https://www.robotstxt.org/¶

• draft-cui-ai-agent-discovery-invocation-01 — Cui, Y. (Tsinghua University), Chao, Y., Du, C. (Zhongguancun Laboratory), "AI Agent Discovery and Invocation Protocol", IETF Individual Submission, February 2026. Expires August 2026. https://datatracker.ietf.org/doc/draft-cui-ai-agent-discovery-invocation/¶

• draft-am-layered-ai-discovery-architecture-00 — Moussa, H., Akhavain, A. (Huawei Canada), "A Layered Approach to AI discovery", IETF Individual Submission, March 2026. Expires September 2026. https://datatracker.ietf.org/doc/draft-am-layered-ai-discovery-architecture/¶

• draft-hood-agtp-discovery-00 — Hood, C. (Nomotic, Inc.), "AGTP Agent Discovery and Name Service", IETF Individual Submission, March¶

  1. Expires September 2026. https://datatracker.ietf.org/doc/draft-hood-agtp-discovery/¶

• draft-mozleywilliams-dnsop-dnsaid-01 — Mozley, J., Williams, N. (Infoblox), Sarikaya, B. (Unaffiliated), Schott, R. (Deutsche Telekom), "DNS for AI Discovery", IETF Individual Submission, March 2026. Expires September 2026. https://datatracker.ietf.org/doc/draft-mozleywilliams-dnsop-dnsaid/¶

• draft-batum-aidre-00 — Batum, F. (Istanbul), "AI Discovery and Retrieval Endpoint (AIDRE)", IETF Individual Submission, April 2026. Expires October 2026. https://datatracker.ietf.org/doc/draft-batum-aidre/¶

• draft-mozley-aidiscovery-01 — Mozley, J., Williams, N. (Infoblox), Sarikaya, B. (Unaffiliated), Schott, R. (Deutsche Telekom), "AI Agent Discovery (AID) Problem Statement", IETF Individual Submission, April¶

  1. Expires October 2026. https://datatracker.ietf.org/doc/draft-mozley-aidiscovery/¶

• draft-pioli-agent-discovery-01 — Pioli, R. (Independent), "Agent Registration and Discovery Protocol (ARDP)", IETF Individual Submission, February 2026. Expires August 2026. https://datatracker.ietf.org/doc/draft-pioli-agent-discovery/¶

• draft-narajala-courtney-ansv2-01 — Courtney, S., Narajala, V.S., Huang, K., Habler, I., Sheriff, A., "Agent Name Service v2 (ANS): A Domain-Anchored Trust Layer for Autonomous AI Agent Identity", IETF Individual Submission, April 2026. Expires October 2026. Supersedes draft-narajala-ans-00. https://datatracker.ietf.org/doc/draft-narajala-courtney-ansv2/¶

• draft-vandemeent-ains-discovery-01 — van de Meent, J., Root AI (Humotica), "AINS: AInternet Name Service - Agent Discovery and Trust Resolution Protocol", IETF Individual Submission, March 2026. Expires September 2026. https://datatracker.ietf.org/doc/draft-vandemeent-ains-discovery/¶

• draft-aiendpoint-ai-discovery-00 — Choi, Y. (AIEndpoint), "The AI Discovery Endpoint: A Structured Mechanism for AI Agent Service Discovery and Capability Exposure", IETF Individual Submission, March 2026. Expires September 2026. https://datatracker.ietf.org/doc/draft-aiendpoint-ai-discovery/¶

• draft-meunier-webbotauth-registry-01 — Guerreiro, M. (Cloudflare), Kirazci, U. (Amazon), Meunier, T. (Cloudflare), "Registry and Signature Agent card for Web bot auth", IETF Individual Submission, October 2025. Expired April 2026; renewal expected. https://datatracker.ietf.org/doc/draft-meunier-webbotauth-registry/¶

• webbotauth IETF Working Group — Chairs: Schinazi, D., Shekh-Yusef, R. AD: Bishop, M. Active WG. https://datatracker.ietf.org/wg/webbotauth/¶

• W3C AI Agent Protocol Community Group — Chairs: Chang, G., Xu, S. Established May 8, 2025. 216 participants as of April 2026. https://www.w3.org/community/agentprotocol/¶

• UDDI Version 3.0.2 — Clement, L., Hately, A., von Riegen, C., Rogers, T. OASIS Committee Draft, October 19, 2004. (Historical reference; see Section 1.3 for analysis of failure modes.) https://www.oasis-open.org/committees/uddi-spec/doc/spec/v3/uddi-v3.0.2-20041019.htm¶