Bot Service Index (BSI): A Global Discovery Infrastructure for Autonomous Agent Services

1. Introduction

1.1. The Bot Ecosystem Gap The internet's foundational infrastructure — HTTP, HTML, DNS, and search engines — was designed with human actors as the primary consumers. Web pages render visual layouts for human eyes. CAPTCHA systems explicitly discriminate against non-human access. Discovery mechanisms such as search engines index content for human-readable navigation. Autonomous agents — software programs that independently execute tasks, consume APIs, and interact with external services without per-action human instruction — are not recognized as legitimate, first-class internet participants in this architecture. They are systematically treated as threats to be filtered, blocked, or rate-limited. This situation is changing. The rapid growth of large language model-based agents, robotic process automation, and programmatic service consumers means that non-human actors now represent a significant and growing proportion of internet traffic. As this proportion increases, internet service providers will increasingly need to serve autonomous agents as a recognized user class alongside humans. The Bot Service Index is premised on this trajectory: bots are becoming first-class internet participants, and the infrastructure to support them — starting with service discovery — does not yet exist.

1.1.1. Motivation: A Concrete Origin The Bot Service Index was not conceived in the abstract. It emerged from a concrete practical failure. A buying bot was built for a private use case: monitoring online shops for a specific product and purchasing it automatically the moment it became available. This is a straightforward task for an autonomous agent — exactly the kind of task agents are well-suited for. The bot failed, not because the task was technically complex, but because the internet's infrastructure is actively hostile to it: HTML-only product pages. Product availability, price, and purchase state were encoded in HTML rendered for a human eye. No machine-readable API existed. The bot had to parse HTML — fragile, maintenance-intensive, and broken by every redesign. Cloudflare Bot Management and equivalent shields. The majority of commercial web services now sit behind bot mitigation infrastructure. Cloudflare Bot Management, and equivalent products from Akamai, Imperva, and others, are deployed specifically to detect and block non-human request patterns. Repeated automated requests — even at modest frequency — trigger rate limiting, CAPTCHA challenges, or silent blocking. A buying bot that polls a product page to detect availability is treated identically to a malicious scraper or a DDoS participant. CAPTCHA payment barriers. Even when product pages were reachable, payment flows required solving CAPTCHAs that explicitly excluded non-human actors. The purchasing step — the final, necessary action — was deliberately made inaccessible to the bot. Proxy network pollution. To work around rate limits and bot detection, the bot required a rotating proxy network — different IP addresses on each request to disguise its automated origin. This is not a solution: it pollutes internet traffic with avoidable requests, raises the cost of operation, and contributes directly to the adversarial dynamic between bots and infrastructure operators. Every proxy request is a wasted roundtrip that a machine-readable API endpoint would have made unnecessary. Polling as the only state-change mechanism. Because the bot had no way to subscribe to product availability events, it had to poll the product page continuously. This is architecturally wasteful: the bot consumes server resources and network bandwidth to repeatedly ask a question whose answer has not changed. If the service had provided a notification registration endpoint — a webhook, a server-sent event stream, or a WebSocket channel — the bot could subscribe once and receive a push notification when the product became available. No polling. No proxy network. No CAPTCHA exposure. These are not edge cases. They are the standard experience for any autonomous agent attempting to consume a commercial internet service today. The buying bot illustrates why the Bot Service Index is necessary: not as an academic exercise, but as the infrastructure layer that makes autonomous agents functional participants in the commercial internet.

1.2. The Discovery Problem When an autonomous agent must fulfill a task that requires an external service, it faces a fundamental discovery problem: how does it find services that can fulfill its requirement? Current approaches are inadequate:

Hardcoded URLs: brittle, require human maintenance, do not adapt to new or changed services.
LLM training data: stale, non-authoritative, not machine-verifiable.
Human-curated lists: do not scale, not machine-navigable, lack structured metadata.
Web search: returns HTML documents designed for humans, not structured service descriptions for agents.

What is needed is a machine-native equivalent of a search engine: a global, always-current, structured index of services that autonomous agents can query by capability, trust level, liveness, and other machine-relevant criteria.

1.3. Lessons from Prior Art The BSI is not the first attempt at a global service registry. Prior efforts must be understood explicitly so that their failure modes are not repeated. UDDI (Universal Description, Discovery and Integration)
UDDI was a SOAP-era standard for a global service registry with the same conceptual goal as BSI, published as an OASIS Committee Draft in October 2004 (editors: Clement, Hately, von Riegen, Rogers). It failed for three reasons: (1) extreme complexity of the XML-based data model; (2) no automatic verification — all data was self-asserted with no crawling or validation; (3) no adoption incentive — there was no commercial model to sustain registration or discovery. BSI addresses all three directly: a simple JSON manifest, automated spider verification, and a commercial tier model. robots.txt (Robots Exclusion Protocol)
Machine-readable, but concerned with exclusion — telling crawlers what not to access — not with discovery of capabilities. Per-domain only. Not a registry. MCP (Model Context Protocol)
Defines tool and capability descriptions for LLM-based agents. Excellent for consumption once a server URL is known. Does not address the discovery problem: there is no index of MCP servers. BSI is complementary to MCP — it can index MCP servers as one supported spec type. Well-Known URIs (RFC 8615)
Per-domain machine-readable metadata at /.well-known/. Useful for per-service metadata but requires the consumer to already know the domain. No cross-service search or global index. DNS
DNS resolves names to addresses but carries no capability semantics. It is an architectural analogy for BSI's federation model, not a comparable system.

1.4. Related IETF and W3C Work As of April 2026, the number of Internet-Drafts working in adjacent areas of agent/bot infrastructure has grown significantly. None addresses the same problem as BSI. This section documents each and states the relationship explicitly. draft-pioli-agent-discovery (ARDP — Agent Registration and Discovery Protocol)
Proposes a federated agent registration and discovery protocol. Agents self-register capabilities (MCP, A2A, HTTP, gRPC) with federated resolvers. Deliberately decentralised — no global registry mandate, no central query URL. Relationship to BSI: complementary. ARDP addresses agent-to-agent capability advertisement within a federation. BSI addresses global, cross-organisation service discovery from a neutral central index. The BSM spec.type vocabulary could reference ARDP-registered agents as a future extension. draft-narajala-courtney-ansv2 (ANS v2 — Agent Name Service v2)
Full title: "Agent Name Service v2 (ANS): A Domain-Anchored Trust Layer for Autonomous AI Agent Identity" (Courtney, Narajala, Huang, Habler, Sheriff; last revised April 2026). Anchors autonomous agent identities to DNS domain names, with Registration Authority verification via ACME, dual certificates, and an append-only Transparency Log compliant with IETF SCITT standards. Defines three verification tiers: Bronze (PKI), Silver (PKI + DANE), and Gold (PKI + DANE + Transparency Log). Focused on agent identity and trust anchoring, not service capability discovery — there is no global index, no capability search, and no liveness model. Relationship to BSI: complementary. ANS v2 could serve as an identity and trust anchoring layer for service operators registered in the BSI, with BSI Organisation levels potentially mapping to ANS verification tiers in a future alignment. (Supersedes the expired draft-narajala-ans-00.) draft-vandemeent-ains-discovery (AINS — AInternet Name Service)
Agent discovery and trust resolution via signed, append-only replication logs. Supports multi-registry federation. No central authority. No commercial sustainability model. Relationship to BSI: different philosophy. AINS prioritises decentralisation and cryptographic verifiability. BSI prioritises a single authoritative global index with a governed trust model. The two approaches represent a genuine design tension that this document's Open Questions section (Section 12, item 8) invites community input on. draft-aiendpoint-ai-discovery (AI Discovery Endpoint)
Defines /.well-known/ai as a per-host machine-readable capability document, analogous to robots.txt for agent consumers. Per-domain only; not a global index. Relationship to BSI: directly complementary. The BSI Spider SHOULD read /.well-known/ai when present on a registered service's domain as an additional source of capability metadata. Service Owners publishing /.well-known/ai documents reduce the Spider's verification burden. draft-batum-aidre (AIDRE — AI Discovery and Retrieval Endpoint)
Defines /.well-known/ai-discovery as a per-origin discovery document exposing collections, metadata, and optional vector-native query interfaces. Decentralised by design — each origin publishes its own endpoint; no central authority aggregates them. Relationship to BSI: complementary. AIDRE and the AI Discovery Endpoint draft (above) both address per-origin machine-readable capability publication. The BSI Spider SHOULD treat /.well-known/ai-discovery as an additional metadata source when present on a registered service's domain. BSI provides the global aggregation and trust verification layer that per-origin endpoints cannot provide alone. draft-cui-ai-agent-discovery-invocation (AI Agent Discovery and Invocation Protocol)
(Cui, Chao, Du — Tsinghua University / Zhongguancun Laboratory, February 2026.) Specifies a metadata format for agent capabilities and a registry-based discovery and invocation mechanism. Explicitly permits multiple coexisting registries; no global authority is defined. No trust model, no commercial sustainability layer, no liveness verification. Relationship to BSI: partially overlapping problem space, different architectural philosophy. Where this draft describes how agents register with and query a registry, BSI specifies what a globally authoritative, commercially sustainable, trust-verified registry looks like and how it is governed. The two are not mutually exclusive; a BSI-registered service could also self-register with local registries described by this protocol. draft-am-layered-ai-discovery-architecture (A Layered Approach to AI Discovery)
(Moussa, Akhavain — Huawei Canada, March 2026.) Proposes a two-layer architecture separating the discovery transport mechanism from the metadata format for the object being discovered. Delegates security and trust to other groups. No global registry, no funding model. Relationship to BSI: architectural framing only. BSI's HATEOAS model is consistent with this layered framing — the Index API provides the transport layer, the BSM provides the discovery object format. draft-hood-agtp-discovery (AGTP Agent Discovery and Name Service)
(Hood — Nomotic, March 2026.) A governance-focused, zone-constrained discovery broker tied to a specific agent protocol (AGTP). Returns ranked agent manifests within trust-gated zones. Multiple independent ANS instances per organisation; optional federation between trusted peers. Relationship to BSI: different scope. AGTP addresses intra-organisation, permission-aware agent orchestration. BSI addresses cross-organisation, open, global service discovery. draft-mozley-aidiscovery (AI Agent Discovery Problem Statement)
(Mozley, Williams — Infoblox; Sarikaya; Schott — Deutsche Telekom, April 2026.) A problem statement that explicitly argues against centralised discovery directories, citing organisational autonomy requirements. Proposes distributed discovery via well-known entry points per organisation. Relationship to BSI: this draft articulates the strongest counter-position to BSI's architecture. The argument from autonomy is valid for intra- organisation agent discovery but does not address the cross-organisation case: an autonomous agent that needs to discover payment, logistics, or data services provided by unknown third parties cannot rely on a decentralised model without a trust anchor. BSI is designed precisely for that cross-boundary, zero-prior-relationship discovery scenario. draft-mozleywilliams-dnsop-dnsaid (DNS for AI Discovery)
(Mozley, Williams — Infoblox; Sarikaya; Schott — Deutsche Telekom, March 2026.) Proposes DNS-AID: using SVCB records under a leaf zone convention (e.g. _agents.example.com) to publish agent service endpoints. Leverages existing DNS infrastructure with DNSSEC for integrity. Supports four discovery models: direct, index-based, multi-domain, and registry-based. Relationship to BSI: complementary at the infrastructure layer. DNS provides routing and namespace anchoring; BSI provides capability search, trust metadata, and liveness verification. A consuming agent could use DNS-AID to resolve a known domain's agent endpoints and use BSI to discover unknown providers by capability. draft-meunier-webbotauth-registry (Registry and Signature Agent card for Web bot auth)
Authored by Maxime Guerreiro (Cloudflare), Ulas Kirazci (Amazon), and Thibault Meunier (Cloudflare). Defines a JSON-based "Signature Agent Card" format for entities originating or forwarding signed HTTP requests to advertise metadata about themselves, and establishes an IANA registry for Signature Agent Card parameters. Focused on bot authentication — how a bot proves who it is to a service — not on service discovery. Related to the active webbotauth IETF Working Group (chaired by David Schinazi and Rifaat Shekh-Yusef, Area Director Mike Bishop), which is currently the most active formal IETF effort in the bot/agent infrastructure space. Relationship to BSI: orthogonal but complementary. webbotauth addresses bot consumer identity; BSI addresses service provider discovery. A future version of BSI may reference webbotauth identity cards as a mechanism for consuming agents to authenticate to the Index API. W3C AI Agent Protocol Community Group
Proposed May 2025, targeting agent interoperability protocols. 216 participants as of April 2026. Pre-specification as of this writing. Relationship to BSI: BSI will monitor this group's outputs and align the BSM capability taxonomy with any vocabulary standardised by the W3C CG where applicable. Positioning
The agent/bot infrastructure space has grown rapidly in early 2026, with over a dozen active Internet-Drafts addressing adjacent problems. The dominant architectural tendency across these drafts is decentralised and federated — distributed registries, DNS-anchored discovery, per-origin well-known endpoints, and zone-based governance. BSI takes a deliberate counter-position for a specific use case: global, cross-organisation, zero-prior-relationship service discovery. This use case cannot be solved by decentralised architectures alone, because decentralisation requires that the discovering agent already knows where to look. BSI provides the answer to "where to look" — a single stable entry point, commercially operated by a neutral foundation, with verified trust metadata and structured capability search. BSI remains the first and only proposed global, bot-native, commercially sustainable service discovery index with a structured multi-dimensional trust model, where the autonomous agent is the primary consumer. The combination of: (1) a single globally queryable central index, (2) capability-based search, (3) a three-dimensional verifiable trust model, and (4) a commercial sustainability layer — does not appear in any other current draft or standard.