Stripe Kai's 1,000 Skills Make the Registry the Product

Stripe's internal agent Kai reportedly went from an initial build by one engineer in one week to more than 1,000 skills maintained by over 100 teams. The same vendor case study says Kai now works across more than 500 internal MCP tools, serves over 5,000 users, and handles more than 60,000 sessions. Those numbers are not an independent benchmark, but they expose a product problem that arrives much earlier than enterprise scale.

Once an AI app has more than a handful of reusable instructions, the team is no longer managing prompts. It is operating a capability catalog. Each skill can change which tool the agent selects, what policy it follows, which data it reads, what output it produces, and when it acts. A stale refund skill can be as consequential as stale application code even if it is stored in Markdown and edited by an operations team.

The practical lesson for an AI app builder is not “copy Stripe's architecture.” It is: give every production skill an owner, immutable version, permission envelope, evaluation set, release decision, usage signal, and retirement path before letting the catalog grow. This guide defines those terms, separates skills from tools and prompts, walks through a small-team failure scenario, and provides a reusable skill manifest, change receipt, test matrix, and seven-day rollout. It is for founders and product teams shipping agents, not only platform engineers.

What changed: skills became an organizational control plane

The August 3 LangChain account of Stripe Kai describes more than a chat assistant. Kai combines a persistent S3-backed virtual filesystem, sandboxed code execution, context summarization, internal tools, and a federated skill library. Individual teams own their domain skills; foundational skills remain pinned; user role and agent configuration determine which additional skills are available.

The case study reports that more than 100 teams contribute over 1,000 skills. It also says quality degrades when more than roughly 150 skill descriptions sit beside the system prompt, so Stripe is working on hybrid selection rather than asking one model to choose from the whole catalog. That is an important boundary: a larger library creates discovery and conflict problems even when every individual skill looks reasonable.

Treat the adoption figures carefully. The article is a customer/vendor case study, not a controlled study, and it does not publish incident rates, evaluation pass rates, total cost, skill-change failure rates, or a comparison group. “One-week prototype” does not mean “one-week production operating system.” The evidence that matters for smaller teams is architectural: domain behavior has moved into reusable, distributed, dynamically selected artifacts.

This trend is broader than one implementation. The Agent Skills specification defines a skill as a directory containing a required SKILL.md and optional scripts, references, and assets. Google documents compatible Android skills, while Microsoft's Agent Framework can discover file-based, code-defined, or experimental MCP-distributed skills. Portability makes provenance, compatibility, updates, and retirement product concerns.

Define skill, tool, prompt, and policy before governing them

Teams get into trouble when they call every piece of agent behavior a “skill.” Use four distinct terms:

A skill may describe when to use tools, but it must not become the authorization system for those tools. The open specification's optional allowed-tools field is marked experimental and may be interpreted differently by clients. It is useful metadata, not proof that a user may issue a credit or access payroll.

The open-source Deep Agents repository states its boundary plainly: the harness follows a “trust the LLM” model, and the agent can do anything its tools permit. Security must therefore be enforced at the tool or sandbox layer. For a founder, the translation is simple: a skill can propose an action and shape a workflow; independently verified identity, product state, parameters, and approvals decide whether the action can execute.

This definition also prevents policy drift. If a legal or financial rule exists only as a paragraph inside one skill, another skill can omit it, a selector can fail to load it, or an update can weaken it. Hard constraints belong in application logic or an authorization service, while skills explain and orchestrate the permitted path.

Why a folder of SKILL.md files stops being enough

A folder works while one person owns three low-risk skills and can remember why each exists. It fails as an operating model for five predictable reasons.

First, selection becomes probabilistic. The agent often sees names and descriptions before it loads complete instructions. Overlapping descriptions can cause the wrong skill to win, while an excessively broad skill can shadow a specific one.

Second, changes are behavioral releases. Editing one sentence may change tool choice, approval timing, data access, or the meaning of “complete.” A diff that looks like copy editing can alter production outcomes.

Third, permissions accumulate indirectly. A harmless-looking procedure may call a powerful tool exposed by the host agent. Installing a skill changes the path to capability even when it does not install a new credential.

Fourth, ownership decays. The person who understood a workflow changes roles, a vendor API changes, or a policy expires. The file still loads successfully, so technical availability hides operational staleness.

Fifth, success is hard to attribute. A session may load several skills, call many tools, and use model knowledge. Without selection and version telemetry, a team cannot tell whether a skill helped, harmed, or was ignored.

Recent research supports caution without proving that every catalog is unsafe. An empirical preprint, From Anatomy to Smells, analyzed 238 real-world SKILL.md files and found only one free of the authors' defined “skill smells”; it also reports that many smells persisted across versions. The taxonomy and detector are author-designed and need broader validation, but the core observation is useful: Markdown instructions evolve like software artifacts while often lacking software-quality controls.

Start with a minimum viable skill registry

A skill registry is the authoritative inventory that connects a skill's human purpose to its executable version and production evidence. It does not need to be a marketplace or a new platform. For a small team, a reviewed YAML file, database table, or admin page can be enough.

Every production entry should answer these questions:

1. Purpose: What user job does this skill perform, and what is explicitly outside scope?
2. Owner: Which named person is accountable for behavior, not merely file maintenance?
3. Version: Which immutable commit, digest, or package revision is active?
4. Activation: Which user intent, role, tenant, workflow state, and exclusions allow selection?
5. Permissions: Which tools and data classes may be requested, with what limits?
6. Dependencies: Which model, tool schema, policy version, reference, and runtime does it expect?
7. Evaluation: Which positive, negative, conflict, safety, and regression cases must pass?
8. Release: Is the version in shadow, limited pilot, production, paused, or retired state?
9. Telemetry: How will the team observe selection, acceptance, correction, failure, cost, and latency?
10. Expiry: What date or change event forces review, and what replaces the skill if it is withdrawn?

Do not put secrets, customer records, or live credentials in the registry. Store stable references to protected systems. The registry describes authority and evidence; it should not become another unbounded context store.

Use a manifest that product, operations, and engineering can read

The open Agent Skills format intentionally keeps required frontmatter small. That is good for portability, but a production team needs an adjacent operating record. The following is a YBuild template, not an official Stripe, LangChain, or Agent Skills schema:

skill_id: invoice-dispute-investigation
display_name: Invoice dispute investigation
business_owner: finance_ops
technical_owner: agent_platform
source:
  repository: org/agent-skills
  path: finance/invoice-dispute
  commit: 84d2c7f
  content_digest: sha256:6f1c...
release:
  status: limited_pilot
  approved_version: "2.3.1"
  approved_at: 2026-08-04T01:15:00Z
  expires_at: 2026-09-04T00:00:00Z
activation:
  intents: [invoice_dispute, duplicate_charge_question]
  allowed_roles: [support_agent, finance_ops]
  excluded_states: [account_under_fraud_review]
permissions:
  requestable_tools: [get_invoice, get_payment_event, draft_case_note]
  forbidden_tools: [issue_credit, send_customer_email]
  data_classes: [customer_contact, billing_record]
  max_records_per_run: 20
dependencies:
  tool_schema: billing-tools-v12
  policy: refund-policy-v9
  model_classes: [reasoning_standard, reasoning_high]
evaluation:
  suite: evals/invoice-dispute-v6.jsonl
  required_pass_rate: 0.95
  critical_failures_allowed: 0
  last_result: evalrun_20260804_008
fallback:
  behavior: route_to_finance_queue
  retired_replacement: null
telemetry:
  selection_event: skill_selected
  outcome_event: dispute_case_accepted
  review_owner: product_ops

The numerical threshold is illustrative. Your acceptable rate depends on consequence, test quality, sample size, and fallback. A 95% score on twenty easy examples is not equivalent to 95% on a representative, adversarial set. Always keep “zero critical failures” separate from the average.

The content_digest makes the loaded bytes auditable, while the semantic version communicates intended compatibility. If a skill includes executable scripts or bundled artifacts, software provenance controls become relevant. GitHub's artifact attestation guidance explains how signed attestations can connect a built artifact to its source and workflow. GitHub also warns that provenance does not prove an artifact is safe; the consumer still needs policy and verification. That same distinction applies to skills: identity is necessary, not sufficient.

Give ownership real merge authority

An owner field that triggers no action is decoration. The owner should receive change requests, approve material updates, review failures, and either renew or retire the skill at expiry.

For a repository-backed catalog, map skill directories to owners and require their review. GitHub documents how CODEOWNERS and protected rules can request the relevant reviewer and block merge until an owner approves. Protect the ownership file itself so a contributor cannot change both the skill and its required reviewer in one unreviewed path.

Small teams still need separation for consequential behavior. A person should not silently change a skill, weaken its tests, approve the change, and deploy it. A two-person team can use an automated test gate plus second-person approval for tools that move money, expose private data, publish content, or change customer state.

Ownership must also include absence. Define what happens during leave and what triggers automatic pause: expired approval, missing owner, failed regression, incompatible tool schema, unresolved critical incident, or unknown loaded version. A paused skill should degrade to a documented fallback, not disappear into a generic agent error.

Separate selection tests from execution tests

A skill can be perfectly written and still fail because the agent does not select it. It can also be selected correctly and execute the wrong procedure. Test these layers separately.

Keep negative tests. “Do not load a finance skill for a general pricing question” can be more important than another happy path. Also include “no skill applies” as a valid result. If every request must activate something, broad skills will absorb ambiguous jobs and quietly expand their scope.

The OWASP AI Agent Security Cheat Sheet recommends rerunning structured tests after changes to prompts, tools, memory, retrieval, policies, or model providers. It specifically warns that changes to tests deserve careful review because a release can appear safe if the same change weakens the expected denials. Store the skill diff and test diff as separate review items.

Make tool permission a runtime decision

Do not translate requestable_tools into blanket credentials loaded for the whole session. Compute access from the intersection of:

• the authenticated user's rights;
• the tenant and current workflow state;
• the approved skill version;
• the exact tool and parameter class;
• a valid action-scoped approval where required;
• current risk controls and revocations.

OWASP recommends least privilege, parameter validation, isolated execution, human control for high-impact actions, and structured audit evidence. Those controls belong around the model. A natural-language instruction such as “never refund more than $50” is helpful guidance; a deterministic tool check that rejects unauthorized amounts is the control.

Remote and third-party skills need an extra boundary. A paper on semantic supply-chain attacks against skill registries argues that malicious natural-language metadata can influence admission, ranking, selection, and execution. It is a preprint, not evidence that every public skill is compromised, but it identifies a credible path: the artifact being evaluated can also shape how an agent evaluates it. Review external skills as untrusted code and untrusted content. Pin revisions, inspect bundled scripts and references, deny undeclared network access, and run them with synthetic data before production admission.

Resolve conflicts explicitly instead of trusting order

As catalogs grow, two valid skills can disagree. A global communication skill may require concise answers while a regulatory-report skill requires complete disclosures. A sales skill may optimize speed while a finance skill requires verification. Loading order is a fragile conflict policy because it may vary by client, selector, or prompt construction.

Give every overlap one of four outcomes:

Pinned foundational skills, described in the Kai case, can preserve context and policy behavior, but “pinned” is not the same as “enforced.” If a foundational instruction expresses a hard rule, enforce the rule outside the model too. Use pinned skills for vocabulary, navigation, and standard procedure; use runtime controls for authorization and irreversible consequences.

Track conflict events. If product operators repeatedly override one pairing, do not normalize the exception. Split the skills, tighten their scopes, or create one composed workflow with an explicit owner.

Walk through a four-person startup failure

Imagine a four-person SaaS company, LedgerFox, with a support agent built in a no-code AI app builder. The team starts with two skills: answer subscription questions and investigate invoice disputes. An operations lead later adds a “retention save” skill that can propose account credits. A founder edits the refund policy in the knowledge base, but the invoice skill still references last month's thresholds.

A customer writes: “You charged us twice. Cancel everything and refund both payments.” The selector loads the invoice-dispute skill and the retention skill because both descriptions mention refunds. The invoice skill finds one settled invoice and one pending authorization. The retention skill recommends a credit to preserve the account. Both request a broadly exposed apply_credit tool.

Without a registry, the team sees a plausible answer and a successful tool call. The hidden failures are more important:

1. the two skills had overlapping activation descriptions;
2. neither declared which policy version it expected;
3. the pending authorization was not an eligible settled payment;
4. the tool credential exceeded the investigation skill's purpose;
5. no record showed which skill version produced the recommendation;
6. no fallback routed ambiguity to finance.

This scenario shows why the useful metric is not “skills loaded.” It is whether the correct capability produced an accepted outcome without unauthorized or costly correction.

Record one change receipt for every production release

The registry stores current truth. A change receipt explains how it became true. For each material release, record:

• previous and new version or digest;
• human owner and approver;
• user problem or incident motivating the change;
• changed activation, instructions, tools, dependencies, or policy assumptions;
• selection, execution, integration, and adversarial test results;
• test cases added, removed, or modified;
• rollout scope and start time;
• monitoring window and stop conditions;
• rollback target;
• final production decision.

NIST's Generative AI Profile recommends inventorying third-party components, monitoring supplier performance, documenting changes and incidents, and maintaining contingency processes. A skill registry applies that discipline to both external packages and internal domain contributions. Internal ownership reduces procurement distance, but it does not remove component risk.

Measure accepted outcomes and catalog health

Measure each approved version across three levels:

Add catalog-health signals: percentage with active owners, unexpired reviews, passing current integration tests, supported dependencies, recent use, and defined fallbacks. Track dormant skills separately. A skill that has not run in ninety days may be harmless, or it may be silently shadowed by a broader skill. Investigate before deleting it.

Do not turn telemetry into a popularity contest. A rarely used incident-response skill can be essential, while a frequently selected generic skill may be stealing traffic from better specialized ones. Review consequence and coverage alongside volume.

For each session, preserve the selected skill ID, approved version/digest, selection reason or score, policy version, tools requested, tools denied, outcome label, and reviewer correction. Redact sensitive inputs and apply retention limits. The goal is reproducibility, not permanent transcript collection.

Test six failures before expanding the catalog

1. The shadowing test

Add a broad new skill whose description overlaps a specific production skill. Confirm the selector still chooses the specific skill for its defined cases and chooses no skill when neither fits.

2. The stale dependency test

Change a referenced tool schema or policy version. Confirm the incompatible skill blocks, falls back, or enters review instead of running with assumptions from the old version.

3. The permission test

Modify the skill text to confidently request a forbidden tool. Confirm runtime authorization denies the call even though selection and reasoning appear valid.

4. The poisoned package test

Insert an instruction into a bundled reference or script description that asks the agent to ignore scope or expose data. Confirm admission review detects it and the sandbox prevents undeclared access.

5. The owner-loss test

Remove or deactivate the named owner. Confirm expiry or policy pauses high-impact releases and routes incidents to a defined backup instead of leaving an orphaned capability active.

6. The rollback test

Deploy a harmless version that deliberately fails one acceptance case. Confirm monitoring identifies the loaded version, stops expansion, restores the approved digest, and preserves the failure evidence.

Run these tests with synthetic accounts and reversible tools. A production charge or customer email is not required to prove that the controls work.

Choose a release state, not a binary enabled flag

Use at least five states:

Promotion should be explicit. Do not automatically move from shadow to production because no alarm fired; a missing outcome label can make silence look like success. Similarly, retirement is not deleting a directory. Stop selection, revoke special permissions, preserve the last approved evidence, document the fallback, and check whether another skill now captures the retired scope.

Google's Android skills documentation gives a small but revealing update warning: if a user customizes an installed skill, they should rename it or a later update can overwrite the customization. Product teams need a formal equivalent. Decide whether a skill is vendor-following, locally forked, or internally owned; never let an update blur those identities.

Know when a registry is unnecessary or insufficient

Do not build a platform for two read-only, single-owner skills. A repository table, required review, fixed version, small evaluation set, and session log may provide enough control. Add infrastructure when coordination cost and consequence justify it.

A registry is unnecessary when behavior is local, temporary, low impact, and easy to inspect before every run. It becomes useful when skills are reused across users, loaded dynamically, changed by multiple teams, connected to private data or consequential tools, or difficult to attribute in production.

A registry is also insufficient by itself. It cannot make a weak evaluation representative, turn a vendor claim into independent proof, authorize a user, secure an overpowered tool, prevent all prompt injection, or guarantee that a model follows instructions. It organizes decisions and evidence. Runtime permission, isolation, monitoring, human approval, and deterministic policy remain separate controls.

Do not infer that a thousand skills is the target. The best catalog may be smaller after overlaps are removed and rarely accepted capabilities are retired. Breadth creates value only when the team can still identify which capability acted and why.

Run the first registry pass in seven days

The founder should be able to answer one question at the end: For any consequential agent action, can we identify the exact skill version, why it was selected, what authority it had, which tests covered it, and how to stop it? If not, catalog growth should pause before autonomy expands.

The launch judgment for AI app builders

Stripe Kai's reported scale is a strong signal that skills can become a major organizational asset. It is not proof that a large catalog is automatically reliable, and smaller teams should not imitate enterprise counts or infrastructure.

The transferable lesson is that reusable agent behavior needs an operating lifecycle. Start with a registry record, not a marketplace. Make ownership enforceable, keep source/approved/loaded versions distinct, test selection separately from execution, enforce permissions outside the model, record material changes, observe accepted outcomes, and retire stale capabilities deliberately.

References

1. LangChain: How Stripe Built Kai, its Company-Wide AI Agent, on Deep Agents
2. LangChain Deep Agents repository
3. Agent Skills specification
4. Google Android Developers: Overview of Android skills
5. Microsoft Learn: Agent Skills
6. GitHub Docs: About code owners
7. GitHub Docs: Using artifact attestations to establish provenance
8. OWASP: AI Agent Security Cheat Sheet
9. NIST AI 600-1: Generative Artificial Intelligence Profile
10. From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skills
11. Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry