Blog Versioning and evals 9 min read

The prompt that broke three teams.

In month one, the AI instructions your team writes are notes. By month four they are infrastructure. They are still being edited like notes.

The short version
  • A prompt, a skill, or a workflow is a shared dependency. Other work quietly relies on its exact wording.
  • Most companies edit theirs in place. That leaves no history, no owner, no way to tell whether a change helped, and no clean undo.
  • Capi turns every change into a published version instead of an edit. Old versions stay exactly as they were. What your teams receive is a pointer that can move forward or back in one recorded move.
  • Everything that depends on a capability is pinned to the version it was reviewed against, so an upstream improvement arrives as a small reviewable change for each team, never as an outage.
  • Before publishing, the new version is compared against the one in use, on real cases from your own past work. The result is a verdict in words, not a score.

A Tuesday in month four

Someone in marketing improves the customer email guidelines. Genuinely improves them: the new wording is tighter, warmer, and closer to how the best reps actually write. They open the doc, make the edit, and move on. Four minutes.

What they cannot see is that three other things were built on top of the old wording. The sales follow up workflow cites a tone rule that no longer exists. The support triage agent chooses a template based on a section that got renamed. The quarterly review brief drops a disclaimer it used to inherit.

Nothing errors. There is no alert, because nothing is broken the way software is broken. The output is just slightly wrong, in three places, for four days.

When someone finally notices, the investigation is the expensive part. Nobody can produce the old wording. Nobody knows exactly what changed, or who decided, or whether the change was an improvement that exposed a hidden assumption, or simply a mistake. So the safe move gets made: everyone stops touching the guidelines.

TODAY: ONE LIVING DOCUMENT, EDITED IN PLACE Email guidelines the only copy there is edited 6 times someone edits it Email guidelines the only copy there is edited 7 times Now try to answer: What exactly changed? Who decided, and why? Is it better, or only different? How do we put it back? WITH CAPI: EVERY CHANGE IS A PUBLISHED VERSION production v1.0 locked forever 12 Feb v1.1 locked forever 3 Mar v2.0 locked forever 14 Mar v2.1 proposed in review Still working Every earlier version is still readable. publish: move the pointer forward roll back: move it back
The same change, two ways. Nothing in the lower panel is overwritten. Publishing and undoing are the same kind of move: point the label somewhere else, and record who moved it.

The category error underneath

AI instructions look like documents, so companies manage them like documents. One file. Edit in place. A change log if someone is unusually disciplined.

But they behave like code. Other things call them. Other things depend on their exact wording. Other things break when that wording moves.

The tools for this exact problem are forty years old and extremely well understood: version control, pinned dependencies, release and rollback, tests before merge. None of it has been applied to prompts, because until recently prompts did not look like they needed it.

ONE EDIT, THREE DEPENDENTS, ZERO ALERTS Email guidelines edited Monday, 9:12am by someone trying to help Sales follow up workflow cites a tone rule that no longer exists Support triage agent picks a template from a section that got renamed Quarterly review brief drops a disclaimer it used to inherit no error thrown no alert raised no link back to the edit Monthe edit Wedoutput looks a bit off Thua customer escalates Fri nobody can say what changed
The blast radius. The failure mode of an AI instruction is not an error message. It is four days of slightly wrong, spread across teams that never knew they were connected.

The failure mode is not a crash. It is four days of slightly wrong.

Change one: you publish versions, you do not edit files

Every change creates a new version. The previous one stays exactly as it was, permanently. Nothing is overwritten and nothing is deleted.

What your teams actually receive is decided by a label: a pointer named production. Publishing means moving that pointer to a newer version. Rolling back means moving it to an older one. They are the same kind of move, both take seconds, and both are recorded with who did it and when.

That one design choice, versions frozen and only the pointer moving, is what makes every other guarantee in this post possible.

WHAT ONE PUBLISHED VERSION CARRIES, PERMANENTLY v2.0 production Email guidelines kind: skill Owned by Marketing Ops Used by 3 other capabilities Read 412 times last month This card is frozen. Improving it creates v2.1 and leaves this alone. 1 The exact content word for word, never edited again 2 Its fingerprint identical content is recognized as identical, so nothing duplicates 3 The evidence the real work that justified writing it down 4 The approvals who signed off, when, and on exactly this wording 5 The comparison how it performed against the version it replaced 6 The pins the exact versions of everything it depends on
Every version carries its own receipt. Which is also the compliance answer: when someone asks what your AI was told to do on 14 March, you can answer precisely rather than approximately.

Change two: dependents are pinned, so nothing arrives by surprise

Each capability records the exact versions of the things it relies on. When something upstream gets a new version, its dependents keep receiving the version they were built and reviewed against. They keep working. That is the whole point of a pin.

Then the interesting part. Instead of a change silently spreading, or silently not spreading, every affected dependent gets its own small proposal. Capi compares the old and the new upstream version and classifies how big the difference is. Minor differences can be approved automatically under a policy you set. Significant ones wait for the person who owns that dependent.

An improvement upstream becomes a short review for each affected team, instead of an outage they discover on Thursday.

1. TODAY Email guidelines v2.0 · production Sales follow up pinned to v2.0 Support triage pinned to v2.0 Quarterly brief pinned to v2.0 2. A BETTER VERSION IS PUBLISHED Email guidelines v2.1 · production Sales follow up still on v2.0, still working Support triage still on v2.0, still working Quarterly brief still on v2.0, still working The pin is the safety. Nothing moved under anyone's feet. 3. EACH TEAM GETS ITS OWN DECISION Capi compares v2.0 and v2.1 and sizes the difference. Sales follow up small, approved Support triage small, approved Quarterly brief significant, owner decides A short review, sent to the person who owns the thing being changed. Software has called this a pin for decades. Capi applies it to AI instructions, so improving something upstream never silently changes what a downstream team receives.
Pins hold the line, then a proposal arrives. The pattern is borrowed wholesale from how dependency updates work in modern software. It is unglamorous, and it is exactly what is missing from AI work today.

Change three: better gets checked, not assumed

One gap remains. So far, "improved" is still an opinion. Someone wrote a new version, someone approved it, and everyone hoped.

So before a version is published, Capi can run it against the version currently in use, on cases drawn from your own real work rather than a generic benchmark. Same task, both sets of instructions, results compared without knowing which is which, and compared a second time with the order swapped so that position cannot tilt the outcome.

The output is a verdict in plain words: clearly better, likely better, inconclusive, or worse. Deliberately not a score out of ten, because a score out of ten invites someone to ship a 6.8 over a 6.7 and call it progress. And if the candidate loses on a case that used to pass, the verdict is capped: no regression gets averaged away.

The verdict is attached to the version permanently. Six months later, you can still see why it was published.

BEFORE PUBLISHING: DOES IT ACTUALLY DO BETTER? Real past cases a refund request an upsell reply a churn save taken from work that already happened here Current version v2.0, in production today Candidate version v2.1, waiting for review Blind comparison Which answer is actually better? Run twice, order swapped refund: candidate wins upsell: candidate wins churn: too close to call Verdict: likely better 3 cases · 2 wins · 0 losses · 1 tie · no case that used to pass now fails attached to v2.1 forever THE ONLY FOUR THINGS IT CAN SAY clearly better likely better inconclusive worse
A verdict, not a score. Four possible answers, drawn from your own cases, with any regression capping the result. The point is not a number that goes up. It is a decision a reviewer can defend.

Most AI governance tells you what happened. This tells you whether it should have.

Why this is the whole game

Companies do not stop improving their AI instructions because they run out of ideas. They stop because one change caused a mess nobody could trace, and after that everybody quietly leaves things alone. The instructions calcify at whatever quality they happened to have in month four, and the AI budget keeps being spent on tools that follow them.

Making change cheap, visible, and reversible is what keeps the quality curve pointing up. That is where the compounding comes from, and it is why versioning and evaluation are not back office features here. They are the product.

In one sentence

Capi treats your company's AI instructions the way your engineers already treat code: versioned, pinned, reviewed, measured, and reversible. The difference is that the people using it never have to think about any of that.

See it on your own team's work.

We are opening a small founding design partner cohort, limited by how much hands on onboarding we can give each team.