# Validation Standard

> When one pass writes the code and the tests, a misread spec produces a wrong build that its own tests confirm. This is the mechanism that catches it.

Source: https://www.koombea.com/ai-pods/standards/validation/

---

## The failure mode, stated plainly

In traditional development, a developer who misunderstood a requirement usually
catches it. Conversation, code review, or QA surfaces the gap.

With an agent the failure looks different. A developer prompts for an
implementation. The agent generates the code and the tests from the same reading
of the spec. The developer ships it. The implementation is wrong, the tests
pass, and the misreading never surfaces.

This is not a defect in the model. It is a structural property of any system that
generates both the artifact and the check in one pass. It is the specific thing
this standard exists to prevent: **code that passes its own tests because the
same misunderstanding generated both.**

Accountability does not move. The agent is the tool, not the deliverer. The
person who signed off is answerable for what shipped.

## Two kinds of validation

**Product validation** proves the build matches the spec.

**System validation** proves the agent behaved correctly, that it interpreted the
spec as intended, did not drift, and produced auditable behavior.

A project needs both. Validation is not a safety net for AI mistakes. It is the
mechanism that makes AI-First delivery auditable at all.

There is a longer arc here. As this way of working matures, the work shifts from
writing code, to reviewing generated code, to designing the systems in which
agents write and review code. At each step the person moves further from the
artifact and catches less by intuition. The validation system has to cover that
growing distance.

## The behavior contract

Quality designs what must be proven. Engineering decides where each scenario
physically lives. That split is deliberate, and it is the opposite of the
traditional bottleneck.

Generating a test at any layer is now cheap. The old constraint, that there was
never time to automate, has collapsed. What is left, and what does not automate,
is deciding what to prove and where an agent is likely to diverge from the spec.

Every scenario in the plan carries four tags.

| Tag | Values | What it answers |
| --- | --- | --- |
| Section | Happy path, validation rules, edge cases, accessibility | What kind of behavior this covers |
| Run purpose | Smoke, functional, regression, usability | When it runs, and in which pipeline |
| Severity | Critical, high, medium, low | What to fix first when it fails |
| Compliance risk | An optional note per scenario | Where an agent could plausibly diverge from the spec |

Run purpose describes when a scenario runs, not how it is built. A scenario can
carry more than one. The point is to enable subset runs.

Severity is independent of the other two. A happy-path scenario can be critical
if it is the purchase flow, or medium if it renders an avatar. An edge case can
be high if it loses data silently, or low if it is cosmetic.

### The compliance risk note is the sharp part

It is required wherever an agent is plausibly likely to ship something that
passes the literal scenario and violates the spec's intent. It names the specific
wrong implementation the agent might choose.

A real one: if the spec says the list endpoint returns items already sorted by
the backend, an agent may sort the response correctly and still render it in
insertion order. So the scenario asserts the visible order, not the shape of the
data.

## Spec compliance testing

Standard tests verify that code does what the code does. Spec compliance tests
verify that code does what the spec said.

The rule: for every feature shipped through an agent-driven build, at least one
test verifies a spec requirement the build's own generated tests did not cover.
A different author, for a different purpose.

Where these earn their keep:

| Scenario | What to test |
| --- | --- |
| Data scoping | A query returns only the authenticated user's records, not all of them |
| Authorization | An action blocked in the interface is also blocked at the API and the data layer |
| Default state | A feature initializes to the specified default, not to the first option or a code-level default |
| State persistence | A selection survives navigation |
| Negative path | Invalid input is rejected, not silently accepted, defaulted, or crashed |
| Contract | Field names, types, sort order and shape match the documented contract, not just a success code |

## Who owns what, and when

Each artifact has one owner and one moment.

| Artifact | When | Owned by |
| --- | --- | --- |
| The spec | Before build | Product |
| The validation plan: scenarios per criterion, tags, edge cases, traceability | Before build | Quality |
| The stack-layer mix: which scenarios run as unit, component, integration, end to end, or visual | In the plan | Engineering |
| The test code | During build | Engineering, generated under supervision |
| Review of whether the suite covers the spec | After build | Engineering |
| Implemented spec compliance tests | After build | Quality, targeting the gaps the compliance-risk notes predicted |
| Manual testing | After build | Quality, as a calibration tool and not as primary coverage |

The boundary that matters: the build agent's tests verify the implementation is
internally consistent. Quality's tests verify it matches the spec. Both are
automated. Different authors, different purpose.

Everyone in this workflow is working with an agent. Product with a spec agent,
quality with a planning agent, engineering with a build agent. When this says
"the build agent's tests", it means the one the developer drove during the build.

## Coverage scales up, never down

Coverage should match the complexity and criticality of the work. A copy change
and a new authentication flow do not need the same suite, but both need one.

When in doubt about whether a behavior needs a test, write it. Generating the test
is cheap now. Missing the behavior is not.

## The gates every change clears

Gates are the baseline, not the coverage. Feature tests prove correctness. Gates
enforce the floor.

| Gate | What it enforces |
| --- | --- |
| Lint | Zero errors, not zero warnings |
| Type check | No type errors and no suppressions |
| Unit suite | Everything passes |
| Coverage threshold | A minimum at project level, higher on security-critical paths |
| Static security scan | Zero warnings |
| Dependency scan | No high-severity vulnerabilities in production dependencies |

A project-level coverage threshold is a lagging indicator. Auth, authorization
middleware, payment processing and key enforcement need near-complete coverage
regardless of what the project average says.

**Gates must fail the build.** A gate that is installed and not enforced is
decoration. If one is disabled, the reason is documented and the date it comes
back is tracked. We have audited a project whose pipeline invoked a test command
for a framework the team did not use: it found nothing, ran nothing, and
reported green on every merge while 93 test files sat invisible. That is what an
unenforced gate looks like from the outside.

## Traceability

Every criterion maps to a test that proves it. The spec's job is to ensure the
criterion exists before the build. The plan's job is to map it. A criterion with
no test is not done, and a test with no criterion is not evidence.

The [specification standard](https://www.koombea.com/ai-pods/standards/specification/) covers the other
half of this, and the
[readiness audit](https://www.koombea.com/ai-pods/ai-readiness-audit/) is how we find out where a
project actually stands against both.