A demo proves nothing. Production proves everything.
AI features are easy to prototype and hard to keep correct. The engineering is in evaluation, guardrails, retrieval quality, and cost that does not surprise you.
The problem
Almost any team can build an AI prototype that impresses people in a meeting. The difficulty starts afterwards, when real users send inputs nobody anticipated and the feature has to be right often enough to trust.
That gap is an engineering problem with known parts. You need an evaluation set before you tune anything, so improvement is measured rather than felt. You need retrieval that returns the correct context, because most wrong answers are retrieval failures wearing a costume. You need guardrails on what the system is allowed to do. And you need to know what a thousand users will cost before you have them.
We build all four as part of the feature rather than after the launch. An AI feature without an evaluation harness is not a feature, it is a hope.
- An evaluation set built before tuning, so quality changes are measured
- Retrieval quality treated as the primary lever on correctness
- Guardrails on actions, not just on wording
- Cost per operation modeled before scale, not discovered during it
What this covers
Applied AI features
Assistants, extraction, classification, and search that work on your data.
Evaluation harness
A repeatable measure of correct, so tuning is engineering.
Guardrails
Constraints on what the system can do and what it can disclose.
Cost and latency control
Modeled per operation and tested at expected volume.
Retrieval and grounding
Getting the right context in front of the model, reliably.
Production plumbing
Queues, caching, fallbacks, and graceful degradation.
From demo to something you can trust
Step 01
Feasibility, honestly
We establish whether the task is one current models are actually good at before anyone commits to a scope. Some are not, and finding that out in week one is the cheapest possible outcome.
Step 02
Build the evaluation set
A labelled set of real cases with expected outputs, agreed with you. This comes before any tuning, because otherwise improvement is a feeling rather than a measurement.
Step 03
Retrieval and grounding
Get the right context in front of the model. Most wrong answers are retrieval failures, so this is where the accuracy actually comes from.
Step 04
Guardrails and fallbacks
Constrain what the system can do and disclose. Add provider fallbacks and modular adapters so one vendor's pricing change is not your outage.
Step 05
Cost and latency modeling
Measure cost per operation and test at the volume you expect, so a thousand users is a forecast rather than a surprise invoice.
Step 06
Monitor and re-evaluate
Model behavior drifts and providers deprecate. The evaluation set runs on a schedule, not once.
What drives the estimate
AI features are the easiest thing to under-scope, because the prototype takes two days and everyone reasons from that. The build is priced from what it takes to make the thing correct often enough to trust, and almost none of that work is prompt writing.
The dominant cost is your data. Retrieval quality is the main lever on correctness, and retrieval quality is mostly a function of whether the underlying content is clean, current, and chunked sensibly. A messy corpus is the single most common reason an AI feature costs double what a client expected.
- The state of the data the answers must come from, which usually needs work before anything else does
- How high the accuracy bar is, and who decides when it has been met
- Whether the system only answers or is allowed to act, because actions need authorization and audit
- The size of the evaluation set, which scales with how many ways the feature can be wrong
- Whether a wrong answer is an inconvenience or a regulated event
"The agent did it" is not an answer
Every team that puts agents into its delivery process eventually hears that sentence, and it has no operational meaning. The person who delegated the work is accountable for what came back: the code, the tests, the document, the design. Naming the tool describes the toolchain, it does not transfer responsibility.
We hold that line internally because you cannot buy a committed scope from a partner who can point at a model when something is wrong. Every artefact we hand you has a person who read it and accepted it.
The second habit matters as much. When output comes back wrong, the instinct is to tell the agent to fix it. We do not do that first. We ask what gap in the specification allowed the misreading, and we fix the specification, because the same gap will produce the same class of error next week and the week after.
That is the real shift. The work stops being writing the software and becomes designing the system in which the software gets written correctly. It is also why our estimates hold: the errors we fix are the ones that would have recurred.
- Every artefact has a named human who reviewed and accepted it
- A wrong output is treated as a specification defect first, an output defect second
- Context is curated, because an agent will treat a superseded document as current
- Superseded decisions are corrected at the source, not remembered selectively
What we will talk you out of
A chatbot on top of documentation nobody maintains. The bot will confidently repeat whatever is stale, and you will have bought an expensive way to distribute a wrong answer faster.
An agent with write access to a production system on day one. Start read-only, prove the judgment, then grant actions one at a time behind authorization and an audit trail.
Fine-tuning as a first move. It is occasionally the right answer and it is almost never the right first answer. Retrieval and prompt design get further for less, and they are reversible.
Anything with no measurable definition of correct. If we cannot agree what a good output looks like, we cannot commit to a scope, and neither of us should pretend otherwise.
Questions worth asking
What happens when the model provider changes pricing or deprecates a model?
How accurate will it be?
Can you add AI to the product we already have?
Is this priced differently from other engineering?
Is your business ready for AI? Find out.
See where your business stands on AI, and what to do next. It takes a few minutes and gives you a score you can act on.
ScopeGen AI already runs this workflow
AI-First delivery is also how we work internally. ScopeGen AI turns a client conversation into an estimated work breakdown, which is why we can price scope before it is built.
Bring us the backlog.
In 30 minutes, we will show you what a Pod would ship first and how we would price it.