# AI Engineering > Building AI features that survive real users: evaluation, guardrails, cost control, and the retrieval work that makes answers correct. Source: https://www.koombea.com/capabilities/ai-engineering/ --- AI features are easy to prototype and hard to keep correct. The engineering is in evaluation, guardrails, retrieval quality, and cost that does not surprise you. ## The problem Almost any team can build an AI prototype that impresses people in a meeting. The difficulty starts afterwards, when real users send inputs nobody anticipated and the feature has to be right often enough to trust. That gap is an engineering problem with known parts. You need an evaluation set before you tune anything, so improvement is measured rather than felt. You need retrieval that returns the correct context, because most wrong answers are retrieval failures wearing a costume. You need guardrails on what the system is allowed to do. And you need to know what a thousand users will cost before you have them. We build all four as part of the feature rather than after the launch. An AI feature without an evaluation harness is not a feature, it is a hope. - An evaluation set built before tuning, so quality changes are measured - Retrieval quality treated as the primary lever on correctness - Guardrails on actions, not just on wording - Cost per operation modeled before scale, not discovered during it ## What this covers - **Applied AI features**: Assistants, extraction, classification, and search that work on your data. - **Evaluation harness**: A repeatable measure of correct, so tuning is engineering. - **Guardrails**: Constraints on what the system can do and what it can disclose. - **Cost and latency control**: Modeled per operation and tested at expected volume. - **Retrieval and grounding**: Getting the right context in front of the model, reliably. - **Production plumbing**: Queues, caching, fallbacks, and graceful degradation. ## From demo to something you can trust Feasibility, honestly: We establish whether the task is one current models are actually good at before anyone commits to a scope. Some are not, and finding that out in week one is the cheapest possible outcome. (Days) Build the evaluation set: A labelled set of real cases with expected outputs, agreed with you. This comes before any tuning, because otherwise improvement is a feeling rather than a measurement. (Before tuning) Retrieval and grounding: Get the right context in front of the model. Most wrong answers are retrieval failures, so this is where the accuracy actually comes from. (The main lever) Guardrails and fallbacks: Constrain what the system can do and disclose. Add provider fallbacks and modular adapters so one vendor's pricing change is not your outage. (Before launch) Cost and latency modeling: Measure cost per operation and test at the volume you expect, so a thousand users is a forecast rather than a surprise invoice. (Before scale) Monitor and re-evaluate: Model behavior drifts and providers deprecate. The evaluation set runs on a schedule, not once. (Ongoing) ## What drives the estimate AI features are the easiest thing to under-scope, because the prototype takes two days and everyone reasons from that. The build is priced from what it takes to make the thing correct often enough to trust, and almost none of that work is prompt writing. The dominant cost is your data. Retrieval quality is the main lever on correctness, and retrieval quality is mostly a function of whether the underlying content is clean, current, and chunked sensibly. A messy corpus is the single most common reason an AI feature costs double what a client expected. - The state of the data the answers must come from, which usually needs work before anything else does - How high the accuracy bar is, and who decides when it has been met - Whether the system only answers or is allowed to act, because actions need authorization and audit - The size of the evaluation set, which scales with how many ways the feature can be wrong - Whether a wrong answer is an inconvenience or a regulated event ## "The agent did it" is not an answer Every team that puts agents into its delivery process eventually hears that sentence, and it has no operational meaning. The person who delegated the work is accountable for what came back: the code, the tests, the document, the design. Naming the tool describes the toolchain, it does not transfer responsibility. We hold that line internally because you cannot buy a committed scope from a partner who can point at a model when something is wrong. Every artefact we hand you has a person who read it and accepted it. The second habit matters as much. When output comes back wrong, the instinct is to tell the agent to fix it. We do not do that first. We ask what gap in the specification allowed the misreading, and we fix the specification, because the same gap will produce the same class of error next week and the week after. That is the real shift. The work stops being writing the software and becomes designing the system in which the software gets written correctly. It is also why our estimates hold: the errors we fix are the ones that would have recurred. - Every artefact has a named human who reviewed and accepted it - A wrong output is treated as a specification defect first, an output defect second - Context is curated, because an agent will treat a superseded document as current - Superseded decisions are corrected at the source, not remembered selectively ## What we will talk you out of A chatbot on top of documentation nobody maintains. The bot will confidently repeat whatever is stale, and you will have bought an expensive way to distribute a wrong answer faster. An agent with write access to a production system on day one. Start read-only, prove the judgment, then grant actions one at a time behind authorization and an audit trail. Fine-tuning as a first move. It is occasionally the right answer and it is almost never the right first answer. Retrieval and prompt design get further for less, and they are reversible. Anything with no measurable definition of correct. If we cannot agree what a good output looks like, we cannot commit to a scope, and neither of us should pretend otherwise. ## Questions worth asking ### What happens when the model provider changes pricing or deprecates a model? This is why we build modular adapters and provider fallbacks rather than wiring one vendor's SDK through your application. Switching provider should be a configuration change and a re-run of the evaluation set, not a rebuild. ### How accurate will it be? Nobody can answer that before seeing your data, and a partner who quotes you a percentage upfront is guessing. We build the evaluation set first, measure where we start, and agree the bar we are committing to hit. ### Can you add AI to the product we already have? Yes, and that is the more common request. It is scoped as a feature against the existing system, with the same evaluation and guardrail work, because bolting it on does not make the correctness problem smaller. ### Is this priced differently from other engineering? The build is priced in story points like anything else. What is different is that we insist on the evaluation harness being in scope. An AI feature without one is not a feature, it is a hope, and we will not fixed-bid a hope. ## ScopeGen AI already runs this workflow AI-First delivery is also how we work internally. ScopeGen AI turns a client conversation into an estimated work breakdown, which is why we can price scope before it is built.