We've spent the last stretch deep in the parts of AI delivery that never make it into a demo. How prompts are designed, where models and data are allowed to live, open weights, the token bill versus the hardware bill, and what happens when a provider has a bad afternoon. Here's the honest, educational version of what we've learned.
A demo shows you the model on its best day. Production shows you everything around it. The five decisions below are the ones that decide whether an AI feature is affordable, dependable, and defensible to a risk committee. None of them are exotic. All of them are easy to skip until the bill or the outage arrives.
Prompt engineering is engineering, not wording
The name undersells it. “Prompt engineering” sounds like finding the magic phrase. In production it's closer to designing an interface against a probabilistic component, and the craft is mostly about being explicit, not being clever.
The prompts that hold up over time share a few habits.
- The contract is explicit. What goes in, what must come out, and in what shape. If you need typed fields, ask for a schema and validate against it. A prompt that says “return JSON” and hopes is not a contract.
- Examples beat adjectives. Two or three worked examples of the input and the output you'd accept move behaviour far more reliably than a paragraph of instructions telling the model to be accurate and thorough.
- Context is curated, not dumped. Give the model what it needs for this task and nothing it doesn't. More context is not more quality. It's more cost, more latency, and more room to get distracted.
- The stable part is separated from the variable part. The instructions and examples rarely change. The user input always does. Keeping that boundary clean lets you cache the stable prefix and cut both cost and latency, which we come back to below.
- Prompts are versioned like code. A prompt is a dependency. It gets a version, a change history, and a test before it ships.
The last habit is the one that separates a system from a science project. A prompt change is a behaviour change, and the only way to make it safely is to measure it. This is where prompt work and evals meet. Without evals a new prompt is a leap of faith. With them, it's a controlled change you can ship on a Tuesday.
“Good prompt engineering doesn't read as clever. It reads as obvious, once someone else has done the work of making the contract explicit.”
Residency, where the model and the data actually live
For a regulated client the first AI question is rarely “is it any good.” It's “where does our data go, and where does the model run.” Get that wrong and the quality of the model is irrelevant, because the feature never clears review.
It helps to separate two things people tend to blur together.
- Data residency is about where data is processed and stored. For UK and EU clients that usually means keeping traffic inside a chosen region and signing terms that rule out training on your data and set retention to zero. The major providers support this through regional endpoints and enterprise agreements, and it's the more common answer.
- Model residency is stronger. The model itself runs inside your own boundary, on your infrastructure, so prompts never leave the perimeter at all. This is where open models come in, and it's the answer when a regional API agreement still isn't enough for the data in question.
The real-world design is rarely all or nothing. A common shape is a frontier API under a zero-retention regional agreement for the genuinely hard work, and a self-hosted open model for the paths that touch the most sensitive data. Residency stops being a blocker and becomes a routing decision, which is a much better place to be.
Experimenting with open models, and when they win
The open-weight models, Llama, Mistral, Qwen, DeepSeek, and now OpenAI's own open releases, have quietly closed most of the gap for well-scoped tasks. Extraction, classification, routing, tagging, and a lot of summarisation no longer need a frontier model. They need a good-enough model you can pin, host, and evaluate.
Three reasons we keep an open model in the toolkit, even when a frontier API is the default.
- Control. A hosted open model doesn't change under you. You pin a version and it behaves the same next month. No silent upgrade, no quiet regression the week after a release.
- Residency. It runs inside your boundary, which answers the model-residency question above outright.
- Cost at volume. For a high, steady workload on a narrow task, a model you host can be dramatically cheaper per request than paying per token to a provider.
The honest catch is that hosting is real work. You take on the serving stack, GPU capacity, quantisation choices that trade a little accuracy for a lot of throughput, and the same eval discipline you'd apply to anything else. Open weights are free to download. Running them well is not. Open doesn't mean free. It means yours.
Tokens or hardware, the question behind the bill
Every AI feature bills one of two ways. Per token to a provider, or per hour for hardware you run. The instinct is to frame this as a model-size question. It's really a utilisation question.
What each one is good at
- Tokens have no fixed cost. You pay for exactly what you use, someone else runs the fleet, and you scale from zero to spike without a capacity conversation. This is the right default for spiky, low, or unpredictable volume, which describes most products for most of their life.
- Hardware costs the same per hour whether it serves one request or a thousand. That makes it painful at low volume and excellent at high steady volume. A GPU you keep busy is cheap per request. A GPU idling overnight is money on fire.
So the break-even isn't about how big the model is. It's about how full you can keep the machine. The directional rule we use.
- Spiky, low, or unpredictable volume, stay on tokens. The flexibility is worth the unit price.
- High, steady, predictable volume on a well-scoped task, hardware starts to win, and if residency already pushed you to self-host, the economics often agree.
And don't price hardware on the GPU alone. The real bill includes ops, on-call, capacity planning, and the engineer-hours to keep a serving stack healthy. Tokens quietly buy you out of all of that, which is a genuine part of their value, not a rounding error.
Before either lever, there's a cheaper one. Caching cuts the token bill without touching hardware. Prompt caching reuses the stable prefix of a request so you're not paying to re-read the same instructions every call. Semantic caching returns a stored answer when a near-identical question comes back. On the right workload those two together take a surprising bite out of the bill before you've changed a single model.
“The AI bill is rarely a model-choice problem. It's a utilisation-and-caching problem wearing a model-choice costume.”
Fallbacks, and why a gateway earns its keep
Providers have outages, rate limits, and slow afternoons. If your product calls one model directly, your product's uptime is that provider's uptime, and you'll find that out at the worst possible time.
A gateway sits between your application and the models. One interface in, many backends out. Whether it's an open-source router you host or a managed one, the value is the same.
- Failover. Primary model returns an error or times out, the request retries on a secondary automatically. The user never sees the wobble.
- Routing. Send easy calls to a cheap model and hard ones to a frontier model, by rule rather than by hope.
- Load balancing. Spread traffic across keys and regions so a single rate limit doesn't become an incident.
- Central control. One place for logging, spend caps, and caching, instead of the same plumbing copied into every service.
There's a discipline that comes with it, and it's easy to miss. A fallback is a quality decision, not just a plumbing one. If your secondary model is weaker, a silent failover can quietly degrade the product in a way that's harder to notice than a clean error and harder to debug. So the fallback path gets evaluated too. You should know, before it fires in anger, that the model you fail over to is one you'd be happy to serve.
Keep the layer thin. A gateway is a routing and resilience layer, not a place to hide business logic. The moment it starts making product decisions, you've moved the hard part somewhere nobody's testing.
How the five fit together
These aren't five separate topics. They're one decision seen from five angles. A clear prompt contract tells you what the task actually needs, which tells you whether a small open model can do it, which decides where the model can live, which drives whether you're really paying for tokens or hardware, and the gateway is what lets you hold all of that together and change your mind later without a rewrite.
If you're commissioning an AI build, these are the questions worth asking early, because they decay slowly and they decide the bill. Which model is the flashy question and the one that changes every few months. How you prompt it, where it runs, what you fall back to, and whether you're paying by the token or by the hour, those are the questions that are still true next year, and the ones that quietly decide whether the thing works.
