From a laptop demo to production - what an AI feature really costs

Most writing about shipping AI stops at the advice layer: define contracts, add observability, build evaluation harnesses. All true, all useless without specifics. So here are the specifics.

We built ShiftGreenBot, a conversational assistant for Shift Green Hub that helps people find community sustainability events and articles. It started as a demo on a laptop running Ollama. It now runs in production on Azure OpenAI behind a serverless API. The provider migration itself took an afternoon and about two hundred lines of code.

The afternoon was not the expensive part. The expensive part was everything that leaked out of the abstraction afterwards, and the things production quietly added to the codebase that nobody ever decided to build.

If you are evaluating what an AI feature will cost you, this is a more honest estimate than most.

Start on a laptop. Seriously.

The first version ran entirely locally: gemma3 for generation, nomic-embed-text for embeddings, both served by Ollama on port 11434, with MongoDB Atlas for vector search. No API key, no quota, no per-request cost, no network latency.

That matters more than it sounds, because most of the work in a retrieval-augmented system is prompt work, and prompt work is a tight-loop activity. You change a sentence, you run it, you read the output, you change it again. Forty times. Doing that against a metered cloud endpoint is like debugging with a thirty-second compile step, and you will do less of it than the feature needs.

The architecture was a fixed three-step pipeline, and it still is:

Step one turns a messy human sentence into something a database can use. Step two retrieves. Step three writes the reply and decides which retrieved items are actually worth showing. Two LLM calls, one embedding call, one database query.

Nothing about that shape is clever, and that is the point. A fixed pipeline is predictable, cheap to reason about, and easy to test. We deliberately did not build an agent with tool-calling, and we still have not, because the product does not yet need the ceiling that architecture buys.

The migration, and where the abstraction leaked

Moving to production meant Azure OpenAI - gpt-4o for chat, text-embedding-3-large for embeddings - behind a NestJS API on AWS Lambda.

We kept the local path alive rather than deleting it, because we still want that tight loop. The provider is resolved at runtime from the environment:

export function getAIProvider(): AIProvider {
  const forced = process.env.AI_PROVIDER?.toLowerCase();
  if (forced === "azure" || forced === "ollama") return forced;
  const env = process.env.ENV;
  return env === "dev" || env === "local" ? "ollama" : "azure";
}

Local and dev get Ollama. Everything else gets Azure. One environment variable overrides both. Underneath, createEmbedding() and the chat helper branch on it, and every caller above them stays provider-agnostic.

That was the theory. Here is where it broke.

Leak one: embeddings are not interchangeable, and the choice is baked into your data

nomic-embed-text produces 768-dimensional vectors. text-embedding-3-large produces 3072. These are not two dialects of the same language - a vector from one model is noise to the other, and an Atlas vector index fixes its dimensionality when you create it.

The consequence is that the provider choice lives in your data, not just your code:

You cannot point a local dev server at the production database and expect search to work - you would be querying a 3072-dimensional index with a 768-dimensional vector. Changing the provider for an environment means re-embedding every document in it and recreating the index from scratch.

This is the single most important thing to understand before you pick an embedding model: the model is a schema decision, not a configuration value. Budget for a full re-index every time you change it.

Leak two: the prompts diverged, so local stopped being a faithful preview

We wanted identical prompts everywhere. We do not have identical prompts anywhere.

The Azure prompt asks the model to curate. It receives the retrieved items pre-numbered and returns index arrays alongside its reply:

{
  "message": "Here are some events I found for you...",
  "recommended_event_indices": [0, 2],
  "recommended_blog_indices": []
}

The backend then filters the retrieved set down to exactly what the model chose. This matters because vector search returns things that are near your query, and near is not the same as appropriate. A query about composting workshops will surface a gardening article with a decent similarity score even when it answers nothing the user asked. gpt-4o is good at dropping those.

A three-billion-parameter local model is not. It returns index arrays that are empty, out of range, hallucinated, or flatly inconsistent with the prose it just wrote in the same response. So the local branch gets a simpler contract - write a message, and the backend shows everything retrieved:

const prompt =
  sharedPromptPrefix +
  (isOllama ? ollamaContentSearchRules : azureContentSearchRules) +
  itemsBlock;

A shared preamble with divergent behaviour rules. It is an honest structure, and we would build it the same way again. But be clear-eyed about what it means: local shows you retrieval quality, production shows you curation quality, and they are different features. A prompt change that improves one can silently damage the other.

Leak three: rate limits, context windows, and whose problem they are

Ollama has no quota. Azure throttles hard on lower tiers, and backfilling embeddings for an entire content library hits that ceiling within seconds.

So the Azure embedding path grew a retry loop with no local equivalent: up to eight attempts, honouring the Retry-After header, falling back to parsing the "retry after N seconds" hint out of the error body, falling back again to exponential backoff - plus a one-second buffer and up to a second of random jitter, so ten concurrent requests in a batch do not all wake at the same instant and re-trigger the limit.

The batch runner carries the same asymmetry in a single line:

const isOllama = getAIProvider() === "ollama";
const delayMs = isOllama ? 0 : 5000;

Full speed locally. Five seconds between batches of ten against Azure.

Then the leak ran the other direction. nomic-embed-text has a roughly 2048-token context window and simply errors on longer input. Blog posts exceed it comfortably. So we added a 6000-character cap on source text before embedding, plus a truncate-and-retry loop that shrinks the input by 40% and tries again whenever Ollama reports an overflow.

Look carefully at what that means. A limitation of our local development model put a permanent 6000-character cap on the text we embed in production, because the text builder is shared code. text-embedding-3-large would happily accept far more context. We are truncating production embeddings to keep a laptop happy.

That is not a bug. It is a real, ongoing quality cost of keeping a dual-provider setup, and it is the kind of thing that never appears in a migration estimate.

The second wave: more collections, and an infrastructure constraint that decided our data model

The next phase was not about infrastructure. Shift Green Hub repositioned - from a marketplace toward a not-for-profit sustainability impact platform. Products stopped being what people come to discover; community events and articles became it.

So the retrieval layer changed shape. Products came out of the assistant entirely. Events and blogs went in. The classifier gained a third job: deciding which content types to search.

The two searches run in parallel, hits are hydrated back into typed groups from their source collections, and the response step may recommend across both in a single answer.

The index budget

The obvious way to add events and blogs is to give each collection its own embedding field and its own Atlas vector index, mirroring what products had.

We could not. Our shared Atlas cluster caps vector and full-text search indexes at three, cluster-wide. We run three databases - dev, test, production. That is the entire budget: one index per database, no room for a per-collection layout, and no room to grow.

So every vector goes into one unified collection instead:

Each row is a derived copy of a source document. refId points home, embedding is the vector, and text records precisely what was fed to the embedding model - which turns out to be the single most valuable debugging field in the system, because "why did this not match?" is almost always answered by reading what you actually embedded rather than what you assumed you embedded. A compound unique index on type and refId means re-running ingestion upserts rather than duplicates.

This is the part I would defend hardest in a review. A number in a pricing tier dictated a data model decision, which sounds like exactly the sort of compromise you apologise for later. But it produced the better architecture anyway: adding a new entity type now costs zero new indexes and roughly twenty lines of code - one text builder and one branch. Had we built three per-collection indexes first and hit the cap later, it would have been a rewrite.

Find your infrastructure limits before you design the data model, not after. Sometimes the constraint is doing you a favour.

Ingestion in an environment with no shell

Locally, generating embeddings is a script you run. In production the API is a Lambda and the database sits behind network rules - there is nowhere to run a script from.

So ingestion became an authenticated endpoint. Two properties make that safe to expose:

• Incremental by default. It looks up which source documents already have a row in the vector store and only processes the ones that do not. Re-running it is a no-op.
• Explicitly destructive on request. A reset flag clears the collection and re-embeds from scratch - needed when the embedding model changes, or to purge stale rows like our old product vectors.

The response reports which provider it used, which is the fastest possible way to catch an environment misconfiguration.

The full request path

Here is what actually happens when someone types a message, end to end:

Two things worth noting in that diagram.

The API key never reaches the browser. The chat call goes through a Next.js server action that holds it, and the endpoint sits behind an API key guard with per-stage CORS origins. That is not sophisticated, but a surprising number of AI demos ship with the provider key in client-side code, and it is the difference between a feature and an incident.

And that note at the bottom - the 29-second ceiling - brings us to the honest part.

What production added when nobody was looking

We told ourselves we were writing environment-agnostic code with one provider switch at the bottom. Reading the codebase back as a list tells a different story.

Deliberate, and correct. Secrets resolve from AWS SSM Parameter Store at deploy time, with the Lambda role granted parameter read and decrypt permissions and nothing wider; endpoints and deployment names live in per-stage config as plain values. The retry-with-jitter on embeddings. The incremental, idempotent ingestion endpoint. The server-side API key.

Absorbed without a decision. This is the interesting bucket.

• A 29-second hard ceiling on the entire pipeline. The Lambda timeout is set to 29 seconds - the API Gateway maximum - against a platform default of 10. That number is the budget for three sequential network round trips. Nobody sized it against measured latency. It is the ceiling because it is the largest number available.
• A warmup schedule. A five-minute scheduled ping to dodge cold starts, added for the API generally, but the assistant is the endpoint that most visibly suffers without it.
• A timeout workaround pointing the other way. The web layer installs a custom HTTP dispatcher with ten-minute timeouts, but only when running locally, because Ollama on a laptop is slower than the default 30-second header timeout. A local-only workaround living permanently in the production code path.
• console.log as the observability strategy. The pipeline logs each step with the classified intent, the generated query, result counts, and the final message. In Lambda those land in CloudWatch, which makes them our de-facto production tracing - and also means we are writing user messages and assistant replies in full to a log aggregator. Nobody decided that. It is the kind of thing that should be a decision, with a retention policy attached.
• Test and production are indistinguishable to Azure. Both stages point at the same resource, the same endpoint, the same deployment names. There is no way to canary a model change in test without also changing production. The only thing separating them is the database.

Never added, and worth naming plainly. There is no retry on the chat calls - embeddings retry on throttling, the two chat calls do not, so a rate-limited request falls straight through to a generic apology. No token cap on generation. No per-user rate limiting on an endpoint that costs money per request. No caching, so identical queries re-embed and re-generate every time. No streaming, so users watch three sequential round trips behind a bouncing-dots animation. And no evaluation suite - every prompt refinement described in this article was validated by typing into the chat and reading the answer, which means no regression test would catch a prompt edit that quietly makes curation worse.

That last one is the real debt. Everything else on the list is an afternoon of work when it becomes a problem. Prompt regressions are invisible until a user tells you, and by then you have shipped forty more changes on top.

What we would tell you before you start

Build the demo locally. The iteration speed on prompts is worth several times what the migration costs, and the migration genuinely is small.

Treat "provider-agnostic" as an aspiration, not a property. The provider leaks into your vector dimensions, your index configuration, your rate-limit handling, your input length caps, and - most insidiously - into your prompts, because a small local model and a frontier model are not capable of following the same instructions. Design for a shared core with explicit, visible per-provider branches. Do not pretend one code path will serve both.

Let infrastructure constraints reach the data model early. Ours forced a unified embedding store, and that turned out to be the architecture we wanted for adding content types cheaply.

Then audit what production added. Not one of the timeouts, warmup schedules, log statements, or retry loops in this system was a designed decision. Each accreted, each locally reasonable. Reading them back as a list is how you find the 29-second ceiling nobody sized and the user messages nobody decided to store.

The advice at the top of most articles about production AI - contracts, observability, evaluation - is correct. But it lands very differently once you can point at the specific line where your laptop's context window started truncating your production data.

---

ShiftGreenBot runs on Next.js and NestJS, MongoDB Atlas Vector Search, and Azure OpenAI - with Ollama still doing the day-to-day work on our laptops. If you are weighing up an AI feature and want an estimate that accounts for the parts nobody puts in the estimate, we are happy to talk.