RAG cost per query: where the money actually goes

By LW Forge — maintainer of LLM Scout · Updated August 31, 2026

Almost every RAG cost estimate you'll read online is wrong in the same direction: it obsesses over the embedding model and waves at the generation call. The arithmetic says the opposite. Embedding a user's question costs a fraction of a cent per thousand queries. The call that turns four retrieved chunks into an answer costs a hundred to a thousand times more. If you want a cheaper RAG system, you tune the second number.

A single query bills three things:

1. Embedding the question — the query text goes through an embedding model to become a vector. Input tokens only, no output.
2. The retrieved context — the chunks you pulled from the vector store are pasted into the prompt and billed as ordinary input tokens.
3. The generation — the model's answer, billed as output tokens, typically at 5-6x the input rate.

Retrieval itself — the nearest-neighbour search — isn't billed per token at all. It's part of whatever your vector database charges, which is a separate line item we'll come back to.

The worked example: 10,000 queries a month

Take a realistic shape: four chunks of 500 tokens each (2,000 tokens of retrieved context), a system prompt and question adding roughly 300 tokens, and a 400-token answer. That's 2,300 input and 400 output tokens per query — 23M input and 4M output tokens across 10,000 queries.

A 25x spread between the top and bottom row, on identical retrieval. Note the Luna figures: OpenAI cut Luna by 80% and Terra by 20% on 2026-07-30, which is recent enough that most RAG cost posts still quote the old numbers. Run your own token sizes through the OpenAI cost calculator or the Claude cost calculator before committing to a tier.

The embedding really is a rounding error

Now add the embedding. OpenAI's text-embedding-3-small is $0.02 per 1M tokens and text-embedding-3-large is $0.13 per 1M (verified on the model pricing pages, August 2026; the Batch API halves both). Budget ~100 tokens per query once you count the question plus any query rewriting, and 10,000 queries is 1M tokens: $0.02 a month on the small model, $0.13 on the large one.

Against $43 on Haiku, that embedding line is 0.05% of the bill. Scale it up and it stays absurd: 500,000 queries a month costs about $1 on 3-small and $6.50 on 3-large. Choosing the more expensive embedding model to gain retrieval quality is one of the cheapest upgrades in the stack — it can add a few dollars a month while removing entire wrong answers.

Where does the money go instead? Within the generation call, output is roughly 45-50% of the cost at 400-token answers and climbs past 60% by 800 tokens, because output bills at 5x input on Claude and 6x on GPT-5.6. Two practical consequences: capping answer length is usually a bigger lever than dropping from four chunks to three, and a model that rambles costs more than its per-token rate suggests. Measure real counts in the token calculator rather than estimating from word counts — why 1,000 words are not 1,000 tokens explains the gap.

The cost nobody budgets for

Change your embedding model, or change your chunking strategy, and every vector in your store is invalid. You re-embed the whole corpus. Teams brace for a big API bill here — and it usually isn't one. Five million documents averaging 800 tokens is 4B tokens; re-embedding costs about $80 on 3-small or $520 on 3-large. Painful, not catastrophic.

The real costs of a re-embed are the ones that don't appear on the model invoice:

• Storage. Vectors are dimensions x 4 bytes each. Ten million chunks at 1,536 dimensions is roughly 61 GB of raw floats; at 3,072 dimensions it doubles to 123 GB, before index overhead. Managed vector stores bill GB-months, so doubling dimensions doubles a recurring line forever. text-embedding-3-large supports truncating to fewer dimensions, which is worth benchmarking against the storage cost.
• The rebuild window. Four billion tokens through a rate-limited endpoint is days, not minutes, and you either run a dual index or serve stale results while it drains.
• Re-evaluation. New vectors mean your retrieval quality changed. If you don't re-run an eval set, you've traded a known system for an unknown one.

Budget the migration as engineering time plus a permanent storage increase, and treat the token cost as the small part. That framing also tells you when to switch: if a better embedding model removes a class of wrong answers, a one-off $520 and a bigger index is trivially worth it.

RAG or just a huge context window?

The tempting alternative is to skip retrieval and paste the entire knowledge base into every prompt. With a 150,000-token base on Claude Sonnet 5, a cached stable prefix bills at about 10% of the input rate — $0.045 per query — plus $0.006 for a 400-token answer.

RAG is 4x cheaper on tokens at every volume, and the gap widens without caching, where stuffing costs $0.45 a query instead of $0.045. But RAG carries fixed costs the table doesn't show: a vector database, an ingestion pipeline and the engineering time to keep both honest. Below a few hundred queries a month, a managed vector store's monthly minimum can exceed the entire token saving, and stuffing a cached prefix is the rational choice. Above a few thousand, retrieval wins on cost and keeps winning.

Two constraints decide it before price does. A base larger than the context window rules out stuffing entirely — check yours in the context window calculator. And caching only pays if the prefix is genuinely stable and hit often enough to survive its TTL; prompt caching savings works through when it does and doesn't.

What to cut first

In order of return: shorten the answer, then drop to a cheaper tier and measure whether quality actually moved, then cache the stable prompt prefix, then batch anything non-interactive at half price, and only then argue about chunk counts. Retrieving three chunks instead of four saves 500 input tokens — $1.50 per million queries on Sonnet. Cutting answers from 800 tokens to 400 saves $6.00 on the same volume.

If you're pricing this from Brazil, remember the invoice arrives in dollars and lands in reais with tax and spread on top — what the OpenAI API costs in reais has that arithmetic. And for the end-to-end version of a conversational product, what a chatbot really costs covers the parts RAG doesn't.