Most teams find out what retrieval-augmented generation costs when the first invoice arrives. That is avoidable, because the cost per answer is a multiplication you can do before writing any code, and the result usually changes a design decision.
Four lines, not one
A single answer bills in four places.
| Line | What it is | Typical share |
|---|---|---|
| Query embedding | Turning the question into a vector | Negligible |
| Input tokens | System prompt + retrieved chunks + question | Usually the largest |
| Output tokens | The generated answer | Moderate, priced higher per token |
| Retries | Anything that failed and was sent again | Invisible until it isn't |
The instinct is to look up the model's price per million tokens and stop there. That number gives you the rate. It tells you nothing about the quantity, and the quantity is set by your retrieval settings rather than by your users.
The multiplication
input_tokens = system_prompt + (chunks × chunk_size) + question
output_tokens = answer_length_cap
cost_per_answer = input_tokens × input_rate
+ output_tokens × output_rate
+ embedding_tokens × embedding_ratePut plausible figures in and the shape is immediately obvious. A 400-token system prompt, eight retrieved chunks of 800 tokens each and a 60-token question comes to 6,860 input tokens, of which 6,400 — about 93% — is retrieved context. The user's question is a rounding error in its own bill.
That is the whole insight, and everything useful about cost control follows from it. You are not paying for questions. You are paying for the context you chose to attach to them.
Then multiply by volume. Ten thousand answers a day is ten thousand times whatever you computed, every day, and that arithmetic is unforgiving in a way a monthly SaaS subscription never prepares anyone for.
Where the money goes
Chunk count is the dominant setting. Retrieving ten chunks instead of five doubles the input bill on every single answer. The quality gain is rarely proportional, because past a handful of genuinely relevant chunks the extra context is mostly material the model has to work to ignore. Measure answer quality at 3, 5 and 10 on a fixed question set before accepting whatever the default was.
Chunk size compounds with it. Chunks sized for recall at indexing time become a per-answer cost at query time, which is a connection people make late. Smaller chunks with a reranker on top usually beat larger chunks passed blindly, because reranking lets you send three good ones instead of ten plausible ones.
The system prompt is paid on every call. A 1,500-token prompt full of instructions and examples is 1,500 tokens per answer, forever. Keep it stable, because providers cache repeated prefixes and a prompt that changes per request cannot be cached at all. Keep it short, because every sentence in it is rent.
Output caps matter more than they look. Generated tokens are priced above input tokens. "Answer in at most 150 words" is a product decision and a cost control at the same time, and it is the rare one that improves the product while saving money.
Retries are real spend. A timeout that is retried bills twice. So does a validation failure that re-prompts. If you have not instrumented your retry rate, your cost model is optimistic by exactly that rate, and idempotency is the prerequisite for retrying safely at all — here with a price attached.
Measure per answer, not per month
A monthly total tells you nothing you can act on. Log the units on every call.
log.info({
event: "rag.answer",
chunks: retrieved.length,
input_tokens: usage.input_tokens,
output_tokens: usage.output_tokens,
cached_input_tokens: usage.cache_read_input_tokens ?? 0,
retries,
ms: Date.now() - started,
});Those field names are Anthropic's, where cached input is reported separately as cache_read_input_tokens alongside input_tokens. Other providers report the same information under different names, so read your own usage object rather than copying this one.
With that logged you can ask the questions that matter. What does the median answer cost, what does the 95th percentile cost, and which queries live in that tail. The expensive tail is usually a small set of broad questions that retrieve the maximum chunks and generate the longest answers, and those are frequently the ones a cached or canned response would serve better anyway.
The levers, in order of return
Retrieve fewer and better chunks. Add reranking, then cut the count. Biggest saving, smallest quality cost, and it is not close.
Cache the system prompt by keeping the prefix byte-identical across calls so the provider can discount it. Cap the output length, which is cheap to do and immediately visible. Cache whole answers for repeated questions, keyed on a normalised form of the question, because support corpora have a long head of identical questions.
Then, maybe, a cheaper model. It is the first lever most teams reach for and the one most likely to cost quality per rupee saved.
Retrieval quality and cost point the same direction here, which is unusual enough to be worth exploiting. Tighter retrieval is cheaper and better at the same time. If you are still choosing a vector store, that is the trade-off to optimise for, and the full pipeline is in building a production RAG chatbot.
Two numbers you choose
The cost of a RAG system is decided by how many chunks you retrieve and how big they are, and almost not at all by how your users phrase their questions.
Do the multiplication before launch. Log tokens per answer from day one, because adding it later means you have no baseline to compare against. And treat the retrieval settings as a cost control as much as a quality control, since they are both.
A system that retrieves three good chunks is cheaper than one that retrieves ten and hopes, and it usually answers better too.