Context is the part of the bill that grows the most without anyone deciding it. Nobody approves “let’s pay triple”: a document gets added to the prompt, then another, and the bill goes up. Every context token is billed on every request, and that is what turns a long window into a cost problem.
Why it gets expensive
A model remembers nothing between requests. Everything you want it to know —instructions, examples, history, documents— travels again with each request and is billed again. If your prompt carries 20,000 context tokens and you make 50,000 requests a month, you pay for a billion input tokens for context alone, whatever the answers turn out to be.
The three usual leaks
| Leak | What causes it | How to fix it |
|---|---|---|
| Full history | The whole conversation is resent every turn | Summarize old turns and send only the recent ones |
| Whole documents | The full PDF is attached “just in case” | Retrieve only the relevant chunks |
| Bloated system prompt | Instructions pile up and never get pruned | Rewrite it and measure whether quality drops |
Retrieve instead of attaching
The structural fix is not putting the corpus in the prompt. You index the documents, retrieve the three or four chunks that answer the question, and send only those. Quality usually goes up —less noise— and context drops by an order of magnitude. We develop this in RAG, fine-tuning and context.
What is worth keeping long
Not every cut is a good cut. Stable instructions and the examples that pin the output format are expensive to remove and cheap to keep if you use prompt caching: the fixed block is cached and billed at a fraction. The practical rule is fixed cached context on top, variable minimal context below.
Put a number on it
Before redesigning anything, measure. Put your current input tokens and volume into the token cost calculator and compare the same work with half the context: you will see the monthly difference at once. And track cost per task before and after to confirm quality has not dropped.
Frequently asked questions
Why does long context raise cost so much? Because context is sent and billed on every request. A large prompt is not paid once: it is paid as many times as you make requests.
How much context is too much? There is no absolute number. The signal is the ratio: if input tokens far exceed output tokens and much of that is documents or history, you have room.
Does trimming context make answers worse? It often makes them better, because the model stops competing with irrelevant information. What you cannot do is trim without measuring cost per task and quality before and after.
At e-ficient we measure the cost per task of companies already running AI in production and tell you how much you would save with each lever at your real volume. The first audit is free if you start a plan with a three-month commitment and returns a diagnosis within 72 hours.