Every hosted large language model bills the same way: per token. A token is roughly ¾ of a word, and you pay separately for the tokens you send in (the prompt) and the tokens the model writes back (the completion). Output tokens usually cost several times more than input tokens. That single fact explains most surprise bills — and most of the ways to lower them.
How LLM API pricing works
- Input tokens: everything in your prompt — system message, context, retrieved documents, chat history.
- Output tokens: what the model generates. Typically 2–5× the input price per token.
- You pay on every call. In a RAG or agent loop, the same context can be re-sent dozens of times.
- Prices vary enormously by model — frontier models can cost 50–100× more per token than small, fast ones.
Why costs balloon in RAG and agents
In a naive RAG setup, every question stuffs large chunks of your documents into the prompt as input tokens. Multiply that by every user and every turn, and the retrieved context — not the model's answer — becomes the dominant cost. Agents make it worse: each step re-sends the running history, so a 10-step task can pay for the same context ten times over.
How to cut your token bill
- Send cleaner context. Converting documents to clean Markdown strips the boilerplate, tags, and formatting cruft that bloat token counts — often cutting input tokens sharply versus raw HTML or PDF text extraction.
- Retrieve less, more precisely. Good retrieval (hybrid search + reranking) puts the right passage first, so you can pass fewer chunks for the same answer quality.
- Right-size the model. Use a small, cheap model for extraction, classification, and routing; reserve a frontier model for the few calls that truly need it.
- Cache repeated context. Re-reading the same source or system prompt shouldn't cost full price every time.
- Cap output. Set max tokens and prompt for concise answers — output is the expensive half.
The cheapest lever is fewer tokens
Switching models saves money once. Sending fewer, cleaner tokens saves money on every call, forever — and it improves answer quality, because the model isn't wading through noise. That's the idea behind Kit for AI: it converts your documents and URLs to token-efficient Markdown, retrieves precisely with hybrid search and reranking, and answers on private local models, so the context you pay for is small and relevant by default.
You don't need to pick between cheap and good. Clean inputs give you both.