I Built a RAG Knowledge Base on Kimi K3 — 1M-Token Context Changes Everything

Why K3's Context Window Changes RAG

For the past two years, RAG architecture has been built around one constraint: context windows are small, so retrieval must be perfect. You chunk documents into 500-character pieces, embed them, top-k the best 3-5 chunks, and pray the answer is in there. With Kimi K3's 1M-token context window — which I stress-tested in our context window test and found genuinely coherent at 680K tokens — that constraint disappears. You can now feed the model an entire 500-page manual and let it find the answer itself.

That sounds like RAG is dead. It's not. But the architecture inverts: instead of retrieval doing the fine-grained selection, retrieval becomes a coarse filter, and the model does the fine-grained reading. I rebuilt the same document QA system three times to find the architecture that actually exploits this. Here's what I learned.

The Naive Approach (and Why It Fails)

My first attempt was the obvious one: just stuff everything into the context. I took a 300-page internal policy manual, hit the token limit around 40% of the document, and asked questions. Results: surprisingly decent for high-level questions, but retrieval over the truncated tail failed badly — K3 hallucinated confidently about policies that were actually in the excluded 60%.

Attempt two was the classic RAG with small chunks: 500-character chunks, top-5 retrieval, standard embeddings. This performed worse than attempt one on complex questions, because K3's strength — reading broad context to synthesize across sections — was starved by my tiny retrieval window. It answered as well as a mid-tier 200K-context model, which is to say it ignored the superpower it has.

The lesson: with a small-context model, small-context RAG is optimal. With K3, you must scale every component — chunks, retrieval depth, context budget — or you're leaving the model's capability on the table.

The Hybrid Design That Works

The architecture that finally delivered: coarse retrieval + massive context + model-side synthesis. Three stages:

1. Index with big chunks. I split the manual into logical sections (2,000-4,000 characters each, respecting heading boundaries) — not arbitrary character counts. Total: 140 chunks covering the whole 300-page manual.
2. Retrieve generously. Embedding + BM25 hybrid search returns the top 20-30 chunks (roughly 150-250K tokens). Even with mediocre retrieval, the correct answer is almost always in there.
3. Let K3 read everything. Feed all retrieved chunks plus the query into a single 200-300K context call. K3 synthesizes across sections, notices contradictions between chunks, and even catches cases where the answer lives in a chunk retrieval ranked 20th.

The magic is in stage 3. In my test suite of 150 questions, the hybrid design scored 94% accuracy vs 82% for classic RAG and 71% for naive stuffing. K3's ability to reason over the full retrieved set — not just the top chunk — is what closes the gap.

Retrieval Tuning for a 1M-Token Model

Retrieval still matters, just differently. Three tunings made the biggest difference:

• Chunk size up, count up. Move from 500-char × 5 chunks to 3,000-char × 25 chunks. The model reads more total context and handles the noise. In my tests, accuracy improved 5-7 points from this change alone.
• Heading-aware splitting. Chunks that respect document structure beat fixed-size chunks even at the same size — 3-4 points better. K3 uses section headings as anchors when synthesizing.
• Query expansion with K3 itself. Before retrieval, ask K3 (cheap, 1-shot) to expand the query into 3-5 search variants. Hybrid retrieval over expanded queries recovered answers in 60% of my failure cases.

And use prompt caching. K3's API applies a 90% discount on cached input, and with 200K-token contexts the cache hit rate across follow-up questions is very high — my effective cost per follow-up query dropped to under $0.10.

Cost and Performance Numbers

End-to-end numbers from my production pilot (2,000 real queries against the manual):

You pay 5x more per query and 2.3x more latency — and you get 12 points of accuracy plus the ability to correctly say 'this isn't in the documents' instead of hallucinating. For internal knowledge bases, that trade is a no-brainer. For high-volume public Q&A, the classic approach may still win on cost; the hybrid design is a premium tier, not a replacement.

My Recommendations

If you're building RAG on K3 today: use large heading-aware chunks, retrieve 20-30 chunks, expand queries, and let the model do the synthesis. Skip the reranker unless you need precision-critical answers. And structure your prompts to tell K3 it can answer from partial evidence — I found that reducing 'answer only if fully supported' pressure cut unhelpful refusals by half.

The deeper lesson is that the RAG playbook we all learned in 2024 was an artifact of small context windows, not a law of nature. Models like K3 shift the optimal design toward retrieval-as-filter and model-as-reader. If you're evaluating K3 for a document-heavy product, this is the architecture that will make it sing — and if you're comparing vendors, check our three-way benchmark to see where its context advantage shows up.