A multi-tenant research assistant that answers questions over a team's documents and the web with inline citations, and has a second model verify each answer. On a 110-case benchmark against my first version, the rebuild answered more accurately, 95.3% against 91.1% on content checks, reached first token in half the time and cost 39% less per turn.
* ONE TURN, END TO END
[01] The problem
Teams want answers from their own documents, with each claim traceable to its source. Document assistants fail quietly in two ways: answers that look cited but are not supported, and refusals of questions the documents can answer. My first version took 11.5 s to its first token at the median, and its quality was judged by reading answers one at a time, which had already missed an unindexed document for days.
[02] The constraints
Most model routes run on free tiers, so quotas, refused keys and dead routes are routine. The runtime has to fail over in seconds, not minutes, and a quota failure must never read as a quality regression in the benchmark.
Guardrails, citation rewriting and streaming all sit behind the chat route, and each had produced a defect invisible one layer below. Every case runs through that route, and deterministic checks on the stored answer decide pass or fail.
A second model checking every answer adds a round trip. The answer streams first, its sources follow, and the grounded badge arrives when the check completes; offline, a check takes a median of 0.49 s.
[03] The approach
I scored my first version and the rebuild side by side on the same 110 cases, audited the harness, fixed fifteen of its defects and scored both again. The rebuilt runtime retrieves in code before the agent's first call, routes turns with rules, has a separate model verify each answer against its evidence and checkpoints only the turns that delegate research.
[04] The trade-offs
[01]
[02]
[03]
[04]
[05] The build
[01]
Answers stream with inline citations to the documents and web pages they draw on, then a sources list, then a grounded badge once the verifier has checked the answer. Users pick Brief, Standard or Deep effort, switch on web search, or hand a question to parallel research agents. The client re-renders only the part of an answer that changed: in a server-side benchmark on a 12 KB answer, render work fell 25-fold.
[02]
For a document question, the rules router sets the route and search runs in code: dense and BM25 results fused by reciprocal rank fusion, then reranked by Cohere, and handed to the LangGraph agent as if it had searched. Searches scoring below a relevance floor of 0.35, set from measured scores, are flagged to the model as weak evidence rather than dropped.
[03]
Chunking and keyword search were reworked. On one test PDF, lines missing from the index fell from 11 to 0 and chunks citing the wrong page from 16 to 0. Keyword search over a synthetic 50,000-chunk index fell from 23.1 ms to 3.8 ms, and a conversation's own documents are now found even when they rank outside the workspace's top 120.
[04]
All 110 cases go through the app's real chat route. Deterministic checks on the stored answer decide pass or fail; one external judge, on a model no arm answers with, scores groundedness afterwards; and every run is appended to a log, never overwritten. An audit of the harness fixed fifteen defects, one of which had let every arm score 100% on citations.
[05]
A separate verifier checks each answer against its evidence, and one without evidence is marked unverified. On 42 labelled answers, it passed none of the 28 planted errors in three runs; the chat-model judge passed 8 in each of its two. In the latest run it scored 97.6%, flagging one correct answer of 14.
[06]
Auxiliary calls have deadlines of 20 s per provider and 45 s per chain, the answering model a 30-second first-byte deadline, and refused keys are benched. With the primary provider failing, a stubbed turn now finishes in 2.1 s instead of 22.1 s; a provider that accepts a call and never answers is abandoned at the first-byte deadline; and when the verifier's credit ran out mid-benchmark, the fallback judge took over unattended.
[07]
Delegated research runs as parallel document and web branches, journalled to MongoDB. In a crash test with a 10-second lease (production uses 45), a turn killed mid-research was answered 53.6 s after the kill, with no one asking again. Checkpoint writes per delegated turn fell from 111 to 4, and their storage from 355 KB to 15 KB.
[06] What broke
!01
The first 110-case run showed the configured primary model answered 0 of 110 turns: every call walked about a dozen failing routes before one answered, and two exhausted rerank keys cost about 10 s a search. Fixing the provider layer cut median latency by 49%, lifted task success by 4.5 points and took failed turns from 2.7% to zero.
!02
Only the real client showed it, because the hand-run tests left the field out: the chat API rejected a null thread ID, so the first message of every new conversation returned a 400. The API now accepts one.
!03
An audit of the harness found citations were scored after the server had stripped invalid markers, so every arm scored 100%. Scored on what the model actually wrote, each arm carried one fabricated marker; the audit fixed fifteen harness defects in all.
[07] The result
95.3%
Answer correctness
* content checks, 110 cases; first version 91.1%
5.5 s
Time to first token, p50
* first version 11.5 s; p95 18.3 s against 31.8 s
−39%
Cost per turn
* $0.00142 against $0.00234, reference prices
0 / 28
Planted errors passed
* verifier, three runs; chat-model judge 8 / 28 twice
Answer correctness, latency and cost come from Run 010 on 24 September 2026: my first version and the rebuild on the same 110 cases, same day and providers, one run each, over a five-document test corpus, with every token priced at one model's list rate. Task success, 101 of 110 against 97, is within noise, since three of the first version's misses were a judge's provider hanging, and groundedness, last judged in Run 005, was level. The rest come from Run 005 for the call and latency gains, Run 012 for routing, three runs over 42 labelled answers for the verifier, stubbed or synthetic benchmarks for rendering, indexing, failover and checkpoints, and one crash test on an earlier build. None of it measures real users; there are none yet.
[08] What I’d do differently
I'd budget three runs per arm from the start. In one re-scoring, two arms two rows apart disagreed on eight rows, so a single pass cannot separate a four-row lead from noise.
Need something like this built?
Next case study [01]
Carder
AI Mini-Site Builder
Read the case study →
