Ask your internal documents, and unsupported sentences get flagged
Every sentence is scored against the retrieved passages, and the ones the sources do not support are flagged in the answer itself. Automatic, and visible.
Why this needs building rather than buying a chat box
The knowledge exists, it is just scattered. Contracts on one drive, process parameters on somebody's laptop, the conclusion from last year's rework sitting in a single email. A new hire asks one question and a senior person spends twenty minutes digging.
Hand the question straight to a language model and it will invent things. Citations make that worse, not better. A citation sitting next to an invented sentence reads as evidence, so the reader stops checking.
So the part worth paying for is not that it answers. It is that the shaky sentence gets pointed at. Almost all the engineering goes there.
Numbers you can reproduce today
| Metric | k=3 | k=5 | k=10 |
|---|---|---|---|
| recall@k | 0.867 | 0.867 | 1.000 |
| MRR | 0.789 | 0.789 | 0.810 |
15 golden questions over 254 chunks of public-domain text. Command npm run eval, run on 2026-08-31. k=3 and k=5 hit the same set, which is why those two columns match. Separately, npm test reports 223 passing and 9 skipped, the ones that need Postgres.
Indexing and retrieval run on your own machine with a local embedding model. No API key, and the documents do not leave. Only the final answer generation calls a model.
Where it gets things wrong
The verifier is a heuristic, not a proof of entailment. These three failure modes are known.
- Negation blindness. "X causes Y" and "X does not cause Y" embed close together, so a contradicted claim can still score as supported.
- False flags. Heavy paraphrase or a claim that combines several passages can fall under the threshold even when it is genuinely grounded.
- It measures traceability, not truth. A sentence that faithfully echoes a wrong source passage scores as supported.
What each side brings
You bring
- A set of documents, md and txt to start
- One concrete situation, such as support staff looking up process parameters
- One person who can judge whether an answer is right
We bring
- Indexing and retrieval on your own machine, no key needed
- A cited answer endpoint: citations you can click back to, and unsupported sentences carry a flag
- One script tag that drops into your existing page template
Not this
- Anything where it makes the decision. It looks things up, it does not sign off
- Stripping the citations and keeping only the conclusion. That voids everything above
