Ask your internal documents, and unsupported sentences get flagged

Every sentence is scored against the retrieved passages, and the ones the sources do not support are flagged in the answer itself. Automatic, and visible.

产品截图:问达尔文的问题,两句带 [1] [2] 引用的回答被判为有据,第三句关于热气球的假话被标成 unsupported,下方读数写着 2 条有据 1 条被标
A screenshot of the product, not a mockup. The question is about Darwin. The first two sentences carry [1] [2]. The third, a fabrication we planted on purpose, is flagged unsupported. The readout below shows 2 supported, 1 flagged, threshold 0.55. The badge says mock model, because that is how we make the planted sentence reproducible every run.

Why this needs building rather than buying a chat box

The knowledge exists, it is just scattered. Contracts on one drive, process parameters on somebody's laptop, the conclusion from last year's rework sitting in a single email. A new hire asks one question and a senior person spends twenty minutes digging.

Hand the question straight to a language model and it will invent things. Citations make that worse, not better. A citation sitting next to an invented sentence reads as evidence, so the reader stops checking.

So the part worth paying for is not that it answers. It is that the shaky sentence gets pointed at. Almost all the engineering goes there.

Numbers you can reproduce today

Retrieval eval: recall and MRR at three values of k
Metric k=3 k=5 k=10
recall@k 0.8670.8671.000
MRR 0.7890.7890.810

15 golden questions over 254 chunks of public-domain text. Command npm run eval, run on 2026-08-31. k=3 and k=5 hit the same set, which is why those two columns match. Separately, npm test reports 223 passing and 9 skipped, the ones that need Postgres.

Indexing and retrieval run on your own machine with a local embedding model. No API key, and the documents do not leave. Only the final answer generation calls a model.

Where it gets things wrong

The verifier is a heuristic, not a proof of entailment. These three failure modes are known.

  • Negation blindness. "X causes Y" and "X does not cause Y" embed close together, so a contradicted claim can still score as supported.
  • False flags. Heavy paraphrase or a claim that combines several passages can fall under the threshold even when it is genuinely grounded.
  • It measures traceability, not truth. A sentence that faithfully echoes a wrong source passage scores as supported.

What each side brings

You bring

  • A set of documents, md and txt to start
  • One concrete situation, such as support staff looking up process parameters
  • One person who can judge whether an answer is right

We bring

  • Indexing and retrieval on your own machine, no key needed
  • A cited answer endpoint: citations you can click back to, and unsupported sentences carry a flag
  • One script tag that drops into your existing page template

Not this

  • Anything where it makes the decision. It looks things up, it does not sign off
  • Stripping the citations and keeping only the conclusion. That voids everything above