What an unsupported sentence means in an internal knowledge assistant, with the numbers
A language model with citations invents more convincingly: the citation sits right there, so the reader assumes the sentence is sourced. So every sentence is scored against the retrieved passages, and the ones that do not hold get flagged on the spot.
How it judges
Four steps: retrieve the relevant passages from the company's material, have the model answer from those passages, split the answer into sentences, and score each sentence separately against the retrieved passages. A sentence below the threshold is marked unsupported in the answer itself.
The screenshot on the service page is the product's own output, not a mockup. The question is about Darwin; the first two sentences carry citations, and the third, "the author wrote it in a hot-air balloon", is a fabrication we planted on purpose. It is flagged. The readout says 2 supported, 1 flagged, threshold 0.55.
How far that goes
It does not make the model more correct. It makes the wrong sentence visible. The threshold is a trade-off: lower it and fewer sentences are flagged but more slip through; raise it and more are flagged, so more need a person's review. That number should be set by the company using it, against its own material, not by us on your behalf.
Retrieval itself can miss. The evaluation numbers below say what proportion of the relevant passages made it into the top three, five and ten, and how high the first relevant passage ranks on average. Those two figures decide whether the scoring step has anything to compare against.
Numbers you can reproduce (run on 31 August 2026)
| Metric | k = 3 | k = 5 | k = 10 |
|---|---|---|---|
| recall@k | 0.867 | 0.867 | 1.000 |
| MRR | 0.789 | 0.789 | 0.810 |
223 automated tests pass, plus 9 that need a Postgres environment. The evaluation runs on a public corpus: npm test and npm run eval.
Five questions before buying one of these
- Can every sentence in an answer be traced to its sourceA string of citations at the end of the answer does not count.
- Does a sentence without a source get flaggedAsk for a live demonstration of a planted false sentence being flagged.
- What are the retrieval recall and ranking figures, and on which corpusNo numbers means it has not been measured.
- Where does the material live, and which model touches which stepIn writing, not as a verbal assurance.
- Who sets the threshold and the handover boundaryIt should be you, against your own material and your tolerance for error.
Tell us which step costs you the most
Contact us