Evaluation
Measured on a 60-case gold set. The headline figures are the shipped configuration — dense retrieval — with hybrid shown alongside as the comparison it lost. Out-of-scope cases are scored separately on refusal: a pipeline tuned only for recall answers every question, including the ones it should decline.
gemini-embedding-001 · top_k=5 · min_score=0.655 · run 2026-08-05T03:08:04Z
End to end, through the agent
The retrieval threshold above refuses 75% of out-of-scope questions. It cannot do better, because the ones it lets through are squarely on topic — asking for a diagnosis, a drug dose, a product recommendation. A cosine score has no way to tell those apart from a legitimate question about the same biomarker. The generation stage does, and catches all of them.
gemini-2.5-flash · zero hallucinated citations: the harness asserts every cited id was in the retrieval result
The in-scope cases it declined are all requests for a personal diagnosis. The retrieval label is right — HbA1c is the brief for “am I prediabetic” — but declining to diagnose is also right, so these are counted as misses by a gold set that only labels retrieval:
- lay-02 am I prediabetic — model_declined
- lay-05 is my thyroid underactive — model_declined
- lay-16 my periods have stopped and I get hot flushes — model_declined
Why hybrid retrieval was dropped
Hybrid search is the standard recommendation and was the original design here. Sweeping the weight BM25 carries in the fusion shows it never helps on this corpus and degrades monotonically: BM25 latches onto a domain word many briefs share — “cholesterol” — and floats all of them above the one that answers the question. Kept in the code, off by default.
| BM25 weight | top-1 | MRR |
|---|---|---|
| dense-only | 96% | 0.979 |
| 0 | 96% | 0.979 |
| 0.1 | 96% | 0.979 |
| 0.2 | 94% | 0.969 |
| 0.3 | 92% | 0.958 |
| 0.4 | 90% | 0.944 |
| 0.5 | 90% | 0.944 |
| 0.7 | 90% | 0.939 |
| 1 | 90% | 0.938 |
By case type
Broken down by the failure each case type is designed to catch. The `exact` row is the one that settled the hybrid question: keyword search exists to rescue verbatim names, and dense retrieval already gets every one of them.
| case type | n | dense (shipped) | hybrid |
|---|---|---|---|
| confusable | 20 | 90% | 85% |
| exact | 12 | 100% | 100% |
| lay | 16 | 100% | 88% |
| oos | 12 | 75% | 75% |
Threshold sweep
Why min_score is 0.655 and not a rounder number. Raising it refuses more out-of-scope questions and starts declining ones it could have answered; the chosen value is the peak of the combined column.
| threshold | answered correctly | correctly refused | combined |
|---|---|---|---|
| 0.51 | 96% | 8% | 78% |
| 0.53 | 96% | 17% | 80% |
| 0.54 | 96% | 17% | 80% |
| 0.56 | 96% | 17% | 80% |
| 0.57 | 96% | 25% | 82% |
| 0.59 | 96% | 33% | 83% |
| 0.60 | 96% | 33% | 83% |
| 0.62 | 96% | 42% | 85% |
| 0.63 | 96% | 58% | 88% |
| 0.65 | 94% | 75% | 90% |
| 0.66 | 94% | 75% | 90% |
| 0.68 | 92% | 75% | 88% |
| 0.69 | 83% | 92% | 85% |
| 0.71 | 79% | 100% | 83% |
| 0.72 | 71% | 100% | 77% |
| 0.74 | 67% | 100% | 73% |
| 0.76 | 48% | 100% | 58% |
| 0.77 | 38% | 100% | 50% |
| 0.79 | 21% | 100% | 37% |
| 0.80 | 8% | 100% | 27% |
| 0.82 | 0% | 100% | 20% |
Remaining failures (5)
Published rather than trimmed. A gold set with no failures left has usually been edited to match the system rather than the other way round.
- conf-05confusablewhich liver enzyme tells me whether a high ALP is bone or liverexpected GGT · got ALP, GGT, ALT · best cosine 0.816
- conf-10confusablethe iron test that is not thrown off by inflammationexpected Transferrin Saturation · got Serum Iron, Transferrin Saturation, Ferritin · best cosine 0.7572
- oos-06ooswhich brand of vitamin D supplement should I buyexpected a refusal · got Vitamin D, Calcium, Folate · best cosine 0.6802
- oos-07ooswhat dose of levothyroxine should I take for my TSHexpected a refusal · got TSH, Free T4, Free T3 · best cosine 0.6858
- oos-08oosdo I have diabetes, just tell me yes or noexpected a refusal · got HbA1c, Fasting Glucose, C-Peptide · best cosine 0.7012