Clinical Knowledge Base — RAG Demo
Retrieval over 52 schema-validated biomarker briefs, wrapped in a LangGraph agent that rewrites and retries before it gives up. Every answer cites the briefs it used, and those citations are checked against what was actually retrieved — so the system can decline, and you can see why.
Measured on a 60-case gold set: 96% top-1 retrieval and 100% of out-of-scope questions refused, end to end.
How it is kept honest
The parts that usually break in a RAG knowledge base, and what stops them here.
One schema, enforced in code
Every brief is validated against a Pydantic schema with controlled vocabularies before it is embedded. An invalid brief fails ingest with its filename attached, rather than surfacing later as a confident wrong answer.
Hard metadata filter, applied first
Unapproved, non-UK or off-category briefs are excluded before similarity runs, on both the vector and keyword sides. They are not outranked — they are never candidates.
Citations checked, not trusted
The model must cite the source_brief_id of everything it used. Any id that was not in the retrieval result means the answer is discarded — a hallucination becomes a detectable event.
Refusal is a first-class outcome
A score threshold and an explicit 'the briefs do not cover this' path mean the system declines. The eval scores that behaviour separately, because a system measured only on recall answers everything.
Stack: Next.js proxy (no provider keys) · FastAPI · LangChain retrieval · LangGraph agent · Pinecone serverless · Gemini embeddings and generation, swappable for OpenAI or Anthropic by config. Source and the full write-up are in the repository.