On the recorded 30-question eval, retrieval recall@8 is 18/24 (75.0%), the answer rate is 8/24, refusal of unanswerable questions is 6/6, and citation precision and faithfulness are 7/9 each.

Ask a question about a PDF corpus and get claims whose quotes are verbatim, highlighted on the page, with an inspectable retrieval trace and a calibrated refusal.

(Python 3.12, FastAPI, PostgreSQL 16 + pgvector, React + Vite, PDF.js, ONNX Runtime)

01 / What it is

What it is

A Python 3.12 FastAPI service with PostgreSQL 16 and pgvector. Ingest refuses a PDF whose SHA-256 differs from corpus.lock.json; retrieval fuses embeddings and lexical search, then reranks; the answer path keeps a claim when its quote appears in the cited chunk and drops it otherwise.

Citation view for the sentence-embedding question: the selected source quote is highlighted on PDF page 3, beside claims 1-3 and the source list. Seeded offline demo data; no model call. — Citation view for the training question: claim 1 quote marked on page 3 in the daylight reader, beside the three-claim answer and navy source list.

Retrieval trail at the second-pass cross-encoder section: rerank rank, score, calibrated evidence probability, kept status, then the final context and decision. Recorded demo trace, not a live request. — Retrieval trail overlay with second-pass rank, score, evidence probability, included status and passage, then the final decision.

Evaluation view of the committed eval-report.json: 2 of 5 checks met, counts and denominators, Wilson 95% ranges, and the stage where each question stopped. — Evaluation page with 2 of 5 checks met, rates, counts, 95% confidence ranges and where each question stopped.

02 / The hard part

The hard part

A retrieval answer is easy to make plausible and hard to check. A claim can cite a passage outside the retrieved set, quote text outside the cited chunk, or survive a question the corpus holds no evidence for. The refusal path is what makes an answer checkable.

03 / The decision

The decision

An uncited claim is dropped, not repaired

Decision Validate each claim against its citations, require a verbatim quote from the cited chunk, and drop the claim on either failure.

Rejected alternative Generate first, then attach the nearest passage; a citation added afterwards can point at text outside the claim.

Cost The validator drops a reply whose quote is not a verbatim substring, so the answer rate is 8/24.

Evidence The citation tests cover the rejection path; the eval records answer rate, citation precision and faithfulness for every run.

04 / The failure it refuses

The failure it refuses

Pathological case

A fluent answer cites a passage outside the retrieved set, or quotes text outside the chunk it names.

Mechanism Every claim is validated against its citations: the cited passage must be in the retrieved context, and the quote must be a verbatim substring of that chunk. A claim that misses either check is dropped before the answer is returned; the refusal threshold is read from eval/calibration.json at boot.

05 / Measured

Measured

The recorded eval measured recall@8 at 18/24 (75.0%, [55.1%, 88.0%]), answer rate at 8/24 (33.3%, [18.0%, 53.3%]) and refusal at 6/6 (100.0%, [61.0%, 100.0%]). Citation precision and faithfulness are 7/9 each. The run took 778.6 s with 26 model requests, 0 retries and 0 rate limits.

Measured results and the method behind each value.
Metric	Value	Method	Artifact
retrieval recall@8, production (hybrid + RRF + rerank)	18/24 (75.0%), Wilson 95% [55.1%, 88.0%]	bash scripts/live.sh eval; generator openai-compatible:openai-fast, fastembed bge-small-en-v1.5 + ms-marco-MiniLM-L-6-v2; Windows 11 Pro 10.0.26200, ONNX CPU	`eval-report.json`
answer rate, production, free endpoint	8/24 (33.3%), Wilson 95% [18.0%, 53.3%]	bash scripts/live.sh eval; generator openai-compatible:openai-fast, fastembed bge-small-en-v1.5 + ms-marco-MiniLM-L-6-v2; Windows 11 Pro 10.0.26200, ONNX CPU	`eval-report.json`
unanswerable refusal, production	6/6 (100.0%), Wilson 95% [61.0%, 100.0%]	bash scripts/live.sh eval; generator openai-compatible:openai-fast, fastembed bge-small-en-v1.5 + ms-marco-MiniLM-L-6-v2; Windows 11 Pro 10.0.26200, ONNX CPU	`eval-report.json`
citation precision and faithfulness, rule-based	7/9 (77.8%), Wilson 95% [45.3%, 93.7%] each	bash scripts/live.sh eval; generator openai-compatible:openai-fast, fastembed bge-small-en-v1.5 + ms-marco-MiniLM-L-6-v2; Windows 11 Pro 10.0.26200, ONNX CPU	`eval-report.json`
production eval run cost	778.6 s; 30 questions; 26 model requests; 0 retries; 0 rate limits	bash scripts/live.sh eval; provenance.run in eval-report.json	`docs/ENV.md`
forced-generation diagnostic (separate diagnostic run)	8/24 (33.3%), Wilson 95% [18.0%, 53.3%]	bash scripts/live.sh eval --force-generation; openai-compatible:openai-fast	`eval-report-forced.json`

Invariants and their tests

Ingest refuses a chunk longer than 448 model tokens rather than storing a truncated vector. tests/test_chunking.py
A claim without an in-context citation, or whose quote is not a normalised substring of the cited chunk, is dropped. tests/test_citations.py
Ingest refuses a PDF whose SHA-256 differs from corpus.lock.json. tests/test_lock.py
The refusal threshold comes from eval/calibration.json at boot and is never hand-tuned. tests/test_refusal.py
Every rate carries a Wilson 95% interval and its denominator; the eval records the earliest stage that produced no answer for each question. tests/test_eval.py
The judge model differs from the generator and is pinned by exact id; its reliability is recorded by the judge check. tests/test_judge_check.py
ONNX work runs in a worker thread behind Semaphore(2). tests/test_embedder.py

Artifacts

Back to the case index