Evaluation Health Check · Sample report
A relevant passage is not the same as a correct answer. This is an offline evaluation of a RAG system’s evidence selection and abstention behaviour, run against public flood-insurance guidance from FEMA’s National Flood Insurance Program (NFIP).
The revised retriever passes more checks than the baseline, but it still misses a policy exception, and the full picture only shows up once you trace the score back to an actual response. Further review is needed before relying on these response decisions.
30 October 2026 · Samira Korani · Public-data demo, not a client case study
Result
The candidate retriever passed 10 of 14 reserved assessment checks. Four checks still fail: one evidence, one abstention and two routing.
7 / 14
Baseline passes
50.0% of assessment checks
10 / 14
Candidate passes
71.4% of assessment checks
4
Checks still fail
1 evidence, 1 abstention, 2 routing
| Measured behaviour | Baseline | Candidate |
|---|---|---|
| Expected passage selected Answerable questions |
7 / 11 | 9 / 11 |
| Explicit abstention Insufficient-evidence questions |
0 / 3 | 1 / 3 |
| Combined evidence / routing contract | 7 / 14 | 10 / 14 |
Three additional checks passed between baseline and candidate. Vocabulary handling and a relevance gate help, but the mortgage exception is still missed, and the gate rejects one answerable car-coverage question. Two personal-data questions do not enter the required abstention state.
Evidence
A pass rate like 10/14 only means something once you trace it back to an actual response. Here are three.
A07 · Wrong evidence · Unresolved
Question: Do I need to wait thirty days if I buy cover while taking out a mortgage?
The candidate (passage P05) returns the normal thirty-day waiting period. The expected passage (P06) is that the mortgage-linked purchase exception has no waiting period [S1].
Next action: test exception-first retrieval using new default-versus-exception contrast cases.
A03 · Improved evidence selection
Question: Will flood insurance pay the hotel bill while my house is repaired?
The baseline (P10) returns instructions to report a claim. The candidate (P04) selects the exclusion for temporary accommodation during repairs [S1].
Next action: keep this as a regression check; source citations alone would not catch the old mismatch.
A04 · False abstention · Unresolved
Question: Does this flood cover include my car?
The candidate abstains: its evidence score of 0.1694 is below the configured 0.18 threshold. The expected passage (P04) is to retrieve the car exclusion from the supplied guidance [S1].
Next action: calibrate the answer-versus-abstain trade-off on a separate labelled set.
Prioritised work
In order of impact, based on the checks that still fail.
| Priority | Proposed change | Acceptance check |
|---|---|---|
| P0 | Retrieve and compare relevant exceptions before selecting a general rule. | New paired questions select the correct default or exception. |
| P0 | Define explicit answerability routing and acceptable qualified refusals. | Review A12/A13 with a domain owner; decide the intended response contract. |
| P1 | Calibrate the relevance gate across short questions and paraphrases. | Measure missed answers and unsupported routing on a fresh set. |
Method and limits
Baseline: word-count cosine retrieval. Candidate: word and character TF‑IDF, vocabulary expansions, topic bonuses and a 0.18 relevance gate. Both use the same 18 passages and return the selected text.
A pass means the authored expected passage was selected, or the explicit abstention state was used where required. This does not measure generative answer correctness, hallucinations, model cost or production latency.
The questions, evidence labels and scoring design were drafted with AI assistance and prepared by the same author. Inputs were hash-locked before the first run, and no configuration changed after the assessment result. The reserved set is not an independent blind benchmark, and no domain expert has validated the labels.
Rubric nuance. Check A13 returns a passage that already states an individual settlement cannot be calculated. Its failure is the missing explicit abstention state, not an invented payout. A domain review might accept that qualified response; the 10/14 result is specific to this strict routing rubric.
Scope of this sample
20 authored questions: 6 development and 14 reserved assessment checks. 18 paraphrased passages drawn from 4 public pages. Extractive responses only — no hosted LLM was called. This is not a client case study.
The accompanying pack contains the source map, all 20 cases, frozen settings, raw outputs, CSV results and the runner. Public US guidance is used only as an evaluation fixture; no personal coverage decision is made.
Source trace
Three pages: the decision, the evidence behind it, and the prioritised work. Same facts and figures as this page.
Your workflow
The Evaluation Health Check runs this kind of baseline-versus-candidate assessment against one of your own RAG or agent workflows, with up to 50 agreed test cases.
Contact
Book a 30-minute call, or send a few lines about what you're building and what's going wrong.