Use the questions your application actually needs to answer. Agree on success criteria before comparing results or committing to a deployment.
Record the document versions, formats, and approximate size. Start with text or Markdown, or agree on conversion during onboarding. Use documents you are authorized to include.
Include direct questions, questions requiring several passages, questions about earlier versions, and questions the documents cannot answer. Record expected supporting passages before looking at the retrieval output.
| Measure | What to record |
|---|---|
| Answer quality | How many answered questions meet the agreed relevance and completeness rubric, out of all answered questions. |
| Coverage | How many answerable questions receive useful evidence, including missed answers and incorrect refusals. |
| Source fidelity | Whether every returned passage matches its identified source. |
| Absent evidence | Correct and incorrect handling of questions without supporting documents. |
| Latency | Median and tail end-to-end timings, measured workload, network conditions, and warm versus cold runs. |
| Updates and history | Indexing pause duration, results after a change, and queries against the same saved version. |
If comparing with a RAG or GraphRAG implementation, record its version, retrieval settings, embedding model, document preparation, and whether an answer-generation step is included. Compare equivalent outputs; an exact passage and a generated summary are different tasks.
Keep failures, exclusions, and ungraded questions in the report. SearchCandy's public 94.4% figure describes 152 of 161 answered questions graded excellent in its documented evaluation. Your pilot needs its own results.