Evaluate a document retrieval pilot
Use the questions your application actually needs to answer. Agree on success criteria before comparing results or committing to a deployment.
1. Define the document set
Record the document versions, formats, and approximate size. Start with text or Markdown, or agree on conversion during onboarding. Use documents you are authorized to include.
2. Write representative questions
Include direct questions, questions requiring several passages, questions about earlier versions, and questions the documents cannot answer. Record expected supporting passages before looking at the retrieval output.
3. Keep a complete scorecard
| Measure | What to record |
|---|---|
| Answer quality | How many answered questions meet the agreed relevance and completeness rubric, out of all answered questions. |
| Coverage | How many answerable questions receive useful evidence, including missed answers and incorrect refusals. |
| Source fidelity | Whether every returned passage matches its identified source. |
| Absent evidence | Correct and incorrect handling of questions without supporting documents. |
| Latency | Median and tail end-to-end timings, measured workload, network conditions, and warm versus cold runs. |
| Updates and history | Indexing pause duration, results after a change, and queries against the same saved version. |
4. Compare equivalent configurations
If comparing with a RAG or GraphRAG implementation, record its version, retrieval settings, embedding model, document preparation, and whether an answer-generation step is included. Compare equivalent outputs; an exact passage and a generated summary are different tasks.
5. Decide with the limitations visible
Keep failures, exclusions, and ungraded questions in the report. Your pilot needs its own results.