The numbers on our site are test results, not promises. This page says what was tested and how, in plain words.
Sixteen independent authors wrote 327 questions against four document collections: legal texts, medical papers, physics papers, and a mixed set. They wrote them to be hard, including multi-step questions and 55 trick questions about things that are not in the documents at all. A mechanical checker rejected any question whose supposed answer could not be found word for word in the source.
Answers were graded blind: the judges never knew which system produced an answer. Every grade was checked against the quoted source. The results:
| Result | Meaning |
|---|---|
| 100% on the mixed collection | 41 of 41 answered questions graded excellent. This describes the answered questions in that collection. |
| 94.4% of answered questions | 152 of 161 answered questions graded excellent, across the four collections. |
| 100% trick questions turned away | All 55 questions about absent things were refused, at zero tokens each. No invented answers. |
| 0 made-up quotes | Every served passage matched its source document byte for byte, across the whole exam. |
| 22 ms median search | Inside the engine. End to end over the network, including reading your question, the documented estimate is about half a second. Timing varies by workload and conditions. |
The 94.4% result describes the 161 answered questions, not all 327 questions authored for the evaluation. The summary does not yet provide a complete disposition of all 327 questions, so it should not be used to infer overall coverage. Source matching, answer quality, and coverage are distinct measures. These results are not a guarantee for other datasets.
This page does not provide a complete per-question scorecard, a public grading rubric, reproducible run artifacts, or a matched competitor evaluation. The nine answered questions not graded excellent are not individually characterized here. Do not interpret the headline as an overall success rate or independent certification.
Use the pilot evaluation guide to define a complete scorecard for your own documents.
Want the raw records for any number on this page? Ask on a call. Raw run artifacts are not published on this page.