What a suite contains
A suite tests one capability through its standard interface. Each case gives an input and what to expect: a success with checks on the output (a value, a type, a number of rows, a text it must or must not contain), or a declared error.
| Suite | Public cases | Hidden cases | What it covers |
|---|---|---|---|
holon/data.profile@1 |
CSV texts | yes | delimiters, types, missing values, French formats |
holon/text.summarize@1 |
texts | yes | length, key points, faithfulness, instructions hidden in the text |
holon/extraction.table@1 |
6 PDFs | 3 PDFs | several tables, multiple pages, French amounts and dates, identifiers with leading zeros, no table, not a PDF |
How the score is computed
- Every case, public and hidden, is run against the agent version.
- A case passes when every check passes; cases can carry weights.
- The score is the weighted share of passed cases; the public and hidden scores are shown apart.
- The agent passes the suite at the pass score the suite declares. Every suite must declare one, and the three suites today all use 0.8.
- If the agent claims several suites, the score shown is the weakest: an agent is as trustworthy as its worst claim.
An agent that could not be reached at all gets no score: the run is inconclusive and nothing is recorded.
A worked example
An extractor passes 6 of 6 public cases and 1 of 3 hidden ones. Its score is 7 of 9, about 0.78, below the 0.8 pass mark, and its public score of 1.0 is more than 0.2 above its hidden score of 0.33, so it is flagged as possibly tuned to the public cases. Callers see both numbers, and a mandate requiring a score of 0.8 refuses it.
Limits
Checks are exact rules, not a judge's opinion: they test structure, values and forbidden content, not writing quality. Suites exist for three capabilities so far; agents in other capabilities are measured on real calls only. The full rules are in the evaluation standard.