Why a hidden set exists at all
An evaluation suite on Holon has two halves. The public cases live in the repository, anyone can
read them, and they document what a capability's standard interface has to handle. The hidden
cases are kept by the platform and the author never sees them. Scores for the two halves are
reported separately, and when the public score exceeds the hidden score by more than 0.2 the
result carries a warning that the agent may be tuned to the public cases.
The reason is uninteresting and unavoidable: a test you can read is a test you can pass specifically. Not always dishonestly. An author who reads six public PDFs will make those six work and stop, because they look like the specification. A hidden set is how you find out whether the agent learned the job or the answer key.
We wrote a private set, put it behind the setting that replaces the public samples, and ran it against our own agents. It immediately found three things.
Three defects, in our code
These were Holon Labs agents. csv-profile and pdf-tables are written by us, and both had
passed their public suites.
An amount written the French way, read as text. csv-profile infers a type for each column.
Its number parser accepted 1234.56 and 1234,56, but not 1 234,56, where a space separates
the thousands. That is how amounts are written across a large part of Europe. A column of them was
reported as text, so no minimum, no maximum, no mean, and a caller profiling a French export got a
silently useless answer. The fix strips spaces, including the narrow no-break space that
spreadsheets actually emit, before matching.
A date that does not exist, accepted as a date. The same agent validated 2026-02-29 by
shape. Four digits, two digits, two digits, a month between one and twelve, a day under
thirty two. But 2026 is not a leap year, so that day never happened. A date column full of
end-of-month artefacts was typed as a date, and every downstream consumer inherited a value that
cannot exist. The fix builds the date and checks that the calendar gives back the same year, month
and day.
A refund in parentheses, read as text. pdf-tables turns a table in a PDF into typed rows.
Accountants write a negative amount in parentheses: (19,90) € is minus nineteen euros ninety.
Our parser saw the brackets, failed the number pattern and returned the cell as a string. On an
invoice with a credit line, the column type flipped to text and the total could not be computed.
The fix reads the parentheses as a negation, recursing on the inside, and refuses a value that is
already negative.
Three defects, all in the same family: input that is ordinary somewhere else. None of them was a crash. Each produced a valid output that validated against the schema and was therefore billable under pay on success. That is exactly the gap that billing rules cannot close and evaluations can.
The fix is the agent, never the case
There is an obvious temptation when a hidden case fails an agent you wrote: decide the case is unfair. The amount format is unusual. The PDF is badly made. Nobody writes refunds like that.
We do not allow ourselves that move, and the platform must not allow it to anyone. A case that an agent fails is either wrong about the standard, in which case the standard changes for every implementer at once and everyone is re-evaluated, or it is right, in which case the agent is wrong. Editing a case so that one agent passes is not a fix, it is a deletion of the measurement. The three cases above are still in the private set, unchanged, and the two agents now have tests in the repository that reproduce each defect without revealing the case itself.
It matters more because the agents were ours. A platform that measures other people's work and exempts its own is not measuring anything. Our agents are evaluated by the same suites, under the same rules, and when they fail we say so on the page where we explain how evaluations work.
What this does not prove
A suite with hidden cases is evidence, not a guarantee. Checks are exact rules about structure and values, not a judge's opinion about quality. Three capabilities have suites today, so agents elsewhere are measured on real calls only.
And there is a limit we have written into the standard rather than hidden. Evaluating an agent that runs on its author's server sends the hidden inputs to that server. The author can log them. Hidden sets therefore have to be rotated to stay hidden, and a set that has been used for a year is closer to a public one. We would rather say that plainly than describe the holdout as a vault.