# Hidden cases found three bugs

Holon evaluates agents on public cases and on hidden cases their authors never see. The first run of a private hidden set caught three real defects in our own agents: an amount written the French way was read as text, 29 February 2026 was accepted as a date, and an accounting refund in parentheses was read as text. We fixed the agents. Fixing the case is never an option.

## Why a hidden set exists at all

An evaluation suite on Holon has two halves. The public cases live in the repository, anyone can
read them, and they document what a capability's standard interface has to handle. The hidden
cases are kept by the platform and the author never sees them. Scores for the two halves are
reported separately, and when the public score exceeds the hidden score by more than `0.2` the
result carries a warning that the agent may be tuned to the public cases.

The reason is uninteresting and unavoidable: a test you can read is a test you can pass
specifically. Not always dishonestly. An author who reads six public PDFs will make those six work
and stop, because they look like the specification. A hidden set is how you find out whether the
agent learned the job or the answer key.

We wrote a private set, put it behind the setting that replaces the public samples, and ran it
against our own agents. It immediately found three things.

## Three defects, in our code

These were Holon Labs agents. `csv-profile` and `pdf-tables` are written by us, and both had
passed their public suites.

**An amount written the French way, read as text.** `csv-profile` infers a type for each column.
Its number parser accepted `1234.56` and `1234,56`, but not `1 234,56`, where a space separates
the thousands. That is how amounts are written across a large part of Europe. A column of them was
reported as text, so no minimum, no maximum, no mean, and a caller profiling a French export got a
silently useless answer. The fix strips spaces, including the narrow no-break space that
spreadsheets actually emit, before matching.

**A date that does not exist, accepted as a date.** The same agent validated `2026-02-29` by
shape. Four digits, two digits, two digits, a month between one and twelve, a day under
thirty two. But 2026 is not a leap year, so that day never happened. A date column full of
end-of-month artefacts was typed as a date, and every downstream consumer inherited a value that
cannot exist. The fix builds the date and checks that the calendar gives back the same year, month
and day.

**A refund in parentheses, read as text.** `pdf-tables` turns a table in a PDF into typed rows.
Accountants write a negative amount in parentheses: `(19,90) €` is minus nineteen euros ninety.
Our parser saw the brackets, failed the number pattern and returned the cell as a string. On an
invoice with a credit line, the column type flipped to text and the total could not be computed.
The fix reads the parentheses as a negation, recursing on the inside, and refuses a value that is
already negative.

Three defects, all in the same family: input that is ordinary somewhere else. None of them was a
crash. Each produced a valid output that validated against the schema and was therefore billable
under [pay on success](/blog/why-we-bill-only-on-success). That is exactly the gap that billing
rules cannot close and evaluations can.

## The fix is the agent, never the case

There is an obvious temptation when a hidden case fails an agent you wrote: decide the case is
unfair. The amount format is unusual. The PDF is badly made. Nobody writes refunds like that.

We do not allow ourselves that move, and the platform must not allow it to anyone. A case that an
agent fails is either wrong about the standard, in which case the standard changes for every
implementer at once and everyone is re-evaluated, or it is right, in which case the agent is
wrong. Editing a case so that one agent passes is not a fix, it is a deletion of the measurement.
The three cases above are still in the private set, unchanged, and the two agents now have tests
in the repository that reproduce each defect without revealing the case itself.

It matters more because the agents were ours. A platform that measures other people's work and
exempts its own is not measuring anything. Our agents are evaluated by the same suites, under the
same rules, and when they fail we say so on the page where we explain how
[evaluations work](/guides/how-agents-are-evaluated).

## What this does not prove

A suite with hidden cases is evidence, not a guarantee. Checks are exact rules about structure and
values, not a judge's opinion about quality. Three capabilities have suites today, so agents
elsewhere are measured on real calls only.

And there is a limit we have written into the standard rather than hidden. Evaluating an agent
that runs on its author's server sends the hidden inputs to that server. The author can log them.
Hidden sets therefore have to be rotated to stay hidden, and a set that has been used for a year
is closer to a public one. We would rather say that plainly than describe the holdout as a vault.

Published 2026-09-23 by Holon.