Test your agent against its examples

What an example promises

In a Holon manifest, examples is not a documentation section that ages quietly. It is the part buyers read before paying, and assistants read before building a call. An example says: give this agent this input, and it returns that output, exactly.

examples:
  - name: a short article
    input: { text: "Holon is a registry of agents. Agents pay per successful call." }
    output: { words: 11, sentences: 2, characters: 62 }
  - name: an empty document
    input: { text: "" }
    error: empty_input

Two kinds of example exist and they are exclusive: one carries output, the other carries error. An example with both is refused. The error code has to be one you declared in interface.errors, because a failure that is not declared is never billed and has different consequences for your record, as explained in declare the errors your agent returns.

What the platform checks, and what it does not

Publishing runs the manifest through validation. On examples, it checks three things:

• every example input validates against interface.input;
• every example output validates against interface.output;
• every example error names a code you declared, and no example sets both output and error.

It also refuses a manifest with no successful example at all. A manifest whose only example is a failure tells a caller nothing about what a good run looks like.

What validation cannot check is reality. Holon does not call your server with your examples before listing you, so a manifest can be internally consistent and still describe an agent that returns something else. That gap is yours to close, and the author kit closes it with one command.

The kit replays every example

agent.test.mjs in the author kit reads holon.yaml and turns every example into a test. It runs each one twice.

First, straight against your function:

for (const example of manifest.examples) {
  test(`example: ${example.name}`, async () => {
    if ('error' in example) {
      await assert.rejects(run(example.input, ctx()), (e) => e.code === example.error);
    } else {
      assert.deepEqual(await run(example.input, ctx()), example.output);
    }
  });
}

Then through the MCP server itself, on a real port: it lists the tools and asserts there is exactly one, named runtime.tool; it calls it with a successful example and compares structuredContent with the declared output; and it calls it with a failing example and checks isError is true and structuredContent.error.code is the declared code.

The second pass is the one that catches transport mistakes: a tool renamed in holon.yaml but not in the server, an output returned as text instead of structuredContent, an error thrown as a crash instead of a declared failure. Those are the same mistakes that make a call fail once the gateway is on the other end, which is why they are worth catching before you deploy.

The comparison is deepEqual, not "looks similar". An extra debug field, a number returned as a string, a rounded value: all failures. That strictness is the point. A caller that reads your example and gets a different shape has to write defensive code, and defensive code around a paid call is how people stop calling.

What breaks when examples drift

Say your profiler returned { "columns": 4, "rows": 120 } and you add a missing count per column. The output schema grows, the code grows, the example stays as it was. Three things happen.

Callers build wrong calls. An assistant that read get_agent writes code for the old shape, gets a field it did not expect or misses one it needed, and reports a failure that is not a failure. You are billed nothing, since only a valid output is billed, but you lost the call anyway.

Your evaluation looks worse than your agent. Suites score an agent against the capability class it claims, and an agent whose declared behaviour and real behaviour differ scores badly for a reason that has nothing to do with quality. The mechanics are in how agents are evaluated.

And the fix is not free. A published version is immutable: you cannot edit the example in place. You bump version, publish again, and yank the old version once callers have moved. That is the strongest argument for running npm test in continuous integration, on every commit, with the manifest in the same repository as the code.

A short checklist before publishing

npm test            # every example, twice: function, then MCP server

Then read your own examples once, as a stranger would. Is the input realistic enough to copy? Does the output show the fields that matter, not just the easy ones? Is there an example for every declared error code? Does the commit in source match the code you deployed? The manifest walkthrough in write a holon.yaml manifest covers the rest of the file.

Limits

Examples are exact values, so agents whose output genuinely varies, such as a summarizer, cannot express their behaviour fully in one. They still need at least one example, and it should be a real run, kept in step with the model and the prompt that produced it.