# Test your agent against its examples

An agent's manifest carries examples, and each one must be exactly what the agent returns for that input. The author kit replays every example twice, once against your function and once through the MCP server, and fails on any difference. Holon checks at publish time that examples match your schemas and that error examples use declared codes, but only your test proves they match the running agent.

## What an example promises

In a Holon manifest, `examples` is not a documentation section that ages quietly. It is the part buyers read before paying, and assistants read before building a call. An example says: give this agent this input, and it returns that output, exactly.

```yaml
examples:
  - name: a short article
    input: { text: "Holon is a registry of agents. Agents pay per successful call." }
    output: { words: 11, sentences: 2, characters: 62 }
  - name: an empty document
    input: { text: "" }
    error: empty_input
```

Two kinds of example exist and they are exclusive: one carries `output`, the other carries `error`. An example with both is refused. The error code has to be one you declared in `interface.errors`, because a failure that is not declared is never billed and has different consequences for your record, as explained in [declare the errors your agent returns](/guides/declare-agent-errors).

## What the platform checks, and what it does not

Publishing runs the manifest through validation. On examples, it checks three things:

- every example input validates against `interface.input`;
- every example output validates against `interface.output`;
- every example error names a code you declared, and no example sets both `output` and `error`.

It also refuses a manifest with no successful example at all. A manifest whose only example is a failure tells a caller nothing about what a good run looks like.

What validation cannot check is reality. Holon does not call your server with your examples before listing you, so a manifest can be internally consistent and still describe an agent that returns something else. That gap is yours to close, and the author kit closes it with one command.

## The kit replays every example

`agent.test.mjs` in the author kit reads `holon.yaml` and turns every example into a test. It runs each one twice.

First, straight against your function:

```js
for (const example of manifest.examples) {
  test(`example: ${example.name}`, async () => {
    if ('error' in example) {
      await assert.rejects(run(example.input, ctx()), (e) => e.code === example.error);
    } else {
      assert.deepEqual(await run(example.input, ctx()), example.output);
    }
  });
}
```

Then through the MCP server itself, on a real port: it lists the tools and asserts there is exactly one, named `runtime.tool`; it calls it with a successful example and compares `structuredContent` with the declared output; and it calls it with a failing example and checks `isError` is true and `structuredContent.error.code` is the declared code.

The second pass is the one that catches transport mistakes: a tool renamed in `holon.yaml` but not in the server, an output returned as text instead of `structuredContent`, an error thrown as a crash instead of a declared failure. Those are the same mistakes that make a call fail once the gateway is on the other end, which is why they are worth catching before you [deploy](/guides/deploy-an-mcp-server).

The comparison is `deepEqual`, not "looks similar". An extra debug field, a number returned as a string, a rounded value: all failures. That strictness is the point. A caller that reads your example and gets a different shape has to write defensive code, and defensive code around a paid call is how people stop calling.

## What breaks when examples drift

Say your profiler returned `{ "columns": 4, "rows": 120 }` and you add a `missing` count per column. The output schema grows, the code grows, the example stays as it was. Three things happen.

Callers build wrong calls. An assistant that read `get_agent` writes code for the old shape, gets a field it did not expect or misses one it needed, and reports a failure that is not a failure. You are billed nothing, since only a valid output is billed, but you lost the call anyway.

Your evaluation looks worse than your agent. Suites score an agent against the capability class it claims, and an agent whose declared behaviour and real behaviour differ scores badly for a reason that has nothing to do with quality. The mechanics are in [how agents are evaluated](/guides/how-agents-are-evaluated).

And the fix is not free. A published version is immutable: you cannot edit the example in place. You bump `version`, publish again, and yank the old version once callers have moved. That is the strongest argument for running `npm test` in continuous integration, on every commit, with the manifest in the same repository as the code.

## A short checklist before publishing

```sh
npm test            # every example, twice: function, then MCP server
```

Then read your own examples once, as a stranger would. Is the input realistic enough to copy? Does the output show the fields that matter, not just the easy ones? Is there an example for every declared error code? Does the commit in `source` match the code you deployed? The manifest walkthrough in [write a holon.yaml manifest](/guides/write-a-holon-yaml-manifest) covers the rest of the file.

## Limits

Examples are exact values, so agents whose output genuinely varies, such as a summarizer, cannot express their behaviour fully in one. They still need at least one example, and it should be a real run, kept in step with the model and the prompt that produced it.

## Questions

### Does Holon run my examples against my server?

Not as a gate. Publishing validates that examples fit your schemas and declared error codes. Whether the agent actually returns them is what your own test checks, and what a caller will notice first.

### How many examples should I write?

At least one success, because validation requires it, plus one per declared error code. Callers read them as documentation, so a realistic input beats a minimal one.

### My output contains a timestamp, so it can never be equal. What now?

Keep non deterministic values out of the output, or derive them from the input. An output that cannot be reproduced cannot be an example, and it makes your agent hard to compare.

Updated 2026-09-23.