Web page to Markdown, for AI agents

What it does

Built for AI agents that need to read the web without paying for tokens of navigation and boilerplate. Keeps the title, author, date and main text of the page (Mozilla Readability, the engine of Firefox reader view), writes it as Markdown with absolute links, and counts the words. Follows up to 5 redirects. Respects robots.txt and identifies itself as HolonBot. Only public http(s) pages: private and internal addresses are refused at every connection. Pages are fetched as served, without running their scripts, so pages that only render with JavaScript may come back empty.

What it is for

Use it from your code

From Claude, once connected

> Read https://example.com/pricing and list the plans with their prices.

curl

curl https://api.useholon.com/v0/calls \
  -H "Authorization: Bearer $HOLON_AGENT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"agent": "holon-labs/web-to-text", "input": {"url": "https://example.com/"}}'

Python

import os, requests

r = requests.post("https://api.useholon.com/v0/calls",
    headers={"Authorization": f"Bearer {os.environ['HOLON_AGENT_KEY']}"},
    json={"agent": "holon-labs/web-to-text", "input": {"url": "https://example.com/"}})
page = r.json()["output"]
print(page["title"], page["word_count"])
print(page["markdown"][:500])

JavaScript

const r = await fetch("https://api.useholon.com/v0/calls", {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.HOLON_AGENT_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({ agent: "holon-labs/web-to-text", input: { url: "https://example.com/" } }),
});
const { output } = await r.json();
console.log(output.title, output.markdown.slice(0, 500));

Call it

claude mcp add --transport http holon https://api.useholon.com/mcp --header "Authorization: Bearer <agent key>"

curl https://api.useholon.com/v0/calls \
  -H "Authorization: Bearer <agent key>" \
  -H "Idempotency-Key: <any unique id>" \
  -d '{"agent":"holon-labs/web-to-text","input":{"url":"https://example.com/"}}'

Interface

{
  "type": "object",
  "additionalProperties": false,
  "required": [
    "url"
  ],
  "properties": {
    "url": {
      "type": "string",
      "maxLength": 2048,
      "pattern": "^https?://",
      "description": "A public http or https URL"
    },
    "max_chars": {
      "type": "integer",
      "minimum": 500,
      "maximum": 500000,
      "default": 100000,
      "description": "Longest Markdown returned; longer pages are truncated"
    }
  }
}

{
  "type": "object",
  "required": [
    "url",
    "final_url",
    "markdown",
    "word_count",
    "truncated"
  ],
  "properties": {
    "url": {
      "type": "string"
    },
    "final_url": {
      "type": "string",
      "description": "After redirects"
    },
    "title": {
      "type": [
        "string",
        "null"
      ]
    },
    "byline": {
      "type": [
        "string",
        "null"
      ]
    },
    "published": {
      "type": [
        "string",
        "null"
      ]
    },
    "language": {
      "type": [
        "string",
        "null"
      ]
    },
    "excerpt": {
      "type": [
        "string",
        "null"
      ]
    },
    "markdown": {
      "type": "string"
    },
    "word_count": {
      "type": "integer",
      "minimum": 0
    },
    "truncated": {
      "type": "boolean"
    }
  }
}

Declared errors

Example: A simple page

{
  "url": "https://example.com/"
}

Data and trust

Declared figures come from the author's manifest. Measured figures come from real calls and evaluations on Holon. How we measure.

Show it in your README

This badge is built from what Holon measured on this agent, and it changes on its own. It says so plainly when there is nothing measured yet.

Markdown

[![measured by Holon](https://api.useholon.com/v0/agents/holon-labs/web-to-text/badge.svg)](https://useholon.com/agents/holon-labs/web-to-text)

HTML

<a href="https://useholon.com/agents/holon-labs/web-to-text"><img src="https://api.useholon.com/v0/agents/holon-labs/web-to-text/badge.svg" alt="measured by Holon"></a>

Anyone can serve it: the image is public, cached for an hour, and loads nothing from anywhere else.

Go further

Compared with other ways to do it

Questions