Platform API Reference

The HTTP contract of the webforai platform. All request and response bodies are JSON.

Base URL

https://platform.webforai.dev

Self-hosted instances expose the same routes under your own host.

Authentication

Every /v1 endpoint except the public demo requires an API key, created on the dashboard and sent as a bearer token.

curl -X POST https://platform.webforai.dev/v1/scrape \
  -H "Authorization: Bearer wfa_..." \
  -H "content-type: application/json" \
  -d '{ "url": "https://example.com/article" }'

Errors

Failures use the appropriate HTTP status and a uniform body:

{
  "error": {
    "code": "payment_required",
    "message": "Free monthly allowance of 1000 credits is used up. Start a subscription at https://platform.webforai.dev/dashboard."
  }
}

Failed operations are never billed. Branch on code, not on message — messages are for people and may change.

In batch and crawl results, a failed page carries the same { code, message } pair in its error field (for example fetch_failed or robots_disallowed); the job itself still completes.

Rate limits

Limits are per account — all of an account's API keys share them. "Paid" means an active subscription.

Counters are approximate, so treat the numbers as a fairness bound rather than exact accounting. The public demo is limited per client IP instead. The CLI's crawl and batch commands retry rate_limited automatically; the TypeScript client's waitForJob waits out a 429, and other calls expose retryAfter and leave retrying to you.

Proxy engines are paid-only

proxy-fetch and proxy-browser — and anything that resolves to them, such as auto with region: "jp" — need an active subscription. Without one they fail with 402 payment_required before anything runs; the free monthly allowance covers fetch, browser and auto. The public demo never runs them.

Region

region is an optional field on /v1/scrape, /v1/batch and /v1/crawl (the demo accepts only auto). It expresses a coarse geographic preference for the egress location:

It only affects the proxy engines (proxy-fetch, proxy-browser), which select a proxy exit node in the requested area; a non-auto region also pins engine: "auto" to the proxy tier so the preference can be honoured — which makes it paid-only. The fetch and browser engines egress from Cloudflare's own network and ignore the field rather than failing, so one default can be set for a mixed-engine workload. Other values are rejected with 400 invalid_request. The region itself adds no credits; you pay for the engine that ran, so auto with a region bills at proxy prices (2–3 credits).

robots.txt

respectRobotsTxt is an optional boolean on /v1/scrape, /v1/batch and /v1/crawl (not the demo). For scrape and batch — URLs you chose — it defaults to false. For crawl — URLs the platform discovers by following links — it defaults to true; send false to crawl without reading robots.txt.

When true, the platform honors the target site's robots.txt rules. Before any engine runs, it reads <origin>/robots.txt and applies the group for the webforai-platform user-agent token, or the * group when there is none. Rules are evaluated per RFC 9309: the longest matching Allow/Disallow path wins (Allow on a tie), with * wildcards and a trailing $ anchor, matched against the path and query string.

• The check applies to every engine, including auto escalation, and to meta-refresh hop targets. HTTP redirects followed during the fetch are not re-checked.
• A disallowed URL fails with 403 robots_disallowed. In batch and crawl jobs that page is recorded as failed; either way it is not billed.
• If robots.txt cannot be read — any 4xx or 5xx status, a network error, a 5-second timeout, or a non-text response — no rules apply and the page is fetched.
• robots.txt responses are cached at Cloudflare's edge for up to an hour, so a change to the file can take that long to be seen. Crawl-delay and Sitemap lines are ignored.

POST /v1/scrape

Synchronous, single URL — fast, but not retried if the connection drops. With "async": true it instead enqueues a 1-URL job (the same shape as batch) and returns 202 { "jobId": "job_..." }.

Request

{
  "url": "https://example.com/article",
  "engine": "auto",              // auto (default) | fetch | browser | proxy-fetch | proxy-browser
  "region": "auto",              // auto | jp (proxy engines only)
  "screenshot": false,           // auto / browser / proxy-browser only (400 otherwise)
  "rehostImages": false,
  "respectRobotsTxt": false,    // opt-in; see robots.txt above
  "async": false,
  "convert": {                   // passthrough to webforai (all optional)
    "extractor": "auto",         // auto | reader | agent | comments | kiwame | takumi | minimal | none — see below
    "frontmatter": true,
    "baseUrl": "..."
  }
}

Response — 200

{
  "url": "https://example.com/article",      // final URL after redirects
  "engine": "fetch",             // the engine that ran — "auto" resolves to a concrete one
  "markdown": "# ...",
  "metadata": { "title": "..." },
  "extraction": { "extractor": "kiwame", "confidence": 0.93 }, // which extractor ran; absent with "none"
  "warning": "...",              // only when the result is probably degraded (unrendered shell)
  "screenshotUrl": "https://...r2...", // when requested; expires
  "images": [{ "original": "https://...", "rehosted": "https://..." }],
  "credits": 1
}

Extractor presets

convert.extractor picks how the main content is selected, with the same presets as the library and the CLI (see How it works):

extraction in the response names what produced the content — adapter, kiwame, or takumi (kiwame's fallback) — and, for kiwame, confidence: an estimate from 0 to 1 of how well the selection matches the page's main content (about the expected token F1). Values below about 0.5 flag pages worth a second look, or a retry with another preset.

The auto engine

auto (the default) starts with the cheapest engine that can satisfy the request — fetch, or proxy-fetch when a non-auto region is set, or a browser engine directly when a screenshot is requested. It escalates to the browser sibling in two cases: when the fetched HTML turns out to be a client-rendered shell (an empty SPA mount point, a noscript-only page, or an anti-bot interstitial), and when the plain fetch is refused in a way a real browser regularly gets past — a bot wall (403/406/429), a challenge served as 503, a Cloudflare edge 52x error, or a network-level failure. Plain origin errors (404, 500, …) are returned as errors without escalation: a browser would see the same thing. engine and credits in the response always reflect the engine that actually produced it, and only that engine is billed.

Explicitly chosen engines never substitute: a fetch/proxy-fetch scrape of a client-rendered page returns its (mostly empty) result with a top-level warning explaining that the page needs JavaScript. The same warning appears when auto could not escalate (for example on a deployment without a browser binding).

Independently of auto, every fetch-tier acquisition follows <meta http-equiv="refresh"> redirect stubs — HTTP 200 pages whose only content is "Redirecting…" — like ordinary redirects: up to 3 hops, http(s) targets only, each target passing the same SSRF guard as the requested URL, billed as one operation. The response url is the final page.

Example

curl -X POST https://platform.webforai.dev/v1/scrape \
  -H "Authorization: Bearer wfa_..." \
  -H "content-type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "engine": "proxy-fetch",
    "region": "jp"
  }'

POST /v1/batch

Asynchronous, a list of URLs. Limit: 100 URLs per job. Counts towards your concurrent job limit. From a shell, npx webforai-cli batch submits, waits and writes one Markdown file per page.

{
  "urls": ["https://a", "https://b"],
  "engine": "auto",
  "region": "auto",
  "screenshot": false,
  "rehostImages": false,
  "respectRobotsTxt": false,     // opt-in for batch; see robots.txt above
  "convert": {}
}

Returns 202:

{ "jobId": "job_..." }

curl -X POST https://platform.webforai.dev/v1/batch \
  -H "Authorization: Bearer wfa_..." \
  -H "content-type: application/json" \
  -d '{ "urls": ["https://example.com/a", "https://example.com/b"] }'

POST /v1/crawl

Asynchronous, recursive same-origin crawl from a seed URL. Counts towards your concurrent job limit. From a shell, npx webforai-cli crawl does the whole round trip and can also write llms.txt / llms-full.txt.

{
  "url": "https://docs.example.com/",
  "engine": "auto",
  "region": "auto",
  "maxDepth": 2,                 // ≤ 5
  "limit": 50,                   // pages, ≤ 500
  "includePaths": ["^/docs"],    // regex on pathname, optional
  "excludePaths": [],
  "sameOrigin": true,            // only true is accepted (sending false is a 400)
  "sitemap": "skip",             // skip (default) | include | only — see below
  "screenshot": false,
  "rehostImages": false,
  "respectRobotsTxt": true,      // default for crawl; false skips robots.txt
  "convert": {}
}

Links are discovered from the <a href> elements of the fetched HTML before extraction, then normalized, deduped, fragment-stripped and same-origin filtered, and walked breadth-first by depth until limit is reached. Returns 202 { "jobId": "job_..." }.

sitemap seeds the crawl from the site's sitemaps — the Sitemap: lines of its robots.txt, or /sitemap.xml when there are none (sitemap indexes and gzipped sitemaps are expanded, up to 8 documents). include adds their same-origin URLs that pass includePaths/excludePaths (up to limit) at depth 1, next to the links found on the seed page, so maxDepth: 0 ignores them; only crawls the seed plus those URLs and follows no links found on pages. A sitemap that cannot be read contributes nothing and the crawl carries on.

curl -X POST https://platform.webforai.dev/v1/crawl \
  -H "Authorization: Bearer wfa_..." \
  -H "content-type: application/json" \
  -d '{ "url": "https://docs.example.com/", "maxDepth": 2, "limit": 50 }'

GET /v1/jobs/

Job status snapshot — cheap to poll.

{
  "jobId": "job_...",
  "type": "crawl",               // batch | crawl (an async scrape is a 1-URL batch)
  "status": "running",           // queued | running | completed | failed
  "total": 50,
  "completed": 12,
  "failed": 1,
  "credits": 13,
  "expiresAt": "..."
}

curl https://platform.webforai.dev/v1/jobs/job_... \
  -H "Authorization: Bearer wfa_..."

GET /v1/jobs/

/results

Paged page results (20 per page). Each item mirrors the synchronous scrape response plus a per-page status:

{
  "jobId": "job_...",
  "status": "completed",         // the job's status at read time
  "results": [
    {
      "url": "https://docs.example.com/intro",
      "status": "ok",            // ok | error
      "engine": "fetch",
      "markdown": "# ...",
      "metadata": { "title": "..." },
      "credits": 1
    },
    {
      "url": "https://docs.example.com/broken",
      "status": "error",
      "engine": "fetch",
      "error": { "code": "fetch_failed", "message": "..." }
    }
  ],
  "cursor": "..."                // absent on the last page
}

Markdown payloads larger than ~100 KiB are stored out-of-band: the item instead carries { "status": "ok", "url", "engine", "credits", "resultUrl": "<expiring URL>" } — fetch resultUrl (a plain GET, no auth header) to get the full item. The TypeScript client's jobResults iterator does this automatically. Results are retained for 7 days; resultUrl/screenshotUrl links expire after ~24 hours.

curl "https://platform.webforai.dev/v1/jobs/job_.../results?cursor=..." \
  -H "Authorization: Bearer wfa_..."

POST /v1/demo/scrape

The public, unauthenticated endpoint behind the demo on the home page. No API key, no account, no credits — it always runs the auto engine and truncates the output near 40,000 characters, cutting at a paragraph boundary and appending a visible …truncated (N more characters) notice inside the markdown.

Request

{
  "url": "https://example.com",
  "region": "auto"               // optional; only "auto" — "jp" is 402 payment_required
}

Response — 200

{
  "url": "https://example.com",
  "region": "auto",              // echoes the requested region
  "engine": "fetch",             // what auto resolved to (browser when the page needed JavaScript)
  "markdown": "# Example Domain\n...",
  "truncated": false,            // true when the markdown was cut at ~40,000 characters
  "title": "Example Domain",     // when the page has one
  "metadata": { "title": "Example Domain" }
}

Response — 429

Rate limited per client IP (IPv6 clients by /64): 3 requests per minute and 5 per 10 minutes, plus a global daily cap. Successful responses are cached for 10 minutes, and a cached answer does not count. The response carries a Retry-After header and repeats the value in the body.

{
  "error": {
    "code": "rate_limited",
    "message": "Too many requests from this IP.",
    "retryAfter": 240
  }
}

curl -X POST https://platform.webforai.dev/v1/demo/scrape \
  -H "content-type: application/json" \
  -d '{ "url": "https://example.com", "region": "auto" }'

Markdown permalinks — GET /https://…

Put any http(s) URL after the host and get the page back as plain Markdown (text/markdown) — handy in a browser, curl, or a tool that only does GET.

# No key: the free demo (same limits, 10-minute cache and 40,000-character cut as above)
curl https://platform.webforai.dev/https://example.com/article
 
# With a key: a normal billed /v1/scrape with the auto engine, full length
curl -H "Authorization: Bearer wfa_..." https://platform.webforai.dev/https://example.com/article

Everything after the first /, query string included, is the target URL. Response headers say which engine ran (X-Webforai-Engine) and, for keyed requests, the credits charged (X-Webforai-Credits). Errors come back as a single code: message line with the usual status code (and Retry-After on a 429).

Behavioural rules

• You are billed only for success: a request refused for billing reasons runs nothing, and a failed page (in a job, per page) is not billed.
• Public URLs only: private IP ranges, localhost and non-standard ports are rejected with 400, for every engine and for redirect targets too.
• robots.txt: checked when respectRobotsTxt is true — the default for crawl only; see robots.txt.
• Rate limits: per account, by tier — see Rate limits. The public demo endpoint is limited per IP instead.
• Spend cap: subscribed accounts stop at their monthly spend cap (402 spend_cap_reached); proxy engines need a subscription.