# Platform API Reference The HTTP contract of the [webforai platform](https://webforai.dev/platform). All request and response bodies are JSON. ## Base URL ```txt https://platform.webforai.dev ``` Self-hosted instances expose the same routes under your own host. ## Authentication Every `/v1` endpoint except the [public demo](https://webforai.dev/platform/api-reference#post-v1demoscrape) requires an API key, created on the [dashboard](https://platform.webforai.dev/dashboard) and sent as a bearer token. ``` curl -X POST https://platform.webforai.dev/v1/scrape \ -H "Authorization: Bearer wfa_..." \ -H "content-type: application/json" \ -d '{ "url": "https://example.com/article" }' ``` ## Errors Failures use the appropriate HTTP status and a uniform body: ``` { "error": { "code": "payment_required", "message": "Free monthly allowance of 1000 credits is used up. Start a subscription at https://platform.webforai.dev/dashboard." } } ``` Failed operations are **never billed**. Branch on `code`, not on `message` — messages are for people and may change. | status | code | meaning | what to do | | - | - | - | - | | 400 | `invalid_request` | The body failed validation: unknown key, bad enum value, invalid regex, more than 100 URLs, … | Fix the request; the message names the field. | | 400 | `invalid_url` | The target is not a public `http(s)` URL (private ranges, localhost and non-standard ports are refused). | Use a public URL. | | 400 | `invalid_cursor` | A results `cursor` was altered or belongs to another job. | Pass cursors back unchanged. | | 401 | `invalid_api_key` | The `Authorization` header is missing, or the key is unknown or revoked. | Create or copy a key on the [dashboard](https://platform.webforai.dev/dashboard). | | 402 | `payment_required` | The free monthly allowance (1,000 credits) is used up, or a [proxy engine](https://webforai.dev/platform/api-reference#proxy-engines-are-paid-only) was requested without a subscription. | Subscribe on the dashboard, or use `fetch` / `browser` / `auto` without a region. | | 402 | `spend_cap_reached` | This month's estimated bill reached your [spend cap](https://webforai.dev/platform/billing#spend-cap) (default \$50). | Raise the cap on the dashboard, or wait for the next month. | | 403 | `robots_disallowed` | `respectRobotsTxt` is on and the site's robots.txt disallows the URL. | Not billed. Respect it, or (for URLs you chose) send `respectRobotsTxt: false`. | | 404 | `job_not_found` | No job with this id for your account (or its results expired after 7 days). | Check the id. | | 429 | `rate_limited` | Over your account's [request rate](https://webforai.dev/platform/api-reference#rate-limits). Carries `Retry-After`. | Wait `Retry-After` seconds, then retry; group URLs into a batch job. | | 429 | `too_many_jobs` | Your account already has its maximum of batch/crawl jobs queued or running. | Wait for one to finish, then resubmit. | | 413 / 415 | `response_too_large` / `unsupported_content_type` | The target answered with an oversized or non-HTML body. | Nothing to retry; the page cannot be converted. | | 502 / 504 | `fetch_failed` / `engine_failed` | The target site failed, refused or timed out (the message carries the upstream status). | Retry later, or try `auto` / a browser engine. | | 503 | `engine_unavailable` | The requested engine is not configured on this deployment (e.g. no proxy credentials). | Pick another engine. | | 503 | `scheduling_unknown` | A job was created but the platform could not confirm it was scheduled; the message carries the job id. | Query that job before submitting a replacement. | | 503 | `result_unavailable` | A stored result could not be read right now. | Retry the same request later. | In batch and crawl results, a failed page carries the same `{ code, message }` pair in its `error` field (for example `fetch_failed` or `robots_disallowed`); the job itself still completes. ## Rate limits Limits are per **account** — all of an account's API keys share them. "Paid" means an active subscription. | limit | free | paid | over the limit | | - | - | - | - | | `/v1` requests per minute (job status/result `GET`s are not counted) | 60 | 600 | `429 rate_limited` with `Retry-After: 60` | | batch + crawl jobs queued or running at once (async scrape included; a job silent for over an hour stops counting) | 3 | 20 | `429 too_many_jobs` | | API keys per account | 50 | 50 | key creation refused | | URLs per batch job | 100 | 100 | `400 invalid_request` | | crawl depth / pages per crawl | 5 / 500 | 5 / 500 | `400 invalid_request` | Counters are approximate, so treat the numbers as a fairness bound rather than exact accounting. The [public demo](https://webforai.dev/platform/api-reference#post-v1demoscrape) is limited per client IP instead. The [CLI](https://webforai.dev/cli)'s `crawl` and `batch` commands retry `rate_limited` automatically; the [TypeScript client](https://webforai.dev/platform/client)'s `waitForJob` waits out a 429, and other calls expose `retryAfter` and leave retrying to you. ## Proxy engines are paid-only `proxy-fetch` and `proxy-browser` — and anything that resolves to them, such as `auto` with `region: "jp"` — need an active subscription. Without one they fail with `402 payment_required` before anything runs; the free monthly allowance covers `fetch`, `browser` and `auto`. The public demo never runs them. ## Region `region` is an optional field on `/v1/scrape`, `/v1/batch` and `/v1/crawl` (the demo accepts only `auto`). It expresses a coarse geographic preference for the egress location: | value | meaning | | - | - | | `auto` | let the platform choose (default) | | `jp` | Japan | It only affects the proxy engines (`proxy-fetch`, `proxy-browser`), which select a proxy exit node in the requested area; a non-`auto` region also pins `engine: "auto"` to the proxy tier so the preference can be honoured — which makes it [paid-only](https://webforai.dev/platform/api-reference#proxy-engines-are-paid-only). The `fetch` and `browser` engines egress from Cloudflare's own network and ignore the field rather than failing, so one default can be set for a mixed-engine workload. Other values are rejected with `400 invalid_request`. The region itself adds no credits; you pay for the engine that ran, so `auto` with a region bills at proxy prices (2–3 credits). ## robots.txt `respectRobotsTxt` is an optional boolean on `/v1/scrape`, `/v1/batch` and `/v1/crawl` (not the demo). For scrape and batch — URLs you chose — it defaults to `false`. For crawl — URLs the platform discovers by following links — it defaults to `true`; send `false` to crawl without reading robots.txt. When `true`, the platform honors the target site's robots.txt rules. Before any engine runs, it reads `/robots.txt` and applies the group for the `webforai-platform` user-agent token, or the `*` group when there is none. Rules are evaluated per [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309): the longest matching `Allow`/`Disallow` path wins (`Allow` on a tie), with `*` wildcards and a trailing `$` anchor, matched against the path and query string. - The check applies to every engine, including `auto` escalation, and to meta-refresh hop targets. HTTP redirects followed during the fetch are not re-checked. - A disallowed URL fails with `403 robots_disallowed`. In batch and crawl jobs that page is recorded as failed; either way it is not billed. - If robots.txt cannot be read — any 4xx or 5xx status, a network error, a 5-second timeout, or a non-text response — no rules apply and the page is fetched. - robots.txt responses are cached at Cloudflare's edge for up to an hour, so a change to the file can take that long to be seen. `Crawl-delay` and `Sitemap` lines are ignored. ## POST /v1/scrape Synchronous, single URL — fast, but not retried if the connection drops. With `"async": true` it instead enqueues a 1-URL job (the same shape as [batch](https://webforai.dev/platform/api-reference#post-v1batch)) and returns `202 { "jobId": "job_..." }`. ### Request ``` { "url": "https://example.com/article", "engine": "auto", // auto (default) | fetch | browser | proxy-fetch | proxy-browser "region": "auto", // auto | jp (proxy engines only) "screenshot": false, // auto / browser / proxy-browser only (400 otherwise) "rehostImages": false, "respectRobotsTxt": false, // opt-in; see robots.txt above "async": false, "convert": { // passthrough to webforai (all optional) "extractor": "auto", // auto | reader | agent | comments | kiwame | takumi | minimal | none — see below "frontmatter": true, "baseUrl": "..." } } ``` ### Response — 200 ``` { "url": "https://example.com/article", // final URL after redirects "engine": "fetch", // the engine that ran — "auto" resolves to a concrete one "markdown": "# ...", "metadata": { "title": "..." }, "extraction": { "extractor": "kiwame", "confidence": 0.93 }, // which extractor ran; absent with "none" "warning": "...", // only when the result is probably degraded (unrendered shell) "screenshotUrl": "https://...r2...", // when requested; expires "images": [{ "original": "https://...", "rehosted": "https://..." }], "credits": 1 } ``` ### Extractor presets `convert.extractor` picks how the main content is selected, with the same presets as the library and the CLI (see [How it works](https://webforai.dev/how-it-works)): | Preset | Output | | - | - | | `auto` (default), `reader` | A site adapter when one applies, otherwise the learned `kiwame` classifier: the main content, as a reader view shows it. | | `agent` | The same main content, then the reader comments and the page's other links grouped by role (Related, Pagination, Section navigation, Breadcrumb, Site navigation) — for AI agents that decide where to go next. | | `comments` | The same main content, then the reader comments, without the links. | | `kiwame` | The learned classifier alone, without site adapters. | | `takumi` | The heuristic, Readability-style extractor. | | `minimal` | Removes only what is clearly not content (metadata, hidden nodes, page chrome). | | `none` | The whole page. | `extraction` in the response names what produced the content — `adapter`, `kiwame`, or `takumi` (kiwame's fallback) — and, for kiwame, `confidence`: an estimate from 0 to 1 of how well the selection matches the page's main content (about the expected token F1). Values below about 0.5 flag pages worth a second look, or a retry with another preset. ### The `auto` engine `auto` (the default) starts with the cheapest engine that can satisfy the request — `fetch`, or `proxy-fetch` when a non-`auto` [region](https://webforai.dev/platform/api-reference#region) is set, or a browser engine directly when a `screenshot` is requested. It escalates to the browser sibling in two cases: when the fetched HTML turns out to be a client-rendered shell (an empty SPA mount point, a noscript-only page, or an anti-bot interstitial), and when the plain fetch is *refused* in a way a real browser regularly gets past — a bot wall (403/406/429), a challenge served as 503, a Cloudflare edge `52x` error, or a network-level failure. Plain origin errors (404, 500, …) are returned as errors without escalation: a browser would see the same thing. `engine` and `credits` in the response always reflect the engine that actually produced it, and only that engine is billed. Explicitly chosen engines never substitute: a `fetch`/`proxy-fetch` scrape of a client-rendered page returns its (mostly empty) result with a top-level `warning` explaining that the page needs JavaScript. The same `warning` appears when `auto` could not escalate (for example on a deployment without a browser binding). Independently of `auto`, every fetch-tier acquisition follows `` redirect stubs — HTTP 200 pages whose only content is "Redirecting…" — like ordinary redirects: up to 3 hops, http(s) targets only, each target passing the same SSRF guard as the requested URL, billed as one operation. The response `url` is the final page. ### Example ``` curl -X POST https://platform.webforai.dev/v1/scrape \ -H "Authorization: Bearer wfa_..." \ -H "content-type: application/json" \ -d '{ "url": "https://example.com/article", "engine": "proxy-fetch", "region": "jp" }' ``` ## POST /v1/batch Asynchronous, a list of URLs. Limit: **100 URLs per job**. Counts towards your [concurrent job limit](https://webforai.dev/platform/api-reference#rate-limits). From a shell, [`npx webforai-cli batch`](https://webforai.dev/cli#batch-and-crawl) submits, waits and writes one Markdown file per page. ``` { "urls": ["https://a", "https://b"], "engine": "auto", "region": "auto", "screenshot": false, "rehostImages": false, "respectRobotsTxt": false, // opt-in for batch; see robots.txt above "convert": {} } ``` Returns `202`: ``` { "jobId": "job_..." } ``` ``` curl -X POST https://platform.webforai.dev/v1/batch \ -H "Authorization: Bearer wfa_..." \ -H "content-type: application/json" \ -d '{ "urls": ["https://example.com/a", "https://example.com/b"] }' ``` ## POST /v1/crawl Asynchronous, recursive same-origin crawl from a seed URL. Counts towards your [concurrent job limit](https://webforai.dev/platform/api-reference#rate-limits). From a shell, [`npx webforai-cli crawl`](https://webforai.dev/cli#batch-and-crawl) does the whole round trip and can also write `llms.txt` / `llms-full.txt`. ``` { "url": "https://docs.example.com/", "engine": "auto", "region": "auto", "maxDepth": 2, // ≤ 5 "limit": 50, // pages, ≤ 500 "includePaths": ["^/docs"], // regex on pathname, optional "excludePaths": [], "sameOrigin": true, // only true is accepted (sending false is a 400) "sitemap": "skip", // skip (default) | include | only — see below "screenshot": false, "rehostImages": false, "respectRobotsTxt": true, // default for crawl; false skips robots.txt "convert": {} } ``` Links are discovered from the `` elements of the fetched HTML *before* extraction, then normalized, deduped, fragment-stripped and same-origin filtered, and walked breadth-first by depth until `limit` is reached. Returns `202 { "jobId": "job_..." }`. `sitemap` seeds the crawl from the site's sitemaps — the `Sitemap:` lines of its robots.txt, or `/sitemap.xml` when there are none (sitemap indexes and gzipped sitemaps are expanded, up to 8 documents). `include` adds their same-origin URLs that pass `includePaths`/`excludePaths` (up to `limit`) at depth 1, next to the links found on the seed page, so `maxDepth: 0` ignores them; `only` crawls the seed plus those URLs and follows no links found on pages. A sitemap that cannot be read contributes nothing and the crawl carries on. ``` curl -X POST https://platform.webforai.dev/v1/crawl \ -H "Authorization: Bearer wfa_..." \ -H "content-type: application/json" \ -d '{ "url": "https://docs.example.com/", "maxDepth": 2, "limit": 50 }' ``` ## GET /v1/jobs/ Job status snapshot — cheap to poll. ``` { "jobId": "job_...", "type": "crawl", // batch | crawl (an async scrape is a 1-URL batch) "status": "running", // queued | running | completed | failed "total": 50, "completed": 12, "failed": 1, "credits": 13, "expiresAt": "..." } ``` ``` curl https://platform.webforai.dev/v1/jobs/job_... \ -H "Authorization: Bearer wfa_..." ``` ## GET /v1/jobs//results Paged page results (20 per page). Each item mirrors the synchronous scrape response plus a per-page status: ``` { "jobId": "job_...", "status": "completed", // the job's status at read time "results": [ { "url": "https://docs.example.com/intro", "status": "ok", // ok | error "engine": "fetch", "markdown": "# ...", "metadata": { "title": "..." }, "credits": 1 }, { "url": "https://docs.example.com/broken", "status": "error", "engine": "fetch", "error": { "code": "fetch_failed", "message": "..." } } ], "cursor": "..." // absent on the last page } ``` Markdown payloads larger than \~100 KiB are stored out-of-band: the item instead carries `{ "status": "ok", "url", "engine", "credits", "resultUrl": "" }` — fetch `resultUrl` (a plain GET, no auth header) to get the full item. The [TypeScript client](https://webforai.dev/platform/client)'s `jobResults` iterator does this automatically. Results are retained for 7 days; `resultUrl`/`screenshotUrl` links expire after \~24 hours. ``` curl "https://platform.webforai.dev/v1/jobs/job_.../results?cursor=..." \ -H "Authorization: Bearer wfa_..." ``` ## POST /v1/demo/scrape The public, unauthenticated endpoint behind the [demo on the home page](https://webforai.dev/platform/api-reference#demo). No API key, no account, no credits — it always runs the [`auto` engine](https://webforai.dev/platform/api-reference#the-auto-engine) and truncates the output near 40,000 characters, cutting at a paragraph boundary and appending a visible `…truncated (N more characters)` notice inside the markdown. ### Request ``` { "url": "https://example.com", "region": "auto" // optional; only "auto" — "jp" is 402 payment_required } ``` ### Response — 200 ``` { "url": "https://example.com", "region": "auto", // echoes the requested region "engine": "fetch", // what auto resolved to (browser when the page needed JavaScript) "markdown": "# Example Domain\n...", "truncated": false, // true when the markdown was cut at ~40,000 characters "title": "Example Domain", // when the page has one "metadata": { "title": "Example Domain" } } ``` ### Response — 429 Rate limited per client IP (IPv6 clients by /64): **3 requests per minute** and **5 per 10 minutes**, plus a global daily cap. Successful responses are cached for 10 minutes, and a cached answer does not count. The response carries a `Retry-After` header and repeats the value in the body. ``` { "error": { "code": "rate_limited", "message": "Too many requests from this IP.", "retryAfter": 240 } } ``` ``` curl -X POST https://platform.webforai.dev/v1/demo/scrape \ -H "content-type: application/json" \ -d '{ "url": "https://example.com", "region": "auto" }' ``` ## Markdown permalinks — GET /https\://… Put any http(s) URL after the host and get the page back as plain Markdown (`text/markdown`) — handy in a browser, `curl`, or a tool that only does GET. ``` # No key: the free demo (same limits, 10-minute cache and 40,000-character cut as above) curl https://platform.webforai.dev/https://example.com/article # With a key: a normal billed /v1/scrape with the auto engine, full length curl -H "Authorization: Bearer wfa_..." https://platform.webforai.dev/https://example.com/article ``` Everything after the first `/`, query string included, is the target URL. Response headers say which engine ran (`X-Webforai-Engine`) and, for keyed requests, the credits charged (`X-Webforai-Credits`). Errors come back as a single `code: message` line with the usual status code (and `Retry-After` on a 429). ## Behavioural rules - **You are billed only for success**: a request refused for billing reasons runs nothing, and a failed page (in a job, per page) is not billed. - **Public URLs only**: private IP ranges, localhost and non-standard ports are rejected with `400`, for every engine and for redirect targets too. - **robots.txt**: checked when `respectRobotsTxt` is `true` — the default for crawl only; see [robots.txt](https://webforai.dev/platform/api-reference#robotstxt). - **Rate limits**: per account, by tier — see [Rate limits](https://webforai.dev/platform/api-reference#rate-limits). The public demo endpoint is limited per IP instead. - **Spend cap**: subscribed accounts stop at their monthly [spend cap](https://webforai.dev/platform/billing#spend-cap) (`402 spend_cap_reached`); proxy engines need a subscription.