Platform API Reference

The HTTP contract of the webforai platform. All request and response bodies are JSON.

Base URL

https://platform.webforai.dev

Self-hosted instances expose the same routes under your own host.

Authentication

Every /v1 endpoint except the public demo requires an API key, created on the dashboard and sent as a bearer token.

curl -X POST https://platform.webforai.dev/v1/scrape \
  -H "Authorization: Bearer wfa_..." \
  -H "content-type: application/json" \
  -d '{ "url": "https://example.com/article" }'

Errors

Failures use the appropriate HTTP status and a uniform body:

{ "error": { "code": "payment_required", "message": "Free allowance exhausted" } }

Failed operations are never billed.

Region

region is an optional field on /v1/scrape, /v1/batch, /v1/crawl and /v1/demo/scrape. It expresses a coarse geographic preference for the egress location:

value	meaning
`auto`	let the platform choose (default)
`us`	United States
`eu`	Europe
`uk`	United Kingdom
`jp`	Japan
`asia`	Asia

It only affects the proxy engines (proxy-fetch, proxy-browser), which select a proxy exit node in the requested area. The fetch and browser engines egress from Cloudflare's own network and ignore the field rather than failing, so one default can be set for a mixed-engine workload. A region is a preference for a representative exit country, not a guarantee. Pricing is unaffected.

POST /v1/scrape

Synchronous, single URL. Runs on the plain Worker — fast, but not durable. With "async": true it instead enqueues a 1-URL job (the same shape as batch) and returns 202 { "jobId": "job_..." }.

Request

{
  "url": "https://example.com/article",
  "engine": "fetch",             // fetch | browser | proxy-fetch | proxy-browser
  "region": "auto",              // auto | us | eu | uk | jp | asia (proxy engines only)
  "screenshot": false,           // browser / proxy-browser only (400 otherwise)
  "rehostImages": false,
  "async": false,
  "convert": {                   // passthrough to webforai (all optional)
    "extractor": "auto",         // auto | takumi | minimal | none
    "frontmatter": true,
    "baseUrl": "..."
  }
}

Response — 200

{
  "url": "https://example.com/article",      // final URL after redirects
  "engine": "fetch",
  "markdown": "# ...",
  "metadata": { "title": "..." },
  "screenshotUrl": "https://...r2...", // when requested; expires
  "images": [{ "original": "https://...", "rehosted": "https://..." }],
  "credits": 1
}

Example

curl -X POST https://platform.webforai.dev/v1/scrape \
  -H "Authorization: Bearer wfa_..." \
  -H "content-type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "engine": "proxy-fetch",
    "region": "jp"
  }'

POST /v1/batch

Asynchronous, a list of URLs. Limit: 100 URLs per job.

{
  "urls": ["https://a", "https://b"],
  "engine": "fetch",
  "region": "auto",
  "screenshot": false,
  "rehostImages": false,
  "convert": {}
}

Returns 202:

{ "jobId": "job_..." }

curl -X POST https://platform.webforai.dev/v1/batch \
  -H "Authorization: Bearer wfa_..." \
  -H "content-type: application/json" \
  -d '{ "urls": ["https://example.com/a", "https://example.com/b"] }'

POST /v1/crawl

Asynchronous, recursive same-origin crawl from a seed URL.

{
  "url": "https://docs.example.com/",
  "engine": "fetch",
  "region": "auto",
  "maxDepth": 2,                 // ≤ 5
  "limit": 50,                   // pages, ≤ 500
  "includePaths": ["^/docs"],    // regex on pathname, optional
  "excludePaths": [],
  "sameOrigin": true,            // only true is accepted (sending false is a 400)
  "screenshot": false,
  "rehostImages": false,
  "convert": {}
}

Links are discovered from the <a href> elements of the fetched HTML before extraction, then normalized, deduped, fragment-stripped and same-origin filtered, and walked breadth-first by depth until limit is reached. Returns 202 { "jobId": "job_..." }.

curl -X POST https://platform.webforai.dev/v1/crawl \
  -H "Authorization: Bearer wfa_..." \
  -H "content-type: application/json" \
  -d '{ "url": "https://docs.example.com/", "maxDepth": 2, "limit": 50 }'

GET /v1/jobs/

Job status snapshot — cheap to poll.

{
  "jobId": "job_...",
  "type": "crawl",               // batch | crawl
  "status": "running",           // queued | running | completed | failed
  "total": 50,
  "completed": 12,
  "failed": 1,
  "credits": 13,
  "expiresAt": "..."
}

curl https://platform.webforai.dev/v1/jobs/job_... \
  -H "Authorization: Bearer wfa_..."

GET /v1/jobs/
/results

Paged page results (20 per page). Each item mirrors the synchronous scrape response plus a per-page status:

{
  "jobId": "job_...",
  "status": "completed",         // the job's status at read time
  "results": [
    {
      "url": "https://docs.example.com/intro",
      "status": "ok",            // ok | error
      "engine": "fetch",
      "markdown": "# ...",
      "metadata": { "title": "..." },
      "credits": 1
    },
    {
      "url": "https://docs.example.com/broken",
      "status": "error",
      "engine": "fetch",
      "error": { "code": "fetch_failed", "message": "..." }
    }
  ],
  "cursor": "..."                // absent on the last page
}

Markdown payloads larger than ~100 KiB are stored out-of-band: the item instead carries { "status": "ok", "url", "engine", "credits", "resultUrl": "<expiring URL>" } — fetch resultUrl (a plain GET, no auth header) to get the full item. The TypeScript client's jobResults iterator does this automatically. Results are retained for 7 days; resultUrl/screenshotUrl links expire after ~24 hours.

curl "https://platform.webforai.dev/v1/jobs/job_.../results?cursor=..." \
  -H "Authorization: Bearer wfa_..."

POST /v1/demo/scrape

The public, unauthenticated endpoint behind the demo on the home page. No API key, no account, no credits — it always runs the proxy-fetch engine and truncates the output.

Request

{
  "url": "https://example.com",
  "region": "auto"               // optional: auto | us | eu | uk | jp | asia
}

Response — 200

{
  "url": "https://example.com",
  "region": "auto",              // echoes the requested region
  "markdown": "# Example Domain\n...",
  "truncated": false,            // true when the markdown was cut at ~8000 characters
  "title": "Example Domain",     // when the page has one
  "metadata": { "title": "Example Domain" }
}

Response — 429

Rate limited to 5 requests per 10 minutes per IP. The response carries a Retry-After header and repeats the value in the body.

{
  "error": {
    "code": "rate_limited",
    "message": "Too many requests from this IP.",
    "retryAfter": 240
  }
}

curl -X POST https://platform.webforai.dev/v1/demo/scrape \
  -H "content-type: application/json" \
  -d '{ "url": "https://example.com", "region": "auto" }'

Behavioural rules

Billing guard runs before side effects, usage is recorded after success only. In async jobs this is per page — a failed page is not billed.
SSRF guard: public http(s) URLs only. Private IP ranges, localhost and non-standard ports are rejected at validation time for every engine, and re-checked inside the container fetcher — redirect targets included.
Rate limits: per-key requests per minute, plus the job-level caps above (≤100 URLs per batch, ≤5 crawl depth, ≤500 crawl pages). The public demo endpoint is limited per IP instead.

Base URL

Authentication

Errors

Region

POST /v1/scrape

Request

Response — 200

Example

POST /v1/batch

POST /v1/crawl

GET /v1/jobs/

GET /v1/jobs//results

POST /v1/demo/scrape

Request

Response — 200

Response — 429

Behavioural rules

GET /v1/jobs/
/results