webforai platform

webforai platform is a hosted crawl→Markdown API built on this library. It takes a URL (or a whole site) and gives you back the same clean Markdown htmlToMarkdown produces, without you running the browsers, proxies and queues yourself.

It is fully open source and self-hostable — the entire service is a single Cloudflare Worker in apps/platform of this repository.

curl -X POST https://platform.webforai.dev/v1/scrape \
  -H "Authorization: Bearer wfa_..." \
  -H "content-type: application/json" \
  -d '{ "url": "https://example.com/article" }'

Try it without an account on the live demo, or just put a URL after the host — https://platform.webforai.dev/https://example.com/article returns the page as Markdown, no key needed (rate-limited and truncated; see Markdown permalinks). The full reference is in the API reference.

Quickstart

1. Create an account at platform.webforai.dev with your email address. We send a 6-digit sign-in code (no password), no card is required, and 1,000 credits per month are free.
2. Mint an API key on the dashboard. Keys look like wfa_... and are shown once at creation.
3. Send the first request — with curl (above), the TypeScript client, or the CLI. The playground runs the same request from the browser and prints the equivalent curl.

Clients

• TypeScript — webforai/platform, a subpath of the webforai package: typed, dependency-free, with job polling and result paging built in.

  import { createPlatformClient } from "webforai/platform";
   
  const platform = createPlatformClient({ apiKey: process.env.WEBFORAI_API_KEY });
  const { markdown } = await platform.scrape({ url: "https://example.com", engine: "browser" });

• CLI — npx webforai-cli uses the platform as a loader, and runs batch and crawl jobs straight into a folder of Markdown files:

  WEBFORAI_API_KEY=wfa_... npx webforai-cli https://example.com --engine browser --json
  WEBFORAI_API_KEY=wfa_... npx webforai-cli crawl https://docs.example.com -o docs --llms-txt

• Anything that speaks HTTP — the API reference documents every route; authentication is one Authorization: Bearer wfa_... header.

Capabilities

• Scrape — a single URL → Markdown + metadata. Synchronous, returns in one request.
• Batch — a list of URLs (≤100 per job) → Markdown per URL. Asynchronous.
• Crawl — a seed URL → recursive same-origin crawl with depth/page limits and include/exclude path patterns → Markdown per page. Asynchronous.

Asynchronous jobs run the exact same core logic on Cloudflare Workflows, so every page gets durable execution and retries. Results are retained for 7 days and read back through the jobs API.

Engines

Every request picks how the HTML is acquired. Engines differ in what they can render, whether they egress through a proxy, and what they cost in credits.

The default auto starts with a plain fetch and reruns in a browser only when the fetched HTML turns out to be a client-rendered shell (or an anti-bot interstitial) — you are billed for the engine that produced the result, and the response names it. Pin fetch for static HTML at a guaranteed 1 credit, the proxy-* engines when a site blocks datacenter IPs or you need a specific region, and either browser engine when the content only exists after JavaScript runs. The proxy engines (and any region other than auto) need an active subscription — on the free allowance they return 402 payment_required; see Billing.

Options

• screenshot — PNG of the rendered page, returned as an expiring URL. Browser engines only (browser, proxy-browser).
• rehostImages — downloads the images referenced by the resulting Markdown, re-hosts them temporarily, and rewrites the Markdown to point at those copies.
• respectRobotsTxt — honor the target site's robots.txt rules. Off by default for scrape and batch, on by default for crawl (send false to turn it off); when on, a URL the rules disallow fails with robots_disallowed and is not billed. See the API reference.
• convert — extraction options passed straight through to webforai (extractor preset — auto, reader, agent, comments, kiwame, takumi, minimal or none —, frontmatter, baseUrl). Results report which extractor ran and, for the learned one, its confidence.

Accounts and pricing

Accounts and API keys live on the dashboard; requests authenticate with Authorization: Bearer wfa_.... Usage is metered in credits with 1,000 credits per month free — past that, a subscription bills from $0.001 per credit (about $1 per 1,000 fetch pages), cheaper at volume, with a $50/month spend cap by default. See Billing for the full schedule. Requests are rate-limited per account (60 per minute free, 600 with a subscription) — see Rate limits.

Self-hosting

Nothing here is proprietary. Deploy your own instance on your own Cloudflare account — D1, KV, R2, Workflows and Browser Rendering bindings, plus your own rotating-proxy and Stripe credentials. The setup steps live in the apps/platform README.