# webforai platform

This section documents the hosted API. The product itself — sign-up, API keys, usage, billing and the playground — lives at [platform.webforai.dev](https://platform.webforai.dev).

**webforai platform** is a hosted crawl→Markdown API built on this library. It takes a URL (or a whole site) and gives you back the same clean Markdown `htmlToMarkdown` produces, without you running the browsers, proxies and queues yourself.

It is fully open source and self-hostable — the entire service is a single Cloudflare Worker in [`apps/platform`](https://github.com/inaridiy/webforai/tree/main/apps/platform) of this repository.

```
curl -X POST https://platform.webforai.dev/v1/scrape \
  -H "Authorization: Bearer wfa_..." \
  -H "content-type: application/json" \
  -d '{ "url": "https://example.com/article" }'
```

Try it without an account on the [live demo](https://webforai.dev/platform/#demo), or just put a URL after the host — `https://platform.webforai.dev/https://example.com/article` returns the page as Markdown, no key needed (rate-limited and truncated; see [Markdown permalinks](https://webforai.dev/platform/api-reference#markdown-permalinks--get-https)). The full reference is in the [API reference](https://webforai.dev/platform/api-reference).

## Quickstart

1. **Create an account** at [platform.webforai.dev](https://platform.webforai.dev/signup) with your email address. We send a 6-digit sign-in code (no password), no card is required, and 1,000 credits per month are free.
2. **Mint an API key** on the [dashboard](https://platform.webforai.dev/dashboard). Keys look like `wfa_...` and are shown once at creation.
3. **Send the first request** — with curl (above), the [TypeScript client](https://webforai.dev/platform/client), or the [CLI](https://webforai.dev/cli). The [playground](https://platform.webforai.dev/playground) runs the same request from the browser and prints the equivalent curl.

## Clients

- **TypeScript** — [`webforai/platform`](https://webforai.dev/platform/client), a subpath of the `webforai` package: typed, dependency-free, with job polling and result paging built in.

  ```ts
  import { createPlatformClient } from "webforai/platform";
   
  const platform = createPlatformClient({ apiKey: process.env.WEBFORAI_API_KEY });
  const { markdown } = await platform.scrape({ url: "https://example.com", engine: "browser" });
  ```

- **CLI** — [`npx webforai-cli`](https://webforai.dev/cli) uses the platform as a loader, and runs batch and crawl jobs straight into a folder of Markdown files:

  ```
  WEBFORAI_API_KEY=wfa_... npx webforai-cli https://example.com --engine browser --json
  WEBFORAI_API_KEY=wfa_... npx webforai-cli crawl https://docs.example.com -o docs --llms-txt
  ```

- **Anything that speaks HTTP** — the [API reference](https://webforai.dev/platform/api-reference) documents every route; authentication is one `Authorization: Bearer wfa_...` header.

## Capabilities

- **Scrape** — a single URL → Markdown + metadata. Synchronous, returns in one request.
- **Batch** — a list of URLs (≤100 per job) → Markdown per URL. Asynchronous.
- **Crawl** — a seed URL → recursive same-origin crawl with depth/page limits and include/exclude path patterns → Markdown per page. Asynchronous.

Asynchronous jobs run the exact same core logic on Cloudflare Workflows, so every page gets durable execution and retries. Results are retained for 7 days and read back through the [jobs API](https://webforai.dev/platform/api-reference#get-v1jobs).

## Engines

Every request picks how the HTML is acquired. Engines differ in what they can render, whether they egress through a proxy, and what they cost in credits.

| engine | how it fetches | proxy | screenshot | plan | credits |
| - | - | - | - | - | - |
| `auto` (default) | `fetch`, escalating to `browser` when the page needs JavaScript | only with a `region` | yes (runs `browser`) | free (subscription with a `region`) | 1–2 (2–3 with a `region`) |
| `fetch` | Workers `fetch()` | no | no | free | 1 |
| `browser` | Browser rendering | no | yes | free | 2 |
| `proxy-fetch` | Plain fetch through a rotating proxy | yes | no | subscription | 2 |
| `proxy-browser` | Browser rendering behind the rotating proxy | yes | yes | subscription | 3 |

The default `auto` starts with a plain fetch and reruns in a browser only when the fetched HTML turns out to be a client-rendered shell (or an anti-bot interstitial) — you are billed for the engine that produced the result, and the response names it. Pin `fetch` for static HTML at a guaranteed 1 credit, the `proxy-*` engines when a site blocks datacenter IPs or you need [a specific region](https://webforai.dev/platform/api-reference#region), and either browser engine when the content only exists after JavaScript runs. The proxy engines (and any `region` other than `auto`) need an active subscription — on the free allowance they return `402 payment_required`; see [Billing](https://webforai.dev/platform/billing#proxy-engines-need-a-subscription).

## Options

- `screenshot` — PNG of the rendered page, returned as an expiring URL. Browser engines only (`browser`, `proxy-browser`).
- `rehostImages` — downloads the images referenced by the resulting Markdown, re-hosts them temporarily, and rewrites the Markdown to point at those copies.
- `respectRobotsTxt` — honor the target site's robots.txt rules. Off by default for scrape and batch, on by default for crawl (send `false` to turn it off); when on, a URL the rules disallow fails with `robots_disallowed` and is not billed. See [the API reference](https://webforai.dev/platform/api-reference#robotstxt).
- `convert` — extraction options passed straight through to webforai (`extractor` preset — `auto`, `reader`, `agent`, `comments`, `kiwame`, `takumi`, `minimal` or `none` —, `frontmatter`, `baseUrl`). Results report which extractor ran and, for the learned one, its confidence.

## Accounts and pricing

Accounts and API keys live on the [dashboard](https://platform.webforai.dev/dashboard); requests authenticate with `Authorization: Bearer wfa_...`. Usage is metered in credits with **1,000 credits per month free** — past that, a subscription bills from **\$0.001 per credit** (about \$1 per 1,000 `fetch` pages), cheaper at volume, with a **\$50/month spend cap** by default. See [Billing](https://webforai.dev/platform/billing) for the full schedule. Requests are rate-limited per account (60 per minute free, 600 with a subscription) — see [Rate limits](https://webforai.dev/platform/api-reference#rate-limits).

## Self-hosting

Nothing here is proprietary. Deploy your own instance on your own Cloudflare account — D1, KV, R2, Workflows and Browser Rendering bindings, plus your own rotating-proxy and Stripe credentials. The setup steps live in the [`apps/platform` README](https://github.com/inaridiy/webforai/tree/main/apps/platform#deploying-your-own-instance).