Skip to content

TypeScript client — webforai/platform

The official typed client for the platform API ships inside the webforai package as the webforai/platform subpath — nothing extra to install. The module pulls in no runtime dependencies of its own and runs anywhere fetch exists: Node ≥18, browsers, Cloudflare Workers and other edge runtimes.

npm i webforai

Scrape

import { createPlatformClient } from "webforai/platform";
 
const platform = createPlatformClient({ apiKey: process.env.WEBFORAI_API_KEY });
 
const page = await platform.scrape({
  url: "https://example.com/article",
  engine: "browser",              // auto (default) | fetch | browser | proxy-fetch | proxy-browser
  convert: { frontmatter: true },
});
 
page.markdown;  // "# …"
page.metadata;  // { title, author, … }
page.credits;   // 2

Self-hosting? Point the client at your own deployment:

const platform = createPlatformClient({ apiKey, baseUrl: "https://platform.your.domain" });

Batch and crawl jobs

Async jobs return a jobId; waitForJob polls until the job reaches a terminal state, and jobResults iterates every page result — following pagination and downloading oversized results (which the API stores out-of-band) transparently.

const { jobId } = await platform.crawl({
  url: "https://docs.example.com/",
  maxDepth: 2,
  limit: 50,
});
 
await platform.waitForJob(jobId, {
  onStatus: (s) => console.log(`${s.status} ${s.completed}/${s.total}`),
});
 
for await (const result of platform.jobResults(jobId)) {
  if (result.status === "ok") {
    console.log(result.url, result.markdown.length);
  } else {
    console.warn(result.url, result.error.code);
  }
}

waitForJob defaults to a 10-minute overall deadline (timeoutMs, rejects with poll_timeout) and a 2-second poll interval (pollIntervalMs). Pass an AbortSignal as signal to cancel polling; the promise rejects with the signal's abort reason.

Lower-level pieces are also exposed: scrapeAsync (a single URL as an async job), getJob(jobId) and getJobResults(jobId, { cursor }) for manual polling and paging.

Error handling

Every non-2xx response throws a PlatformApiError carrying the API's error code, the HTTP status, and retryAfter (seconds) when the server provided one. The client never retries on its own.

import { PlatformApiError } from "webforai/platform";
 
try {
  await platform.scrape({ url });
} catch (error) {
  if (error instanceof PlatformApiError) {
    switch (error.code) {
      case "rate_limited":      // 60/min free, 600/min paid
      case "too_many_jobs":     // 3 (free) / 20 (paid) batch+crawl jobs already active
        await new Promise((r) => setTimeout(r, (error.retryAfter ?? 60) * 1000));
        return platform.scrape({ url }); // retry once
      case "payment_required":  // free allowance used up, or a proxy engine without a subscription
      case "spend_cap_reached": // monthly spend cap hit; raise it on the dashboard
        throw new Error(`Billing: ${error.message}`);
    }
  }
  throw error;
  }
}

401 and 402 messages end with a pointer to the deployment's dashboard (<baseUrl>/dashboard). The full list of codes, with what to do about each, lives in the API reference.

Custom fetch (Cloudflare Workers and other special runtimes)

All I/O goes through a single injectable fetch. By default the client binds the global fetch, but you can supply your own — the expected type (FetchLike) is structural, so it does not depend on lib.dom, @types/node or @cloudflare/workers-types agreeing about what fetch is:

import { createPlatformClient, type FetchLike } from "webforai/platform";
 
// Cloudflare Workers: route the client through a service binding
const platform = createPlatformClient({
  apiKey: env.WEBFORAI_API_KEY,
  fetch: (url, init) => env.PLATFORM.fetch(url, init),
});
 
// Or wrap for instrumentation / retries / a corporate proxy (undici, node-fetch, …)
const logged: FetchLike = async (url, init) => {
  console.log(init.method, url);
  return fetch(url, init);
};

Anything that accepts (url: string, init: { method, headers?, body?, signal? }) and resolves to an object with ok, status, headers.get() and json() qualifies — a plain object is fine, no Response class required. On runtimes without a global fetch, constructing a client without fetch throws a clear error instead of failing inside the first request.

Demo endpoint

The public, keyless demo endpoint is available too (rate-limited, truncated output):

const demo = await createPlatformClient().demoScrape({ url: "https://example.com" });