Benchmarks

This page compares webforai with common open-source HTML→Markdown pipelines, including Defuddle and Trafilatura, with a self-hosted Firecrawl and with Cloudflare's toMarkdown, on the same cached pages. The main-content comparison runs locally from the repository's evaluation harness, with no network access and no API keys at measure time; the link-following benchmark for agent mode renders live pages and asks models, as described in its section. The corpus numbers below are from the run on 2026-10-09 at commit 297b7df; the raw summary is evals/benchmarks/compare-summary.json.

Read the limitations first. The corpus is small (60 captures of 46 sites) and was assembled while developing webforai. Its quality assertions were written against webforai's output and its failures. The results are therefore likely to favour webforai, and they are not a general measure of extraction quality.

The chart summarises three columns of the tables below; its F1 axis starts at 0.5, and webforai's bar runs on in a lighter shade from the default preset's score to the comments preset's. After a new run, regenerate it (for the README and this page) with pnpm --filter site benchmark-chart; the WCEB scores are constants in site/scripts/benchmark-chart.mjs, the rest is read from the committed summary.

Pipelines

Turndown is configured with ATX headings, fenced code blocks and - bullets, the usual setup for GFM output. Defuddle runs with its defaults except useAsync: false, which stops it from calling third-party APIs (YouTube transcripts, for one) when a page has no local content — every pipeline converts the same HTML offline. Trafilatura runs in Python, so its Markdown is produced once by trafilatura and read from a cache, like the service pipelines; its time is measured in Python around the extract call. The two full-page pipelines are baselines: they show what conversion without content extraction produces. Firecrawl runs as a service, so its Markdown is produced once by firecrawl-oss and read from a cache; it is the hosted product's open-source core, not the hosted API, which may differ. Cloudflare toMarkdown also runs as a service (on Cloudflare, under the evaluator's account); its output is produced once by cf-tomarkdown and read from a cache, and since Cloudflare does not version it, the run date identifies it. Rendering was disabled so every pipeline converts identical HTML — in a browser the cached pages' scripts could rewrite the DOM.

Metrics

The metrics were fixed before the recorded run. A one-round dry run of the harness found one measurement bug (code lines rendered as <div> elements inside <pre> were joined into one line); it was fixed for all pipelines before the recorded run.

Revision note (2026-10-01): headings, code blocks and tables are now counted from a CommonMark + GFM parse of each output instead of line patterns. Adding Firecrawl showed the patterns measured syntax, not structure: they missed setext headings and indented code blocks, counted # comment lines inside code as headings, and credited tables whose rows a Markdown parser does not accept (Turndown's output for Sphinx tables). Every pipeline is measured with the new definitions; the Python-docs heading floor was corrected from 20 to its 19 real content headings.

Revision note (2026-10-11, latest): the block classifier, the page-confidence model and the agent extractor's role models were retrained on labels made with Codex (none from WCEB); the role models also cover more pages and gained a model that decides which links to keep. WCEB over pages moved from 0.892 to 0.891 with the default extraction and from 0.918 to 0.923 with the comments preset, which takes its comments from the role models. At this release the corpus checks, code blocks and tables are unchanged and webforai's empty outputs fall from 4 to 2; the corpus table below is still the 2026-10-09 run. Following links, a benchmark for agent mode, was added.

Revision note (2026-10-09): Trafilatura was added as a pipeline (corpus and WCEB), the corpus comparison was re-run for all pipelines, and webforai at 297b7df writes tables without alignment padding (median output 8,461 → 8,207 characters; no check, code or table figure changed). The WCEB table gains webforai's comments preset, which keeps reader comments as Trafilatura does by default; it was added after this comparison showed where the two differ (see below). Its comments come from the agent extractor's role models, which are not trained on WCEB.

Revision note (2026-10-09, later): Cloudflare toMarkdown was added as a pipeline (corpus and WCEB, run 2026-10-09), full-page Turndown was scored on WCEB for context, and the corpus comparison was re-run for all pipelines at the same time.

Revision note (2026-10-09): Defuddle was added as a pipeline, and the corpus and WCEB were re-run with webforai at 386ca9b, which keeps small specification tables and no longer prints screen-reader text twice — two changes adapted from Defuddle after this comparison (credited in the source and in the package's THIRD_PARTY_NOTICES.md). They add two kept tables (Amazon) and change 5 WCEB pages (mean F1 0.892067 → 0.892069). webforai's empty outputs rose from 1 to 4 between the two runs, before these changes (84e86fd gives the same 4): the Medium capture, a 404 page, is now reduced to its short not-found message. The heuristic-extractor WCEB row is from the 2026-10-01 run. The other pipelines' versions and the captures are unchanged.

• Checks passed — the existing corpus assertions in evals/src/assertions.ts, applied to every pipeline's output: minimum headings, code blocks and length; maximum share of link-only lines; and short presence/absence anchors such as Wikipedia's [edit] links. 75 checks over 22 captures. 14 further checks are webforai-specific and are excluded: 10 check which webforai site adapter claims a page, and 4 match the Markdown format that webforai's YouTube and Hacker News adapters emit (- Channel: lines, a ## Comments heading, nested blockquotes for replies). webforai passes 89/89 when they are included.
• Crashes / empty — a thrown error, or trimmed output shorter than 200 characters.
• Nav leak — captures where more than 20% of non-blank output lines are nothing but a link. This is the harness's main signal that navigation survived.
• Boilerplate — captures whose output contains any of 14 fixed page-chrome phrases ("skip to content", "privacy policy", "cookie preferences", "all rights reserved", …; the full list is in the summary JSON). These phrases can occur in real content, so this is a signal, not proof.
• Code fenced — for each <pre> in the source, its longest line of 12–200 characters must appear as a line inside a code block of the output (fenced or indented). Pooled over all 748 such blocks. It measures recall only: keeping code from page chrome is not penalized.
• Tables — source tables with a <th> and no nested table, against tables a GFM parser finds in the output (capped per capture). Pooled over 64 tables. Recall only.
• Time — one warm-up round, then five measured rounds. Each conversion is timed on its own, pipelines are interleaved per capture and the starting pipeline rotates each round. The figure is the sum over all captures of each capture's median, including HTML parsing. Node 24.15.0 on an AMD Ryzen 9 8945HX, not CPU-pinned. Firecrawl's figure is the wall time of its scrape requests (8 concurrent), including its queue and fetching the locally served page — a service latency, not comparable with in-process conversion.

Results

60 captures, 24.17 MiB of HTML. Nav leak, boilerplate and empty count captures out of 60. Times were measured with a load average of about 3 (recorded in the summary); Trafilatura's is the sum of each page's median over 5 timed extract calls in Python, run alone, one page at a time; pipelines are interleaved per capture, so compare them as ratios. Firecrawl's figure is unchanged from the 2026-10-01 run, because its outputs are read from the cache that run produced. Cloudflare toMarkdown's is the sum of each page's toMarkdown call (6 concurrent), including the round trip from this machine to Cloudflare; like Firecrawl's, it is a service latency.

What the numbers show

• Where webforai is behind. Full-page Turndown is about 2.5× faster, because it does no extraction, and Trafilatura, timed in Python in a separate run, is faster too. Firecrawl and node-html-markdown keep the most tables: they remove little or nothing, so Wikipedia's navboxes and sidebars (navigation tables built from <th> cells; 30 on the Japanese capture) count as kept. Most of the tables webforai drops are those, plus a PostgreSQL navigation header, GitHub's repository file list and a two-row table on the Amazon product page; it keeps the infoboxes and article tables. One table is laid out as blocks on purpose: Wikipedia's Markdown-versus-HTML comparison, whose cells are multi-line code samples, becomes labelled code blocks. webforai leaks navigation on 5 captures, all news, index or gated pages (the BBC and The Verge front pages, NHK, ICS MEDIA).
• Defuddle. It keeps more tables than webforai (35 against 25), but mostly the same Wikipedia navboxes: 14 of the 33 on the Japanese capture, against webforai's 3, where only 2 are article tables. Its two failed checks are link-only-line ratios on GIGAZINE and Japanese Wikipedia, and navigation leaks on 8 captures — the Medium capture (a 404 page) comes out whole, inline SVG logo included, and on NHK the related-article lists make the output about 25 times longer than webforai's. It normalises bare <pre> into code blocks, so it fences far more code than the Turndown pipelines; most of its misses are on the TypeScript handbook, Drizzle, viem and the English Wikipedia's code samples. It prints a stack trace to the console for a page whose schema.org JSON is malformed (the CNN.co.jp capture), without failing.
• Trafilatura. It is the most conservative extractor here: no capture leaks navigation and its median output is the shortest (5,630 characters). Its 11 failed checks are mostly content it leaves out: code blocks on Tailwind, the TypeScript handbook and Overreacted, headings on Tailwind and Kubernetes, most of the shadcn/ui page and of Japanese Wikipedia, and the page title as an # heading on YouTube and Hacker News. It fences 40.6% of the code probes, the fewest of the extracting pipelines; links and images are dropped by default.
• Firecrawl OSS. Its main-content step removes a fixed list of selectors (header, nav, .sidebar, .ad, …), so pages whose chrome uses other names keep it: navigation leaks on 15 captures and page-chrome phrases appear on 22. Code is emitted as indented blocks, which the parse counts; on CodeMirror-style pages (React, Sandpack) each block's lines are joined into one, so those probes are missed. The one crash is the Reddit capture (a client-rendered shell), which its fetch engine reported as a failure.
• Cloudflare toMarkdown. It removes <header> and <footer> but nothing else, so it behaves like the full-page baselines with less chrome: navigation leaks on 13 captures and page-chrome phrases on 22. Of its 13 failed checks, 6 are link-only-line ratios (shadcn/ui, Hatena, the Cloudflare blog, WordPress, GIGAZINE, English Wikipedia), 3 are Wikipedia chrome left in ("Jump to content", [edit], "Retrieved from"), 3 are missing headings on YouTube and Hacker News (the page title goes into the front matter, not an # heading) and 1 is the TypeScript handbook's code-block count. It fences code about as often as the Turndown pipelines (53.6%) and keeps the same share of tables as node-html-markdown, navboxes included. Every page starts with front matter built from the <meta> tags and ends with the page's JSON-LD, if any.
• Checks. Readability + Turndown misses on Wikipedia (heading count, [edit] links), on YouTube's watch page (no output), on heading counts for the Python docs, Substack and Hacker News pages, and on link-only-line ratios for two documentation pages (shadcn/ui, TypeScript handbook). The full-page pipelines fail mainly on the link-only-line ratio, as expected without extraction.
• Code. Turndown only fences a <pre> that contains a <code> element. Python, Go and PostgreSQL documentation use bare <pre>, so every Turndown-based pipeline scores 0 on those pages. Readability also drops some code blocks before conversion (Next.js, Prisma). On the React docs every Turndown-based pipeline keeps only 2 of 23 blocks. webforai's remaining misses are mostly on the TypeScript handbook. Several of the handbook's probe lines exist only when a compiler error box, its hidden duplicate and the "Try" link are run together into one line; webforai writes the error as a // error TS2345: … line instead, which the probe does not match. The probe also treats every <div> inside <pre> as a line break, so on Vite and Hono it picks a fragment of a Twoslash hover card, which webforai keeps on the token's line.
• Robustness. Turndown threw on one capture (Amazon, rendered). Every pipeline produced empty output for the Reddit capture, which is a client-rendered shell; Readability and Defuddle also returned nothing for the YouTube watch page (Defuddle's YouTube extractor needs the network, which is off here). webforai's other two empty outputs are the Medium capture, a 404 page, which it reduces to the page's 188-character not-found message.
• Speed. Readability's time includes building a jsdom document, Defuddle's a linkedom one. node-html-markdown is slow on the largest pages (about 5 s for Japanese Wikipedia alone).

Against reference text

The checks above are proxies. The evaluation harness also scores extraction against reference main-content text — token precision, recall and F1 per page, averaged — on WCEB (Bevendorff et al., SIGIR 2023): 3,985 pages from eight published datasets, mostly news from 2007–2017. No model in webforai is trained on WCEB.

Readability + Turndown is ahead on three of the eight WCEB datasets — its own test set (F1 0.98 against webforai's 0.95), Scrapinghub and L3S-GN1 — and webforai on the other five. Of the pipelines that extract main content, Firecrawl recovers the most of the reference text but keeps the most besides it: its precision is the lowest of them on every dataset. Defuddle is level with webforai on Scrapinghub (0.920 against 0.920) and behind it on the other seven, by most on L3S-GN1 (0.716 against 0.923) and CleanEval (0.784 against 0.914). It scores more than 0.2 above webforai on 89 pages and more than 0.2 below on 430. Of the 89, most are wins on recall and 46 are Dragnet pages; the ones inspected are pages where Defuddle keeps the whole page and the reference counts more than the article as content — reader comments, which Dragnet's references include, or a page that is mostly a list of links. Defuddle is designed for a browser, where it also reads the page's mobile stylesheet to find clutter; under linkedom there are no stylesheets, so that signal is absent here.

Trafilatura is the strongest of the other pipelines on WCEB, and with webforai's default extraction the comparison is split: Trafilatura is ahead on the mean over pages (0.900 against 0.891; paired bootstrap, stratified by dataset, webforai − Trafilatura −0.009 [−0.015, −0.003]) and behind on the mean over datasets (0.902 against 0.910; +0.007 [+0.001, +0.013]). The difference over pages is Dragnet, the largest dataset (1,379 pages), whose references include reader comments: Trafilatura keeps comments by default, webforai's default does not. On the 227 Dragnet pages where Trafilatura scores more than 0.2 higher, webforai's recall is 0.32 against Trafilatura's 0.88, and nearly all of their references contain comment text. With the comments preset webforai keeps them too and leads on both means: 0.923 against 0.900 over pages (+0.023 [+0.019, +0.027]) and 0.919 against 0.902 over datasets (+0.016 [+0.011, +0.022]). It also gains on CETD (0.918 → 0.952), whose references keep comments too, and gives up a little where they do not (L3S-GN1 0.923 → 0.916, Readability's set 0.950 → 0.943). With the preset webforai is ahead of Trafilatura on seven datasets and behind on Scrapinghub (0.911 against 0.948).

Cloudflare toMarkdown converts nearly all of the page, so it has the highest recall in the table (0.985) and, after full-page Turndown, the lowest precision (0.604). It removes the header and footer, which puts it above full-page Turndown on every dataset, but below Firecrawl OSS on every dataset and below webforai on all eight; it scores more than 0.2 above webforai on 230 pages and more than 0.2 below on 1,765. Its output also starts with the page metadata as front matter and ends with any JSON-LD, which the reference text does not contain. Full-page Turndown threw on 84 WCEB pages, scored as empty.

Following links (agent mode)

The comparisons above score what a pipeline keeps of an article. An agent that browses also needs the page's links: the related article, the next part, the menu entry that leads to the answer. The agent preset (agentExtractor) writes the default output and then the page's other links grouped by role (related, pagination, section and site navigation, breadcrumbs), leaving out the links its role models classify as boilerplate. This section measures whether a model can still find the link it needs in that output, on sites set aside from development, and at what size.

Set. related-hops-2, built on 2026-10-10 and never used to choose a change: 290 tasks on 290 English sites. Each task is a start page from Common Crawl (CC-NEWS for August 2026, and blog, news and documentation paths of CC-MAIN-2026-34), one same-site link on it — a related article (199), the previous or next article (86), a "read next" (3) or a part of a series (2) — and a question answered by a fact on the linked page. The link was picked by Sonnet 5.5 from the page's list of anchors with each anchor's enclosing elements, not from any pipeline's output. Sonnet 5.5 wrote the question; Opus 5.5 then checked, independently, that the linked page supports the answer and that the start page, teasers and link texts included, does not answer it. Its sites are kept apart from the pages agent mode's role models were trained on, from WebWalkerQA (below) and from a first batch built the same way (related-hops, now 279 tasks), which became a development set when a bug was traced on it. Five tasks were removed after its first measurement — three on a site that a redirect also put in the first batch, two about private people — and the numbers below are for the remaining 290, measured again at the release. Pages were rendered in headless Chromium, and every pipeline converts the same rendered HTML. The task list — URLs, the link, the question and the answer — is committed as evals/benchmarks/related-hops-2/tasks.jsonl; the pages are not.

• Link kept — the output links to the target page (inline, reference or autolink; URLs normalised).
• Link chosen — a model gets the question, the page's URL and one pipeline's output, and answers with the one URL it would open next (structured output, no tools). The answer is correct when it is the target and is a link in the output: a model may not navigate from memory. Haiku 5.5 and Sonnet 5.5 through claude -p with a fixed system prompt, three repetitions each; the table gives the mean.
• Size — median o200k_base tokens of the output, per page.

webforai's outputs were produced at commit a90c29a. "webforai + every link" is webforai's default output followed by every other link of the page in document order: the simplest way to give an agent all the links, and the baseline the role models have to beat.

• The same choices in fewer tokens. With the agent output, models choose the right link as often as with every link or the whole page converted: agent − every link +1.3 points [−0.2, +3.0] (Haiku) and −0.3 [−1.7, +1.1] (Sonnet); agent − node-html-markdown +1.1 and −0.3, intervals across zero (paired, 95%, bootstrapped over sites). The output is 16% smaller than listing every link and about 45% smaller than converting the whole page.
• Grouping does not make the choice easier. The same links without the role headings score the same (agent − without headings +0.0 [−0.9, +0.8] and −1.1 [−2.9, +0.3]). What agent mode adds is size, not accuracy.
• Reader-mode extraction cannot do this task. Readability, Defuddle, Trafilatura and webforai's default preset keep the article and drop the rails around it, so the link is usually gone — as they are designed to. An agent that browses needs links on top of the article; the question is only how many.

WebWalkerQA (development set). WebWalkerQA (Wu et al., 2025) gives click paths through conference, society and organisation sites. Followed on the live sites, 249 clicks of its English tasks resolve, mostly through site and section menus. The agent output keeps 98.8% of the clicked links (every link 100%, node-html-markdown 94.0%) at 2,770 median tokens against 2,994 for every link — 7% fewer: these sites carry few boilerplate links. This set and the first related-hops batch were used to find agent mode's missing links and to choose its operating point, so they flatter it. The model choice on WebWalkerQA, measured at 3e9e429 before the last two changes, was 0.5–2.9 points below every link, with intervals reaching zero.

Limitations

• Selection bias. The corpus URLs were chosen while developing webforai, and many assertions pin regressions webforai once had. A corpus chosen by someone else would likely give different results.
• Small sample. 46 sites; static and rendered captures of the same site are correlated, not independent. Pooled code and table figures are dominated by a few large pages (Effective Go alone has 151 of the 748 code blocks).
• Proxies, not ground truth. None of the metrics compares against a reference document, and there is no human or model judgement. Recall metrics reward keeping everything; leak metrics reward removing things. Output length is reported, not scored: shorter is not automatically better.
• Configuration. Each competitor runs with a typical, untuned configuration. Different Turndown rules (for example a rule for bare <pre>), a different DOM implementation or a Readability fallback would change their numbers. Defuddle on jsdom instead of linkedom scored the same on a 20-page WCEB sample (F1 0.917 against 0.915) at about four times the time.
• Stale captures. Captures are cached snapshots and may differ from the live sites. Page HTML is not committed (third-party content), so reproducing the exact numbers requires the same captures; the corpus fingerprint in the summary identifies them.
• Link tasks. The link and the question of each related-hops-2 task are written and checked by models, not people; the tasks are English and one click deep; and start pages were kept only if they have a same-site related or sequence link, so pages without one are not represented. Link choice measures one decision, not a whole browsing session. The pages are live sites: re-rendering them later gives slightly different pages, and the "every link" row shows how many targets are still linked.
• Timing scope. Warmed, single-process, local conversion only. Trafilatura's time is measured in Python, in a separate run, so it is comparable as an order of magnitude, not as a ratio. No network, browser rendering, cold start or memory measurement. The host was not isolated.

Reproduce

pnpm install
pnpm --filter @webforai/evals corpus:fetch   # populate evals/.cache (not committed)
pnpm --filter @webforai/evals bench:compare  # add -- --rounds=7 or -- --no-summary as needed

The two service pipelines are read from caches that are filled first. Firecrawl OSS needs a self-hosted Firecrawl (firecrawl-oss, see evals/README.md); Cloudflare toMarkdown needs wrangler login and a helper Worker run locally, whose AI binding calls Cloudflare:

pnpm --filter platform exec wrangler dev --config ../../evals/workers/cf-tomarkdown/wrangler.jsonc --port 8799
CF_TOMARKDOWN_URL=http://localhost:8799 pnpm --filter @webforai/evals cf-tomarkdown -- --sets=corpus,wceb

The link benchmark renders the task list's start pages again (through the proxy pool configured in .env; --direct fetches without one), lists the links each pipeline keeps, then asks the models through the local claude CLI:

pnpm --filter @webforai/evals exec tsx src/cli/related-hops-build.ts --set=related-hops-2 --stage=fetch
pnpm --filter @webforai/evals webwalker:links -- --set=related-hops-2
pnpm --filter @webforai/evals webwalker:choose -- --set=related-hops-2 --models=haiku,sonnet --reps=3
pnpm --filter @webforai/evals webwalker:choose -- --set=related-hops-2 --models=haiku,sonnet --reps=3 --report

The run writes the full per-capture results, a Markdown report and every pipeline's output to evals/.reports/<date>-compare/ (gitignored), and rewrites the committed aggregate evals/benchmarks/compare-summary.json. The metric definitions live in evals/src/compare-metrics.ts and the pipelines in evals/src/competitors.ts. jsdom 30 requires Node 22.22.2+ or 24.15.0+.