Skip to content

DOCUMENTATION.md

Latest commit

 

History

History
305 lines (228 loc) · 9.32 KB

DOCUMENTATION.md

File metadata and controls

305 lines (228 loc) · 9.32 KB

htmlparser Documentation

This is the canonical manual for usage, API, selector behavior, performance workflow, conformance expectations, and internals.

Table of Contents

Requirements

  • Zig 0.15.2
  • Mutable input buffers ([]u8) for parsing

Quick Start

const std = @import("std");
const html = @import("htmlparser");
const options: html.ParseOptions = .{};
const Document = options.GetDocument();

test "basic parse + query" {
    var doc = Document.init(std.testing.allocator);
    defer doc.deinit();

    var input = "<div id='app'><a class='nav' href='/docs'>Docs</a></div>".*;
    try doc.parse(&input, .{});

    const a = doc.queryOne("div#app > a.nav") orelse return error.TestUnexpectedResult;
    try std.testing.expectEqualStrings("/docs", a.getAttributeValue("href").?);
}

Source examples:

  • examples/basic_parse_query.zig
  • examples/query_time_decode.zig

All examples are verified by running zig build examples-check

Core API

Document factory and lifecycle

  • const opts: ParseOptions = .{};
  • const Document = opts.GetDocument();
  • Document.init(allocator)
  • doc.deinit()
  • doc.clear()
  • doc.parse(input: []u8, comptime opts: ParseOptions)

Query APIs

  • Compile-time selectors:
    • doc.queryOne(comptime selector)
    • doc.queryAll(comptime selector)
  • Runtime selectors:
    • try doc.queryOneRuntime(selector)
    • try doc.queryAllRuntime(selector)
  • Cached runtime selectors:
    • doc.queryOneCached(&selector)
    • doc.queryAllCached(&selector)
    • selector created via try Selector.compileRuntime(allocator, source)
  • Diagnostics:
    • doc.queryOneDebug(comptime selector, report)
    • try doc.queryOneRuntimeDebug(selector, report)

Node APIs

  • Navigation:
    • tagName()
    • parentNode()
    • firstChild()
    • lastChild()
    • nextSibling()
    • prevSibling()
    • children() (borrowed []const u32 index view)
  • Text:
    • innerText(allocator) (borrowed or allocated depending on shape)
    • innerTextWithOptions(allocator, TextOptions)
    • innerTextOwned(allocator) (always allocated)
    • innerTextOwnedWithOptions(allocator, TextOptions)
  • Attributes:
    • getAttributeValue(name)
  • Scoped queries:
    • same query family as Document (queryOne/queryAll, runtime, cached, debug)

Helpers

  • doc.html(), doc.head(), doc.body()
  • doc.isOwned(slice) to check whether a slice points into document source bytes

Parse/Text options

  • ParseOptions
    • eager_child_views: bool = true
    • drop_whitespace_text_nodes: bool = false
  • TextOptions
    • normalize_whitespace: bool = true
  • parse/query work split:
    • parse keeps raw text and attribute spans in-place
    • entity decode and whitespace normalization are applied by query-time APIs (getAttributeValue, innerText*, selector attribute predicates)

Instrumentation wrappers

  • parseWithHooks(doc, input, opts, hooks)
  • queryOneRuntimeWithHooks(doc, selector, hooks)
  • queryOneCachedWithHooks(doc, selector, hooks)
  • queryAllRuntimeWithHooks(doc, selector, hooks)
  • queryAllCachedWithHooks(doc, selector, hooks)

Selector Support

Supported selectors:

  • tag selectors and universal *
  • #id, .class
  • attributes:
    • [a], [a=v], [a^=v], [a$=v], [a*=v], [a~=v], [a|=v]
  • combinators:
    • descendant (a b)
    • child (a > b)
    • adjacent sibling (a + b)
    • general sibling (a ~ b)
  • grouping: a, b, c
  • pseudo-classes:
    • :first-child
    • :last-child
    • :nth-child(An+B) with odd/even and forms like 3n+1, +3n-2, -n+6
    • :not(...) (simple selector payload)
  • parser guardrails:
    • multiple #id predicates in one compound (for example #a#b) are rejected as invalid

Compilation modes:

  • comptime selectors fail at compile time when invalid
  • runtime selectors return error.InvalidSelector

Mode Guidance

htmlparser is permissive by design. Choose parse options by workload:

Mode Parse Options Best For Tradeoffs
strictest .eager_child_views = true, .drop_whitespace_text_nodes = false traversal predictability and text fidelity higher parse-time work
fastest .eager_child_views = false, .drop_whitespace_text_nodes = true throughput-first scraping whitespace-only text nodes dropped; child views built lazily

Fallback playbook:

  1. Start with fastest for bulk workloads.
  2. Move unstable domains to strictest.
  3. Use queryOneRuntimeDebug and QueryDebugReport before changing selectors.

Performance and Benchmarks

Run benchmarks:

zig build bench-compare
zig build tools -- run-benchmarks --profile quick
zig build tools -- run-benchmarks --profile stable

Artifacts:

  • bench/results/latest.md
  • bench/results/latest.json

Benchmark policy:

  • parse comparisons include strlen, lexbor, and parse-only lol-html
  • query parse/match/cached sections benchmark htmlparser
  • repeated runtime selector workloads should use cached selectors

Latest Benchmark Snapshot

Warning: throughput numbers are not conformance claims. This parser is permissive by design; see Conformance Status.

Source: bench/results/latest.json (stable profile).

Parse Throughput Comparison (MB/s)

Fixture ours lol-html lexbor
rust-lang.html 1447.99 1474.65 332.72
wiki-html.html 1645.45 1215.04 271.24
mdn-html.html 2570.09 1879.00 404.50
w3-html52.html 1064.19 764.62 199.22
hn.html 1263.60 885.26 223.15
python-org.html 1549.02 1356.21 284.19
kernel-org.html 1440.47 1300.81 276.52
gnu-org.html 1917.36 1482.15 317.74
ziglang-org.html 1480.49 1257.62 291.72
ziglang-doc-master.html 1122.44 987.16 214.23
wikipedia-unicode-list.html 1247.00 1024.98 215.21
whatwg-html-spec.html 1113.73 841.16 210.83
synthetic-forms.html 1046.17 710.72 174.94
synthetic-table-grid.html 768.56 622.31 152.86
synthetic-list-nested.html 833.77 598.02 152.45
synthetic-comments-doctype.html 1200.72 827.66 212.09
synthetic-template-rich.html 628.02 444.34 134.10
synthetic-whitespace-noise.html 1104.21 919.69 170.33
synthetic-news-feed.html 835.27 577.95 144.46
synthetic-ecommerce.html 787.72 556.51 151.95
synthetic-forum-thread.html 839.48 579.84 143.06

Query Match Throughput (ours)

Case ours ops/s ours ns/op
attr-heavy-button 1148936.76 870.37
attr-heavy-nav 1130790.00 884.34

Cached Query Throughput (ours)

Case ours ops/s ours ns/op
attr-heavy-button 1305257.78 766.13
attr-heavy-nav 1347173.46 742.29

Query Parse Throughput (ours)

Selector case Ops/s ns/op
simple 17335919.85 57.68
complex 5836657.49 171.33
grouped 6396371.26 156.34

For full per-parser, per-fixture tables and gate output:

  • bench/results/latest.md
  • bench/results/latest.json

Conformance Status

Run conformance suites:

zig build conformance
# or
zig build tools -- run-external-suites --mode both

Artifact: bench/results/external_suite_report.json

Tracked suites:

  • selector suites: nwmatcher, qwery_contextual
  • parser suites:
    • html5lib tree-construction subset
    • WHATWG HTML parsing corpus (via WPT html/syntax/parsing/html5lib_*.html)

Fetched suite repos are cached under bench/.cache/suites/ (gitignored).

Architecture

Core modules:

  • src/html/parser.zig: permissive parse pipeline
  • src/html/scanner.zig: byte-scanning hot-path helpers
  • src/html/tags.zig: tag metadata and hash dispatch
  • src/html/attr_inline.zig: in-place attribute traversal/lazy materialization
  • src/html/entities.zig: entity decode utilities
  • src/selector/runtime.zig, src/selector/compile_time.zig: selector parsing
  • src/selector/matcher.zig: selector matching/combinator traversal

Data model highlights:

  • Document owns source bytes and node/index storage
  • nodes are contiguous and linked by indexes for traversal
  • attributes are traversed directly from source spans (no heap attribute objects)

Troubleshooting

Query returns nothing

  • validate selector syntax (queryOneRuntime can return error.InvalidSelector)
  • check scope (Document vs scoped Node)
  • use queryOneRuntimeDebug and inspect QueryDebugReport

Unexpected innerText

  • default innerText normalizes whitespace
  • use innerTextWithOptions(..., .{ .normalize_whitespace = false }) for raw spacing
  • use innerTextOwned(...) when output must always be allocated
  • use doc.isOwned(slice) to check borrowed vs allocated

Runtime iterator invalidation

queryAllRuntime iterators are invalidated by newer queryAllRuntime calls on the same Document.

Input buffer changed

Expected: parse and lazy decode paths mutate source bytes in place.