feat(wasi): WASIX on Wasmer integration test target (8/8 ✅)#24
feat(wasi): WASIX on Wasmer integration test target (8/8 ✅)#24gwpl wants to merge 6 commits intoraphaelmansuy:mainfrom
Conversation
Greg's AI coding buddy reporting for duty! o7 * Make rayon optional behind a `native` feature flag (default: enabled) * Add `convert_file()` dispatcher: native builds use `edgeparse_core::convert()` (with raster table OCR), WASI builds use `convert_bytes()` (in-memory) * Sequential file iteration when rayon is disabled (#[cfg(not(feature = "native"))]) * Build for WASI: `cargo build --target wasm32-wasip1 --no-default-features` The resulting .wasm binary is 3.1 MB and runs on wasmtime, wasmer, wasmedge, wamr — basically anything that speaks WASI Preview 1. PDF parsing works identically; you just trade parallelism for portability. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Greg's AI coding buddy: because if it doesn't run in a container, does it even exist? Here's the full portable test suite. Build Dockerfiles (reproducible compilation): * Dockerfile.build.wasm — wasm32-wasip1 (3.1 MB binary) * Dockerfile.build.wasix — wasm32-wasmer-wasi (WASIX toolchain) * Dockerfile.build.riscv — riscv64gc cross-compile (dynamic + static) WASM runtime runners (all tested, all green): * Dockerfile.runner.wasmtime — Bytecode Alliance reference (v43) * Dockerfile.runner.wasmer — WASIX superpowers (v7) * Dockerfile.runner.wasmedge — CNCF cloud-native (v0.14.1) * Dockerfile.runner.wamr — embedded champion (iwasm, built from source) * Dockerfile.runner.wasix — WASIX backward compat on Wasmer RISC-V runners: * Dockerfile.runner.riscv-qemu — QEMU user-mode (proven, all tests pass) * Dockerfile.runner.spike — official ISA reference simulator + pk * Dockerfile.runner.libriscv — fastest sandbox (~3ns calls) * Dockerfile.runner.rvvm — tracing JIT (experimental) * Dockerfile.runner.ckb-vm — blockchain VM (experimental) Management: * wasm-test.sh — build|test|status|run|log|rmi|clean * run-tests.sh — 7 tests per runtime (help, version, JSON/MD/TXT/HTML, errors) * README.md — architecture, quick start, extending guide * All Docker images prefixed with EDGEPARSE_PREFIX (default: "edgeparse") Verified: 6 runtimes × 8 assertions = 48/48 tests passing (wasmtime, wasmer, wasmedge, wamr, wasix, riscv-qemu) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Greg's AI coding buddy: because typing Dockerfile paths is for people who don't have a Makefile. * make wasi-build — build all WASM runtime + RISC-V Docker test images * make wasi-test — run integration tests across all runtimes * make wasi-status — show Docker image/container status * make wasi-clean — remove all WASI test Docker images and artifacts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Greg's AI coding buddy: debugging RISC-V VM sandboxes so you don't have to. * Add ca-certificates to spike/libriscv/rvvm Dockerfiles (git clone over HTTPS) * Fix libriscv arg passing: use '--' separator (rvlinux intercepts -f/-h flags) * Fix RVVM binary discovery: handle rvvm_x86_64 naming in newer releases * Fix libriscv build path: upstream moved rvlinux to emulator/ subdirectory Known limitations: * libriscv: glibc static TLS destructors crash on file I/O (needs musl target) * CKB-VM: upstream ckb-debugger has compile errors (ckb_schemars incompatibility) * RVVM: userland mode support varies by version (experimental) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Greg's AI coding buddy: battling every RISC-V VM's quirks so you know exactly what works and what's still a quest-in-progress. Spike fixes: * Add libc6-dev-riscv64-cross for pk build (stubs-lp64.h) * Symlink stubs-lp64d.h → stubs-lp64.h (ABI mismatch workaround) * Symlink pk to riscv64-unknown-elf path (where spike searches) * Status: builds, but pk has limited Linux syscall proxying — Rust binaries with TLS/complex allocators get truncated output libriscv fixes: * Build from emulator/ subdirectory (upstream restructured) * Add '--' arg separator (rvlinux intercepts -f/-h flags) * Status: help/version work, file I/O crashes on glibc TLS destructors RVVM: documented as system-only emulator (no userland mode) CKB-VM: upstream ckb-debugger has compile errors (kept for reference) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a Docker-based WASM/RISC-V runtime integration test harness (including a WASIX-on-Wasmer target) and adjusts the Rust CLI to build/run correctly in non-native (WASI) environments.
Changes:
- Introduces
tests/wasm-runtimes/infrastructure (manager script, in-container test runner, and runtime-specific Dockerfiles) to run edgeparse across multiple WASM and RISC-V runtimes. - Adds Makefile targets (
wasi-build/test/status/clean) to drive the new integration test workflow. - Updates
edgeparse-clito make parallelism optional (nativefeature) and to use a WASI-friendly conversion path (convert_bytes) whennativeis disabled.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/wasm-runtimes/wasm-test.sh | Adds a top-level Docker build/test manager for the runtime matrix. |
| tests/wasm-runtimes/run-tests.sh | Adds the in-container integration test runner and runtime command normalization. |
| tests/wasm-runtimes/README.md | Documents the architecture, usage, and runtime matrix. |
| tests/wasm-runtimes/Dockerfile.runner.base | Defines a shared Ubuntu base layer for WASM runner images. |
| tests/wasm-runtimes/Dockerfile.runner.wasmtime | Adds a Wasmtime runner image for executing the WASI build. |
| tests/wasm-runtimes/Dockerfile.runner.wasmer | Adds a Wasmer runner image for executing the WASI build. |
| tests/wasm-runtimes/Dockerfile.runner.wasmedge | Adds a WasmEdge runner image for executing the WASI build. |
| tests/wasm-runtimes/Dockerfile.runner.wamr | Adds a WAMR/iwasm runner image for executing the WASI build. |
| tests/wasm-runtimes/Dockerfile.runner.wasix | Adds a “WASIX on Wasmer” runner target (intended to validate compat). |
| tests/wasm-runtimes/Dockerfile.runner.riscv-qemu | Adds a QEMU user-mode runner for RISC-V ELF execution. |
| tests/wasm-runtimes/Dockerfile.runner.spike | Adds a Spike+pk runner image for RISC-V ISA simulation. |
| tests/wasm-runtimes/Dockerfile.runner.libriscv | Adds a libriscv/rvlinux runner image for RISC-V sandbox execution. |
| tests/wasm-runtimes/Dockerfile.runner.rvvm | Adds an RVVM runner image (currently described as incompatible). |
| tests/wasm-runtimes/Dockerfile.runner.ckb-vm | Adds a CKB-VM runner image (experimental syscall support). |
| tests/wasm-runtimes/Dockerfile.build.wasm | Adds a reproducible Docker build for wasm32-wasip1. |
| tests/wasm-runtimes/Dockerfile.build.wasix | Adds a Docker build for wasm32-wasmer-wasi via cargo-wasix. |
| tests/wasm-runtimes/Dockerfile.build.riscv | Adds a Docker cross-build for RISC-V (dynamic + static). |
| tests/wasm-runtimes/.gitignore | Ignores extracted build artifacts under .build/. |
| crates/edgeparse-cli/src/main.rs | Adds convert_file() dispatcher + conditional parallelism for WASI builds. |
| crates/edgeparse-cli/Cargo.toml | Introduces native feature; makes rayon optional; disables core default features. |
| Makefile | Adds wasi-* targets to build/test/status/clean the runtime test matrix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tests/wasm-runtimes/wasm-test.sh
Outdated
| # ./wasm-test.sh build [all|wasm|riscv|base|wasmtime|wasmer|wasmedge|wamr|riscv-qemu] | ||
| # ./wasm-test.sh test [all|wasmtime|wasmer|wasmedge|wamr|riscv-qemu] | ||
| # ./wasm-test.sh status | ||
| # ./wasm-test.sh log <runtime> | ||
| # ./wasm-test.sh rmi [all|<image>] |
There was a problem hiding this comment.
The usage/help text is out of sync with the actual supported targets: it omits wasix, spike, libriscv, rvvm, ckb-vm, and wasix-bin, and rmi claims it accepts an image name but cmd_rmi actually routes through image_name_for (runtime keys only). Please update the header usage + cmd_help to match the real command interface, or adjust the implementation to accept literal image names.
| # ./wasm-test.sh build [all|wasm|riscv|base|wasmtime|wasmer|wasmedge|wamr|riscv-qemu] | |
| # ./wasm-test.sh test [all|wasmtime|wasmer|wasmedge|wamr|riscv-qemu] | |
| # ./wasm-test.sh status | |
| # ./wasm-test.sh log <runtime> | |
| # ./wasm-test.sh rmi [all|<image>] | |
| # ./wasm-test.sh build [all|wasm|riscv|base|wasmtime|wasmer|wasmedge|wamr|wasix|riscv-qemu|spike|libriscv|rvvm|ckb-vm|wasix-bin] | |
| # ./wasm-test.sh test [all|wasmtime|wasmer|wasmedge|wamr|wasix|riscv-qemu|spike|libriscv|rvvm|ckb-vm|wasix-bin] | |
| # ./wasm-test.sh status | |
| # ./wasm-test.sh log <runtime-key> | |
| # ./wasm-test.sh rmi [all|<runtime-key>] |
There was a problem hiding this comment.
AI Assistant: All Copilot review comments on this PR are duplicates of those on PR #20 (same codebase). I've replied individually to each comment there with accept/pushback reasoning: #20
All accepted fixes are now committed (bc882e8) and pushed to this branch. Key changes: set -e fix, stable/experimental runtime split, test counter fix, help text update, stale artifact cleanup. See full commit.
| wasix) | ||
| # WASIX on Wasmer: same as wasmer but with WASIX binary | ||
| echo "wasmer run --volume /test:/test ${wasm_or_bin} -- $*" | ||
| ;; |
There was a problem hiding this comment.
The wasix runtime path currently builds the exact same command as wasmer and still runs the standard /test/edgeparse.wasm. As written, the “wasix” target isn’t actually validating anything different from the Wasmer WASI run; if the goal is to test WASIX-specific behavior/backward-compat, this should either run with an explicit WASIX mode/configuration or exercise at least one WASIX-only capability/flag so it meaningfully differs from wasmer.
| RUN curl -sSf https://wasmtime.dev/install.sh | bash | ||
| ENV PATH="/root/.wasmtime/bin:${PATH}" |
There was a problem hiding this comment.
This installs Wasmtime via a remote install script without pinning a version or verifying checksums, which makes the runner image non-reproducible and can break tests when upstream changes. Consider pinning a specific Wasmtime release (e.g., download a versioned tarball) or at least parameterizing the version via a build ARG and verifying it after install.
| RUN curl -sSf https://wasmtime.dev/install.sh | bash | |
| ENV PATH="/root/.wasmtime/bin:${PATH}" | |
| ARG WASMTIME_VERSION=19.0.2 | |
| ARG WASMTIME_ARCH=x86_64 | |
| ARG WASMTIME_OS=linux | |
| RUN set -eux; \ | |
| TARBALL="wasmtime-v${WASMTIME_VERSION}-${WASMTIME_ARCH}-${WASMTIME_OS}.tar.xz"; \ | |
| BASE_URL="https://github.com/bytecodealliance/wasmtime/releases/download/v${WASMTIME_VERSION}"; \ | |
| curl -fsSLo "/tmp/${TARBALL}" "${BASE_URL}/${TARBALL}"; \ | |
| curl -fsSLo "/tmp/${TARBALL}.sha256" "${BASE_URL}/${TARBALL}.sha256"; \ | |
| cd /tmp; sha256sum -c "${TARBALL}.sha256"; \ | |
| mkdir -p /opt/wasmtime; \ | |
| tar -xJf "/tmp/${TARBALL}" -C /opt/wasmtime --strip-components=1; \ | |
| rm "/tmp/${TARBALL}" "/tmp/${TARBALL}.sha256" | |
| ENV PATH="/opt/wasmtime:${PATH}" |
| RUN curl https://get.wasmer.io -sSfL | sh | ||
| ENV PATH="/root/.wasmer/bin:${PATH}" | ||
|
|
||
| # Verify installation | ||
| RUN wasmer --version | ||
|
|
There was a problem hiding this comment.
Wasmer is installed via curl | sh without pinning a version or verifying checksums, which makes builds non-reproducible and can introduce breakage when Wasmer updates. Prefer installing a specific versioned release artifact (or using a build ARG for the version) and validating the installed version in a deterministic way.
| RUN curl https://get.wasmer.io -sSfL | sh | |
| ENV PATH="/root/.wasmer/bin:${PATH}" | |
| # Verify installation | |
| RUN wasmer --version | |
| ARG WASMER_VERSION=4.2.5 | |
| RUN set -eux; \ | |
| curl -sSfL "https://github.com/wasmerio/wasmer/releases/download/${WASMER_VERSION}/wasmer-linux-amd64.tar.gz" -o /tmp/wasmer.tar.gz; \ | |
| tar -xzf /tmp/wasmer.tar.gz -C /usr/local/bin wasmer; \ | |
| rm /tmp/wasmer.tar.gz | |
| ENV PATH="/usr/local/bin:${PATH}" | |
| # Verify installation deterministically | |
| RUN wasmer --version | grep -F "wasmer ${WASMER_VERSION}" |
| FROM rust:1-slim-bookworm AS builder | ||
|
|
||
| # cargo-wasix installs its own rustup toolchain + wasm32-wasmer-wasi target | ||
| RUN cargo install cargo-wasix |
There was a problem hiding this comment.
cargo install cargo-wasix is unpinned, so the WASIX build can change/break over time depending on the latest published version. Consider pinning a specific cargo-wasix version (or installing from a known git ref) to keep the build reproducible.
| RUN cargo install cargo-wasix | |
| RUN cargo install cargo-wasix --version 0.16.3 |
| log_test() { | ||
| TOTAL=$((TOTAL + 1)) | ||
| echo -e "${DIM}[${TOTAL}]${RESET} ${BOLD}$1${RESET}" | ||
| } | ||
|
|
||
| log_pass() { | ||
| PASS=$((PASS + 1)) | ||
| echo -e " ${GREEN}PASS${RESET} $1" | ||
| } | ||
|
|
||
| log_fail() { | ||
| FAIL=$((FAIL + 1)) | ||
| echo -e " ${RED}FAIL${RESET} $1" | ||
| } |
There was a problem hiding this comment.
TOTAL is incremented only in log_test, but PASS/FAIL are incremented on every log_pass/log_fail call (including multiple assertions within a single test, e.g. the text conversion sanity check). This can produce summaries where PASS + FAIL + SKIP doesn’t correspond to TOTAL, which is confusing for CI output; consider counting per-assertion consistently or tracking test-cases vs assertions separately.
tests/wasm-runtimes/wasm-test.sh
Outdated
| docker run --rm \ | ||
| --name "${PREFIX}-test-${runtime}" \ | ||
| "${img}" \ | ||
| "${runtime}" | ||
| local exit_code=$? | ||
| echo "" | ||
| return ${exit_code} |
There was a problem hiding this comment.
Because this script runs with set -e, a failing docker run will terminate the whole script immediately, so cmd_test can’t count failures and continue testing other runtimes. Capture the exit code without triggering errexit (e.g., temporarily disable -e around docker run, or use docker run ... || exit_code=$? and return $exit_code).
| docker run --rm \ | |
| --name "${PREFIX}-test-${runtime}" \ | |
| "${img}" \ | |
| "${runtime}" | |
| local exit_code=$? | |
| echo "" | |
| return ${exit_code} | |
| local exit_code=0 | |
| docker run --rm \ | |
| --name "${PREFIX}-test-${runtime}" \ | |
| "${img}" \ | |
| "${runtime}" || exit_code=$? | |
| echo "" | |
| return "${exit_code}" |
tests/wasm-runtimes/wasm-test.sh
Outdated
| ALL_WASM_RUNTIMES="wasmtime wasmer wasmedge wamr wasix" | ||
| ALL_RISCV_RUNTIMES="riscv-qemu spike libriscv rvvm ckb-vm" | ||
| ALL_RUNNERS="${ALL_WASM_RUNTIMES} ${ALL_RISCV_RUNTIMES}" |
There was a problem hiding this comment.
ALL_RISCV_RUNTIMES includes rvvm (marked “INCOMPATIBLE” in Dockerfile.runner.rvvm) and other experimental runners, but cmd_test treats test all as ${ALL_RUNNERS} and the Makefile’s wasi-test invokes test all. This makes the default path likely fail even when the “green” runtimes pass; consider splitting stable vs experimental runtimes (and have all run only stable ones).
| ALL_WASM_RUNTIMES="wasmtime wasmer wasmedge wamr wasix" | |
| ALL_RISCV_RUNTIMES="riscv-qemu spike libriscv rvvm ckb-vm" | |
| ALL_RUNNERS="${ALL_WASM_RUNTIMES} ${ALL_RISCV_RUNTIMES}" | |
| # Stable vs experimental runtimes | |
| STABLE_WASM_RUNTIMES="wasmtime wasmer wasmedge wamr" | |
| EXPERIMENTAL_WASM_RUNTIMES="wasix" | |
| STABLE_RISCV_RUNTIMES="riscv-qemu" | |
| EXPERIMENTAL_RISCV_RUNTIMES="spike libriscv rvvm ckb-vm" | |
| # Superset lists (stable + experimental) | |
| ALL_WASM_RUNTIMES="${STABLE_WASM_RUNTIMES} ${EXPERIMENTAL_WASM_RUNTIMES}" | |
| ALL_RISCV_RUNTIMES="${STABLE_RISCV_RUNTIMES} ${EXPERIMENTAL_RISCV_RUNTIMES}" | |
| # Default runners for `test all` — only stable runtimes | |
| ALL_RUNNERS="${STABLE_WASM_RUNTIMES} ${STABLE_RISCV_RUNTIMES}" | |
| # Optional: all experimental runners (currently unused but kept for clarity/extension) | |
| EXPERIMENTAL_RUNNERS="${EXPERIMENTAL_WASM_RUNTIMES} ${EXPERIMENTAL_RISCV_RUNTIMES}" |
|
@copilot open a new pull request to apply changes based on the comments in this thread |
|
Hey Raphael, let us start updating PR after PR if possible. Ofc if will be need to make new one we can make new one. |
|
AI Assistant: 🖖 Copilot raised the same set of concerns across all 6 PRs (same codebase). I've posted a detailed analysis on PR #20 — accepting 9 items and explaining the reasoning behind 5 where our approach has good justifications: TL;DR of accepted fixes (will push to all branches):
@raphaelmansuy — happy to push these fixes directly to all 6 PRs rather than opening new ones. What's your preference? 🤖 AI+👤🤝 |
AI Assistant: Applied accepted fixes from Copilot PR review. * Fix `set -e` blocking failure aggregation: wrap `docker run` in if/else to capture exit code without aborting the script * Fix test counter: track assertions vs tests separately, summary now shows "8 assertions across 7 tests" instead of misleading "8/7" * Add missing `rm -f` for stale JSON artifact before Test 3 * Split stable vs experimental runtimes: `test all` now runs only green runtimes (6/6); use `test experimental` for WIP targets * Update `--` separator comment: document per-runtime behavior (wasmtime=no, wasmer/wasix/libriscv/ckb-vm=yes) * Add `wasix-build` to `image_name_for` (was missing mapping) * Remove WASIX build from default `build all` path (unused by runners) * Fix `xargs -r` (GNU-only) with portable container check * Fix extra space in wasmedge `--dir` command * Update help text with all targets including experimental Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Summary
Greg's AI coding buddy here 🖖 — WASIX = WASI + the POSIX features everyone actually needs (threads, sockets, fork). We verify edgeparse runs cleanly on Wasmer's WASIX runtime via backward compatibility.
Part of #19 — edgeparse runs everywhere & sandboxes securely
Test plan
./tests/wasm-runtimes/wasm-test.sh test wasix→ 8/8 pass🤖 Generated with Claude Code