Skip to content

feat(wasi): WASIX on Wasmer integration test target (8/8 ✅)#24

Open
gwpl wants to merge 6 commits intoraphaelmansuy:mainfrom
VariousForks:i19-target-wasix
Open

feat(wasi): WASIX on Wasmer integration test target (8/8 ✅)#24
gwpl wants to merge 6 commits intoraphaelmansuy:mainfrom
VariousForks:i19-target-wasix

Conversation

@gwpl
Copy link
Copy Markdown

@gwpl gwpl commented Mar 25, 2026

Summary

Greg's AI coding buddy here 🖖 — WASIX = WASI + the POSIX features everyone actually needs (threads, sockets, fork). We verify edgeparse runs cleanly on Wasmer's WASIX runtime via backward compatibility.

  • Tests the standard wasm32-wasip1 binary on Wasmer's WASIX runtime
  • Validates backward compatibility: WASI p1 binaries run on WASIX without recompilation
  • 8/8 tests passing: POSIX runtime treats our WASI binary just fine

Part of #19 — edgeparse runs everywhere & sandboxes securely

Test plan

  • ./tests/wasm-runtimes/wasm-test.sh test wasix → 8/8 pass

🤖 Generated with Claude Code

gwpl and others added 5 commits March 25, 2026 08:45
Greg's AI coding buddy reporting for duty! o7

* Make rayon optional behind a `native` feature flag (default: enabled)
* Add `convert_file()` dispatcher: native builds use `edgeparse_core::convert()`
  (with raster table OCR), WASI builds use `convert_bytes()` (in-memory)
* Sequential file iteration when rayon is disabled (#[cfg(not(feature = "native"))])
* Build for WASI: `cargo build --target wasm32-wasip1 --no-default-features`

The resulting .wasm binary is 3.1 MB and runs on wasmtime, wasmer,
wasmedge, wamr — basically anything that speaks WASI Preview 1.
PDF parsing works identically; you just trade parallelism for portability.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Greg's AI coding buddy: because if it doesn't run in a container,
does it even exist? Here's the full portable test suite.

Build Dockerfiles (reproducible compilation):
* Dockerfile.build.wasm   — wasm32-wasip1 (3.1 MB binary)
* Dockerfile.build.wasix  — wasm32-wasmer-wasi (WASIX toolchain)
* Dockerfile.build.riscv  — riscv64gc cross-compile (dynamic + static)

WASM runtime runners (all tested, all green):
* Dockerfile.runner.wasmtime  — Bytecode Alliance reference (v43)
* Dockerfile.runner.wasmer    — WASIX superpowers (v7)
* Dockerfile.runner.wasmedge  — CNCF cloud-native (v0.14.1)
* Dockerfile.runner.wamr      — embedded champion (iwasm, built from source)
* Dockerfile.runner.wasix     — WASIX backward compat on Wasmer

RISC-V runners:
* Dockerfile.runner.riscv-qemu — QEMU user-mode (proven, all tests pass)
* Dockerfile.runner.spike      — official ISA reference simulator + pk
* Dockerfile.runner.libriscv   — fastest sandbox (~3ns calls)
* Dockerfile.runner.rvvm       — tracing JIT (experimental)
* Dockerfile.runner.ckb-vm     — blockchain VM (experimental)

Management:
* wasm-test.sh  — build|test|status|run|log|rmi|clean
* run-tests.sh  — 7 tests per runtime (help, version, JSON/MD/TXT/HTML, errors)
* README.md     — architecture, quick start, extending guide
* All Docker images prefixed with EDGEPARSE_PREFIX (default: "edgeparse")

Verified: 6 runtimes × 8 assertions = 48/48 tests passing
(wasmtime, wasmer, wasmedge, wamr, wasix, riscv-qemu)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Greg's AI coding buddy: because typing Dockerfile paths is for people
who don't have a Makefile.

* make wasi-build   — build all WASM runtime + RISC-V Docker test images
* make wasi-test    — run integration tests across all runtimes
* make wasi-status  — show Docker image/container status
* make wasi-clean   — remove all WASI test Docker images and artifacts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Greg's AI coding buddy: debugging RISC-V VM sandboxes so you don't have to.

* Add ca-certificates to spike/libriscv/rvvm Dockerfiles (git clone over HTTPS)
* Fix libriscv arg passing: use '--' separator (rvlinux intercepts -f/-h flags)
* Fix RVVM binary discovery: handle rvvm_x86_64 naming in newer releases
* Fix libriscv build path: upstream moved rvlinux to emulator/ subdirectory

Known limitations:
* libriscv: glibc static TLS destructors crash on file I/O (needs musl target)
* CKB-VM: upstream ckb-debugger has compile errors (ckb_schemars incompatibility)
* RVVM: userland mode support varies by version (experimental)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Greg's AI coding buddy: battling every RISC-V VM's quirks so you
know exactly what works and what's still a quest-in-progress.

Spike fixes:
* Add libc6-dev-riscv64-cross for pk build (stubs-lp64.h)
* Symlink stubs-lp64d.h → stubs-lp64.h (ABI mismatch workaround)
* Symlink pk to riscv64-unknown-elf path (where spike searches)
* Status: builds, but pk has limited Linux syscall proxying — Rust
  binaries with TLS/complex allocators get truncated output

libriscv fixes:
* Build from emulator/ subdirectory (upstream restructured)
* Add '--' arg separator (rvlinux intercepts -f/-h flags)
* Status: help/version work, file I/O crashes on glibc TLS destructors

RVVM: documented as system-only emulator (no userland mode)
CKB-VM: upstream ckb-debugger has compile errors (kept for reference)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a Docker-based WASM/RISC-V runtime integration test harness (including a WASIX-on-Wasmer target) and adjusts the Rust CLI to build/run correctly in non-native (WASI) environments.

Changes:

  • Introduces tests/wasm-runtimes/ infrastructure (manager script, in-container test runner, and runtime-specific Dockerfiles) to run edgeparse across multiple WASM and RISC-V runtimes.
  • Adds Makefile targets (wasi-build/test/status/clean) to drive the new integration test workflow.
  • Updates edgeparse-cli to make parallelism optional (native feature) and to use a WASI-friendly conversion path (convert_bytes) when native is disabled.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
tests/wasm-runtimes/wasm-test.sh Adds a top-level Docker build/test manager for the runtime matrix.
tests/wasm-runtimes/run-tests.sh Adds the in-container integration test runner and runtime command normalization.
tests/wasm-runtimes/README.md Documents the architecture, usage, and runtime matrix.
tests/wasm-runtimes/Dockerfile.runner.base Defines a shared Ubuntu base layer for WASM runner images.
tests/wasm-runtimes/Dockerfile.runner.wasmtime Adds a Wasmtime runner image for executing the WASI build.
tests/wasm-runtimes/Dockerfile.runner.wasmer Adds a Wasmer runner image for executing the WASI build.
tests/wasm-runtimes/Dockerfile.runner.wasmedge Adds a WasmEdge runner image for executing the WASI build.
tests/wasm-runtimes/Dockerfile.runner.wamr Adds a WAMR/iwasm runner image for executing the WASI build.
tests/wasm-runtimes/Dockerfile.runner.wasix Adds a “WASIX on Wasmer” runner target (intended to validate compat).
tests/wasm-runtimes/Dockerfile.runner.riscv-qemu Adds a QEMU user-mode runner for RISC-V ELF execution.
tests/wasm-runtimes/Dockerfile.runner.spike Adds a Spike+pk runner image for RISC-V ISA simulation.
tests/wasm-runtimes/Dockerfile.runner.libriscv Adds a libriscv/rvlinux runner image for RISC-V sandbox execution.
tests/wasm-runtimes/Dockerfile.runner.rvvm Adds an RVVM runner image (currently described as incompatible).
tests/wasm-runtimes/Dockerfile.runner.ckb-vm Adds a CKB-VM runner image (experimental syscall support).
tests/wasm-runtimes/Dockerfile.build.wasm Adds a reproducible Docker build for wasm32-wasip1.
tests/wasm-runtimes/Dockerfile.build.wasix Adds a Docker build for wasm32-wasmer-wasi via cargo-wasix.
tests/wasm-runtimes/Dockerfile.build.riscv Adds a Docker cross-build for RISC-V (dynamic + static).
tests/wasm-runtimes/.gitignore Ignores extracted build artifacts under .build/.
crates/edgeparse-cli/src/main.rs Adds convert_file() dispatcher + conditional parallelism for WASI builds.
crates/edgeparse-cli/Cargo.toml Introduces native feature; makes rayon optional; disables core default features.
Makefile Adds wasi-* targets to build/test/status/clean the runtime test matrix.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +14 to +18
# ./wasm-test.sh build [all|wasm|riscv|base|wasmtime|wasmer|wasmedge|wamr|riscv-qemu]
# ./wasm-test.sh test [all|wasmtime|wasmer|wasmedge|wamr|riscv-qemu]
# ./wasm-test.sh status
# ./wasm-test.sh log <runtime>
# ./wasm-test.sh rmi [all|<image>]
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The usage/help text is out of sync with the actual supported targets: it omits wasix, spike, libriscv, rvvm, ckb-vm, and wasix-bin, and rmi claims it accepts an image name but cmd_rmi actually routes through image_name_for (runtime keys only). Please update the header usage + cmd_help to match the real command interface, or adjust the implementation to accept literal image names.

Suggested change
# ./wasm-test.sh build [all|wasm|riscv|base|wasmtime|wasmer|wasmedge|wamr|riscv-qemu]
# ./wasm-test.sh test [all|wasmtime|wasmer|wasmedge|wamr|riscv-qemu]
# ./wasm-test.sh status
# ./wasm-test.sh log <runtime>
# ./wasm-test.sh rmi [all|<image>]
# ./wasm-test.sh build [all|wasm|riscv|base|wasmtime|wasmer|wasmedge|wamr|wasix|riscv-qemu|spike|libriscv|rvvm|ckb-vm|wasix-bin]
# ./wasm-test.sh test [all|wasmtime|wasmer|wasmedge|wamr|wasix|riscv-qemu|spike|libriscv|rvvm|ckb-vm|wasix-bin]
# ./wasm-test.sh status
# ./wasm-test.sh log <runtime-key>
# ./wasm-test.sh rmi [all|<runtime-key>]

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Assistant: All Copilot review comments on this PR are duplicates of those on PR #20 (same codebase). I've replied individually to each comment there with accept/pushback reasoning: #20

All accepted fixes are now committed (bc882e8) and pushed to this branch. Key changes: set -e fix, stable/experimental runtime split, test counter fix, help text update, stale artifact cleanup. See full commit.

Comment on lines +93 to +96
wasix)
# WASIX on Wasmer: same as wasmer but with WASIX binary
echo "wasmer run --volume /test:/test ${wasm_or_bin} -- $*"
;;
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wasix runtime path currently builds the exact same command as wasmer and still runs the standard /test/edgeparse.wasm. As written, the “wasix” target isn’t actually validating anything different from the Wasmer WASI run; if the goal is to test WASIX-specific behavior/backward-compat, this should either run with an explicit WASIX mode/configuration or exercise at least one WASIX-only capability/flag so it meaningfully differs from wasmer.

Copilot uses AI. Check for mistakes.
Comment on lines +13 to +14
RUN curl -sSf https://wasmtime.dev/install.sh | bash
ENV PATH="/root/.wasmtime/bin:${PATH}"
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This installs Wasmtime via a remote install script without pinning a version or verifying checksums, which makes the runner image non-reproducible and can break tests when upstream changes. Consider pinning a specific Wasmtime release (e.g., download a versioned tarball) or at least parameterizing the version via a build ARG and verifying it after install.

Suggested change
RUN curl -sSf https://wasmtime.dev/install.sh | bash
ENV PATH="/root/.wasmtime/bin:${PATH}"
ARG WASMTIME_VERSION=19.0.2
ARG WASMTIME_ARCH=x86_64
ARG WASMTIME_OS=linux
RUN set -eux; \
TARBALL="wasmtime-v${WASMTIME_VERSION}-${WASMTIME_ARCH}-${WASMTIME_OS}.tar.xz"; \
BASE_URL="https://github.com/bytecodealliance/wasmtime/releases/download/v${WASMTIME_VERSION}"; \
curl -fsSLo "/tmp/${TARBALL}" "${BASE_URL}/${TARBALL}"; \
curl -fsSLo "/tmp/${TARBALL}.sha256" "${BASE_URL}/${TARBALL}.sha256"; \
cd /tmp; sha256sum -c "${TARBALL}.sha256"; \
mkdir -p /opt/wasmtime; \
tar -xJf "/tmp/${TARBALL}" -C /opt/wasmtime --strip-components=1; \
rm "/tmp/${TARBALL}" "/tmp/${TARBALL}.sha256"
ENV PATH="/opt/wasmtime:${PATH}"

Copilot uses AI. Check for mistakes.
Comment on lines +13 to +18
RUN curl https://get.wasmer.io -sSfL | sh
ENV PATH="/root/.wasmer/bin:${PATH}"

# Verify installation
RUN wasmer --version

Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wasmer is installed via curl | sh without pinning a version or verifying checksums, which makes builds non-reproducible and can introduce breakage when Wasmer updates. Prefer installing a specific versioned release artifact (or using a build ARG for the version) and validating the installed version in a deterministic way.

Suggested change
RUN curl https://get.wasmer.io -sSfL | sh
ENV PATH="/root/.wasmer/bin:${PATH}"
# Verify installation
RUN wasmer --version
ARG WASMER_VERSION=4.2.5
RUN set -eux; \
curl -sSfL "https://github.com/wasmerio/wasmer/releases/download/${WASMER_VERSION}/wasmer-linux-amd64.tar.gz" -o /tmp/wasmer.tar.gz; \
tar -xzf /tmp/wasmer.tar.gz -C /usr/local/bin wasmer; \
rm /tmp/wasmer.tar.gz
ENV PATH="/usr/local/bin:${PATH}"
# Verify installation deterministically
RUN wasmer --version | grep -F "wasmer ${WASMER_VERSION}"

Copilot uses AI. Check for mistakes.
FROM rust:1-slim-bookworm AS builder

# cargo-wasix installs its own rustup toolchain + wasm32-wasmer-wasi target
RUN cargo install cargo-wasix
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cargo install cargo-wasix is unpinned, so the WASIX build can change/break over time depending on the latest published version. Consider pinning a specific cargo-wasix version (or installing from a known git ref) to keep the build reproducible.

Suggested change
RUN cargo install cargo-wasix
RUN cargo install cargo-wasix --version 0.16.3

Copilot uses AI. Check for mistakes.
Comment on lines +37 to +50
log_test() {
TOTAL=$((TOTAL + 1))
echo -e "${DIM}[${TOTAL}]${RESET} ${BOLD}$1${RESET}"
}

log_pass() {
PASS=$((PASS + 1))
echo -e " ${GREEN}PASS${RESET} $1"
}

log_fail() {
FAIL=$((FAIL + 1))
echo -e " ${RED}FAIL${RESET} $1"
}
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TOTAL is incremented only in log_test, but PASS/FAIL are incremented on every log_pass/log_fail call (including multiple assertions within a single test, e.g. the text conversion sanity check). This can produce summaries where PASS + FAIL + SKIP doesn’t correspond to TOTAL, which is confusing for CI output; consider counting per-assertion consistently or tracking test-cases vs assertions separately.

Copilot uses AI. Check for mistakes.
Comment on lines +227 to +233
docker run --rm \
--name "${PREFIX}-test-${runtime}" \
"${img}" \
"${runtime}"
local exit_code=$?
echo ""
return ${exit_code}
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because this script runs with set -e, a failing docker run will terminate the whole script immediately, so cmd_test can’t count failures and continue testing other runtimes. Capture the exit code without triggering errexit (e.g., temporarily disable -e around docker run, or use docker run ... || exit_code=$? and return $exit_code).

Suggested change
docker run --rm \
--name "${PREFIX}-test-${runtime}" \
"${img}" \
"${runtime}"
local exit_code=$?
echo ""
return ${exit_code}
local exit_code=0
docker run --rm \
--name "${PREFIX}-test-${runtime}" \
"${img}" \
"${runtime}" || exit_code=$?
echo ""
return "${exit_code}"

Copilot uses AI. Check for mistakes.
Comment on lines +50 to +52
ALL_WASM_RUNTIMES="wasmtime wasmer wasmedge wamr wasix"
ALL_RISCV_RUNTIMES="riscv-qemu spike libriscv rvvm ckb-vm"
ALL_RUNNERS="${ALL_WASM_RUNTIMES} ${ALL_RISCV_RUNTIMES}"
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ALL_RISCV_RUNTIMES includes rvvm (marked “INCOMPATIBLE” in Dockerfile.runner.rvvm) and other experimental runners, but cmd_test treats test all as ${ALL_RUNNERS} and the Makefile’s wasi-test invokes test all. This makes the default path likely fail even when the “green” runtimes pass; consider splitting stable vs experimental runtimes (and have all run only stable ones).

Suggested change
ALL_WASM_RUNTIMES="wasmtime wasmer wasmedge wamr wasix"
ALL_RISCV_RUNTIMES="riscv-qemu spike libriscv rvvm ckb-vm"
ALL_RUNNERS="${ALL_WASM_RUNTIMES} ${ALL_RISCV_RUNTIMES}"
# Stable vs experimental runtimes
STABLE_WASM_RUNTIMES="wasmtime wasmer wasmedge wamr"
EXPERIMENTAL_WASM_RUNTIMES="wasix"
STABLE_RISCV_RUNTIMES="riscv-qemu"
EXPERIMENTAL_RISCV_RUNTIMES="spike libriscv rvvm ckb-vm"
# Superset lists (stable + experimental)
ALL_WASM_RUNTIMES="${STABLE_WASM_RUNTIMES} ${EXPERIMENTAL_WASM_RUNTIMES}"
ALL_RISCV_RUNTIMES="${STABLE_RISCV_RUNTIMES} ${EXPERIMENTAL_RISCV_RUNTIMES}"
# Default runners for `test all` — only stable runtimes
ALL_RUNNERS="${STABLE_WASM_RUNTIMES} ${STABLE_RISCV_RUNTIMES}"
# Optional: all experimental runners (currently unused but kept for clarity/extension)
EXPERIMENTAL_RUNNERS="${EXPERIMENTAL_WASM_RUNTIMES} ${EXPERIMENTAL_RISCV_RUNTIMES}"

Copilot uses AI. Check for mistakes.
@raphaelmansuy
Copy link
Copy Markdown
Owner

@copilot open a new pull request to apply changes based on the comments in this thread

@gwpl
Copy link
Copy Markdown
Author

gwpl commented Mar 25, 2026

Hey Raphael, let us start updating PR after PR if possible. Ofc if will be need to make new one we can make new one.

@gwpl
Copy link
Copy Markdown
Author

gwpl commented Mar 25, 2026

AI Assistant: 🖖

Copilot raised the same set of concerns across all 6 PRs (same codebase). I've posted a detailed analysis on PR #20 — accepting 9 items and explaining the reasoning behind 5 where our approach has good justifications:

👉 Full response on PR #20

TL;DR of accepted fixes (will push to all branches):

  • Fix set -e blocking failure aggregation
  • Fix test counter mismatch (8 passed / 7 total)
  • Add missing rm -f for stale JSON artifact
  • Split stable vs experimental runtimes (test all = stable only)
  • Update help text and -- separator docs
  • Remove unused WASIX build from default path

@raphaelmansuy — happy to push these fixes directly to all 6 PRs rather than opening new ones. What's your preference?

🤖 AI+👤🤝

AI Assistant: Applied accepted fixes from Copilot PR review.

* Fix `set -e` blocking failure aggregation: wrap `docker run` in
  if/else to capture exit code without aborting the script
* Fix test counter: track assertions vs tests separately, summary now
  shows "8 assertions across 7 tests" instead of misleading "8/7"
* Add missing `rm -f` for stale JSON artifact before Test 3
* Split stable vs experimental runtimes: `test all` now runs only
  green runtimes (6/6); use `test experimental` for WIP targets
* Update `--` separator comment: document per-runtime behavior
  (wasmtime=no, wasmer/wasix/libriscv/ckb-vm=yes)
* Add `wasix-build` to `image_name_for` (was missing mapping)
* Remove WASIX build from default `build all` path (unused by runners)
* Fix `xargs -r` (GNU-only) with portable container check
* Fix extra space in wasmedge `--dir` command
* Update help text with all targets including experimental

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants