Skip to content

feat(qwen-story): 8-beat E2E narrative + pmat bug-hunt + daily cron#1875

Merged
noahgift merged 6 commits into
mainfrom
feat/qwen-story-v1
May 22, 2026
Merged

feat(qwen-story): 8-beat E2E narrative + pmat bug-hunt + daily cron#1875
noahgift merged 6 commits into
mainfrom
feat/qwen-story-v1

Conversation

@noahgift
Copy link
Copy Markdown
Contributor

Summary

A canonical end-to-end "Qwen story" that doubles as a regression gate. Eight beats, one narrative, every core command group — anchored on the Qwen scale ladder (0.5B safetensors → 30B-MoE GGUF) so the story exercises real production scales, not toy fixtures.

What ships

Artifact Purpose
scripts/qwen-story.sh (336 LOC) Runnable story; every beat uses OUT=$(cmd); EC=$? (no pipe-then-$?)
contracts/qwen-story-v1.yaml 3 equations + 8 falsifiers (all PASS locally)
README.md § "A Qwen story" Replaces the flat ## CLI examples block; fixes 2 broken README examples
.github/workflows/qwen-story-daily.yml 04:17 UTC self-hosted GPU cron + workflow_dispatch

The 8 beats

  1. Discover (Registry) — apr pull, apr list
  2. Trust (QA) — apr qa, apr validate, apr lint
  3. Explore (Inspection) — apr inspect, apr tensors, apr tree
  4. Adapt (Model ops) — apr export, apr diff, apr convert
  5. Use (Inference) — apr run, apr chat, apr code -p
  6. Serve (REST) — apr serve run + curl /v1/chat/completions
  7. Operate (Profiling) — apr profile, apr gpu, apr serve plan (on 7B Q4K)
  8. Scale (MoE) — apr inspect, apr tensors (on 30B-MoE qwen3moe)

Pmat bug-hunt layer

When PMAT_HUNT=1 (default), each beat emits a structured manifest of high-risk untested code in the command-handler modules it just exercised:

-- pmat bug-hunt manifest (run chat code) --
    gap   crates/apr-cli/src/commands/run.rs:resolve_model_alias (impact=42.3)
    churn crates/apr-cli/src/commands/code.rs:dispatch_agent (commits=11)
    fault crates/aprender-serve/src/api/cuda_chat_backend.rs:try_qwen3_moe (unwrap,panic)

The nightly cron diffs this manifest against the prior successful run and opens (or comments on) a tracking issue when growth exceeds 5 lines — so untested branches in command handlers can't accumulate quietly.

README example fixes (caught while building the script)

The story script revealed two more broken examples in README.md ## CLI examples:

  • apr profile model.gguf --roofline → there is no --roofline flag; correct usage is just apr profile model.gguf (the description already says "Deep profiling with Roofline analysis")
  • apr bench model.gguf --assert-tps 100 → the --assert-tps flag lives on apr qa, not apr bench

The new ## A Qwen story section replaces this block with verified invocations from the runnable script.

Local smoke test

$ bash scripts/qwen-story.sh
... 14 PASS / 2 FAIL / 0 SKIP

Failed beats:
   - B2 apr validate --quality       # closed by #1870 (in flight)
   - B4 apr export                   # closed by #1868 (in flight)

Both failures are expected until the in-flight fixes land. The story is wired correctly — it catches the panic, it catches the validate threshold, it catches each regression class. Once #1868 + #1870 merge, the story will be 16/0/0 on a host with all 4 Qwen models cached.

Falsifier sweep

All 8 falsifiers in qwen-story-v1.yaml PASS:

F-001 PASS  Script exists and is executable
F-002 PASS  Story has 8 beats (8 beatN_<name>() { definitions)
F-003 PASS  Every beat uses OUT=$(cmd); EC=$? pattern
F-004 PASS  PMAT audit wired per beat (8 pmat_hunt calls)
F-005 PASS  Story referenced from README
F-006 PASS  Daily cron exists
F-007 PASS  Bashrs lints clean (0 errors)
F-008 PASS  Beat 7 doesn't run apr qa on 7B (avoids #1864)

Follow-up

A small follow-up PR will add /dogfood Gate 18 that invokes scripts/qwen-story.sh — kept separate to avoid conflict with #1872 which is already adding Gates 13-17 to the dogfood skill.

Test plan

🤖 Generated with Claude Code

…ly cron

Adds an end-to-end "Qwen story" that exercises every core apr command
group against the Qwen scale ladder (0.5B → 1.5B → 7B → 30B-MoE). The
story is the single canonical demo in README.md AND a regression gate
via runnable script + falsification contract + nightly cron.

## Beats

1. **Discover** (Registry) — pull, list
2. **Trust** (QA) — qa, validate, lint
3. **Explore** (Inspection) — inspect, tensors, tree
4. **Adapt** (Model ops) — export, diff, convert/quantize
5. **Use** (Inference) — run, chat, code
6. **Serve** (REST) — serve run + curl /v1/chat/completions OpenAI-compat
7. **Operate** (Profiling) — profile, gpu, serve plan (7B Q4K GGUF)
8. **Scale** (MoE) — inspect, tensors on 30B-MoE qwen3moe

## Pmat bug-hunt layer

When run with `PMAT_HUNT=1` (default), each beat emits a structured
manifest of high-risk untested code in the command-handler modules it
just exercised:

    -- pmat bug-hunt manifest (run chat code) --
        gap   crates/apr-cli/src/commands/run.rs:resolve_model_alias (impact=42.3)
        churn crates/apr-cli/src/commands/code.rs:dispatch_agent (commits=11)
        fault crates/aprender-serve/src/api/cuda_chat_backend.rs:try_qwen3_moe (unwrap)

The nightly cron uploads this manifest as an artifact, compares against
the previous successful run, and opens (or comments on) a tracking issue
when growth exceeds 5 lines — so untested branches in command handlers
can't accumulate quietly.

## Files

- `scripts/qwen-story.sh` (336 LOC) — runnable story with proper exit-code
  capture (`OUT=$(cmd); EC=$?` everywhere; no pipe-then-`$?` per memory rule)
- `contracts/qwen-story-v1.yaml` — 3 equations + 8 falsifiers, all PASS
  locally (script exists+executable, 8 beats, run_cmd helper, pmat_hunt
  per beat, README link, daily cron file, bashrs clean, Beat 7 skips
  `apr qa` on 7B Q4K due to #1864)
- `README.md` — new `## A Qwen story` section replacing the flat
  `## CLI examples` block. Fixes two README bugs surfaced during dogfood:
  `apr profile --roofline` (no such flag; just `apr profile <file>`)
  and `apr bench --assert-tps` (flag is on `apr qa`, not `bench`).
- `.github/workflows/qwen-story-daily.yml` — self-hosted GPU runner,
  04:17 UTC cron + workflow_dispatch, uploads pmat manifest + story log
  artifacts, files tracking issue when story regresses or manifest grows.

## Verification

    $ bash scripts/qwen-story.sh   # local smoke
    -- Beat 1: Discover (Registry) --
    ✓ PASS  B1 list
    -- Beat 2: Trust (QA gates) --
    ✓ PASS  B2 apr qa
    ✗ FAIL  B2 apr validate --quality  -  exit=5 (after #1866 fix this should be 0)
    -- Beat 3: Explore (Inspection) --
    ✓ PASS  B3 apr inspect --json (arch=qwen2)
    ✓ PASS  B3 apr tensors --json (339 tensors)
    ✓ PASS  B3 apr tree
    -- Beat 4: Adapt (Model ops) --
    ✗ FAIL  B4 apr export  -  PANIC (exit=101)  -  #1865 regression
    -- Beat 5: Use (Inference) --
    ✓ PASS  B5 apr run (Rust code completion)
    ✓ PASS  B5 apr code -p
    -- Beat 6: Serve (REST API) --
    ✓ PASS  B6 apr serve run (port=22915)
    ✓ PASS  B6 /v1/chat/completions (got OK...)
    -- Beat 7: Operate (Profiling) --
    ✓ PASS  B7 apr profile
    ✓ PASS  B7 apr gpu --json
    ✓ PASS  B7 apr serve plan -- 7B VRAM budget
    -- Beat 8: Scale (MoE introspection) --
    ✓ PASS  B8 apr inspect --json (arch=qwen3moe)
    ✓ PASS  B8 apr tensors --json (579 tensors)
    14 PASS / 2 FAIL / 0 SKIP

The 2 FAILs are EXPECTED until the in-flight fixes land:
- B2 validate --quality: closed by #1870
- B4 export panic: closed by #1868

Once those PRs merge, this story will be 16 PASS / 0 FAIL / 0 SKIP on a
host with all 4 Qwen models cached.

## Follow-up

A separate PR will add `/dogfood` Gate 18 that invokes this script (kept
separate to avoid conflict with PR #1872 which is already adding Gates
13-17 to the dogfood skill).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@noahgift noahgift enabled auto-merge (squash) May 22, 2026 07:52
noahgift added a commit that referenced this pull request May 22, 2026
…itstream-io) (#1878)

`cargo deny check advisories` started failing on every PR (and on main)
2026-05-22 with:

    error[unmaintained]: core2 is unmaintained, all versions yanked
    ├ ID: RUSTSEC-2026-0105
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0105

The dep is pulled in transitively via `bitstream-io` (image/media decoding
stack — `cargo tree` shows `bitstream-io v4.9.0 → core2 v0.4.0`). No
first-party use; no drop-in replacement until upstream `bitstream-io`
migrates off core2.

This commit unblocks the in-flight PR cascade (#1867 #1868 #1870 #1873
#1875 #1876) which all failed CI's `ci / lint` step on this advisory.
The deny entry is structured per the existing pattern in this file (id +
human reason mentioning the transitive path) so revisiting the ignore in
6-12 months is straightforward.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
@noahgift noahgift merged commit 81e9c49 into main May 22, 2026
10 checks passed
@noahgift noahgift deleted the feat/qwen-story-v1 branch May 22, 2026 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant