CI Draft: Integrate VPTO Test CI

[and0d0]

草稿: 融合vpto的测试ci

[gemini-code-assist]

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

[reedhecre]

Codex Review

该评论由 review 机器人自动更新。

• PR: CI Draft: Integrate VPTO Test CI #702 CI Draft: Integrate VPTO Test CI
• Author: and0d0
• Base/Head: main / ci-vpto-merge
• Head SHA: 47bf3f274bc2
• Trigger: PR 有新提交
• Generated At: 2026-05-27T03:17:07Z
• Previous Head SHA: f53b840e2941
• Status: completed

Summary

检查到 3 个 CI 逻辑问题：分支检测脚本会直接失败、VPTO LLVM 与主线 LLVM 共享缓存键、手动触发永远不会运行新增的 VPTO 自托管作业。

Findings

1. P1 未定义的 `HEAD_REF` 会让 `build-and-test` 在 checkout 前直接失败 .github/workflows/ci.yml:111

新增的 Detect PR source fork 步骤在 set -u 下读取 ${HEAD_REF}，但 GitHub Actions 文档里提供的是 GITHUB_HEAD_REF，而且它也只会在 PR 事件中出现：https://docs.github.com/en/actions/reference/workflows-and-actions/variables 。当前这个变量名在所有事件里都是未定义的，因此这里会先报 unbound variable，后面的两个 checkout 步骤都不会执行。结果是整个 build-and-test 作业被回归打坏；remote-npu-validation 还依赖这个作业，所以 schedule / workflow_dispatch 也会被一并阻断。

2. P1 VPTO LLVM 分支和主线 LLVM 复用了同一个缓存键 .github/workflows/ci.yml:172

这次改动虽然在 Resolve LLVM source SHA 里区分了 VPTO LLVM 的源码提交，但缓存恢复/保存仍然只用 env.LLVM_COMMIT 这个主线 LLVM SHA 做 key。也就是说，IS_VPTO_FORK=true 时，工作流会把源码切到 vpto-dev/llvm-project:feature-vpto，却可能直接复用之前为主线 LLVM 生成的 llvm/build-assert；反过来，如果第一次写入缓存的是 VPTO 构建，后续普通 CI 也会拿到 VPTO 的二进制。这样 CI 实际验证的 LLVM/MLIR 工具链与源码分支不一致，既可能制造假红，也可能把真实集成问题放过去。

3. P2 `workflow_dispatch` 永远不会运行新增的 `vpto-sim-validation` 作业 .github/workflows/ci.yml:329

这个 job 的 if 明确把 workflow_dispatch 列为允许事件，但后半段又要求 startsWith(github.event.pull_request.head.ref, 'feature-vpto-')。手动触发并没有 pull_request payload，所以这部分条件在 workflow_dispatch 下不可能成立。GitHub 文档也说明 head ref 只在 PR 类事件里提供：https://docs.github.com/en/actions/reference/workflows-and-actions/variables 。结果就是手动触发时，这个新的 VPTO 自托管验证作业会被一直跳过，无法用于人工回归或补跑。

[HecreReed]

/run a5

[reedhecre]

已接收 /run a5，A5 板测器会处理这条请求。

• 进度页：http://154.9.227.233/ptoas-board-dashboard/#board-a5
• 当前状态：板测器空闲，这条请求会在本轮轮询启动。

页面会自动刷新，可以直接看当前阶段、排队情况和最近结果。

[reedhecre]

A5 板测失败

• 触发方式：manual
• 源码提交：33a00d20d05b
• 结果汇总：OK 214 / FAIL 5 / SKIP 1
• 日志：/root/ptoas-board-monitor-a5/logs/20260528_171406_manual_pr702.log
• 手动指令：/run a5
• 触发人：HecreReed
• 触发评论：CI Draft: Integrate VPTO Test CI #702 (comment)
• 失败阶段：board-validation / exit=1

失败用例

• tprefetch_async_binding (run, exit=1)
• syncall_binding (run, exit=1)
• set_validshape (run, exit=139)
• cmps (run, exit=2)
• cmp (run, exit=2)

[reedhecre]

A5 板测失败详情：PR #702

tprefetch_async_binding

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507035 (/tmp/ptoas-board-monitor-a5/runs/20260528_171406_manual_pr702/npu_validation/TPrefetchAsync/tprefetch_async_binding/main.cpp:91)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 717236] 2026-05-28-17:21:05.422.138 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 13, there is an aivec error exception, core id is 0, error code = 271, dump info: pc start: 0x100040800000, current: 0x1000408001a0, sc error info: 0xffffffffffff, su error info: 0xe6ffd23d139c0059,0xcc3fd0e010009bf5, mte error info: 0x1fd3f5c60007eff1, vec error info: 0xe7d1ff9e0013db24, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
        TraceBack (most recent call last):
       The extend info: errcode:(271) errorStr: The MPU address access is invalid. subErrType: 0x4.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:583]
       Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1728]
       AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=62, report_stream_id=62, task_id=0, flip_num=0, fault kernel_name=_Z30tprefetch_async_binding_kernelPfPa, fault kernel info ext=_Z30tprefetch_async_binding_kernelPfPa, program id=0, hash=1899772384034012286.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       rtStreamSynchronize execution failed, reason=vector core exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-05-28 17:21:11] ERROR: testcase failed (exit 1): tprefetch_async_binding

syncall_binding

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507015 (/tmp/ptoas-board-monitor-a5/runs/20260528_171406_manual_pr702/npu_validation/SyncAll/syncall_binding/main.cpp:84)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 724176] 2026-05-28-17:22:33.422.383 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 14, there is an aicore error exception, core id is 0, error code = 259, dump info: pc start: 0x100040800000, current: 0x100040800110, sc error info: 0xffffffffffff, su error info: 0xfeffffee1efe0030,0x72042208f8007fff, mte error info: 0xd91ddef00007beb6, vec error info: 0, cube error info: 0, l1 error info: 0xfff600167e8e, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
        TraceBack (most recent call last):
       The extend info: errcode:(259) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. subErrType: 0x4.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:583]
       Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1728]
       AICORE Kernel task happen error, retCode=0x26.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=62, report_stream_id=62, task_id=0, flip_num=0, fault kernel_name=_Z22syncall_binding_kernelPii, fault kernel info ext=_Z22syncall_binding_kernelPii, program id=0, hash=9475521060208115623.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       rtStreamSynchronize execution failed, reason=aicore exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507015[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-05-28 17:22:39] ERROR: testcase failed (exit 1): syncall_binding

set_validshape

stage=run info=exit=139

./test/npu_validation/scripts/run_remote_npu_validation.sh: line 366: 782132 Segmentation fault      (core dumped) LD_LIBRARY_PATH="${LD_LIBRARY_PATH_NPU}" ./build/${testcase}
[2026-05-28 17:34:32] ERROR: testcase failed (exit 139): set_validshape

cmps

stage=run info=exit=2

[ERROR] Packed mask mismatch: golden_v2.bin vs v2.bin, idx=4 (golden=98, out=0)
[ERROR] compare failed
[2026-05-28 18:13:25] ERROR: testcase failed (exit 2): cmps

cmp

stage=run info=exit=2

[ERROR] Packed mask mismatch: golden_v3.bin vs v3.bin, idx=4 (golden=49, out=0)
[ERROR] compare failed
[2026-05-28 18:13:37] ERROR: testcase failed (exit 2): cmp