Add strict EVEX dispatch for AVX512VL forms backed by existing AVX handlers

[necat101]

dynarec/arm64: add strict EVEX dispatch for AVX-compatible AVX512VL forms

Add a shared EVEX-to-VEX decoder for EVEX instructions that Box64 can
represent with the existing 128/256-bit AVX register model and opcode
handlers.

The decoder accepts only k0/no-mask, no-broadcast, no-SAE/rounding,
non-ZMM, non-high-register EVEX forms, then routes them through the
existing AVX interpreter and ARM64 dynarec paths. Unsupported EVEX forms
still fail cleanly instead of being misdecoded as legacy BOUND or
silently producing incorrect results.

Also improve missing-opcode diagnostics to distinguish EVEX from VEX/AVX.

Tested with:

• avx_intrinsics ctest
• ARM64 dynarec native_pass0..3 build
• manual EVEX vaddps smoke test through interpreter and ARM64 dynarec

[ptitSeb]

First: please remove the youtube channel link: this github repo is about box64 and nothing else.

Now, about this PR. Thanks for it, for but is the end plan here? Do you plan to add full AVX512VL support. If yes, how, because register pressure on Arm64 backend will get very high, not sure it can actually work with current register assiment method. If now, that what is the point of partial EVEX decoding, does it improve compatibility somehow?

[necat101]

Thanks for looking at it!

The end plan for this PR is not full AVX512VL support. I agree that full AVX512VL, especially masks, zero/merge semantics, high registers, and anything that increases vector live state, needs a much larger design discussion. I do not think this PR solves that, and I do not want it to imply Box64 is now advertising or supporting AVX512.

The narrower point here is compatibility for EVEX encodings that are already representable by the current Box64 AVX model: VL128/VL256, k0/no mask, no broadcast, no SAE/rounding, no ZMM state, no high ZMM16-31 registers. For those cases the EVEX instruction is effectively equivalent to an existing VEX/AVX instruction that Box64 already implements. The PR just decodes that strict subset and sends it through the existing AVX interpreter/dynarec path.

So register pressure should not increase for accepted instructions, because this does not add new guest vector state or new backend allocation requirements. It reuses the current XMM/YMM path and rejects anything that would need more state or mask handling.

Compatibility-wise, this helps code that reaches EVEX-encoded 128/256-bit forms even though the operation itself does not need true AVX512 behavior. Before this, those instructions were treated as 0x62/BOUND or illegal in long mode and would fail immediately. After this, the safe subset can run with the same behavior/performance class as the existing AVX implementation. Unsupported EVEX forms still fail rather than being guessed.

also, sorry for the YT link, i was a little tipsied drunk after i got done with the PR and pushed it.

[ptitSeb]

Ok, fine, lets see were it goes.