[Bug]: VK_KHR_video_{dec,enc}ode_av1 & VK_KHR_video_encode_h26x issues in FFmpeg on Navi33

[nyanmisaka]

This issue isn't specific to AMF, but it's related to the VCN codecs. I think the AMF team might be more familiar with VCN than the GFX team, so I mirrored GPUOpen-Drivers/AMD-Gfx-Drivers#99 here, hoping AMF team can take a look. Note that this issue does not exist in Mesa RADV driver.

---

GPU RDNA3 Navi33
Driver version 26.5.1 WHQL (32.0.31007.1017)
Windows 10 22H2 19045

FFmpeg version 8.1:
https://github.com/BtbN/FFmpeg-Builds/releases/download/autobuild-2026-03-31-13-11/ffmpeg-n8.1-7-ga3475e2554-win64-gpl-shared-8.1.zip

For comparison, D3D11VA decode and AMF decode/encode do not have the following issues:

• VK_KHR_video_decode_av1
  Slow decoding certain AV1 video, usually downloaded from youtube, "Video Codec 0" shows 100% but the speed is slow:
  COSTA RICA IN 4K 60fps HDR (ULTRA HD) [LXb3EKWsInQ].zip

./ffmpeg -hide_banner -init_hw_device vulkan=vk -filter_hw_device vk -hwaccel vulkan -hwaccel_output_format vulkan -i 'E:\Sample Video\AV1\COSTA RICA IN 4K 60fps HDR (ULTRA HD) [LXb3EKWsInQ].webm' -an -sn -dn -f null -
Input #0, matroska,webm, from 'E:\Sample Video\AV1\COSTA RICA IN 4K 60fps HDR (ULTRA HD) [LXb3EKWsInQ].webm':
  Metadata:
    COMPATIBLE_BRANDS: iso6av01mp41
    MAJOR_BRAND     : dash
    MINOR_VERSION   : 0
    ENCODER         : Lavf62.3.100
  Duration: 00:05:13.81, start: 0.000000, bitrate: 25001 kb/s
  Stream #0:0: Video: av1 (libdav1d) (Main), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2160 [SAR 1:1 DAR 16:9], 59.94 fps, 59.94 tbr, 1k tbn (default)
    Metadata:
      HANDLER_NAME    : ISO Media file produced by Google Inc.
      VENDOR_ID       : [0][0][0][0]
      DURATION        : 00:05:13.780000000
    Side data:
      Content light level metadata: MaxCLL=1100, MaxFALL=180
      Mastering display metadata: has_primaries:1 has_luminance:1 r(0.6780,0.3220) g(0.2450,0.7030) b(0.1380 0.0520) wp(0.3127, 0.3290) min_luminance=0.000100, max_luminance=1000.000000
  Stream #0:1(eng): Audio: opus, 48000 Hz, stereo, fltp, start 0.007000 (default)
    Metadata:
      DURATION        : 00:05:13.808000000
Stream mapping:
  Stream #0:0 -> #0:0 (av1 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    COMPATIBLE_BRANDS: iso6av01mp41
    MAJOR_BRAND     : dash
    MINOR_VERSION   : 0
    encoder         : Lavf62.12.101
  Stream #0:0: Video: wrapped_avframe, vulkan(tv, bt2020nc/bt2020/smpte2084, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 59.94 fps, 59.94 tbn (default)
    Metadata:
      encoder         : Lavc62.28.101 wrapped_avframe
      HANDLER_NAME    : ISO Media file produced by Google Inc.
      VENDOR_ID       : [0][0][0][0]
      DURATION        : 00:05:13.780000000
    Side data:
      Mastering display metadata: has_primaries:1 has_luminance:1 r(0.6780,0.3220) g(0.2450,0.7030) b(0.1380 0.0520) wp(0.3127, 0.3290) min_luminance=0.000100, max_luminance=1000.000000
      Content light level metadata: MaxCLL=1100, MaxFALL=180
frame= 1861 fps= 28 q=-0.0 size=N/A time=00:00:31.04 bitrate=N/A speed=0.463x elapsed=0:01:07.09

• VK_KHR_video_encode_av1
  Artifacts in encoded AV1 Main 10bit video:
  test_av1_1080p_10b.zip
  

./ffmpeg -hide_banner -init_hw_device vulkan=vk -filter_hw_device vk -f lavfi -i testsrc2=s=1920x1080,format=p010le -vf hwupload -c:v av1_vulkan -rc_mode:v vbr -b:v 5M -t 5 -y test_av1_1080p_10b.mp4
Input #0, lavfi, from 'testsrc2=s=1920x1080,format=p010le':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, p010le, 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> av1 (av1_vulkan))
Press [q] to stop, [?] for help
Output #0, mp4, to 'test_av1_1080p_10b.mp4':
  Metadata:
    encoder         : Lavf62.12.101
  Stream #0:0: Video: av1 (Main) (av01 / 0x31307661), vulkan(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 5000 kb/s, 25 fps, 12800 tbn
    Metadata:
      encoder         : Lavc62.28.101 av1_vulkan
[out#0/mp4 @ 0000024c561c2380] video:3038KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.036294%
frame=  125 fps=0.0 q=-0.0 Lsize=    3039KiB time=00:00:05.00 bitrate=4978.9kbits/s speed=16.8x elapsed=0:00:00.29

• VK_KHR_video_encode_h264
  VBR/CBR rate control failed:

./ffmpeg -hide_banner -init_hw_device vulkan=vk -filter_hw_device vk -f lavfi -i testsrc2=s=1920x1080,format=nv12 -vf hwupload -c:v h264_vulkan -rc_mode:v cbr -b:v 5M -f null -
Input #0, lavfi, from 'testsrc2=s=1920x1080,format=nv12':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, nv12, 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> h264 (h264_vulkan))
Press [q] to stop, [?] for help
[h264_vulkan @ 000002c9680f0a00] rbsp_stop_one_bit out of range: 0, but must be in [1,1].
[h264_vulkan @ 000002c9680f0a00] Failed to read unit 0 (type 7): Invalid data found when processing input.
[h264_vulkan @ 000002c9680f0a00] Unable to parse feedback units, bad drivers: Invalid data found when processing input
[vost#0:0/h264_vulkan @ 000002c9680f0780] [enc:h264_vulkan @ 000002c9631fa6c0] Error while opening encoder - maybe incorrect parameters such as bit_rate, rate, width or height.
[vf#0:0 @ 000002c9680f5540] Error sending frames to consumers: Invalid data found when processing input
[vf#0:0 @ 000002c9680f5540] Task finished with error code: -1094995529 (Invalid data found when processing input)
[vf#0:0 @ 000002c9680f5540] Terminating thread with return code -1094995529 (Invalid data found when processing input)
[vost#0:0/h264_vulkan @ 000002c9680f0780] [enc:h264_vulkan @ 000002c9631fa6c0] Could not open encoder before EOF
[vost#0:0/h264_vulkan @ 000002c9680f0780] Task finished with error code: -22 (Invalid argument)
[vost#0:0/h264_vulkan @ 000002c9680f0780] Terminating thread with return code -22 (Invalid argument)
[out#0/null @ 000002c967f77c40] Nothing was written into output file, because at least one of its streams received no packets.
frame=    0 fps=0.0 q=0.0 Lsize=       0KiB time=N/A bitrate=N/A speed=N/A elapsed=0:00:00.00
Conversion failed

• VK_KHR_video_encode_h265
  CQP/VBR/CBR rate control failed:

./ffmpeg -hide_banner -init_hw_device vulkan=vk -filter_hw_device vk -f lavfi -i testsrc2=s=1920x1080,format=nv12 -vf hwupload -c:v hevc_vulkan -rc_mode:v cqp -qp:v 10 -f null -
Input #0, lavfi, from 'testsrc2=s=1920x1080,format=nv12':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, nv12, 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> hevc (hevc_vulkan))
Press [q] to stop, [?] for help
[hevc_vulkan @ 00000201f60c3e40] Invalid ue-golomb code at cpb_cnt_minus1[i]: bitstream ended.
[hevc_vulkan @ 00000201f60c3e40] Failed to read unit 0 (type 32): Invalid data found when processing input.
[hevc_vulkan @ 00000201f60c3e40] Unable to parse feedback units, bad drivers: Invalid data found when processing input
[vost#0:0/hevc_vulkan @ 00000201f60c3bc0] [enc:hevc_vulkan @ 00000201f60b7b40] Error while opening encoder - maybe incorrect parameters such as bit_rate, rate, width or height.
[vf#0:0 @ 00000201f60c4200] Error sending frames to consumers: Invalid data found when processing input
[vf#0:0 @ 00000201f60c4200] Task finished with error code: -1094995529 (Invalid data found when processing input)
[vf#0:0 @ 00000201f60c4200] Terminating thread with return code -1094995529 (Invalid data found when processing input)
[vost#0:0/hevc_vulkan @ 00000201f60c3bc0] [enc:hevc_vulkan @ 00000201f60b7b40] Could not open encoder before EOF
[vost#0:0/hevc_vulkan @ 00000201f60c3bc0] Task finished with error code: -22 (Invalid argument)
[vost#0:0/hevc_vulkan @ 00000201f60c3bc0] Terminating thread with return code -22 (Invalid argument)
[out#0/null @ 00000201f5f57880] Nothing was written into output file, because at least one of its streams received no packets.
frame=    0 fps=0.0 q=0.0 Lsize=       0KiB time=N/A bitrate=N/A speed=N/A elapsed=0:00:00.01
Conversion failed!

• VK_KHR_video_encode_h265
  No P010 10bit Main10 profile support, only NV12 8bit Main profile:

./ffmpeg -hide_banner -init_hw_device vulkan=vk -filter_hw_device vk -f lavfi -i testsrc2=s=1920x1080,format=p010le -vf hwupload -c:v hevc_vulkan -b:v 5M -f null -
Input #0, lavfi, from 'testsrc2=s=1920x1080,format=p010le':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, p010le, 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> hevc (hevc_vulkan))
Press [q] to stop, [?] for help
[hevc_vulkan @ 00000241c435fe80] No supported profiles for given format
[vost#0:0/hevc_vulkan @ 00000241c435fc00] [enc:hevc_vulkan @ 00000241ff43dc80] Error while opening encoder - maybe incorrect parameters such as bit_rate, rate, width or height.
[vf#0:0 @ 00000241c4360240] Error sending frames to consumers: Error number -129 occurred
[vf#0:0 @ 00000241c4360240] Task finished with error code: -129 (Error number -129 occurred)
[vf#0:0 @ 00000241c4360240] Terminating thread with return code -129 (Error number -129 occurred)
[vost#0:0/hevc_vulkan @ 00000241c435fc00] [enc:hevc_vulkan @ 00000241ff43dc80] Could not open encoder before EOF
[vost#0:0/hevc_vulkan @ 00000241c435fc00] Task finished with error code: -22 (Invalid argument)
[vost#0:0/hevc_vulkan @ 00000241c435fc00] Terminating thread with return code -22 (Invalid argument)
[out#0/null @ 00000241c41f6bc0] Nothing was written into output file, because at least one of its streams received no packets.
frame=    0 fps=0.0 q=0.0 Lsize=       0KiB time=N/A bitrate=N/A speed=N/A elapsed=0:00:00.00
Conversion failed!

[MikhailAMD]

I pinged the right people in AMD. Just curios, do you see similar issues with AMF on Linux?

[nyanmisaka]

As of Mesa 26.0, VK video on RADV has no such issues. So we haven't received any similar reports regarding AMF via RADV. But I'm not sure about AMF via AMDVLK, since on Windows, VK video seem to be provided by AMDVLK, which is affected.

[MikhailAMD]

I am asking because on Linux AMF will decode via RADV but encode using internal implementation, not using VK encode extension.

[nyanmisaka]

This is the first time I've heard of this hybrid combination. I guess AMF encoder has added extra advanced rate control algorithms, thus RADV's encoder cannot be used. VK video is currently more like VAAPI, only exposing basic features but it's more cross-platform/vendor friendly.

I believe AMF's proprietary encoder implementation has undergone more testing. VK video is still relatively new, its extensions only appeared in Windows drivers last year or the year before.

https://www.amd.com/en/resources/support-articles/release-notes/RN-RAD-WIN-VULKAN.html

[nyanmisaka]

Another request is, if possible, please backport the fixes to the older branches of drivers used by RDNA1+2, at least to get FFmpeg working properly. Although I'm not sure if they were also affected by these bugs.