Add VK_EXT_ycbcr_3plane_16bit_lsb_formats extension

[rmader]

This extension adds support for 10/12bit YCbCr formats used by software decoders like ffmpeg, dav1d and libvpx.

See also:

• torvalds/linux@2271e0a
• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34303
• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35576
• https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/9251
• https://www.ffmpeg.org/doxygen/1.0/pixfmt_8h.html

---

This branch as well as the related Mesa and GTK4 MRs have been in draft status for a while and already got a lot of testing. With the recent Gstreamer 1.28 release (shipped in distros like Fedora 44 and Ubuntu 26.04) they can be tested and used - e.g. the default Gnome Video player will make use of the new formats when playing HDR videos (for various codecs, including AV1, HEVC, H266, Pro-Res and DNxHR).

@gfxstrand I'd be super happy if you could have a look at this and, if you find time, help me get this over the line 😅 Most importantly point me to where I should elaborate more.

[oddhack]

@rmader please rebase on github main branch, which should fix the CI CTS framework issue, and address the other issues raised by CI around copyright dates and naming.

[rmader]

@rmader please rebase on github main branch, which should fix the CI CTS framework issue, and address the other issues raised by CI around copyright dates and naming.

Thanks for the review! I think I addressed all issues - and hope the change to append _EXT to all formats is correct. The goal of course being that they'll get promoted at some point going forward.

[rmader]

I added one more change for completeness - 14bit formats. While those are very uncommon AFAIK, they are supported by ffmpeg. Thus let's include them while on it, so we don't have to revisit this in a couple of years.

[gfxstrand]

Sorry this took me nearly a month to get to. Between a Khronos meeting and vacation, it kind of got lost in my e-mail.

[gfxstrand]

This is a tricky assertion to make. Right now, we assume that the X bits are garbage and (mostly) ignored by sampling. I say "mostly" because it's kind of okay if they're not since they're the low bits and any garbage in those bits will show up as noise which likely isn't perceptible to the human eye. With theese formats, however, your assertion that they're easy for shaders by doing a bit of multiplication assumes that the top bits are always zero. While this is fairly easy for software to guarantee when outputting such an image, it's very hard for the driver to defend against. We could make it invalid usage or unknown results if you have non-zero bits in the high bits when sampling but we'd need to be very explicit about that in the API.

[rmader]

Indeed. So on the DRM side we solved that by using z instead of x: https://github.com/torvalds/linux/blob/master/include/uapi/drm/drm_fourcc.h#L403-L405

We could do the same here by turning

VK_FORMAT_X6G10_X6B10_X6R10_3PLANE_420_UNORM_3PACK16_EXT

into

VK_FORMAT_Z6G10_Z6B10_Z6R10_3PLANE_420_UNORM_3PACK16_EXT

which would hopefully make this clear. WDYT?

[gfxstrand]

We could do that. Since no one will ever be writing to these formats, it's not like we would need hardware to support that. I'd feel a little better about that than saying it's ignored. Again, I think this is where @fluppeteer will have good opinions. As might @spencer-lunarg since he did the format table in the first place.

[rmader]

Since no one will ever be writing to these formats

For completeness: these formats are also the native/optimal input formats for common software encoders, meaning there is a chance that at some point we'd want native driver support for writing into shared buffers that can directly be fed into those sw-encoders. Not super likely, just to keep in mind.

[rmader]

Did this for now (X -> Z).

[fluppeteer]

Sorry for the delay - I tried to write this a couple of weeks ago and GitHub ate it.

I don't love Z as a choice for the same depth-related reasons as @gfxstrand (and arguably should we decide to use WXYZ in a format at any point), but it may be the least bad option - N for "null" may be an alternative, but that may conflict with the common writing of "nnnn" to mean a series of bits. As you say, this feels like a 16-bit format with a different decode path for scaling, since we're saying the "must be zero" bits are actually significant and just expected to be zero; I'm not 100% clear that our hardware, at least, would be able to support this, since our hardware matrix transform has limited inputs. At least we avoided using Z for depth. (Fortunately a recent KDFS update can express mandatory 0/1 bits, I hope as is needed here.)

For reference it would be nice to include these formats in the 56.1 descriptions. C.f.:

VK_FORMAT_G10X6_B10X6R10X6_2PLANE_420_UNORM_3PACK16 specifies an unsigned normalized multi-planar format that has a 10-bit G component in the top 10 bits of each 16-bit word of plane 0, and a two-component, 32-bit BR plane 1 consisting of a 10-bit B component in the top 10 bits of the word in bytes 0..1, and a 10-bit R component in the top 10 bits of the word in bytes 2..3, with the bottom 6 bits of each word unused. The horizontal and vertical dimensions of the BR plane are halved relative to the image dimensions, and each R and B value is shared with the G components for which ⌊iG×0.5⌋=[iB]=[iR] and ⌊jG×0.5⌋=[jB]=[jR]. The location of each plane when this image is in linear layout can be determined via vkGetImageSubresourceLayout, using VK_IMAGE_ASPECT_PLANE_0_BIT for the G plane, and VK_IMAGE_ASPECT_PLANE_1_BIT for the BR plane. This format only supports images with a width and height that is a multiple of two.

I don't think I see that in this PR? (Unless they're autogenerated these days.)

[rmader]

I don't think I see that in this PR? (Unless they're autogenerated these days.)

Ouch, I'm sorry - this was indeed missing and arguably a very important part 😅

Added that now.

I don't love Z as a choice for the same depth-related reasons as @gfxstrand (and arguably should we decide to use WXYZ in a format at any point), but it may be the least bad option

Thanks! IIUC that means unless somebody else voices resistance we can go forward with that.

I'm not 100% clear that our hardware, at least, would be able to support this

For the record: even if the formats can only be supported via a shader fallback path it would still be a useful for multimedia apps, freeing them from having to carry custom shaders themselves.

[gfxstrand]

@fluppeteer should probably also give this a read.

[oddhack]

I got tired with adding explanatory comments, but there are what appear to be a bunch of minor markup errors in the new VUs which I have suggested fixes for.

[oddhack]

(Sorry, accidentally clicked the comment & close button).

[oddhack]

Github is borked ATM (presumably because it is a day ending in 'y') and won't let me comment on the diff for some reason, but features.adoc line 1845 introduced some leading white space before the ' ifdef::' which should be removed.

[rmader]

Github is borked ATM (presumably because it is a day ending in 'y') and won't let me comment on the diff for some reason, but features.adoc line 1845 introduced some leading white space before the ' ifdef::' which should be removed.

Whops, fixed. Also should have addressed all other issues from the last CI run (and ran the corresponding scripts locally to validate that)

[rmader]

Sorry for the noise, just some more minor fixes.

[fluppeteer]

Feedback from two weeks ago...

[fluppeteer]

Sorry for the delay - I tried to write this a couple of weeks ago and GitHub ate it.

I don't love Z as a choice for the same depth-related reasons as @gfxstrand (and arguably should we decide to use WXYZ in a format at any point), but it may be the least bad option - N for "null" may be an alternative, but that may conflict with the common writing of "nnnn" to mean a series of bits. As you say, this feels like a 16-bit format with a different decode path for scaling, since we're saying the "must be zero" bits are actually significant and just expected to be zero; I'm not 100% clear that our hardware, at least, would be able to support this, since our hardware matrix transform has limited inputs. At least we avoided using Z for depth. (Fortunately a recent KDFS update can express mandatory 0/1 bits, I hope as is needed here.)

For reference it would be nice to include these formats in the 56.1 descriptions. C.f.:

VK_FORMAT_G10X6_B10X6R10X6_2PLANE_420_UNORM_3PACK16 specifies an unsigned normalized multi-planar format that has a 10-bit G component in the top 10 bits of each 16-bit word of plane 0, and a two-component, 32-bit BR plane 1 consisting of a 10-bit B component in the top 10 bits of the word in bytes 0..1, and a 10-bit R component in the top 10 bits of the word in bytes 2..3, with the bottom 6 bits of each word unused. The horizontal and vertical dimensions of the BR plane are halved relative to the image dimensions, and each R and B value is shared with the G components for which ⌊iG×0.5⌋=[iB]=[iR] and ⌊jG×0.5⌋=[jB]=[jR]. The location of each plane when this image is in linear layout can be determined via vkGetImageSubresourceLayout, using VK_IMAGE_ASPECT_PLANE_0_BIT for the G plane, and VK_IMAGE_ASPECT_PLANE_1_BIT for the BR plane. This format only supports images with a width and height that is a multiple of two.

I don't think I see that in this PR? (Unless they're autogenerated these days.)

[rmader]

One open question from my side: is it ok if all three 10-bit formats share class="10-bit 3-plane LSB" - or do they need to have a own class each, with the chroma included/repeated?