feat: support nydus image format #566
Conversation
Signed-off-by: fatelei <fatelei@gmail.com>
|
Packit jobs failed. @containers/packit-build please check. |
|
✅ A new PR has been created in buildah to vendor these changes: containers/buildah#6610 |
mtrmac
left a comment
mtrmac
left a comment
There was a problem hiding this comment.
Thanks!
Per https://github.com/containers/container-libs/blob/main/CONTRIBUTING.md#sign-your-prs :
Use a real name (sorry, no anonymous contributions)
(please read on for more nuance to that.)
Cc: @giuseppe (Referencing https://github.com/dragonflyoss/nydus/blob/master/docs/nydus-design.md, that says:
- The image format has an
artifactTypein the index, but not in the manifest itself?! - It is not really an artifact, it has the usual OCI config type
- … it’s “just” a new structure of the layer blobs.
Reading this PR, I don’t think much of these changes is actually necessary. Do copy operations fail otherwise??
Apart from pure unchanged copies, long-term we need to have some idea of what this format is from the end-user workflow perspective, and I think it’s necessary to think about that now, to shape the API / UI appropriately.
Is it a compression format, like zstd:chunked, where each individual layer can be ~losslessly converted between tar and the other compression? (Seems not to be the case, with the bootstrap/blob differentiation.)
Is it a ~different manifest format, really distinct from OCI? (Looking at the metadata format suggests “no”; in particular, returning NonImageArtifactError from OCIConfig seems inappropriate.)
And yet, it is ultimately significantly different, at least in that, right now, podman pull would fail.
How would a conversion from a tar format to Nydus, or back, work? Skopeo is (well, currently) not set up to do conversions that need to download and significantly manipulate large blobs — conversions between v2s1/v2s2/OCI are metadata operations + compression/decompression where we can stream the data with little local storage. Would these conversions ~forever be delegated to specialized tools? Or is the idea that podman build+podman push should be able to somehow directly convert to Nydus on the fly?
| ociencspec.MediaTypeLayerNonDistributableEnc, ociencspec.MediaTypeLayerNonDistributableGzipEnc, ociencspec.MediaTypeLayerNonDistributableZstdEnc: | ||
| return nil, fmt.Errorf("during manifest conversion: encrypted layers (%q) are not supported in docker images", layers[idx].MediaType) | ||
| case internalManifest.NydusBootstrapLayerMediaType, internalManifest.NydusBlobLayerMediaType: | ||
| layers[idx].MediaType = manifest.DockerV2Schema2ForeignLayerMediaType |
There was a problem hiding this comment.
I can’t see how this works. The consumer is still going to interpret the blob as a compressed tarball.
| imgspecv1.MediaTypeLayoutHeader, | ||
| ociencspec.MediaTypeLayerEnc, ociencspec.MediaTypeLayerGzipEnc: | ||
| ociencspec.MediaTypeLayerEnc, ociencspec.MediaTypeLayerGzipEnc, | ||
| manifest.NydusBootstrapLayerMediaType, manifest.NydusBlobLayerMediaType: |
There was a problem hiding this comment.
This function is deprecated and has no known users (at least within the Podman+friends tools), so I don’t think there is any point in changing it.
| return fmt.Errorf("preparing updated manifest, layer %q: %w", info.Digest, err) | ||
| // Nydus layer types don't support compression/decompression operations | ||
| // They should only be preserved as-is | ||
| if mimeType == manifest.NydusBootstrapLayerMediaType || mimeType == manifest.NydusBlobLayerMediaType { |
There was a problem hiding this comment.
This is way too late to make a decision like that, the compression/decompression has already happened. There’s CanChangeLayerCompression for making decisions like that. (And I suspect that already returns the correct value, and updatedMIMEType correctly fails, on MIME types not present in oci1CompressionMIMETypeSets).
| github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw= | ||
| github.com/chainguard-dev/clog v1.7.0/go.mod h1:4+WFhRMsGH79etYXY3plYdp+tCz/KCkU8fAr0HoaPvs= | ||
| github.com/checkpoint-restore/go-criu/v8 v8.0.0/go.mod h1:GtCbuv5kNUGKXLEhHwURnHwhHf7VApBd4k0+5cqsRmA= | ||
| github.com/checkpoint-restore/go-criu/v7 v7.2.0/go.mod h1:u0LCWLg0w4yqqu14aXhiB4YD3a1qd8EcCEg7vda5dwo= |
3 participants
resolve #565
support nydus image format