Extend copy operations with tarball extraction capabilities

[stefanprodan]

Flux sources such as GitRepos and Buckets can contain manifests packaged with flux build artifact or with helm package. To allow the reconciliation of such manifests, we should implement a copy strategy that extracts the contents of the archive into the destination artifact. The implementation should use fluxcd/pkg/tar.Untar.

Example

Assuming that in CI we run the following:

flux build artifact --path ./manifests/ -o manifests.tgz
aws s3 cp manifests.tgz s3://my-fleet/clusters/prod-1/manifests.tgz

We can define a generator that extracts the manifests from the tarball and places them in the root of the generated artifact:

apiVersion: source.extensions.fluxcd.io/v1beta1
kind: ArtifactGenerator
metadata:
  name: cluster-config-unpack
  namespace: flux-system
spec:
  sources:
    - alias: bucket
      kind: Bucket
      name: fleet
  artifacts:
    - name: cluster-config
      originRevision: "@bucket"
      copy:
        - from: "@bucket/clusters/prod-1/manifests.tgz"
          to: "@artifact/"
          strategy: Extract

To reconcile the manifests with a Flux Kustomization:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: cluster-sync
  namespace: flux-system
spec:
  interval: 15m
  sourceRef:
    kind: ExternalArtifact
    name: cluster-config
  path: "./"
  prune: true

[chris-morandi]

Is this to match a single archive literal reference only e.g. @bucket/clusters/prod-1/manifests.tgz or should it also match multiple archives via patterns?

[matheuscscp]

I can see pattern matching being useful 🤔 Would it make sense to extract a list of tarballs in the same target path?

[chris-morandi]

I can see pattern matching being useful 🤔 Would it make sense to extract a list of tarballs in the same target path?

That's how I've solved it for now. If you think perhaps we should be extracting into the same path they were matched that could be an alternative. Not sure which is the better option.

[matheuscscp]

I'm honestly not sure what's best here 🤔 I tried to think about some error scenarios in the first choice to maybe justify going with the second choice, but the same error scenarios could currently happen with the plain simple copy operation without any extraction. At the end of the day you need to have an understanding of the file paths and be able to debug these scenarios... I'm trying to think about a real-world use case for pattern matching...

[stealthybox]

I'm trying to think about a real-world use case for pattern matching...

I can maybe see multiple teams pushing Flux Artifacts to some central place where a single ArtifactGenerator does a fanout, combining artifacts based on some partial file-name?

ex: Platform team exposes a Bucket where the top level folders are namespaces.
The artifact generator is dynamically updated via some Namespace as a Service approach.
Teams are free to publish any number of tarballs under their namespace in the bucket, and the ArtifactGenerator will combine them into a single Source for their namespace. The team is now free to split up their Kustomizations and HelmReleases using any of the files in their source. Each tarball is considered its own atomic boundary for a changeset.

[matheuscscp]

I'm trying to think about a real-world use case for pattern matching...

I can maybe see multiple teams pushing Flux Artifacts to some central place where a single Artifact Generator does a fanout, combining artifacts based on some partial file-name?

That makes sense

[stealthybox]

edit: added an example