Extend copy operations with tarball extraction capabilities

[chris-morandi]

Update generator Builder to support Extraction of tarball archives

Assumptions made

1. Pattern matching inherited from current copy operation including recursive **
2. If matching multiple archives they are all extracted to the same destination
3. None-archive files are ignored when specifying Extract
4. Extraction over existing files will always be overwrite

Fixes #301

Changes

• Updates to ArtifactGenerator API and builder module to incorporate the Extract strategy into copy operations
• Switched over the FS.Glob call to doublestar.Glob for the Extract strategy to avoid walking the entire tree
• Tests around Extract.

Testing (other than unit testing)

1. Deploy controller in kind cluster

kind create cluster
make dev-deploy
make docker-build
kind load docker-image fluxcd/source-watcher:latest

5. Create bucket as the initial source (using Minio internal server)

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: Bucket
metadata:
  name: fleet
  namespace: flux-system
spec:
  interval: 1m0s
  provider: generic
  bucketName: my-bucket
  endpoint: minio.minio.svc.cluster.local:9000
  insecure: true
  secretRef:
    name: minio-bucket-creds

6. Upload archive including a deployment of podinfo generated using

flux build artifact --path ./podinfo/ -o podinfo.tgz

7. Create ArtifactGenerator to generate an ExternalArtifact from the Bucket

apiVersion: source.extensions.fluxcd.io/v1beta1
kind: ArtifactGenerator
metadata:
  name: artifactgenerator-podinfo
  namespace: flux-system
spec:
  sources:
    - alias: bucket
      kind: Bucket
      name: fleet
  artifacts:
    - name: podinfo
      originRevision: "@bucket"
      copy:
        - from: "@bucket/podinfo.t*"
          to: "@artifact/"
          strategy: Extract

➜  source-watcher-tests kc get ag  -n flux-system 
NAME                        AGE   READY   STATUS
artifactgenerator-podinfo   67m   True    reconciliation succeeded, generated 1 artifact(s)

➜  source-watcher-tests kc get externalartifact -n flux-system
NAME      AGE   READY   STATUS              SOURCE
podinfo   68m   True    Artifact is ready   artifactgenerator-podinfo


 source-watcher-tests curl -L http://localhost:8080/externalartifact/flux-system/podinfo/2114082096.tar.gz | tar -tzv    
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Handling connection for 8080
100  1191  100  1191    0     0  66302      0 --:--:-- --:--:-- --:--:-- 70058
drwxr-x---  0 0      0           0 Jan  1  1970 .
-rw-------  0 0      0        1786 Jan  1  1970 deployment.yaml
-rw-------  0 0      0         306 Jan  1  1970 kustomization.yaml
-rw-------  0 0      0         142 Jan  1  1970 namespace.yaml
drwxr-x---  0 0      0           0 Jan  1  1970 rbac
-rw-------  0 0      0         473 Jan  1  1970 rbac/role.yaml
-rw-------  0 0      0         343 Jan  1  1970 rbac/rolebinding.yaml
-rw-------  0 0      0         168 Jan  1  1970 rbac/serviceaccount.yaml
-rw-------  0 0      0         396 Jan  1  1970 service.yaml

8. Kustomization to read the ExternalArtifact

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: deploy-podinfo
  namespace: flux-system
spec:
  interval: 15m
  sourceRef:
    kind: ExternalArtifact
    name: podinfo
  path: "./"
  prune: true

➜  source-watcher-tests kc get ks -n flux-system
NAME             AGE   READY   STATUS
deploy-podinfo   91m   True    Applied revision: latest@sha256:21a9ea0dc9cb3fee65e9e5bc774c1f4a1ec611337a124b339207661472da454f

Name:         deploy-podinfo
Namespace:    flux-system
Labels:       <none>
Annotations:  <none>
API Version:  kustomize.toolkit.fluxcd.io/v1
Kind:         Kustomization
Metadata:
  Creation Timestamp:  2025-12-28T21:22:33Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:        1
  Resource Version:  13699
  UID:               e4b9bd8a-ac16-49ab-a58f-816a1891c6a1
Spec:
  Force:     false
  Interval:  15m
  Path:      ./
  Prune:     true
  Source Ref:
    Kind:  ExternalArtifact
    Name:  podinfo
Status:
  Conditions:
    Last Transition Time:  2025-12-28T22:43:45Z
    Message:               Applied revision: latest@sha256:21a9ea0dc9cb3fee65e9e5bc774c1f4a1ec611337a124b339207661472da454f
    Observed Generation:   1
    Reason:                ReconciliationSucceeded
    Status:                True
    Type:                  Ready
  History:
    Digest:                    sha256:a9c5df0e811c812bc8aa0178e5e6f66bfd6d13103624a16c23b63bf35a2e9ca8
    First Reconciled:          2025-12-28T21:25:39Z
    Last Reconciled:           2025-12-28T22:43:45Z
    Last Reconciled Duration:  222.521166ms
    Last Reconciled Status:    ReconciliationSucceeded
    Metadata:
      Origin Revision:      sha256:b214ef672140efce168eb91ef42fe2e2263080d267ff9593aa69c83a146c694d
      Revision:             latest@sha256:21a9ea0dc9cb3fee65e9e5bc774c1f4a1ec611337a124b339207661472da454f
    Total Reconciliations:  5
  Inventory:
    Entries:
      Id:                        _podinfo__Namespace
      V:                         v1
      Id:                        podinfo_podinfo__ServiceAccount
      V:                         v1
      Id:                        podinfo_podinfo_rbac.authorization.k8s.io_Role
      V:                         v1
      Id:                        podinfo_podinfo_rbac.authorization.k8s.io_RoleBinding
      V:                         v1
      Id:                        podinfo_podinfo__Service
      V:                         v1
      Id:                        podinfo_podinfo_apps_Deployment
      V:                         v1
  Last Applied Origin Revision:  sha256:b214ef672140efce168eb91ef42fe2e2263080d267ff9593aa69c83a146c694d
  Last Applied Revision:         latest@sha256:21a9ea0dc9cb3fee65e9e5bc774c1f4a1ec611337a124b339207661472da454f
  Last Attempted Revision:       latest@sha256:21a9ea0dc9cb3fee65e9e5bc774c1f4a1ec611337a124b339207661472da454f
  Observed Generation:           1
Events:
  Type    Reason                   Age   From                  Message
  ----    ------                   ----  ----                  -------
  Normal  ReconciliationSucceeded  73m   kustomize-controller  Reconciliation finished in 159.470667ms, next run in 15m0s

➜  source-watcher-tests kc get all -n podinfo
NAME                          READY   STATUS    RESTARTS   AGE
pod/podinfo-cf995b8f9-9sb9w   1/1     Running   0          89m
pod/podinfo-cf995b8f9-knfb6   1/1     Running   0          89m

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/podinfo   ClusterIP   10.96.132.241   <none>        9898/TCP,9999/TCP   89m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/podinfo   2/2     2            2           89m

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/podinfo-cf995b8f9   2         2         2       89m

[chris-morandi]

**For initial comments. Going to do some further testing on the original operations for merge/overwrite strategies incase the change of globing has impacted them. Unit tests run but probably needs a better look. **

[chris-morandi]

Tested with previous Copy operations. Resulted that I needed to change the Glob function depending on the strategy. fs.Glob for Overwrite/Merge std files and doublestar.Glob for tarballs. With Extract strategy we don't need to walk the tree

[stefanprodan]

LGTM

Thanks @chris-morandi 🥇

[matheuscscp]

LGTM! 🚀

Just a few nits

[matheuscscp]

Can you please amend the commit message to match the PR title?

[chris-morandi]

Seems a garbage collection test is failing. Not seeing this locally. I'll have a look later.

[matheuscscp]

Seems a garbage collection test is failing. Not seeing this locally. I'll have a look later.

The tests seem to have ran fine in CI, looks like you just missed regenerating the CRD YAML after changing the struct comment:

https://github.com/fluxcd/source-watcher/actions/runs/20846092767/job/59890249872?pr=302

[chris-morandi]

Yeap fixed that. Yeah dunno what happened with test.