Bug: crds: CreateReplace fails when chart's crds/ directory contains non-CRD resources (e.g. ValidatingAdmissionPolicy)

[p-zany]

Upstream: envoyproxy/gateway#9015

Bug description

When a Helm chart bundles non-CRD resources (such as ValidatingAdmissionPolicy) inside its crds/ directory (or a sub-chart's crds/ directory), setting crds: CreateReplace on a HelmRelease causes the install/upgrade to fail with an error like:

failed to apply CustomResourceDefinitions: failed to update CustomResourceDefinition(s): no ValidatingAdmissionPolicy with the name "safe-upgrades.gateway.networking.k8s.io" found

Reproduction

Using gateway-helm chart from oci://docker.io/envoyproxy/gateway-helm at version 1.8.0. This chart bundles ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding resources inside charts/crds/crds/gatewayapi-crds.yaml (the crds/ directory of the crds sub-chart).

HelmRelease:

spec:
  install:
    crds: CreateReplace
  upgrade:
    crds: CreateReplace

Error on install when the ValidatingAdmissionPolicy does not yet exist in the cluster:

failed to apply CustomResourceDefinitions: failed to update CustomResourceDefinition(s):
no ValidatingAdmissionPolicy with the name "safe-upgrades.gateway.networking.k8s.io" found

Root cause

internal/action/crds.go in the CreateReplace branch processes all resources found in a chart's crds/ directories. When it encounters a non-CustomResourceDefinition resource (like ValidatingAdmissionPolicy), it attempts an update (PUT) operation on it. If the resource does not yet exist in the cluster, the update fails with "not found" instead of falling back to a create.

Expected behavior

CreateReplace should handle non-CRD resources found in crds/ directories using upsert semantics (create if not exists, replace if exists) — the same as Server-Side Apply — rather than failing when the resource is absent.

Alternatively, non-CRD resources in crds/ could be skipped with a warning, since placing them there is unconventional (see the related upstream issue in envoyproxy/gateway).

Environment

• flux2: v2.x (helm-controller using helm.toolkit.fluxcd.io/v2)
• Chart: gateway-helm v1.8.0 from oci://docker.io/envoyproxy
• Kubernetes: v1.32.7+k3s1

[matheuscscp]

The best we can do in helm-controller is ignore non-CRD objects with a loud warning. The CreateReplace code path was firmly designed on top of the assumption that everything under crds/ is a CRD. You need to open an issue in envoyproxy/gateway and ask them to split the VAP off from crds/ and move it into templates/, like I did in this other Gateway API project I have:

• CRDs under crds/: https://github.com/matheuscscp/cloudflare-gateway-controller/blob/main/charts/cloudflare-gateway-controller/crds/gateway-api.yaml
• VAP under templates/: https://github.com/matheuscscp/cloudflare-gateway-controller/blob/main/charts/cloudflare-gateway-controller/templates/gateway-api.yaml
• How to disable: https://github.com/matheuscscp/cloudflare-gateway-controller/blob/main/charts/cloudflare-gateway-controller/README.md
• Automation for the split: https://github.com/matheuscscp/cloudflare-gateway-controller/blob/main/.github/workflows/upgrade-gateway-api.yaml

Edit: I see you already opened one, thanks.

[matheuscscp]

I filed a PR to print a warning log and ignore non-CRD objects. This fix should land in a patch release next week.

Here's a release candidate for helm-controller 1.5.5 with the fix (branch), it should unblock you from installing envoyproxy/gateway's gateway-helm chart in your clusters (we are also migrating to Envoy Gateway in our clusters and are now using this fix):

image: ghcr.io/fluxcd/helm-controller
tag: rc-70c4bf56
digest: sha256:ed9d0fd6654aa3b206aa590eed35c4a11a3bdb82fdc4544344e85d8e473fd926