fix(ci): pin --server-side=true alongside --force-conflicts for helm 4 upgrade#1698
Open
zanejohnson-azure wants to merge 1 commit into
Open
fix(ci): pin --server-side=true alongside --force-conflicts for helm 4 upgrade#1698zanejohnson-azure wants to merge 1 commit into
zanejohnson-azure wants to merge 1 commit into
Conversation
…4 upgrade PR #1667 added --force-conflicts to handle SSA field-ownership conflicts, but Helm 4's `helm upgrade --server-side` defaults to "auto", which resolves to false when the existing release's ApplyMethod is "client-side apply" (e.g., a release originally created by Helm 3 and never re-applied with SSA). With auto -> false and forceConflicts=true, Helm rejects the combination client-side before contacting the cluster: invalid client update option(s): forceConflicts enabled when serverSideApply disabled This is exactly what failed Stage_3 against Monitoring-Model-Cluster-WCUS / -WEU on 2026-05-22, while clusters whose last revision was already SSA succeeded. Fix: pass `--server-side=true` explicitly so SSA is engaged whenever --force-conflicts is set. The `=true` form (not bare `--server-side`) is required because: - `helm upgrade` defines --server-side as a StringVar ("true", "false", or "auto"); bare --server-side would consume the next token as the value. - `helm install` defines --server-side as a BoolVar. The explicit `=true` form is unambiguous for both subcommands. Side-effect on first run per cluster: Helm performs a field-manager handover from `kubectl-client-side-apply` to `helm` (SSA), which is why --force-conflicts is needed alongside. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Stage_3 of ci-aks-prod-release.yaml failed on 2026-05-22 for
Monitoring-Model-Cluster-WCUS(andMonitoring-Model-Cluster-WEU) with:Root cause
PR #1667 added
--force-conflictsto handle SSA field-ownership conflicts, but Helm 4'shelm upgrade --server-sidedefaults toauto, which resolves to:ApplyMethodautoresolves tossa(server-side apply)true✅client-side apply(e.g., release created by Helm 3)false❌When
auto → falseandforceConflicts=true, Helm rejects the combination client-side, before reaching the cluster (seepkg/kube/client.go:729).That perfectly explains the observed asymmetry: clusters whose last revision was already SSA (e.g.,
ci-logs-prod-aks-work-load-identityat rev 49) succeeded;Monitoring-Model-Cluster-*releases (originally Helm 3,ApplyMethod = client-side apply) failed.Fix
Add
--server-side=trueso SSA is always engaged when--force-conflictsis set.Why
=true(not bare--server-side)helm upgradedefines--server-sideas a StringVar ("true"|"false"|"auto"). Bare--server-sidewould consume the next token as the value.helm installdefines--server-sideas a BoolVar.=trueform is unambiguous for both subcommands.Production cluster recovery
Already manually re-deployed both failed clusters out-of-band using the corrected command:
As a side-effect, both releases'
ApplyMethodflipped CSA → SSA, so any futureautoresolution on those releases will producetrueand would succeed even without this fix. This PR is still needed for any other cluster whose release history is still CSA (and to keep behavior deterministic for new clusters).Test plan
The next prod-release pipeline run will exercise this path against both prod clusters; both are now on SSA, so the fix is verified in production via the manual recovery.