Skip to content

Conversation

@aditmeno
Copy link
Contributor

Summary

Fixes helmfile/helmfile#2291

This PR fixes an issue where strategicMergePatches causes CRDs from templates/crds/ to be relocated to the root crds/ directory, changing how Helm manages them and causing helm diff to show CRDs being removed.

Problem

When using strategicMergePatches with charts like KEDA that place CRDs in templates/crds/:

  1. Before patching: CRDs exist in templates/crds/ (managed as regular resources)
  2. After chartify processing: CRDs relocated to root crds/ directory (Helm special handling)
  3. Result: helm diff shows CRDs being removed, even though patches don't reference CRDs

Why This Matters

Chart authors intentionally use templates/crds/ instead of root crds/ for:

  • Conditional rendering: {{- if .Values.crds.install }}
  • Template features: .Release.Namespace, labels, annotations
  • Upgrade capability: CRDs that need updates during helm upgrade

Helm's Different Treatment

Aspect templates/crds/ crds/
Templating ✅ Go templates work ❌ Plain YAML only
On upgrade ✅ Updated ❌ Immutable (install-only)
On uninstall ✅ Deleted ❌ Orphaned

Root Cause

In patch.go:342-364, CRDs were unconditionally relocated:

if r.IsHelm3() || r.IsHelm4() {
    crdsDir = filepath.Join(tempDir, "crds")  // ❌ Always moves to crds/
}

Solution

Detect if CRDs originally came from templates/crds/ and preserve that location:

// Detect if CRDs originally came from templates/ directory
crdsFromTemplates := false
for _, f := range generatedManifestFiles {
    relPath := strings.Replace(f, tempDir+string(filepath.Separator), "", 1)
    relPath = filepath.ToSlash(relPath)
    if strings.HasPrefix(relPath, "templates/crds/") {
        crdsFromTemplates = true
        r.Logf("Detected CRDs in templates/ directory - will preserve location")
        break
    }
}

// Preserve original location to maintain chart author's intent
if crdsFromTemplates {
    crdsDir = filepath.Join(tempDir, "templates", "crds")  // ✅ Keep original
    r.Logf("Preserving CRDs in templates/crds/ (original location)")
} else if r.IsHelm3() || r.IsHelm4() {
    crdsDir = filepath.Join(tempDir, "crds")  // Standard location
    r.Logf("Placing CRDs in crds/ (standard Helm 3/4 location)")
}

Changes

patch.go

  • Lines 48-62: Added CRD source location detection logic

    • Checks if any generated files are in templates/crds/ subdirectory
    • Uses filepath.ToSlash() for cross-platform compatibility
    • Logs detection result for debugging
  • Lines 342-368: Modified CRD placement logic

    • Preserves templates/crds/ location when detected
    • Falls back to standard crds/ location for Helm 3/4
    • Maintains Helm 2 compatibility
    • Added logging for transparency
  • Comments: Added detailed explanations referencing issue #2291

patch_test.go (new file)

Added comprehensive test coverage:

  1. TestPatch_PreserveCRDLocation: End-to-end scenarios

    • ✅ CRDs from templates/crds/ stay in templates/crds/
    • ✅ CRDs from root crds/ go to crds/
    • ✅ Mixed resources without CRDs in templates/
  2. TestPatch_CRDLocationDetection: Detection logic validation

    • ✅ Correct detection of templates/crds/ files
    • ✅ No false positives for crds/ or other paths
    • ✅ Cross-platform path handling
  3. TestPatch_KEDA_RealWorld: Real-world KEDA scenario

    • ✅ Mimics actual KEDA chart structure (6 CRDs in templates/crds/)
    • ✅ Validates the fix for issue #2291

Testing

# Run new tests
go test -v -run TestPatch_PreserveCRDLocation
go test -v -run TestPatch_CRDLocationDetection
go test -v -run TestPatch_KEDA_RealWorld

# Run all tests
go test -v ./...

Test Results

All tests pass and cover:

  • ✅ CRD location preservation logic
  • ✅ Path normalization (Windows/Unix)
  • ✅ Edge cases (no CRDs, mixed resources)
  • ✅ Real-world chart structures (KEDA)

Verification

A complete reproduction test case demonstrating the issue is available with:

  • Automated verification script showing CRD relocation bug
  • KEDA chart structure analysis (v2.18.1)
  • Before/after comparison of chart structure
  • Complete technical documentation with code references

Run verification:

# See /tmp/keda-issue-test/README.md for full reproduction steps
cd /tmp/keda-issue-test
./verify-issue.sh

Backward Compatibility

No breaking changes

  • Charts with CRDs in root crds/same behavior as before
  • Charts with CRDs in templates/crds/now correctly preserved
  • Charts without CRDs → no impact
  • All existing functionality maintained

Impact

Affected Charts

Any chart with CRDs in templates/ subdirectories:

Benefits

  • ✅ Preserves chart author's intent and design
  • ✅ Fixes helm diff showing incorrect CRD deletions
  • ✅ Maintains upgrade capability for template-based CRDs
  • ✅ Supports conditional CRD installation patterns
  • ✅ Prevents accidental CRD management behavior changes

Example: KEDA Chart

Before this fix:

helm template + strategicMergePatches
  → CRDs moved from templates/crds/ to crds/
  → helm diff shows 6 CRDs being removed ❌
  → Patches on Deployments unexpectedly affect CRD management

After this fix:

helm template + strategicMergePatches
  → CRDs preserved in templates/crds/
  → helm diff shows only intended Deployment changes ✅
  → CRD management behavior unchanged

Related Issues

Review Guide

Key areas for reviewers:

  1. Detection Logic (patch.go:48-62)

    • Correctly identifies files in templates/crds/ subdirectory
    • Handles path separators cross-platform
    • Early detection before processing
  2. Preservation Logic (patch.go:342-368)

    • Places CRDs in correct location based on source
    • Maintains backward compatibility
    • Clear logging for debugging
  3. Test Coverage (patch_test.go)

    • Comprehensive scenario coverage
    • Real-world KEDA structure validation
    • Cross-platform compatibility
  4. Documentation

    • Clear comments explaining the "why"
    • References to original issue
    • Helm behavior differences documented

Ready for review

This fix resolves a critical issue where patches unintentionally change CRD management behavior, making strategicMergePatches safe to use with charts like KEDA.

Fixes #2291

Problem:
When using strategicMergePatches with charts that have CRDs in
templates/crds/ (like KEDA), chartify relocates all CRDs to the
root crds/ directory. This changes how Helm manages these CRDs,
causing helm diff to show CRDs being removed even though patches
don't reference CRDs.

Root Cause:
The Patch() function unconditionally moves all CRDs to crds/ for
Helm 3/4, regardless of their original location. This breaks charts
that intentionally place CRDs in templates/crds/ for:
- Conditional rendering ({{- if .Values.crds.install }})
- Template features (.Release.Namespace, labels, annotations)
- CRD updates during helm upgrade

Helm treats these locations differently:
- templates/crds/: Regular resources, updated on upgrade, deleted on uninstall
- crds/: Install-only, immutable on upgrade, not deleted on uninstall

Solution:
Detect if CRDs originally came from templates/crds/ subdirectory
and preserve that location. This maintains the chart author's intent
and ensures helm diff behaves correctly.

Implementation:
- Added detection logic to check if any generated files are in templates/crds/
- When detected, CRDs are placed back in templates/crds/ after patching
- Otherwise, standard Helm 3/4 behavior (crds/) is preserved
- Cross-platform compatible using filepath.ToSlash() for path normalization

Testing:
Added comprehensive test suite covering:
- CRDs from templates/crds/ preserved in original location
- CRDs from root crds/ go to standard crds/ location
- Mixed resources without CRDs in templates/
- KEDA real-world scenario validation
- Cross-platform path handling

This preserves backward compatibility while fixing the issue for
affected charts.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
aditmeno added a commit to aditmeno/helmfile that referenced this pull request Nov 24, 2025
Add comprehensive integration test to validate that strategicMergePatches
do not cause CRDs to be incorrectly relocated or removed.

## Changes

1. **Integration Test** (test/integration/test-cases/issue-2291.sh)
   - Verifies CRDs in templates/crds/ are correctly templated
   - Validates strategicMergePatches don't trigger CRD relocation
   - Ensures helm diff doesn't show CRDs as being removed
   - Confirms patches are applied only to intended resources

2. **Test Chart** (test/integration/test-cases/issue-2291/input/)
   - Minimal chart with CRD in templates/crds/ directory
   - Uses conditional rendering with .Values.crds.install
   - Includes Deployment resource to be patched
   - Strategic merge patch for DNS configuration

3. **Test Suite Integration** (test/integration/run.sh)
   - Added issue-2291 test to main test suite

4. **Chartify Dependency** (go.mod)
   - Added replace directive to use chartify PR helmfile#160
   - Points to fix/preserve-crd-location-issue-2291 branch
   - Allows testing the fix before upstream merge

## Test Validation

The test validates the complete workflow:
- Step 1: Template chart and verify CRD is included
- Step 2: Apply chart with strategicMergePatches
- Step 3: Verify CRD was installed to cluster
- Step 4: Run diff - should NOT show CRD removal (regression check)
- Step 5: Verify strategic merge patch applied to Deployment

## Related

- Issue: helmfile#2291
- Chartify PR: helmfile/chartify#160

The test will fail without the chartify fix, and pass with it,
providing regression protection for this critical bug.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
aditmeno added a commit to aditmeno/helmfile that referenced this pull request Nov 24, 2025
Add comprehensive integration test to validate that strategicMergePatches
do not cause CRDs to be incorrectly relocated or removed.

## Changes

1. **Integration Test** (test/integration/test-cases/issue-2291.sh)
   - Verifies CRDs in templates/crds/ are correctly templated
   - Validates strategicMergePatches don't trigger CRD relocation
   - Ensures helm diff doesn't show CRDs as being removed
   - Confirms patches are applied only to intended resources

2. **Test Chart** (test/integration/test-cases/issue-2291/input/)
   - Minimal chart with CRD in templates/crds/ directory
   - Uses conditional rendering with .Values.crds.install
   - Includes Deployment resource to be patched
   - Strategic merge patch for DNS configuration

3. **Test Suite Integration** (test/integration/run.sh)
   - Added issue-2291 test to main test suite

4. **Chartify Dependency** (go.mod)
   - Added replace directive to use chartify PR helmfile#160
   - Points to fix/preserve-crd-location-issue-2291 branch
   - Allows testing the fix before upstream merge

## Test Validation

The test validates the complete workflow:
- Step 1: Template chart and verify CRD is included
- Step 2: Apply chart with strategicMergePatches
- Step 3: Verify CRD was installed to cluster
- Step 4: Run diff - should NOT show CRD removal (regression check)
- Step 5: Verify strategic merge patch applied to Deployment

## Related

- Issue: helmfile#2291
- Chartify PR: helmfile/chartify#160

The test will fail without the chartify fix, and pass with it,
providing regression protection for this critical bug.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
aditmeno added a commit to aditmeno/helmfile that referenced this pull request Nov 24, 2025
Add comprehensive integration test to validate that strategicMergePatches
do not cause CRDs to be incorrectly relocated or removed.

1. **Integration Test** (test/integration/test-cases/issue-2291.sh)
   - Verifies CRDs in templates/crds/ are correctly templated
   - Validates strategicMergePatches don't trigger CRD relocation
   - Ensures helm diff doesn't show CRDs as being removed
   - Confirms patches are applied only to intended resources

2. **Test Chart** (test/integration/test-cases/issue-2291/input/)
   - Minimal chart with CRD in templates/crds/ directory
   - Uses conditional rendering with .Values.crds.install
   - Includes Deployment resource to be patched
   - Strategic merge patch for DNS configuration

3. **Test Suite Integration** (test/integration/run.sh)
   - Added issue-2291 test to main test suite

4. **Chartify Dependency** (go.mod)
   - Added replace directive to use chartify PR helmfile#160
   - Points to fix/preserve-crd-location-issue-2291 branch
   - Allows testing the fix before upstream merge

The test validates the complete workflow:
- Step 1: Template chart and verify CRD is included
- Step 2: Apply chart with strategicMergePatches
- Step 3: Verify CRD was installed to cluster
- Step 4: Run diff - should NOT show CRD removal (regression check)
- Step 5: Verify strategic merge patch applied to Deployment

- Issue: helmfile#2291
- Chartify PR: helmfile/chartify#160

The test will fail without the chartify fix, and pass with it,
providing regression protection for this critical bug.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
aditmeno added a commit to aditmeno/helmfile that referenced this pull request Nov 24, 2025
Add comprehensive integration test to validate that strategicMergePatches
do not cause CRDs to be incorrectly relocated or removed.

1. **Integration Test** (test/integration/test-cases/issue-2291.sh)
   - Verifies CRDs in templates/crds/ are correctly templated
   - Validates strategicMergePatches don't trigger CRD relocation
   - Ensures helm diff doesn't show CRDs as being removed
   - Confirms patches are applied only to intended resources

2. **Test Chart** (test/integration/test-cases/issue-2291/input/)
   - Minimal chart with CRD in templates/crds/ directory
   - Uses conditional rendering with .Values.crds.install
   - Includes Deployment resource to be patched
   - Strategic merge patch for DNS configuration

3. **Test Suite Integration** (test/integration/run.sh)
   - Added issue-2291 test to main test suite

4. **Chartify Dependency** (go.mod)
   - Added replace directive to use chartify PR helmfile#160
   - Points to fix/preserve-crd-location-issue-2291 branch
   - Allows testing the fix before upstream merge

The test validates the complete workflow:
- Step 1: Template chart and verify CRD is included
- Step 2: Apply chart with strategicMergePatches
- Step 3: Verify CRD was installed to cluster
- Step 4: Run diff - should NOT show CRD removal (regression check)
- Step 5: Verify strategic merge patch applied to Deployment

- Issue: helmfile#2291
- Chartify PR: helmfile/chartify#160

The test will fail without the chartify fix, and pass with it,
providing regression protection for this critical bug.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
aditmeno added a commit to aditmeno/helmfile that referenced this pull request Nov 24, 2025
Add comprehensive integration test to validate that strategicMergePatches
do not cause CRDs to be incorrectly relocated or removed.

1. **Integration Test** (test/integration/test-cases/issue-2291.sh)
   - Verifies CRDs in templates/crds/ are correctly templated
   - Validates strategicMergePatches don't trigger CRD relocation
   - Ensures helm diff doesn't show CRDs as being removed
   - Confirms patches are applied only to intended resources

2. **Test Chart** (test/integration/test-cases/issue-2291/input/)
   - Minimal chart with CRD in templates/crds/ directory
   - Uses conditional rendering with .Values.crds.install
   - Includes Deployment resource to be patched
   - Strategic merge patch for DNS configuration

3. **Test Suite Integration** (test/integration/run.sh)
   - Added issue-2291 test to main test suite

4. **Chartify Dependency** (go.mod)
   - Added replace directive to use chartify PR helmfile#160
   - Points to fix/preserve-crd-location-issue-2291 branch
   - Allows testing the fix before upstream merge

The test validates the complete workflow:
- Step 1: Template chart and verify CRD is included
- Step 2: Apply chart with strategicMergePatches
- Step 3: Verify CRD was installed to cluster
- Step 4: Run diff - should NOT show CRD removal (regression check)
- Step 5: Verify strategic merge patch applied to Deployment

- Issue: helmfile#2291
- Chartify PR: helmfile/chartify#160

The test will fail without the chartify fix, and pass with it,
providing regression protection for this critical bug.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
aditmeno added a commit to aditmeno/helmfile that referenced this pull request Nov 24, 2025
Add comprehensive integration test to validate that strategicMergePatches
do not cause CRDs to be incorrectly relocated or removed.

1. **Integration Test** (test/integration/test-cases/issue-2291.sh)
   - Verifies CRDs in templates/crds/ are correctly templated
   - Validates strategicMergePatches don't trigger CRD relocation
   - Ensures helm diff doesn't show CRDs as being removed
   - Confirms patches are applied only to intended resources

2. **Test Chart** (test/integration/test-cases/issue-2291/input/)
   - Minimal chart with CRD in templates/crds/ directory
   - Uses conditional rendering with .Values.crds.install
   - Includes Deployment resource to be patched
   - Strategic merge patch for DNS configuration

3. **Test Suite Integration** (test/integration/run.sh)
   - Added issue-2291 test to main test suite

4. **Chartify Dependency** (go.mod)
   - Added replace directive to use chartify PR helmfile#160
   - Points to fix/preserve-crd-location-issue-2291 branch
   - Allows testing the fix before upstream merge

The test validates the complete workflow:
- Step 1: Template chart and verify CRD is included
- Step 2: Apply chart with strategicMergePatches
- Step 3: Verify CRD was installed to cluster
- Step 4: Run diff - should NOT show CRD removal (regression check)
- Step 5: Verify strategic merge patch applied to Deployment

- Issue: helmfile#2291
- Chartify PR: helmfile/chartify#160

The test will fail without the chartify fix, and pass with it,
providing regression protection for this critical bug.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
aditmeno added a commit to aditmeno/helmfile that referenced this pull request Nov 24, 2025
Add comprehensive integration test to validate that strategicMergePatches
do not cause CRDs to be incorrectly relocated or removed.

1. **Integration Test** (test/integration/test-cases/issue-2291.sh)
   - Verifies CRDs in templates/crds/ are correctly templated
   - Validates strategicMergePatches don't trigger CRD relocation
   - Ensures helm diff doesn't show CRDs as being removed
   - Confirms patches are applied only to intended resources

2. **Test Chart** (test/integration/test-cases/issue-2291/input/)
   - Minimal chart with CRD in templates/crds/ directory
   - Uses conditional rendering with .Values.crds.install
   - Includes Deployment resource to be patched
   - Strategic merge patch for DNS configuration

3. **Test Suite Integration** (test/integration/run.sh)
   - Added issue-2291 test to main test suite

4. **Chartify Dependency** (go.mod)
   - Added replace directive to use chartify PR helmfile#160
   - Points to fix/preserve-crd-location-issue-2291 branch
   - Allows testing the fix before upstream merge

The test validates the complete workflow:
- Step 1: Template chart and verify CRD is included
- Step 2: Apply chart with strategicMergePatches
- Step 3: Verify CRD was installed to cluster
- Step 4: Run diff - should NOT show CRD removal (regression check)
- Step 5: Verify strategic merge patch applied to Deployment

- Issue: helmfile#2291
- Chartify PR: helmfile/chartify#160

The test will fail without the chartify fix, and pass with it,
providing regression protection for this critical bug.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
@aditmeno
Copy link
Contributor Author

@yxxhero PR is ready for review

@zhaque44
Copy link
Contributor

@yxxhero @mumoshu are we sure we want this change?

@yxxhero
Copy link
Member

yxxhero commented Nov 24, 2025

@aditmeno LGTM.

@yxxhero yxxhero merged commit a15915f into helmfile:master Nov 24, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use of strategicMergePatches causes KEDA chart to uninstall its CRDs

3 participants