Skip to content

Fix ServiceAccount secret accumulation on OpenShift#3627

Open
ChrisJBurns wants to merge 3 commits intomainfrom
fix/openshift-serviceaccount-secret-accumulation
Open

Fix ServiceAccount secret accumulation on OpenShift#3627
ChrisJBurns wants to merge 3 commits intomainfrom
fix/openshift-serviceaccount-secret-accumulation

Conversation

@ChrisJBurns
Copy link
Collaborator

Summary

  • Fix unbounded ServiceAccount secret accumulation on OpenShift/OKD clusters
  • Preserve existing Secrets and ImagePullSecrets fields when the operator doesn't explicitly set them
  • Add tests for OpenShift compatibility and explicit overwrite scenarios

Problem

On OpenShift/OKD 4.15, the operator's reconciliation loop caused unbounded creation of kubernetes.io/service-account-token and kubernetes.io/dockercfg secrets for ServiceAccounts. Over ~1 week, hundreds to thousands of orphaned secrets accumulated per MCPServer instance (~90 orphaned secret pairs/day).

Root cause: The upsertServiceAccount function unconditionally overwrote Secrets and ImagePullSecrets fields with nil values. On OpenShift, the openshift-controller-manager automatically manages these fields, so when cleared, OpenShift created new secrets while old ones became orphaned.

Solution

Modified upsertServiceAccount to only overwrite Secrets and ImagePullSecrets when explicitly specified (non-nil). When nil, existing values are preserved.

This follows the pattern recommended in: operator-framework/operator-sdk#6494

Impact

  • OpenShift/OKD: Fixes the secret accumulation bug
  • Standard Kubernetes: No change in behavior (these fields are typically empty)
  • Future compatibility: If explicit management of these fields is ever needed, passing a non-nil value (including empty slice []) will apply the desired state

Test plan

  • Added test: preserves_existing_Secrets_and_ImagePullSecrets_when_not_specified_(OpenShift_compatibility)
  • Added test: overwrites_Secrets_and_ImagePullSecrets_when_explicitly_specified
  • All existing tests pass
  • Manual verification on OpenShift cluster (reporter can verify)

Fixes #3622

🤖 Generated with Claude Code

On OpenShift/OKD, the openshift-controller-manager automatically manages
the Secrets and ImagePullSecrets fields on ServiceAccounts by creating
kubernetes.io/service-account-token and kubernetes.io/dockercfg secrets.

Previously, the operator's upsertServiceAccount function unconditionally
overwrote these fields with nil during reconciliation. This caused
OpenShift to detect the SA as "missing dockercfg" and create new secrets,
while the old ones became orphaned. Over time, this led to unbounded
secret accumulation (~90 orphaned secret pairs/day).

This fix preserves existing Secrets and ImagePullSecrets fields when the
desired values are nil, following the pattern recommended in:
operator-framework/operator-sdk#6494

The fix is safe for standard Kubernetes clusters where these fields are
typically empty anyway.

Fixes #3622

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions github-actions bot added the size/S Small PR: 100-299 lines changed label Feb 5, 2026
@codecov
Copy link

codecov bot commented Feb 5, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.77%. Comparing base (533821b) to head (c70231d).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3627      +/-   ##
==========================================
- Coverage   65.78%   65.77%   -0.02%     
==========================================
  Files         411      411              
  Lines       40727    40729       +2     
==========================================
- Hits        26794    26791       -3     
- Misses      11850    11857       +7     
+ Partials     2083     2081       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions github-actions bot added size/S Small PR: 100-299 lines changed and removed size/S Small PR: 100-299 lines changed labels Feb 5, 2026
@github-actions github-actions bot added size/S Small PR: 100-299 lines changed and removed size/S Small PR: 100-299 lines changed labels Feb 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/S Small PR: 100-299 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] OKD 4.15 (Openshift). Toolhive Operator. MCPServer CRD. Abnormal, constantly growing number of secrets for a service account

2 participants