fix(reftracker): copy refs map to avoid concurrent iteration panic#1825
Open
SAY-5 wants to merge 2 commits into
Open
fix(reftracker): copy refs map to avoid concurrent iteration panic#1825SAY-5 wants to merge 2 commits into
SAY-5 wants to merge 2 commits into
Conversation
…n panic AppsForRef and RefsForApp returned the internal map directly, so callers iterated outside the tracker lock while ReconcileRefs and RemoveAppFromAllRefs mutated the same maps under the lock. Under load this triggered fatal 'concurrent map iteration and map write' panics in the controller. Returning a copied set lets callers iterate safely; the map values are unused so the shallow copy is sufficient. Fixes carvel-dev#1812 Signed-off-by: Sai Asish Y <say.apm35@gmail.com>
Signed-off-by: Sai Asish Y <say.apm35@gmail.com>
SAY-5
force-pushed
the
fix-apprefttracker-concurrent-iteration
branch
from
c714a84 to
7215232
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
AppRefTracker.AppsForRefandRefsForAppreturned the internal map directly. Callers (SecretHandler.enqueueAppsForUpdate,ConfigMapHandler.enqueueAppsForUpdate) then iterated those maps outside the tracker lock whileReconcileRefsandRemoveAppFromAllRefswere mutating the same maps under the lock. On large clusters with high churn this triggered Go's fatalconcurrent map iteration and map writepanic, taking down the controller.The fix is to return a shallow copy of the set; the map values are unused so a copy of the keys is sufficient. Callers can iterate the snapshot without holding the lock.
Which issue(s) this PR fixes:
Fixes #1812
Does this PR introduce a user-facing change?
Additional Notes for your reviewer:
The new test
Test_AppsForRef_SafeForConcurrentIterationreproduces the original panic when run with-raceagainst the previous implementation and passes after the fix.Review Checklist:
a link to that PR
change
Additional documentation e.g., Proposal, usage docs, etc.: