Skip to content

Fix multus kubeconfig brackets before binary upgrade#3000

Open
sdodson wants to merge 1 commit intoopenshift:release-4.14from
sdodson:fix-multus-kubeconfig-brackets-4.14
Open

Fix multus kubeconfig brackets before binary upgrade#3000
sdodson wants to merge 1 commit intoopenshift:release-4.14from
sdodson:fix-multus-kubeconfig-brackets-4.14

Conversation

@sdodson
Copy link
Copy Markdown
Member

@sdodson sdodson commented May 8, 2026

Summary

  • Adds an init container to the multus DaemonSet that strips brackets from non-IPv6 hostnames in the existing multus kubeconfig before the main container copies the new binary
  • Fixes upgrade failures caused by the Go bump for CVE-2025-47912 where net/url now rejects https://[hostname]:6443 format URLs
  • The init container is a no-op on fresh installs and already-fixed nodes; IPv6 addresses in brackets are preserved

Problem

During 4.13→4.14 upgrades, the old multus kubeconfig on nodes (written by the 4.13 shell entrypoint which unconditionally wraps KUBERNETES_SERVICE_HOST in brackets) contains server URLs like https://[hostname]:6443. When the 4.14 multus DaemonSet rolls out, cnibincopy.sh copies the new binary before multus-daemon rewrites the kubeconfig. CRI-O immediately uses the new binary for pod sandbox teardowns (DEL), reads the old bracketed kubeconfig, and fails:

Multus: error getting k8s client: host must be a URL or a host:port pair:
"https://[api-int.ci-op-...]:6443"

This blocks all pod termination (500+ FailedKillPod events), stalls the dns-default DaemonSet rollout, causes DNS operator degradation, and fails the upgrade. The gcp-ovn-rt-upgrade-4.14-minor blocking job has been failing for 5+ consecutive payloads due to this issue.

Fix

Add a fix-cni-kubeconfig init container that runs before kube-multus:

  1. Checks if the kubeconfig exists (exits on fresh install)
  2. If the server URL has brackets around a non-IPv6 hostname, strips them with sed
  3. Main container then safely copies the new binary — CRI-O uses new binary + fixed kubeconfig

Test plan

  • Verify gcp-ovn-rt-upgrade-4.14-minor (4.13→4.14 upgrade) passes with this change
  • Verify fresh 4.14 install succeeds (init container should exit cleanly with no kubeconfig to fix)
  • Verify IPv6 dual-stack clusters preserve bracketed IPv6 addresses in kubeconfigs

🤖 Generated with Claude Code

The Go bump for CVE-2025-47912 causes net/url to reject non-IPv6
hostnames wrapped in brackets. During 4.13->4.14 upgrades, the old
multus kubeconfig on nodes contains server URLs like
https://[hostname]:6443 (written by the 4.13 entrypoint which
unconditionally wraps KUBERNETES_SERVICE_HOST in brackets).

When the 4.14 multus DaemonSet rolls out, cnibincopy.sh copies the
new binary to the node before multus-daemon rewrites the kubeconfig.
CRI-O immediately uses the new binary for pod sandbox teardowns, reads
the old bracketed kubeconfig, and every DEL call fails with:

  Multus: error getting k8s client: host must be a URL or a host:port
  pair: "https://[hostname]:6443"

This blocks all pod termination, stalls the dns-default DaemonSet
rollout, and causes DNS operator degradation — failing the upgrade.

Add an init container that reads the existing kubeconfig and strips
brackets from non-IPv6 hostnames before the main container copies the
new binary. IPv6 addresses in brackets are preserved. The init
container is a no-op on fresh installs (no kubeconfig exists) and on
already-fixed nodes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 8, 2026

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 244e6e6b-16f0-460f-8b85-a98f6618d850

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Comment @coderabbitai help to get the list of available commands and usage tips.

@sdodson
Copy link
Copy Markdown
Member Author

sdodson commented May 8, 2026

/test ci/prow/4.14-upgrade-from-stable-4.13-images

@sdodson
Copy link
Copy Markdown
Member Author

sdodson commented May 8, 2026

/retest-required

KC="/host/etc/cni/net.d/multus.d/multus.kubeconfig"
[ -f "$KC" ] || exit 0
if grep -q 'server:.*://\[' "$KC"; then
sed -i -E 's|(server: https?://)(\[)([^]]+[^0-9:.])(\])|\1\3|g' "$KC"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sed -i -E 's|(server: https?://)\[([^]:]+)\]|\1\2|g' "$KC"
replace with this so that we dont break on dual stack clusters

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants