CNTRLPLANE-642: Handle missing resource group gracefully during Azure cluster deletion #7394

bryan-cox · 2025-12-16T12:38:57Z

What this PR does / why we need it:

Fixes Azure HostedCluster deletion when the resource group is already gone (404).

Two fixes:

CLI: hypershift destroy cluster azure now handles missing resource groups gracefully instead of erroring
Operator: Implements DeleteOrphanedMachines for Azure to remove CAPZ finalizers from orphaned AzureMachines, unblocking the Machine → NodePool → HostedCluster deletion chain

Key changes:

Refactor Azure 404 error detection to use SDK-recommended errors.As pattern
Add resourceGroupChecker interface with fail-safe approach (only act on definitive 404)
Add unit/integration tests with Azure SDK mock patterns

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/CNTRLPLANE-642

Special notes for your reviewer:

The operator fix uses a fail-safe approach: finalizers are only removed when Azure definitively returns 404. Other errors (403, network issues) assume the resource group exists to avoid orphaning cloud resources.

Checklist:

Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

openshift-ci-robot · 2025-12-16T12:39:02Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target either version "4.22." or "openshift-4.22.", but it targets "openshift-4.16" instead.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-12-16T12:39:02Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

coderabbitai · 2025-12-16T12:39:04Z

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Excluded labels (none allowed) (1)

do-not-merge/work-in-progress

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

openshift-ci-robot · 2025-12-16T12:39:06Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target either version "4.22." or "openshift-4.22.", but it targets "openshift-4.16" instead.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-12-16T12:39:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [bryan-cox]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2025-12-16T12:40:56Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target either version "4.22." or "openshift-4.22.", but it targets "openshift-4.16" instead.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-12-16T12:41:00Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target either version "4.22." or "openshift-4.22.", but it targets "openshift-4.16" instead.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-12-16T12:56:17Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-12-16T12:56:53Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-12-16T14:12:48Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-12-16T14:12:52Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-12-16T14:27:56Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-12-16T14:28:11Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-12-16T14:55:43Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-12-16T14:55:47Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

bryan-cox · 2025-12-16T20:33:52Z

/test all

bryan-cox · 2025-12-16T20:52:46Z

/retest

bryan-cox · 2025-12-17T09:34:24Z

/test e2e-aks

bryan-cox · 2025-12-17T10:48:44Z

/auto-cc

jparrill

Some minor comments, looks good overall

jparrill · 2025-12-17T11:06:23Z

cmd/cluster/azure/destroy.go

 	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources"

-	"k8s.io/apimachinery/pkg/util/errors"
+	kerrors "k8s.io/apimachinery/pkg/util/errors"


Rename not needed

Done. Removed the kerrors alias since there's no conflicting import.

AI-assisted response via Claude Code

jparrill · 2025-12-17T11:06:43Z

cmd/cluster/azure/destroy.go

 		inputErrors = append(inputErrors, fmt.Errorf("location is required"))
 	}
-	if err := errors.NewAggregate(inputErrors); err != nil {
+	if err := kerrors.NewAggregate(inputErrors); err != nil {


Not needed too

Done. Updated to use errors.NewAggregate directly.

AI-assisted response via Claude Code

jparrill · 2025-12-17T11:26:05Z

cmd/infra/azure/destroy.go

 	} else {
 		logger.Info("Deleting main resource group", "resource-group", mainResourceGroup)
 		destroyFuture, err = resourceGroupClient.BeginDelete(ctx, mainResourceGroup, nil)
 		if err != nil {


I see many if/else in this block (from if o.PreserveResourceGroup {. Is possible to enhance it a bit?

Done. Extracted the resource group deletion logic into a deleteResourceGroup helper method that handles the 404 case gracefully. This reduces nesting and makes the code more readable.

AI-assisted response via Claude Code

jparrill · 2025-12-17T11:27:32Z

support/testutil/azure.go

+//   - Azure SDK Design Guidelines for Go: https://azure.github.io/azure-sdk/golang_introduction.html
+//   - Azure SDK internal mock package: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/internal/mock
+//   - Azure SDK fake server examples: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/samples/fakes
+//   - policy.Transporter interface: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore/policy#Transporter
+//   - azcore.TokenCredential interface: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore#TokenCredential


Pretty useful 👍

Thanks! Glad these utilities are helpful.

AI-assisted response via Claude Code

jparrill · 2025-12-17T11:30:16Z

support/testutil/azure.go

+}
+
+// Do returns a mock HTTP response with the configured status code and body.
+func (m *MockAzureTransport) Do(req *http.Request) (*http.Response, error) {


Could these mocks be changed by mockgen ones? (Maybe not due to the best practices things, but just confirming)

The Azure SDK for Go recommends mocking at the HTTP transport level rather than using interface mocks. This is because the SDK's client types aren't designed around mockable interfaces - they use concrete types that wrap the transport layer.

The Azure SDK team provides guidance on this approach:

Azure SDK Design Guidelines for Go

Azure SDK fake server examples

Using policy.Transporter gives us control at the HTTP level, which is where Azure recommends testing. This allows us to test the actual client code paths including error handling, retries, and response parsing - things that would be bypassed with interface mocks.

That said, if the team prefers mockgen for consistency with other parts of the codebase, we could explore wrapping the client behind an interface, though that would require additional abstraction layers.

AI-assisted response via Claude Code

…deletion When a user-provided resource group name does not exist in Azure (404), the destroy operation now continues gracefully rather than failing. This prevents cluster destroy from getting stuck when the Azure infrastructure has already been manually deleted or was never fully created. Changes: - Add IsNotFoundError helper to check for Azure 404 responses using Azure SDK best practices (errors.As with azcore.ResponseError) - Add deleteResourceGroup helper to consolidate resource group deletion logic - Check resource group existence before attempting deletion when using user-provided resource group names - Handle 404 errors gracefully by logging and continuing with cluster deletion The helper function follows Azure SDK guidelines for error handling: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore#ResponseError 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add integration-style tests that verify the actual DestroyInfraOptions.Run behavior when Azure returns a 404 ResourceGroupNotFound error. This follows the Azure SDK testing pattern of injecting mock transports at the HTTP level. Changes: - Add clientOptions and azureCredential fields to DestroyInfraOptions for dependency injection during testing - Add shared Azure test utilities in support/testutil/azure.go: - FakeAzureCredential: implements azcore.TokenCredential - MockAzureTransport: implements policy.Transporter - MockAzureResourceGroupTransport: handles HEAD/DELETE methods properly - Factory functions for common error scenarios - Add TestDestroyInfraRun test that verifies: - Resource group exists: deletion succeeds - 404 errors are handled gracefully (no error returned) - Other errors (403) are properly propagated References: - Azure SDK Design Guidelines: https://azure.github.io/azure-sdk/golang_introduction.html - Azure SDK internal mock package: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/internal/mock - Azure SDK fake server examples: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/samples/fakes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

bryan-cox · 2025-12-17T16:38:53Z

Tested this out. Here is what previously happened.

% ./bin/hypershift destroy cluster azure --name $CLUSTER_NAME --azure-creds $AZURE_CREDS --resource-group-name ${MANAGED_RG_NAME}
{"level":"info","ts":"2025-12-17T10:55:06-05:00","msg":"Found hosted cluster","namespace":"clusters","name":"brcox-sm-hc"}
{"level":"info","ts":"2025-12-17T10:55:06-05:00","msg":"Using credentials from file","path":"/Users/brcox/.azure/credentials"}
{"level":"error","ts":"2025-12-17T10:55:07-05:00","msg":"Failed to destroy cluster","error":"failed to get resource group name, 'brcox-sm-managed-rg': GET https://management.azure.com/subscriptions/5f99720c-6823-4792-8a28-69efb0719eea/resourcegroups/brcox-sm-managed-rg\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: ResourceGroupNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ResourceGroupNotFound\",\n    \"message\": \"Resource group 'brcox-sm-managed-rg' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n","stacktrace":"github.com/openshift/hypershift/cmd/cluster/azure.NewDestroyCommand.func1\n\t/Users/brcox/bryan-cox/hypershift_ws/hypershift/cmd/cluster/azure/destroy.go:53\ngithub.com/spf13/cobra.(*Command).execute\n\t/Users/brcox/bryan-cox/hypershift_ws/hypershift/vendor/github.com/spf13/cobra/command.go:1019\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/Users/brcox/bryan-cox/hypershift_ws/hypershift/vendor/github.com/spf13/cobra/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\t/Users/brcox/bryan-cox/hypershift_ws/hypershift/vendor/github.com/spf13/cobra/command.go:1071\ngithub.com/spf13/cobra.(*Command).ExecuteContext\n\t/Users/brcox/bryan-cox/hypershift_ws/hypershift/vendor/github.com/spf13/cobra/command.go:1064\nmain.main\n\t/Users/brcox/bryan-cox/hypershift_ws/hypershift/main.go:78\nruntime.main\n\t/opt/homebrew/Cellar/go/1.24.5/libexec/src/runtime/proc.go:283"}

Here is what happens now with this PR:

% ./bin/hypershift destroy cluster azure --name $CLUSTER_NAME --azure-creds $AZURE_CREDS --resource-group-name ${MANAGED_RG_NAME}
{"level":"info","ts":"2025-12-17T10:57:43-05:00","msg":"Found hosted cluster","namespace":"clusters","name":"brcox-sm-hc"}
{"level":"info","ts":"2025-12-17T10:57:43-05:00","msg":"Using credentials from file","path":"/Users/brcox/.azure/credentials"}
{"level":"info","ts":"2025-12-17T10:57:44-05:00","msg":"Resource group not found, skipping Azure resource cleanup","resource-group":"brcox-sm-managed-rg"}

When deleting a HostedCluster after the Azure resource group has been removed externally, the cluster deletion gets stuck because CAPZ AzureMachine finalizers prevent Machine deletion, which blocks NodePool and HostedCluster cleanup. This implements the platform.OrphanDeleter interface for Azure: - Check if the Azure resource group exists via Azure SDK - If gone (404), remove finalizers from all deleting AzureMachines - This unblocks the deletion chain The implementation uses a fail-safe approach: only remove finalizers when the Azure API definitively returns 404. For any other error (permissions, network issues, etc.), assume the resource group exists to avoid orphaning cloud resources that would continue to incur costs. This extends CNTRLPLANE-642 with operator-side logic to complement the CLI-side fix for handling missing resource groups during cluster deletion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

openshift-ci-robot · 2025-12-17T17:05:13Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-12-17T17:05:52Z

@bryan-cox: This pull request references CNTRLPLANE-642 which is a valid jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

bryan-cox · 2025-12-17T17:14:24Z

/test all

bryan-cox · 2025-12-18T11:49:51Z

/retest-required

openshift-ci · 2025-12-18T13:58:17Z

@bryan-cox: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-4-21	`aac2b64`	link	true	`/test e2e-aws-4-21`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Dec 16, 2025

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 16, 2025

openshift-ci bot added do-not-merge/needs-area area/cli Indicates the PR includes changes for CLI area/platform/azure PR/issue for Azure (AzurePlatform) platform and removed do-not-merge/needs-area labels Dec 16, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 16, 2025

openshift-ci bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release labels Dec 16, 2025

bryan-cox force-pushed the CNTRLPLANE-642 branch from 0eafb40 to 78f9262 Compare December 16, 2025 14:27

bryan-cox force-pushed the CNTRLPLANE-642 branch from 78f9262 to f61eac0 Compare December 16, 2025 14:52

bryan-cox force-pushed the CNTRLPLANE-642 branch 2 times, most recently from 59fce85 to c7aa13f Compare December 16, 2025 15:03

openshift-ci bot requested review from enxebre and jparrill December 17, 2025 10:48

jparrill reviewed Dec 17, 2025

View reviewed changes

bryan-cox and others added 2 commits December 17, 2025 06:47

bryan-cox force-pushed the CNTRLPLANE-642 branch from c7aa13f to 380fcea Compare December 17, 2025 11:47

CNTRLPLANE-642: Handle missing resource group gracefully during Azure cluster deletion #7394

Are you sure you want to change the base?

CNTRLPLANE-642: Handle missing resource group gracefully during Azure cluster deletion #7394

Uh oh!

Conversation

bryan-cox commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

openshift-ci-robot commented Dec 16, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

openshift-ci bot commented Dec 16, 2025

Uh oh!

coderabbitai bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

openshift-ci-robot commented Dec 16, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

openshift-ci bot commented Dec 16, 2025

Uh oh!

openshift-ci-robot commented Dec 16, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

openshift-ci-robot commented Dec 16, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

openshift-ci-robot commented Dec 16, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

openshift-ci-robot commented Dec 16, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

openshift-ci-robot commented Dec 16, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

openshift-ci-robot commented Dec 16, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

openshift-ci-robot commented Dec 16, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

bryan-cox commented Dec 16, 2025 •

edited

Loading

openshift-ci-robot commented Dec 16, 2025 •

edited by openshift-ci bot

Loading

coderabbitai bot commented Dec 16, 2025 •

edited

Loading

openshift-ci-robot commented Dec 16, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 16, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 16, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 16, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 16, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 16, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 16, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 16, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 16, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 16, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 16, 2025 •

edited by openshift-ci bot

Loading