feat: multi-cluster replication and remote cleanup support

[Amar-08]

Pull Request: Multi-Cluster Replication and Remote Cleanup Support

Description

This PR introduces support for Istio resource replication across multiple clusters, specifically targeting Multi-Primary (Master-Master) Istio deployments. It enhances Flagger's ability to sync and clean up VirtualService and DestinationRule resources across remote Kubernetes clusters.

Key Features

• ClusterManager: A new component that discovers remote clusters via Kubernetes Secrets (matching Istio's secret-based discovery pattern).
• IstioRouter Enhancements: The IstioRouter now maintains multiple clients and replicates routing changes to all discovered clusters simultaneously.
• Robust Remote Cleanup: Implemented explicit deletion of remote resources during the Canary finalization phase. This overcomes the limitation of cross-cluster OwnerReferences.
• Automated Provisioning: Flagger now automatically creates and labels (istio-injection: enabled) target namespaces on remote clusters if they are missing.
• Finalizer Enforcement: The Canary controller now enforces finalizers in multi-cluster mode regardless of the revertOnDeletion setting to ensure a leak-free environment.

Motivation

Standard Flagger is cluster-local, which makes it challenging to use in global service meshes where routing must be consistent across multiple independent control planes. This PR allows Flagger to act as a "Global Traffic Orchestrator" for Istio.

Technical Details

• Added pkg/router/multi_cluster.go for cluster discovery and client management.
• Updated IstioRouter to use MultiClusterClient.
• Modified finalize logic in pkg/controller/finalizer.go to handle readiness checks more robustly during cleanup.
• Added flags:
  • --istio-multicluster-enabled
  • --istio-multicluster-secret-label
  • --istio-multicluster-secret-namespace

Verification Results

Unit Tests

A new test suite has been added to verify multi-cluster behaviors:

• TestController_MultiClusterFinalizer: Verifies that finalizers are added/triggered and remote resources are cleaned up.
• TestClusterManager_ResolveNamespace: Verifies automated namespace creation and labeling.
• TestIstioRouter_ReplicateToRemote: Verifies concurrent replication to multiple clusters.

Status: All tests passed (go test -v ./pkg/controller/... ./pkg/router/...)

Manual Verification

Tested on a live AWS EKS environment across multiple accounts using a private Flagger build. Verified that:

• Deleting a Canary in the primary cluster correctly purged Istio resources in the secondary cluster.
• Promoting a Canary correctly updated weights globally.