Cluster update stucks because hook "BeforeClusterUpgrade" is blocking

[Nils98Ar]

/kind bug

What steps did you take and what happened:

1. Create Cluster testcluster-01 with ClusterStack openstack-scs-1-30-v1 (openstack/scs) / .spec.topology.version: v1.30.2
2. Update testcluster-01 to a new ClusterStack openstack-scs-1-30-v2 / .spec.topology.version: v1.30.6.

The cluster object reports:

kubectl get cluster -n testcluster testcluster-01 -o yaml | grep -i blocking
    message: 'hook "BeforeClusterUpgrade" is blocking: '
    reason: LifecycleHookBlocking

Logs of cso:

kubectl logs -n cso-system cso-controller-manager-57d8cc48df-dnzgf | tail
{"level":"INFO","time":"2025-01-16T17:18:19.390Z","file":"extension/hooks.go:56","message":"DoBeforeClusterUpgrade is called"}
{"level":"INFO","time":"2025-01-16T17:18:29.421Z","file":"extension/hooks.go:56","message":"DoBeforeClusterUpgrade is called"}
{"level":"INFO","time":"2025-01-16T17:18:39.448Z","file":"extension/hooks.go:56","message":"DoBeforeClusterUpgrade is called"}
{"level":"INFO","time":"2025-01-16T17:18:49.480Z","file":"extension/hooks.go:56","message":"DoBeforeClusterUpgrade is called"}
{"level":"INFO","time":"2025-01-16T17:18:59.509Z","file":"extension/hooks.go:56","message":"DoBeforeClusterUpgrade is called"}
{"level":"INFO","time":"2025-01-16T17:19:05.689Z","file":"extension/hooks.go:56","message":"DoBeforeClusterUpgrade is called"}
{"level":"INFO","time":"2025-01-16T17:19:09.545Z","file":"extension/hooks.go:56","message":"DoBeforeClusterUpgrade is called"}
{"level":"INFO","time":"2025-01-16T17:19:19.575Z","file":"extension/hooks.go:56","message":"DoBeforeClusterUpgrade is called"}
{"level":"INFO","time":"2025-01-16T17:19:29.616Z","file":"extension/hooks.go:56","message":"DoBeforeClusterUpgrade is called"}
{"level":"INFO","time":"2025-01-16T17:19:39.647Z","file":"extension/hooks.go:56","message":"DoBeforeClusterUpgrade is called"}

kubectl logs -n cso-system cso-controller-manager-57d8cc48df-6htsv | grep -i testcluster
{"level":"INFO","time":"2025-01-16T17:23:51.998Z","file":"external/tracker.go:56","message":"Adding watch on external object \"infrastructure.clusterstack.x-k8s.io/v1alpha1, Kind=OpenStackClusterStackRelease\"","controller":"clusterstackrelease","controllerGroup":"clusterstack.x-k8s.io","controllerKind":"ClusterStackRelease","ClusterStackRelease":{"name":"openstack-scs-1-30-v1","namespace":"testcluster"},"namespace":"testcluster","name":"openstack-scs-1-30-v1","reconcileID":"02d985b6-fc66-4767-b484-01c0a1abb1e7"}

Logs of cspo:

kubectl logs -n cspo-system cspo-controller-manager-7c6cdfcf59-dzs42 | tail
2025-01-16T17:24:12Z    INFO    OpenStackNodeImageRelease **ready** - image is **ACTIVE**.      {"controller": "openstacknodeimagerelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackNodeImageRelease", "OpenStackNodeImageRelease": {"name":"openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1", "reconcileID": "cfef15e2-4cc1-46d0-975b-425775998acf", "name": "ubuntu-capi-image-v1.30.6", "ID": "c0b835fb-d152-48a7-98a0-b85b6c9f46b1"}
2025-01-16T17:24:12Z    DEBUG   events  image status is ACTIVE "ubuntu-capi-image-v1.30.6", ID "c0b835fb-d152-48a7-98a0-b85b6c9f46b1"   {"type": "Normal", "object": {"kind":"OpenStackNodeImageRelease","namespace":"testcluster","name":"openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1","uid":"5c8571af-fd8e-40ca-bb96-9461f1fb637c","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69428969"}, "reason": "OpenStackImageActive"}
2025-01-16T17:24:13Z    INFO    OpenStackNodeImageRelease **ready** - image is **ACTIVE**.      {"controller": "openstacknodeimagerelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackNodeImageRelease", "OpenStackNodeImageRelease": {"name":"openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1", "reconcileID": "1f4aa932-bb0c-4ddb-a24e-5bc68b2fb6a3", "name": "ubuntu-capi-image-v1.30.6", "ID": "c0b835fb-d152-48a7-98a0-b85b6c9f46b1"}
2025-01-16T17:24:13Z    DEBUG   events  image status is ACTIVE "ubuntu-capi-image-v1.30.6", ID "c0b835fb-d152-48a7-98a0-b85b6c9f46b1"   {"type": "Normal", "object": {"kind":"OpenStackNodeImageRelease","namespace":"testcluster","name":"openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1","uid":"5c8571af-fd8e-40ca-bb96-9461f1fb637c","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69428993"}, "reason": "OpenStackImageActive"}
2025-01-16T17:24:40Z    INFO    OpenStackClusterStackRelease **ready**  {"controller": "openstackclusterstackrelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackClusterStackRelease", "OpenStackClusterStackRelease": {"name":"openstack-scs-1-30-v2","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-v2", "reconcileID": "d79f391e-5124-4b6b-913d-f51d369ce77a"}
2025-01-16T17:24:40Z    DEBUG   events  OpenStackNodeImageRelease objects are ready     {"type": "Normal", "object": {"kind":"OpenStackClusterStackRelease","namespace":"testcluster","name":"openstack-scs-1-30-v2","uid":"e11d5b19-e2a5-4db4-a773-bf6800de5433","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69428973"}, "reason": "OpenStackNodeImageReleasesReady"}
2025-01-16T17:24:40Z    INFO    OpenStackClusterStackRelease **ready**  {"controller": "openstackclusterstackrelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackClusterStackRelease", "OpenStackClusterStackRelease": {"name":"openstack-scs-1-30-v2","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-v2", "reconcileID": "30bcada7-3286-4873-97f5-4a8ffa3c1a49"}
2025-01-16T17:24:40Z    DEBUG   events  OpenStackNodeImageRelease objects are ready     {"type": "Normal", "object": {"kind":"OpenStackClusterStackRelease","namespace":"testcluster","name":"openstack-scs-1-30-v2","uid":"e11d5b19-e2a5-4db4-a773-bf6800de5433","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69429112"}, "reason": "OpenStackNodeImageReleasesReady"}
2025-01-16T17:24:40Z    INFO    OpenStackClusterStackRelease **ready**  {"controller": "openstackclusterstackrelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackClusterStackRelease", "OpenStackClusterStackRelease": {"name":"openstack-scs-1-30-v2","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-v2", "reconcileID": "170a100e-b3bd-4c08-a443-d377ed7246de"}
2025-01-16T17:24:40Z    DEBUG   events  OpenStackNodeImageRelease objects are ready     {"type": "Normal", "object": {"kind":"OpenStackClusterStackRelease","namespace":"testcluster","name":"openstack-scs-1-30-v2","uid":"e11d5b19-e2a5-4db4-a773-bf6800de5433","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69429114"}, "reason": "OpenStackNodeImageReleasesReady"}

 kubectl logs -n cspo-system cspo-controller-manager-7c6cdfcf59-dzs42 | grep testcluster
2025-01-16T17:24:10Z    DEBUG   events  successfully downloaded ClusterStackReleaseAssets "openstack-scs-1-30-v1"       {"type": "Normal", "object": {"kind":"OpenStackClusterStackRelease","namespace":"testcluster","name":"openstack-scs-1-30-v1","uid":"8df2a92f-6a2d-4907-a6d9-405fe17e2145","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69404207"}, "reason": "ClusterStackReleaseAssetsReady"}
2025-01-16T17:24:10Z    DEBUG   events  successfully downloaded ClusterStackReleaseAssets "openstack-scs-1-30-v2"       {"type": "Normal", "object": {"kind":"OpenStackClusterStackRelease","namespace":"testcluster","name":"openstack-scs-1-30-v2","uid":"e11d5b19-e2a5-4db4-a773-bf6800de5433","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69425897"}, "reason": "ClusterStackReleaseAssetsReady"}
2025-01-16T17:24:10Z    INFO    OpenStackClusterStackRelease **ready**  {"controller": "openstackclusterstackrelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackClusterStackRelease", "OpenStackClusterStackRelease": {"name":"openstack-scs-1-30-v1","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-v1", "reconcileID": "dda4b578-30ad-4023-82c0-03b032decfe6"}
2025-01-16T17:24:10Z    DEBUG   events  OpenStackNodeImageRelease objects are ready     {"type": "Normal", "object": {"kind":"OpenStackClusterStackRelease","namespace":"testcluster","name":"openstack-scs-1-30-v1","uid":"8df2a92f-6a2d-4907-a6d9-405fe17e2145","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69428961"}, "reason": "OpenStackNodeImageReleasesReady"}
2025-01-16T17:24:10Z    INFO    OpenStackClusterStackRelease **not ready** yet - waiting for OpenStackNodeImageRelease to be ready      {"controller": "openstackclusterstackrelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackClusterStackRelease", "OpenStackClusterStackRelease": {"name":"openstack-scs-1-30-v2","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-v2", "reconcileID": "fd1f13eb-95d9-4a0b-be0e-50d190e84d8f", "name:": "openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1", "ready:": false}
2025-01-16T17:24:10Z    DEBUG   events  successfully created OpenStackNodeImageRelease object "openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1" {"type": "Normal", "object": {"kind":"OpenStackClusterStackRelease","namespace":"testcluster","name":"openstack-scs-1-30-v2","uid":"e11d5b19-e2a5-4db4-a773-bf6800de5433","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69428963"}, "reason": "OpenStackNodeImageReleaseCreated"}
2025-01-16T17:24:10Z    INFO    OpenStackClusterStackRelease **ready**  {"controller": "openstackclusterstackrelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackClusterStackRelease", "OpenStackClusterStackRelease": {"name":"openstack-scs-1-30-v1","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-v1", "reconcileID": "32de89a1-bb55-4c24-9c13-976a0ca787b4"}
2025-01-16T17:24:10Z    DEBUG   events  OpenStackNodeImageRelease objects are ready     {"type": "Normal", "object": {"kind":"OpenStackClusterStackRelease","namespace":"testcluster","name":"openstack-scs-1-30-v1","uid":"8df2a92f-6a2d-4907-a6d9-405fe17e2145","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69428968"}, "reason": "OpenStackNodeImageReleasesReady"}
2025-01-16T17:24:10Z    INFO    OpenStackClusterStackRelease **not ready** yet - waiting for OpenStackNodeImageRelease to be ready      {"controller": "openstackclusterstackrelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackClusterStackRelease", "OpenStackClusterStackRelease": {"name":"openstack-scs-1-30-v2","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-v2", "reconcileID": "f68ed6a5-90bb-4511-ad49-2a14464d3599", "name:": "openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1", "ready:": false}
2025-01-16T17:24:12Z    INFO    OpenStackNodeImageRelease **ready** - image is **ACTIVE**.      {"controller": "openstacknodeimagerelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackNodeImageRelease", "OpenStackNodeImageRelease": {"name":"openstack-scs-1-30-ubuntu-capi-image-v1.30.2-v1","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-ubuntu-capi-image-v1.30.2-v1", "reconcileID": "f5083258-3ab2-46af-a78e-c5393f49a704", "name": "ubuntu-capi-image-v1.30.2", "ID": "c40e3a9e-7be2-4959-b190-ace16bbef0bc"}
2025-01-16T17:24:12Z    DEBUG   events  image status is ACTIVE "ubuntu-capi-image-v1.30.2", ID "c40e3a9e-7be2-4959-b190-ace16bbef0bc"   {"type": "Normal", "object": {"kind":"OpenStackNodeImageRelease","namespace":"testcluster","name":"openstack-scs-1-30-ubuntu-capi-image-v1.30.2-v1","uid":"058bea72-2621-411a-b072-be7748f4dfa3","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69404081"}, "reason": "OpenStackImageActive"}
2025-01-16T17:24:12Z    INFO    OpenStackNodeImageRelease **ready** - image is **ACTIVE**.      {"controller": "openstacknodeimagerelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackNodeImageRelease", "OpenStackNodeImageRelease": {"name":"openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1", "reconcileID": "cfef15e2-4cc1-46d0-975b-425775998acf", "name": "ubuntu-capi-image-v1.30.6", "ID": "c0b835fb-d152-48a7-98a0-b85b6c9f46b1"}
2025-01-16T17:24:12Z    DEBUG   events  image status is ACTIVE "ubuntu-capi-image-v1.30.6", ID "c0b835fb-d152-48a7-98a0-b85b6c9f46b1"   {"type": "Normal", "object": {"kind":"OpenStackNodeImageRelease","namespace":"testcluster","name":"openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1","uid":"5c8571af-fd8e-40ca-bb96-9461f1fb637c","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69428969"}, "reason": "OpenStackImageActive"}
2025-01-16T17:24:13Z    INFO    OpenStackNodeImageRelease **ready** - image is **ACTIVE**.      {"controller": "openstacknodeimagerelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackNodeImageRelease", "OpenStackNodeImageRelease": {"name":"openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1", "reconcileID": "1f4aa932-bb0c-4ddb-a24e-5bc68b2fb6a3", "name": "ubuntu-capi-image-v1.30.6", "ID": "c0b835fb-d152-48a7-98a0-b85b6c9f46b1"}
2025-01-16T17:24:13Z    DEBUG   events  image status is ACTIVE "ubuntu-capi-image-v1.30.6", ID "c0b835fb-d152-48a7-98a0-b85b6c9f46b1"   {"type": "Normal", "object": {"kind":"OpenStackNodeImageRelease","namespace":"testcluster","name":"openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1","uid":"5c8571af-fd8e-40ca-bb96-9461f1fb637c","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69428993"}, "reason": "OpenStackImageActive"}
2025-01-16T17:24:40Z    INFO    OpenStackClusterStackRelease **ready**  {"controller": "openstackclusterstackrelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackClusterStackRelease", "OpenStackClusterStackRelease": {"name":"openstack-scs-1-30-v2","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-v2", "reconcileID": "d79f391e-5124-4b6b-913d-f51d369ce77a"}
2025-01-16T17:24:40Z    DEBUG   events  OpenStackNodeImageRelease objects are ready     {"type": "Normal", "object": {"kind":"OpenStackClusterStackRelease","namespace":"testcluster","name":"openstack-scs-1-30-v2","uid":"e11d5b19-e2a5-4db4-a773-bf6800de5433","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69428973"}, "reason": "OpenStackNodeImageReleasesReady"}
2025-01-16T17:24:40Z    INFO    OpenStackClusterStackRelease **ready**  {"controller": "openstackclusterstackrelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackClusterStackRelease", "OpenStackClusterStackRelease": {"name":"openstack-scs-1-30-v2","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-v2", "reconcileID": "30bcada7-3286-4873-97f5-4a8ffa3c1a49"}
2025-01-16T17:24:40Z    DEBUG   events  OpenStackNodeImageRelease objects are ready     {"type": "Normal", "object": {"kind":"OpenStackClusterStackRelease","namespace":"testcluster","name":"openstack-scs-1-30-v2","uid":"e11d5b19-e2a5-4db4-a773-bf6800de5433","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69429112"}, "reason": "OpenStackNodeImageReleasesReady"}
2025-01-16T17:24:40Z    INFO    OpenStackClusterStackRelease **ready**  {"controller": "openstackclusterstackrelease", "controllerGroup": "infrastructure.clusterstack.x-k8s.io", "controllerKind": "OpenStackClusterStackRelease", "OpenStackClusterStackRelease": {"name":"openstack-scs-1-30-v2","namespace":"testcluster"}, "namespace": "testcluster", "name": "openstack-scs-1-30-v2", "reconcileID": "170a100e-b3bd-4c08-a443-d377ed7246de"}
2025-01-16T17:24:40Z    DEBUG   events  OpenStackNodeImageRelease objects are ready     {"type": "Normal", "object": {"kind":"OpenStackClusterStackRelease","namespace":"testcluster","name":"openstack-scs-1-30-v2","uid":"e11d5b19-e2a5-4db4-a773-bf6800de5433","apiVersion":"infrastructure.clusterstack.x-k8s.io/v1alpha1","resourceVersion":"69429114"}, "reason": "OpenStackNodeImageReleasesReady"}

Cluster Stacks related ressources:

kubectl get clusterstack -n testcluster
NAME                 PROVIDER    CLUSTERSTACK   K8S    CHANNEL   AUTOSUBSCRIBE   USABLE   LATEST                            AGE   REASON   MESSAGE
openstack-scs-1-30   openstack   scs            1.30   stable    false           v1, v2   openstack-scs-1-30-v2 | v1.30.6   92m

kubectl get clusterstackreleases.clusterstack.x-k8s.io -n testcluster
NAME                    K8S VERSION   READY   AGE   REASON   MESSAGE
openstack-scs-1-30-v1   v1.30.2       true    94m
openstack-scs-1-30-v2   v1.30.6       true    51m

kubectl get openstacknodeimagereleases.infrastructure.clusterstack.x-k8s.io -n testcluster
NAME                                              READY   AGE   REASON   MESSAGE
openstack-scs-1-30-ubuntu-capi-image-v1.30.2-v1   true    96m
openstack-scs-1-30-ubuntu-capi-image-v1.30.6-v1   true    6s

kubectl get clusterclasses.cluster.x-k8s.io -n testcluster
NAME                    AGE
openstack-scs-1-30-v1   96m
openstack-scs-1-30-v2   53m

kubectl get cluster -n testcluster
NAME             CLUSTERCLASS            PHASE         AGE   VERSION
testcluster-01   openstack-scs-1-30-v2   Provisioned   88m   v1.30.6

kubectl get clusteraddons.clusterstack.x-k8s.io -n testcluster
NAME                           CLUSTER          HOOK   READY   AGE   REASON   MESSAGE
cluster-addon-testcluster-01   testcluster-01          true    89m

kubectl get openstackclusterstackreleases -n testcluster
NAME                    READY   AGE   REASON   MESSAGE
openstack-scs-1-30-v1   true    99m
openstack-scs-1-30-v2   true    55m

What did you expect to happen:

CAPI starts updating.

Anything else you would like to add:

The Cluster Stacks are released to a private OCI registry based on https://github.com/SovereignCloudStack/cluster-stacks/tree/main/providers/openstack/scs.

Environment:

• cluster-stack-operator version: v0.1.0-alpha.8
• cluster-stack-provider-openstack version: cspo:v0.1.0-alpha.5
• Kubernetes version: v1.29.2
• OS (e.g. from /etc/os-release): Debian GNU/Linux 12

[janiskemper]

what's the status of the ClusterAddon custom resource?

[Nils98Ar]

Do you need more than this?

kubectl get clusteraddons.clusterstack.x-k8s.io -n testcluster
NAME                           CLUSTER          HOOK   READY   AGE   REASON   MESSAGE
cluster-addon-testcluster-01   testcluster-01          true    89m

I can give more detailed status information from kubectl describe later.

[Nils98Ar]

Is it expected that there is only one ClusterAddon for two versions of a ClusterStack?

apiVersion: clusterstack.x-k8s.io/v1alpha1
kind: ClusterAddon
metadata:
  annotations:
    ClusterAddonStage: created
  creationTimestamp: "2025-01-16T15:56:43Z"
  finalizers:
  - clusteraddon.clusterstack.x-k8s.io
  generation: 13
  name: cluster-addon-testcluster-01
  namespace: testcluster
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta1
    kind: Cluster
    name: testcluster-01
    uid: ab0bb2d7-9a1d-4588-8a5a-c5a9c553efc6
  resourceVersion: "69428915"
  uid: 0396e51a-503a-4a07-a305-bf23778126f2
spec:
  clusterRef:
    apiVersion: cluster.x-k8s.io/v1beta1
    kind: Cluster
    name: testcluster-01
    namespace: testcluster
    uid: ab0bb2d7-9a1d-4588-8a5a-c5a9c553efc6
  clusterStack: openstack-scs-1-30-v2
  version: v2
status:
  conditions:
  - lastTransitionTime: "2025-01-16T17:24:00Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-01-16T15:59:39Z"
    status: "True"
    type: ClusterReady
  - lastTransitionTime: "2025-01-16T17:24:00Z"
    status: "True"
    type: ClusterStackReleaseDownloaded
  ready: true
  resources:
  - group: apps
    kind: DaemonSet
    name: cilium
    namespace: kube-system
    status: synced
    version: v1
  - kind: ServiceAccount
    name: cilium-envoy
    namespace: kube-system
    status: synced
    version: v1
  - group: apps
    kind: Deployment
    name: cilium-operator
    namespace: kube-system
    status: synced
    version: v1
  - group: apps
    kind: Deployment
    name: metrics-server
    namespace: kube-system
    status: synced
    version: v1
  - kind: ServiceAccount
    name: csi-cinder-controller-sa
    namespace: kube-system
    status: synced
    version: v1
  - kind: ServiceAccount
    name: csi-cinder-node-sa
    namespace: kube-system
    status: synced
    version: v1
  - kind: ServiceAccount
    name: cloud-controller-manager
    namespace: kube-system
    status: synced
    version: v1
  - kind: Secret
    name: cilium-ca
    namespace: kube-system
    status: synced
    version: v1
  - kind: Secret
    name: hubble-server-certs
    namespace: kube-system
    status: synced
    version: v1
  - kind: ConfigMap
    name: cilium-config
    namespace: kube-system
    status: synced
    version: v1
  - kind: ConfigMap
    name: cilium-envoy-config
    namespace: kube-system
    status: synced
    version: v1
  - group: storage.k8s.io
    kind: StorageClass
    name: csi-cinder-sc-delete
    status: synced
    version: v1
  - group: storage.k8s.io
    kind: StorageClass
    name: csi-cinder-sc-retain
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: cilium
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: cilium-operator
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRole
    name: system:metrics-server-aggregated-reader
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: system:metrics-server
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRole
    name: csi-attacher-role
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRole
    name: csi-provisioner-role
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRole
    name: csi-snapshotter-role
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRole
    name: csi-resizer-role
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRole
    name: csi-nodeplugin-role
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: system:cloud-controller-manager
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: cilium
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: cilium-operator
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: metrics-server:system:auth-delegator
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: system:metrics-server
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: csi-attacher-binding
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: csi-provisioner-binding
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: csi-snapshotter-binding
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: csi-resizer-binding
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: csi-nodeplugin-binding
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: system:cloud-controller-manager
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: RoleBinding
    name: cilium-config-agent
    namespace: kube-system
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: RoleBinding
    name: cilium-config-agent
    namespace: kube-system
    status: synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: RoleBinding
    name: metrics-server-auth-reader
    namespace: kube-system
    status: synced
    version: v1
  - kind: Service
    name: cilium-envoy
    namespace: kube-system
    status: synced
    version: v1
  - kind: Service
    name: hubble-peer
    namespace: kube-system
    status: synced
    version: v1
  - group: apps
    kind: Deployment
    name: metrics-server
    namespace: kube-system
    status: synced
    version: v1
  - group: apps
    kind: DaemonSet
    name: cilium
    namespace: kube-system
    status: synced
    version: v1
  - group: apps
    kind: DaemonSet
    name: cilium-envoy
    namespace: kube-system
    status: synced
    version: v1
  - group: apps
    kind: DaemonSet
    name: openstack-cinder-csi-nodeplugin
    namespace: kube-system
    status: synced
    version: v1
  - group: apps
    kind: DaemonSet
    name: openstack-cloud-controller-manager
    namespace: kube-system
    status: synced
    version: v1
  - group: apps
    kind: Deployment
    name: cilium-operator
    namespace: kube-system
    status: synced
    version: v1
  - group: apps
    kind: Deployment
    name: metrics-server
    namespace: kube-system
    status: synced
    version: v1
  - group: apps
    kind: Deployment
    name: openstack-cinder-csi-controllerplugin
    namespace: kube-system
    status: synced
    version: v1
  - group: apiregistration.k8s.io
    kind: APIService
    name: v1beta1.metrics.k8s.io
    status: synced
    version: v1
  - group: storage.k8s.io
    kind: CSIDriver
    name: cinder.csi.openstack.org
    status: synced
    version: v1

[janiskemper]

yes and this seems to be all ready on v2. Maybe you have an issue with this hook -server? What about CAPI logs? and maybe the full cluster object?

[Nils98Ar]

kubectl get cluster -n testcluster testcluster-01  -o yaml

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cluster.x-k8s.io/v1beta1","kind":"Cluster","metadata":{"annotations":{},"labels":{"managed-secret":"cloud-config"},"name":"testcluster-01","namespace":"testcluster"},"spec":{"clusterNetwork":{"pods":{"cidrBlocks":["10.251.0.0/16"]},"serviceDomain":"cluster.local","services":{"cidrBlocks":["10.252.0.0/16"]}},"topology":{"class":"openstack-scs-1-30-v2","controlPlane":{"replicas":3},"variables":[{"name":"controller_flavor","value":"SCS-4V-8-20s"},{"name":"worker_flavor","value":"SCS-4V-8-20s"},{"name":"external_id","value":"e44ef66c-709f-4963-9a9c-475477e23b6f"},{"name":"node_cidr","value":"10.253.2.0/24"},{"name":"dns_nameservers","value":["XXX.XX.XX.XX","XXX.XX.XX.XX"]},{"name":"ssh_key","value":"arnold-nils"}],"version":"v1.30.6","workers":{"machineDeployments":[{"class":"default-worker","failureDomain":"nova","name":"worker","replicas":3}]}}}}
  creationTimestamp: "2025-01-16T15:56:43Z"
  finalizers:
  - cluster.cluster.x-k8s.io
  generation: 4
  labels:
    cluster.x-k8s.io/cluster-name: testcluster-01
    managed-secret: cloud-config
    topology.cluster.x-k8s.io/owned: ""
  name: testcluster-01
  namespace: testcluster
  resourceVersion: "69428916"
  uid: ab0bb2d7-9a1d-4588-8a5a-c5a9c553efc6
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 10.251.0.0/16
    serviceDomain: cluster.local
    services:
      cidrBlocks:
      - 10.252.0.0/16
  controlPlaneEndpoint:
    host: XXX.XX.XX.XXX
    port: 6443
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: testcluster-01-fhcgj
    namespace: testcluster
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: OpenStackCluster
    name: testcluster-01-7rn8n
    namespace: testcluster
  topology:
    class: openstack-scs-1-30-v2
    controlPlane:
      metadata: {}
      replicas: 3
    variables:
    - name: controller_flavor
      value: SCS-4V-8-20s
    - name: worker_flavor
      value: SCS-4V-8-20s
    - name: external_id
      value: e44ef66c-709f-4963-9a9c-475477e23b6f
    - name: node_cidr
      value: 10.253.2.0/24
    - name: dns_nameservers
      value:
      - XXX.XX.XX.XX
      - XXX.XX.XX.XX
    - name: ssh_key
      value: arnold-nils
    - name: apiserver_loadbalancer
      value: octavia-amphora
    - name: certSANs
      value: []
    - name: cloud_name
      value: openstack
    - name: controller_server_group_id
      value: ""
    - name: openstack_security_groups
      value: []
    - name: secret_name
      value: openstack
    - name: worker_root_disk
      value: 25
    - name: worker_server_group_id
      value: ""
    - name: workload_loadbalancer
      value: octavia-amphora
    version: v1.30.6
    workers:
      machineDeployments:
      - class: default-worker
        failureDomain: nova
        metadata: {}
        name: worker
        replicas: 3
status:
  conditions:
  - lastTransitionTime: "2025-01-16T16:02:21Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-01-16T15:59:36Z"
    status: "True"
    type: ControlPlaneInitialized
  - lastTransitionTime: "2025-01-16T16:02:21Z"
    status: "True"
    type: ControlPlaneReady
  - lastTransitionTime: "2025-01-16T15:58:04Z"
    status: "True"
    type: InfrastructureReady
  - lastTransitionTime: "2025-01-16T17:24:00Z"
    message: 'hook "BeforeClusterUpgrade" is blocking: '
    reason: LifecycleHookBlocking
    severity: Info
    status: "False"
    type: TopologyReconciled
  controlPlaneReady: true
  failureDomains:
    nova:
      controlPlane: true
  infrastructureReady: true
  observedGeneration: 4
  phase: Provisioned

[Nils98Ar]

kubectl logs -n capi-system capi-controller-manager-6547d5777f-dm874

I0116 21:50:03.957946       1 reconcile_state.go:66] "Reconciling state for topology owned objects" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="758be91e-15ac-489c-81fc-494d69202406"
I0116 21:50:13.961997       1 client.go:219] "Calling all extensions of hook \"BeforeClusterUpgrade\"" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="6450a7cd-4e1d-47ae-a0fb-6bb0db3b98bd" hook="BeforeClusterUpgrade"
I0116 21:50:13.962068       1 client.go:325] "Calling extension handler \"before-cluster-upgrade.cso-hook-server-extensionconfig\"" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="6450a7cd-4e1d-47ae-a0fb-6bb0db3b98bd" hook="BeforeClusterUpgrade" extensionHandler="before-cluster-upgrade.cso-hook-server-extensionconfig" hook="BeforeClusterUpgrade"
I0116 21:50:13.969860       1 client.go:366] "extension handler returned blocking response with retryAfterSeconds of 10" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="6450a7cd-4e1d-47ae-a0fb-6bb0db3b98bd" hook="BeforeClusterUpgrade" extensionHandler="before-cluster-upgrade.cso-hook-server-extensionconfig" hook="BeforeClusterUpgrade"
I0116 21:50:13.969908       1 desired_state.go:530] "Cluster upgrade to version \"v1.30.6\" is blocked by \"BeforeClusterUpgrade\" hook" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="6450a7cd-4e1d-47ae-a0fb-6bb0db3b98bd"
I0116 21:50:13.984842       1 reconcile_state.go:66] "Reconciling state for topology owned objects" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="6450a7cd-4e1d-47ae-a0fb-6bb0db3b98bd"
I0116 21:50:18.179861       1 extensionconfig_controller.go:232] "Injecting CA Bundle into ExtensionConfig from secret \"cso-system/cso-hook-server-server-cert\"" controller="extensionconfig" controllerGroup="runtime.cluster.x-k8s.io" controllerKind="ExtensionConfig" ExtensionConfig="cso-hook-server-extensionconfig" namespace="" name="cso-hook-server-extensionconfig" reconcileID="2563bb04-9bf2-4b10-94ff-19f92d9b61b6"
I0116 21:50:18.183632       1 client.go:120] "Performing discovery for ExtensionConfig" controller="extensionconfig" controllerGroup="runtime.cluster.x-k8s.io" controllerKind="ExtensionConfig" ExtensionConfig="cso-hook-server-extensionconfig" namespace="" name="cso-hook-server-extensionconfig" reconcileID="2563bb04-9bf2-4b10-94ff-19f92d9b61b6"
I0116 21:50:18.191771       1 extensionconfig_controller.go:152] "Registering ExtensionConfig information into registry" controller="extensionconfig" controllerGroup="runtime.cluster.x-k8s.io" controllerKind="ExtensionConfig" ExtensionConfig="cso-hook-server-extensionconfig" namespace="" name="cso-hook-server-extensionconfig" reconcileID="2563bb04-9bf2-4b10-94ff-19f92d9b61b6"
I0116 21:50:23.989274       1 client.go:219] "Calling all extensions of hook \"BeforeClusterUpgrade\"" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="3860c5b0-bcec-4353-a486-5cca308aae11" hook="BeforeClusterUpgrade"
I0116 21:50:23.989334       1 client.go:325] "Calling extension handler \"before-cluster-upgrade.cso-hook-server-extensionconfig\"" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="3860c5b0-bcec-4353-a486-5cca308aae11" hook="BeforeClusterUpgrade" extensionHandler="before-cluster-upgrade.cso-hook-server-extensionconfig" hook="BeforeClusterUpgrade"
I0116 21:50:23.999280       1 client.go:366] "extension handler returned blocking response with retryAfterSeconds of 10" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="3860c5b0-bcec-4353-a486-5cca308aae11" hook="BeforeClusterUpgrade" extensionHandler="before-cluster-upgrade.cso-hook-server-extensionconfig" hook="BeforeClusterUpgrade"
I0116 21:50:23.999339       1 desired_state.go:530] "Cluster upgrade to version \"v1.30.6\" is blocked by \"BeforeClusterUpgrade\" hook" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="3860c5b0-bcec-4353-a486-5cca308aae11"
I0116 21:50:24.017616       1 reconcile_state.go:66] "Reconciling state for topology owned objects" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="3860c5b0-bcec-4353-a486-5cca308aae11"
I0116 21:50:34.022526       1 client.go:219] "Calling all extensions of hook \"BeforeClusterUpgrade\"" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="da0e7ceb-3210-4e15-9fcd-9db0cfca12a6" hook="BeforeClusterUpgrade"
I0116 21:50:34.022571       1 client.go:325] "Calling extension handler \"before-cluster-upgrade.cso-hook-server-extensionconfig\"" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="da0e7ceb-3210-4e15-9fcd-9db0cfca12a6" hook="BeforeClusterUpgrade" extensionHandler="before-cluster-upgrade.cso-hook-server-extensionconfig" hook="BeforeClusterUpgrade"
I0116 21:50:34.030268       1 client.go:366] "extension handler returned blocking response with retryAfterSeconds of 10" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="da0e7ceb-3210-4e15-9fcd-9db0cfca12a6" hook="BeforeClusterUpgrade" extensionHandler="before-cluster-upgrade.cso-hook-server-extensionconfig" hook="BeforeClusterUpgrade"
I0116 21:50:34.030337       1 desired_state.go:530] "Cluster upgrade to version \"v1.30.6\" is blocked by \"BeforeClusterUpgrade\" hook" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="da0e7ceb-3210-4e15-9fcd-9db0cfca12a6"
I0116 21:50:34.055524       1 reconcile_state.go:66] "Reconciling state for topology owned objects" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="testcluster/testcluster-01" namespace="testcluster" name="testcluster-01" reconcileID="da0e7ceb-3210-4e15-9fcd-9db0cfca12a6"

[Nils98Ar]

I have upgraded cso and cspo by like this. Not sure if this is important.

curl -sSL https://github.com/SovereignCloudStack/cluster-stack-operator/releases/latest/download/cso-infrastructure-components.yaml | envsubst | kubectl apply -f -
curl -sSL https://github.com/sovereignCloudStack/cluster-stack-provider-openstack/releases/latest/download/cspo-infrastructure-components.yaml | envsubst | kubectl apply -f -

kubectl patch deployment -n cso-system cso-controller-manager --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value": ["/manager", "-source", "oci"]}]'
kubectl patch deployment -n cspo-system cspo-controller-manager --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value": ["/manager", "-source", "oci"]}]'

[chess-knight]

I had probably the same problem in the past #259 - but could't reproduce it anymore.

[janiskemper]

There is some inconsistency that I see but I don't know where it comes from. Apparently, the ClusterAddon is missing the condition "HelmChartApplied". This is the condition that is important to the hook server. It doesn't see it and therefore doesn't know that the update of the cluster addons is probably done already.

You can, if I'm not mistaken, just retrigger the update process of the ClusterAddon, by changing the spec of it to "v1" and the first cluster stack. If you change both of that, the controller will have to think it should do the upgrade again, re-apply everything in v2, and then ideally shouldn't again have an inconsistent state.

[Nils98Ar]

Indeed. Now the HelmChartApplied condition exists and is true.
And capi is starting the rolling upgrade, @janiskemper thanks! Let's see if it completes successfully :)

Maybe I restarted cso at the wrong time while cso was doing the “pre-upgrade steps”? Not sure. It would still be cool if cso could check for started/unfinished upgrades during pod startup and restart them, or at least log a hint.

[janiskemper]

Glad that it worked! I really didn't understand yet why this happened. Maybe it was the CSO restart. However, I'd say if you ever encounter this again (or anyone else reading this), then I'd appreciate to know if something like a container restart happened, etc.

Looking at the code I didn't find a reason why the container restart should lead to this though.

Should we close this now knowing there is a manual workaround, or should we keep it open to make it easier to find for others encountering this problem @Nils98Ar ?

[Nils98Ar]

I would say we can close this. Then others see that there is a workaround 👍

[Nils98Ar]

@janiskemper The same issue (and solution) again for another cluster today. No restarts of any cso/cspo pods since yesterday.

The clusters where originally created with an older cso version and 2 weeks ago cso was updated.