fix: EBS CSI driver crashes on AL2023 — missing IRSA role on addon

[asmacdo]

Summary

The EBS CSI addon was created without service_account_role_arn in #3166, leaving the controller without AWS credentials (no IRSA, no IMDS fallback). This caused the controller to crashloop with no EC2 IMDS role found, preventing all PVC provisioning. One-line fix: wire the existing IRSA role to the addon.

There is no viable manual workaround — attaching the role via aws eks update-addon creates drift that causes nebari to fail with Cross-account pass role is not allowed on the next deploy.

After applying this fix (and correcting various drift from manual debugging), the staging cluster was fully recovered. Keycloak data was lost due to PVC recreation during recovery but that's a non-issue on staging.

cc @viniciusdc

Debug logs

EBS CSI controller had no AWS creds

kubectl logs -n kube-system ebs-csi-controller-<pod> --all-containers --tail=50

Failed health check (verify network connection and IAM credentials):
dry-run EC2 API call failed:
operation error EC2: DescribeAvailabilityZones,
get identity: get credentials:
failed to refresh cached credentials,
no EC2 IMDS role found

CSI controller crashlooping

kubectl get pods -n kube-system | grep ebs

ebs-csi-controller-xxxxx   1/6   CrashLoopBackOff

PVCs stuck waiting for provisioner

kubectl describe pvc <pvc-name>

Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com'

Addon missing IRSA

aws eks describe-addon --cluster-name dandi-hub-staging --addon-name aws-ebs-csi-driver

"status": "DEGRADED"
// no serviceAccountRoleArn present

Manual role attach causes drift on next deploy

aws eks update-addon --service-account-role-arn arn:aws:iam::<acct>:role/...

Controller recovers, but next nebari deploy fails:

Error: updating EKS Add-On: AccessDeniedException: Cross-account pass role is not allowed.

Test plan

• Tested against live AWS EKS deployment upgrading from AL2 to AL2023
• Confirmed removing and re-adding IRSA reproduces and fixes the issue deterministically

🤖 Generated with Claude Code

[asmacdo]

@Adam-D-Lewis maybe you'd like to have a look also?

[asmacdo]

One reason this was tricky to debug: our working prod cluster also has no IRSA on the EBS CSI addon, and it works fine on AL2. So the missing IRSA looked "normal" when comparing (I assume theres a node policy fallback?). It's unclear exactly why that stops working on latest main + AL2023, but the IRSA wiring was is what seems to be intended by #3166, and adding it to the EBS CSI addon did fix the issue.

[asmacdo]

I've successfully finished upgrading production cluster with this patch. I had a number of issues, mostly related to AZs since the main node group was destroyed. But I can confirm this works :)