Skip to content

[sushy_emulator] Add retry logic for container image pull #3564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
hjensas merged 1 commit into from
Dec 22, 2025
Merged

[sushy_emulator] Add retry logic for container image pull #3564

hjensas merged 1 commit into from
Dec 22, 2025
+4 −0

Conversation

@Valkyrie00
Copy link
Contributor

@Valkyrie00 Valkyrie00 commented Dec 17, 2025 •
edited by openshift-ci bot
Loading

The sushy emulator container image pull task could fail due to temporary network issues, causing the entire job to fail and requiring manual re-triggering.

Solution

Added retry logic to the Pull Sushy Emulator container image task:

  • retries: 3 - Retry up to 3 times on failure
  • delay: 5 - Wait 5 seconds between retries
  • until: is success - Continue retrying until the task succeeds

This follows the same pattern already used in the tempest role.

What does it solve?

  • More resilient to temporary network failures
  • Reduces manual intervention and job re-triggers
  • Saves CI resources

Ref: OSPRH-23772

@Valkyrie00 Valkyrie00 self-assigned this Dec 17, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 17, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 17, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from valkyrie00. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Valkyrie00 Valkyrie00 force-pushed the feat/add-retry-sushy-pull-task branch from c0d8916 to cfb3ae5 Compare December 17, 2025 11:09
@Valkyrie00 Valkyrie00 changed the title [DNM] feat(sushy_emulator): add retry logic for container image pull [DNM] [sushy_emulator] Add retry logic for container image pull Dec 17, 2025
@softwarefactory-project-zuul
Copy link

softwarefactory-project-zuul bot commented Dec 17, 2025

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/279533f0e3e74c6f82efc691df9d7413

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 57m 53s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 20m 09s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 26m 53s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 1h 43m 28s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 24s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 03s
✔️ cifmw-pod-pre-commit SUCCESS in 7m 43s
✔️ build-push-container-cifmw-client SUCCESS in 25m 04s
✔️ cifmw-molecule-reproducer SUCCESS in 14m 43s
❌ cifmw-molecule-sushy_emulator FAILURE in 3m 41s

@Valkyrie00
Copy link
Contributor Author

Valkyrie00 commented Dec 17, 2025

recheck

@softwarefactory-project-zuul
Copy link

softwarefactory-project-zuul bot commented Dec 17, 2025

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/93cf5e9c7cec4a969b6e83836f78265f

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 33m 36s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 22m 12s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 31m 43s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 16m 35s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 00s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 10s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 21s
❌ build-push-container-cifmw-client FAILURE in 24m 52s
❌ cifmw-molecule-reproducer FAILURE in 3m 34s
✔️ cifmw-molecule-sushy_emulator SUCCESS in 19m 13s

@Valkyrie00 Valkyrie00 force-pushed the feat/add-retry-sushy-pull-task branch from cfb3ae5 to da0f4c6 Compare December 18, 2025 08:27
@softwarefactory-project-zuul
Copy link

softwarefactory-project-zuul bot commented Dec 18, 2025

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/3f5fea2b160245e1b69ce21094f587fa

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 27m 16s
❌ podified-multinode-edpm-deployment-crc RETRY_LIMIT in 8m 51s
❌ cifmw-crc-podified-edpm-baremetal NODE_FAILURE Node request 099-0008119591 failed in 0s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 11m 33s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 39s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 57s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 06s
✔️ cifmw-molecule-reproducer SUCCESS in 16m 03s
✔️ cifmw-molecule-sushy_emulator SUCCESS in 18m 53s

@Valkyrie00
[sushy_emulator] Add retry logic for container image pull
ff3e84f 
Add retries, delay, and until parameters to the podman_image task to handle temporary network failures when pulling the sushy emulator container image from the registry.

Signed-off-by: Vito Castellano <vcastell@redhat.com>
@Valkyrie00 Valkyrie00 force-pushed the feat/add-retry-sushy-pull-task branch from da0f4c6 to ff3e84f Compare December 18, 2025 11:03
@softwarefactory-project-zuul
Copy link

softwarefactory-project-zuul bot commented Dec 18, 2025

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/59e5b2ae881442f58fb768e4109923e0

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 39m 12s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 24m 19s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 39m 16s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 24m 04s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 37s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 29s
✔️ cifmw-pod-pre-commit SUCCESS in 7m 55s
✔️ cifmw-molecule-reproducer SUCCESS in 15m 43s
❌ cifmw-molecule-sushy_emulator FAILURE in 13m 04s

@Valkyrie00
Copy link
Contributor Author

Valkyrie00 commented Dec 18, 2025

recheck

@softwarefactory-project-zuul
Copy link

softwarefactory-project-zuul bot commented Dec 18, 2025

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/df68931374a647fbb3ca514a190919f8

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 39m 37s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 36m 09s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 51m 13s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 19m 56s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 01s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 06s
✔️ cifmw-pod-pre-commit SUCCESS in 7m 46s
✔️ cifmw-molecule-reproducer SUCCESS in 14m 39s
❌ cifmw-molecule-sushy_emulator FAILURE in 14m 01s

@Valkyrie00
Copy link
Contributor Author

Valkyrie00 commented Dec 19, 2025

recheck

@softwarefactory-project-zuul
Copy link

softwarefactory-project-zuul bot commented Dec 19, 2025

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/3094dfe469a64c53a9b26a4f937f1d2c

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 55m 58s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 29m 55s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 1h 18m 55s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 38m 07s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 27s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 18s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 21s
✔️ cifmw-molecule-reproducer SUCCESS in 15m 21s
✔️ cifmw-molecule-sushy_emulator SUCCESS in 19m 01s

@Valkyrie00
Copy link
Contributor Author

Valkyrie00 commented Dec 19, 2025

recheck

@Valkyrie00 Valkyrie00 changed the title [DNM] [sushy_emulator] Add retry logic for container image pull [sushy_emulator] Add retry logic for container image pull Dec 19, 2025
@Valkyrie00 Valkyrie00 marked this pull request as ready for review December 19, 2025 13:36
@Valkyrie00 Valkyrie00 requested review from a team and arxcruz December 19, 2025 13:36
@hjensas hjensas enabled auto-merge (rebase) December 19, 2025 18:45
@hjensas hjensas merged commit afeece8 into openstack-k8s-operators:main Dec 22, 2025
9 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hjensas hjensas hjensas approved these changes

@evallesp evallesp evallesp approved these changes

@arxcruz arxcruz Awaiting requested review from arxcruz

Assignees

@Valkyrie00 Valkyrie00

Labels

Ready For Review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Valkyrie00 @hjensas @evallesp