openstack-test: migrate CI config to OTE (openshift-tests-extension)#79487
openstack-test: migrate CI config to OTE (openshift-tests-extension)#79487mandre wants to merge 1 commit into
Conversation
Update the openstack-test main CI configuration to support the OTE migration in openshift/openstack-test#260. Changes: - binary_build_commands: build OTE extension binary via 'make tests-ext-build', package as tar.gz, and place at /usr/bin/ for payload discovery by openshift-tests - build test: verify the extension binary with --help instead of the removed openstack-tests binary - test/ccpmso jobs: use new openstack-test-openstack-ote step ref - dualstack jobs: override test phase with openshift-e2e-test + openstack-test-openstack-ote (replacing the old chain that used openstack-test-openstack) New step registry ref openstack-test-openstack-ote: - Runs from 'tests' image (openshift-tests) instead of 'openstack-tests' - Discovers the openstack-test extension from the release payload - Grants image-puller access for extension discovery - Supports same env vars: OS_CLOUD, OPENSTACK_TEST_SKIPS, CONFIG_TYPE - Runs 'openshift-tests run openstack-test/all' suite The old openstack-test-openstack step ref is preserved for older release branches that still use the standalone openstack-tests binary.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mandre The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
WalkthroughThis pull request introduces a new ChangesOpenStack OTE Step Registry Integration
🎯 2 (Simple) | ⏱️ ~12 minutes Suggested labels: 🚥 Pre-merge checks | ✅ 12✅ Passed checks (12 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsGit: Failed to clone repository. Please run the Comment |
|
[REHEARSALNOTIFIER]
Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals. Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift/openstack-test/openshift-openstack-test-main.yaml`:
- Around line 1-2: The current binary_build_commands creates a tarball instead
of delivering the executable expected by OTE; update the build step referenced
by binary_build_commands so the actual Go binary (openstack-test-tests-ext) ends
up in /usr/bin/ — either extract the tarball and copy the extracted binary into
/usr/bin/ or skip tarball creation and copy bin/openstack-test-tests-ext
directly to /usr/bin/ after running make tests-ext-build.
In
`@ci-operator/step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-commands.sh`:
- Line 49: The script line that pipes TESTS into grep and assigns exit_code uses
"return $exit_code" which is invalid outside a function; change that final
"return $exit_code" to "exit $exit_code" so the script terminates properly when
no tests match OPENSTACK_TEST_SKIPS (the line referencing TESTS,
OPENSTACK_TEST_SKIPS and exit_code).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 4b31c67c-254c-476e-87de-b50a6f7db460
📒 Files selected for processing (5)
ci-operator/config/openshift/openstack-test/openshift-openstack-test-main.yamlci-operator/step-registry/openstack/test/openstack-ote/OWNERSci-operator/step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-commands.shci-operator/step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-ref.metadata.jsonci-operator/step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-ref.yaml
| binary_build_commands: make tests-ext-build && cd bin && tar -czvf openstack-test-test-extension.tar.gz | ||
| openstack-test-tests-ext && cp openstack-test-test-extension.tar.gz /usr/bin/ |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Search for similar tests-ext or extension packaging patterns in CI configs
rg -n -C3 'tests-ext.*tar.*usr/bin' --type yaml
# Also check for any documentation on openshift-tests extension discovery
rg -n 'openshift-tests.*extension' --type yaml -g 'ci-operator/**' | head -20Repository: openshift/release
Length of output: 3188
🏁 Script executed:
#!/bin/bash
# Search for openshift-tests extension documentation or discovery code
rg -n 'extension.*discover\|extension.*load\|extension.*tar\|\.tar\.gz.*extension' --type yaml -g 'ci-operator/**' | head -30
# Look for environment variables or configurations related to extensions
rg -n 'OPENSHIFT.*EXTENSION\|TEST.*EXTENSION' --type yaml -g 'ci-operator/**' | head -20
# Check the actual step registry files for extension-related commands
fd -e 'sh' -e 'yaml' 'extension' ci-operator/step-registry/ -x grep -l 'extension' {} \; | head -10Repository: openshift/release
Length of output: 326
🏁 Script executed:
#!/bin/bash
# Read the extension commands script to understand how extensions are discovered
cat ci-operator/step-registry/openshift-tests-extension/admission-crd-install/openshift-tests-extension-admission-crd-install-commands.sh
# Search for any documentation about openshift-tests extension format
rg -n 'extension' --type md -g '**' | grep -i 'format\|install\|package\|tar\|extract' | head -20Repository: openshift/release
Length of output: 4353
🏁 Script executed:
#!/bin/bash
# Search for any documentation or code that shows extension discovery patterns
rg -n '/usr/bin.*extension\|extension.*discover\|OPENSHIFT_TEST.*EXTENSION' --type yaml --type sh | head -30
# Search for references to bin directories and extensions
rg -n 'bin.*extension\|extension.*bin' --type yaml -g 'ci-operator/**' | head -20
# Look for openshift-tests references to understand how it loads extensions
rg -n 'openshift-tests' --type sh -g 'ci-operator/**' | grep -i 'extension\|usr/bin' | head -20Repository: openshift/release
Length of output: 3930
🏁 Script executed:
#!/bin/bash
# Search for how extensions are expected to be discovered in openshift-tests
rg -n 'OPENSHIFT.*TEST.*PATH\|TEST.*EXTENSION.*PATH\|extension.*path' --type yaml --type sh | head -30
# Look for environment variables related to external tests
rg -n 'OPENSHIFT_SKIP_EXTERNAL_TESTS\|EXTERNAL_TESTS\|TEST_EXTENSION' --type yaml --type sh -g 'ci-operator/**' | head -20
# Check if there's any Makefile or build documentation in openstack-test references
rg -n 'tests-ext-build\|tests-ext' --type yaml -g 'ci-operator/**' -C 3 | head -40Repository: openshift/release
Length of output: 43
🏁 Script executed:
#!/bin/bash
# Read the full config file to understand the complete context
cat ci-operator/config/openshift/openstack-test/openshift-openstack-test-main.yaml
# Search for any references to extension binary naming patterns
rg -n 'test.*extension\|extension.*tar\|tar.*extension' --type sh --type yaml -i | head -30Repository: openshift/release
Length of output: 2389
🏁 Script executed:
#!/bin/bash
# Find the openstack-test-openstack-ote step definition
fd -e yaml 'openstack-test-openstack-ote' ci-operator/step-registry/
# If not found, search for the step in the registry
rg -n 'openstack-test-openstack-ote' --type yaml | head -10Repository: openshift/release
Length of output: 920
🏁 Script executed:
#!/bin/bash
# Read the step commands to see how the extension is loaded
cat ci-operator/step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-commands.shRepository: openshift/release
Length of output: 2240
🏁 Script executed:
#!/bin/bash
# Search for how openshift-tests discovers external test suites or extensions
rg -n 'external.*test\|openstack-test' --type sh -g 'ci-operator/step-registry/**' | grep -i 'discover\|load\|pull\|image' | head -30
# Check if there's documentation on test suite naming or discovery
rg -n 'openshift-tests.*run\|test.*suite' --type sh -g 'ci-operator/step-registry/**' | head -20
# Look for how the extension binary is expected to be found
rg -n '/usr/bin' --type sh -g 'ci-operator/step-registry/**' | grep -i 'extension\|test\|discover' | head -20Repository: openshift/release
Length of output: 3280
🏁 Script executed:
#!/bin/bash
# Search for how extensions are named or packaged for openshift-tests
rg -n 'extension.*tar\|\.tar\.gz' --type yaml -g 'ci-operator/config/**' | head -20
# Look for any patterns showing how openshift-tests loads extensions
rg -n 'extension' --type yaml -g 'ci-operator/step-registry/**' -C 2 | grep -i 'path\|bin\|tar\|discover\|load' | head -30
# Check if there's documentation in step registry about extensions
fd -e md 'extension' ci-operator/step-registry/ -x cat {} \; | head -50Repository: openshift/release
Length of output: 5674
🏁 Script executed:
#!/bin/bash
# Read the full step definition with documentation
cat ci-operator/step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-ref.yamlRepository: openshift/release
Length of output: 1362
🏁 Script executed:
#!/bin/bash
# Check how the binary is used in the image promotion/build
cat ci-operator/config/openshift/openstack-test/openshift-openstack-test-main.yaml | grep -A 10 -B 10 "images\|promotion"
# Search for how extensions are packaged for release payloads
rg -n 'from.*bin\|to.*bin.*extension' --type yaml -g 'ci-operator/config/**' -C 2 | head -50Repository: openshift/release
Length of output: 709
🏁 Script executed:
#!/bin/bash
# Check if there's any issue tracking or documentation about OTE extension format
rg -n 'openstack-test-test-extension\|openstack-test-tests-ext' --type yaml --type sh | head -30
# Look for how other test extensions are packaged
rg -n 'extension.*tar\|tar.*extension' --type yaml -g 'ci-operator/config/**' -C 3 | head -50
# Check if there's any reference to how extensions should be formatted
rg -n 'TestExtensionAdmission\|extension.*format' --type yaml | head -20Repository: openshift/release
Length of output: 43
🏁 Script executed:
#!/bin/bash
# Search for documentation or code about openshift-tests extension format expectations
web_search "openshift-tests extension format tarball binary"
# Also check for any comments in the openstack-test repository or related files
rg -n 'extension.*binary\|binary.*extension\|tar.*extension\|extension.*extract' --type yaml --type sh --type md | head -40Repository: openshift/release
Length of output: 110
🌐 Web query:
openshift-tests extension binary format discovery
💡 Result:
The OpenShift Tests Extension (OTE) framework is a standardized system designed to allow external repositories to contribute tests to the main OpenShift test suite [1]. It does not rely on a specific proprietary binary format for the extension itself; rather, it utilizes standard, executable Go binaries that implement a specific interface provided by the openshift-tests-extension Go module [1][2]. Key aspects of the OTE binary format and discovery include: 1. Binary Structure: An OTE-compatible extension is a standalone Go binary (e.g., cluster-etcd-operator-tests-ext, oc-tests-ext) [3][4]. These binaries are typically built using the Go toolchain and are often packaged within container images for CI/CD integration [3][4][5]. 2. Framework Integration: The binary uses the openshift-tests-extension framework, specifically the registry pattern, to register itself [3][6]. Developers use the extension.NewRegistry and extension.NewExtension functions to define the extension's identity (e.g., "openshift", "payload", "component-name") [3][7]. 3. Discovery and Execution: The OTE framework provides a standardized interface for test discovery and execution [1]. The binary must support commands such as listing available test suites and individual tests, as well as executing them and reporting results in standardized formats like JUnit XML [3][5][2]. 4. Admission and Extraction: Recent developments have introduced infrastructure for the discovery and extraction of non-payload OTE binaries [8]. This includes the TestExtensionAdmission CRD (testextension.redhat.io/v1), which allows the system to manage, classify, and validate permitted versus unpermitted extensions [8]. In summary, there is no unique binary format; OTE extensions are standard executable binaries that adhere to the OTE framework's Go-based API for registration, discovery, and execution [1][2].
Citations:
- 1: https://github.com/openshift-eng/openshift-tests-extension
- 2: TRT-1835: Add an openshift-tests-extension compatible test binary kubernetes#2105
- 3: openshift/cluster-etcd-operator@b549b29
- 4: openshift/oc@037dbde
- 5: https://github.com/openshift/openshift-controller-manager
- 6: CNTRLPLANE-2167:Integrate OpenShift Test Extension (OTE) into oc oc#2176
- 7: openshift/cloud-credential-operator@cfb019e
- 8: TRT-2295: admit and extract non-payload OTE binaries origin#30863
🏁 Script executed:
#!/bin/bash
# Check how the binary from /usr/bin/ is included in the release payload
# Look at the docker image definition and promotion
# Search for how other extensions are packaged in images
rg -n 'COPY.*usr/bin\|ADD.*usr/bin' --type yaml -g 'ci-operator/config/**' -C 3 | head -40
# Check if there are any dockerfile literals that reference /usr/bin/
rg -n 'dockerfile_literal' --type yaml -g 'ci-operator/config/**' -A 20 | grep -E 'RUN|COPY|ADD|usr/bin' | head -30
# Look for how extensions are actually delivered in the image
rg -n 'openstack-test' --type yaml -g 'ci-operator/config/**' -B 5 -A 5Repository: openshift/release
Length of output: 50374
Extract the extension binary for delivery instead of packaging as tarball.
The OTE framework expects executable Go binaries, not tarballs. The current build commands create a tarball (openstack-test-test-extension.tar.gz) and copy it to /usr/bin/, but this needs to be the extracted binary instead. Either extract the tarball before copying to /usr/bin/:
binary_build_commands: make tests-ext-build && cd bin && tar -czvf openstack-test-test-extension.tar.gz openstack-test-tests-ext && tar -xzf openstack-test-test-extension.tar.gz -C /usr/bin/
Or copy the binary directly without creating the tarball intermediate:
binary_build_commands: make tests-ext-build && cp bin/openstack-test-tests-ext /usr/bin/
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/config/openshift/openstack-test/openshift-openstack-test-main.yaml`
around lines 1 - 2, The current binary_build_commands creates a tarball instead
of delivering the executable expected by OTE; update the build step referenced
by binary_build_commands so the actual Go binary (openstack-test-tests-ext) ends
up in /usr/bin/ — either extract the tarball and copy the extracted binary into
/usr/bin/ or skip tarball creation and copy bin/openstack-test-tests-ext
directly to /usr/bin/ after running make tests-ext-build.
| TESTS="$(openshift-tests run --dry-run --provider "${TEST_PROVIDER}" "${TEST_SUITE}")" | ||
| echo "${TESTS}" | grep -v "${OPENSTACK_TEST_SKIPS}" >/tmp/tests | ||
| echo "Skipping tests:" | ||
| echo "${TESTS}" | grep "${OPENSTACK_TEST_SKIPS}" || { exit_code=$?; echo 'Error: no tests were found matching the OPENSTACK_TEST_SKIPS regex:'; echo "$OPENSTACK_TEST_SKIPS"; return $exit_code; } |
There was a problem hiding this comment.
Use exit instead of return in script context.
Line 49 uses return outside of a function. In a script (non-function context), return is invalid and will cause an error. Use exit to terminate the script with an exit code.
🐛 Proposed fix
- echo "${TESTS}" | grep "${OPENSTACK_TEST_SKIPS}" || { exit_code=$?; echo 'Error: no tests were found matching the OPENSTACK_TEST_SKIPS regex:'; echo "$OPENSTACK_TEST_SKIPS"; return $exit_code; }
+ echo "${TESTS}" | grep "${OPENSTACK_TEST_SKIPS}" || { exit_code=$?; echo 'Error: no tests were found matching the OPENSTACK_TEST_SKIPS regex:'; echo "$OPENSTACK_TEST_SKIPS"; exit $exit_code; }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| echo "${TESTS}" | grep "${OPENSTACK_TEST_SKIPS}" || { exit_code=$?; echo 'Error: no tests were found matching the OPENSTACK_TEST_SKIPS regex:'; echo "$OPENSTACK_TEST_SKIPS"; return $exit_code; } | |
| echo "${TESTS}" | grep "${OPENSTACK_TEST_SKIPS}" || { exit_code=$?; echo 'Error: no tests were found matching the OPENSTACK_TEST_SKIPS regex:'; echo "$OPENSTACK_TEST_SKIPS"; exit $exit_code; } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-commands.sh`
at line 49, The script line that pipes TESTS into grep and assigns exit_code
uses "return $exit_code" which is invalid outside a function; change that final
"return $exit_code" to "exit $exit_code" so the script terminates properly when
no tests match OPENSTACK_TEST_SKIPS (the line referencing TESTS,
OPENSTACK_TEST_SKIPS and exit_code).
|
@mandre: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
Update the openstack-test main CI configuration to support the OTE migration in openshift/openstack-test#260.
Changes:
New step registry ref openstack-test-openstack-ote:
The old openstack-test-openstack step ref is preserved for older release branches that still use the standalone openstack-tests binary.
OpenStack Test CI Migration to OTE (OpenShift Tests Extension)
This PR migrates the OpenStack test CI configuration from the traditional approach to the OpenShift Tests Extension (OTE) framework. The changes update how OpenStack tests are built, packaged, and executed in the OpenShift release pipeline.
Build Configuration Changes
The main CI config for openstack-test is updated to:
make tests-ext-buildinstead of the traditional OpenStack test binary buildbin/directory and place the binary at/usr/bin/for discovery by openshift-tests--helpflag instead of checking for the removed openstack-tests binaryNew Step Registry Entry
A new step registry entry
openstack-test-openstack-oteis introduced that:testsimage (openshift-tests) instead of the separate openstack-tests imagesystem:image-pullerpermissions for extension discoveryOS_CLOUD,OPENSTACK_TEST_SKIPS, andCONFIG_TYPEopenshift-tests run openstack-test/allThe implementation script handles OpenStack client configuration, proxy setup, test filtering via regex patterns, and cluster role grants for image pulling from the build farm.
Job Configuration Updates
Test/ccpmso jobs and dualstack jobs are updated to:
openstack-test-openstackstep with the newopenstack-test-openstack-otestep referenceopenshift-e2e-testalongsideopenstack-test-openstack-oteThe existing
openstack-test-openstackstep reference is preserved for compatibility with older release branches.