Skip to content

openstack-test: migrate CI config to OTE (openshift-tests-extension) #79487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mandre wants to merge 1 commit into
base: main
Choose a base branch
from
Open

openstack-test: migrate CI config to OTE (openshift-tests-extension) #79487

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions ci-operator/config/openshift/openstack-test/openshift-openstack-test-main.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
binary_build_commands: go build -o /bin/openstack-tests ./cmd/openshift-tests
binary_build_commands: make tests-ext-build && cd bin && tar -czvf openstack-test-test-extension.tar.gz
openstack-test-tests-ext && cp openstack-test-test-extension.tar.gz /usr/bin/
Comment on lines +1 to +2
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Search for similar tests-ext or extension packaging patterns in CI configs
rg -n -C3 'tests-ext.*tar.*usr/bin' --type yaml

# Also check for any documentation on openshift-tests extension discovery
rg -n 'openshift-tests.*extension' --type yaml -g 'ci-operator/**' | head -20

Repository: openshift/release

Length of output: 3188

🏁 Script executed:

#!/bin/bash
# Search for openshift-tests extension documentation or discovery code
rg -n 'extension.*discover\|extension.*load\|extension.*tar\|\.tar\.gz.*extension' --type yaml -g 'ci-operator/**' | head -30

# Look for environment variables or configurations related to extensions
rg -n 'OPENSHIFT.*EXTENSION\|TEST.*EXTENSION' --type yaml -g 'ci-operator/**' | head -20

# Check the actual step registry files for extension-related commands
fd -e 'sh' -e 'yaml' 'extension' ci-operator/step-registry/ -x grep -l 'extension' {} \; | head -10

Repository: openshift/release

Length of output: 326

🏁 Script executed:

#!/bin/bash
# Read the extension commands script to understand how extensions are discovered
cat ci-operator/step-registry/openshift-tests-extension/admission-crd-install/openshift-tests-extension-admission-crd-install-commands.sh

# Search for any documentation about openshift-tests extension format
rg -n 'extension' --type md -g '**' | grep -i 'format\|install\|package\|tar\|extract' | head -20

Repository: openshift/release

Length of output: 4353

🏁 Script executed:

#!/bin/bash
# Search for any documentation or code that shows extension discovery patterns
rg -n '/usr/bin.*extension\|extension.*discover\|OPENSHIFT_TEST.*EXTENSION' --type yaml --type sh | head -30

# Search for references to bin directories and extensions
rg -n 'bin.*extension\|extension.*bin' --type yaml -g 'ci-operator/**' | head -20

# Look for openshift-tests references to understand how it loads extensions
rg -n 'openshift-tests' --type sh -g 'ci-operator/**' | grep -i 'extension\|usr/bin' | head -20

Repository: openshift/release

Length of output: 3930

🏁 Script executed:

#!/bin/bash
# Search for how extensions are expected to be discovered in openshift-tests
rg -n 'OPENSHIFT.*TEST.*PATH\|TEST.*EXTENSION.*PATH\|extension.*path' --type yaml --type sh | head -30

# Look for environment variables related to external tests
rg -n 'OPENSHIFT_SKIP_EXTERNAL_TESTS\|EXTERNAL_TESTS\|TEST_EXTENSION' --type yaml --type sh -g 'ci-operator/**' | head -20

# Check if there's any Makefile or build documentation in openstack-test references
rg -n 'tests-ext-build\|tests-ext' --type yaml -g 'ci-operator/**' -C 3 | head -40

Repository: openshift/release

Length of output: 43

🏁 Script executed:

#!/bin/bash
# Read the full config file to understand the complete context
cat ci-operator/config/openshift/openstack-test/openshift-openstack-test-main.yaml

# Search for any references to extension binary naming patterns
rg -n 'test.*extension\|extension.*tar\|tar.*extension' --type sh --type yaml -i | head -30

Repository: openshift/release

Length of output: 2389

🏁 Script executed:

#!/bin/bash
# Find the openstack-test-openstack-ote step definition
fd -e yaml 'openstack-test-openstack-ote' ci-operator/step-registry/

# If not found, search for the step in the registry
rg -n 'openstack-test-openstack-ote' --type yaml | head -10

Repository: openshift/release

Length of output: 920

🏁 Script executed:

#!/bin/bash
# Read the step commands to see how the extension is loaded
cat ci-operator/step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-commands.sh

Repository: openshift/release

Length of output: 2240

🏁 Script executed:

#!/bin/bash
# Search for how openshift-tests discovers external test suites or extensions
rg -n 'external.*test\|openstack-test' --type sh -g 'ci-operator/step-registry/**' | grep -i 'discover\|load\|pull\|image' | head -30

# Check if there's documentation on test suite naming or discovery
rg -n 'openshift-tests.*run\|test.*suite' --type sh -g 'ci-operator/step-registry/**' | head -20

# Look for how the extension binary is expected to be found
rg -n '/usr/bin' --type sh -g 'ci-operator/step-registry/**' | grep -i 'extension\|test\|discover' | head -20

Repository: openshift/release

Length of output: 3280

🏁 Script executed:

#!/bin/bash
# Search for how extensions are named or packaged for openshift-tests
rg -n 'extension.*tar\|\.tar\.gz' --type yaml -g 'ci-operator/config/**' | head -20

# Look for any patterns showing how openshift-tests loads extensions
rg -n 'extension' --type yaml -g 'ci-operator/step-registry/**' -C 2 | grep -i 'path\|bin\|tar\|discover\|load' | head -30

# Check if there's documentation in step registry about extensions
fd -e md 'extension' ci-operator/step-registry/ -x cat {} \; | head -50

Repository: openshift/release

Length of output: 5674

🏁 Script executed:

#!/bin/bash
# Read the full step definition with documentation
cat ci-operator/step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-ref.yaml

Repository: openshift/release

Length of output: 1362

🏁 Script executed:

#!/bin/bash
# Check how the binary is used in the image promotion/build
cat ci-operator/config/openshift/openstack-test/openshift-openstack-test-main.yaml | grep -A 10 -B 10 "images\|promotion"

# Search for how extensions are packaged for release payloads
rg -n 'from.*bin\|to.*bin.*extension' --type yaml -g 'ci-operator/config/**' -C 2 | head -50

Repository: openshift/release

Length of output: 709

🏁 Script executed:

#!/bin/bash
# Check if there's any issue tracking or documentation about OTE extension format
rg -n 'openstack-test-test-extension\|openstack-test-tests-ext' --type yaml --type sh | head -30

# Look for how other test extensions are packaged
rg -n 'extension.*tar\|tar.*extension' --type yaml -g 'ci-operator/config/**' -C 3 | head -50

# Check if there's any reference to how extensions should be formatted
rg -n 'TestExtensionAdmission\|extension.*format' --type yaml | head -20

Repository: openshift/release

Length of output: 43

🏁 Script executed:

#!/bin/bash
# Search for documentation or code about openshift-tests extension format expectations
web_search "openshift-tests extension format tarball binary"

# Also check for any comments in the openstack-test repository or related files
rg -n 'extension.*binary\|binary.*extension\|tar.*extension\|extension.*extract' --type yaml --type sh --type md | head -40

Repository: openshift/release

Length of output: 110

🌐 Web query:

openshift-tests extension binary format discovery

💡 Result:

The OpenShift Tests Extension (OTE) framework is a standardized system designed to allow external repositories to contribute tests to the main OpenShift test suite [1]. It does not rely on a specific proprietary binary format for the extension itself; rather, it utilizes standard, executable Go binaries that implement a specific interface provided by the openshift-tests-extension Go module [1][2]. Key aspects of the OTE binary format and discovery include: 1. Binary Structure: An OTE-compatible extension is a standalone Go binary (e.g., cluster-etcd-operator-tests-ext, oc-tests-ext) [3][4]. These binaries are typically built using the Go toolchain and are often packaged within container images for CI/CD integration [3][4][5]. 2. Framework Integration: The binary uses the openshift-tests-extension framework, specifically the registry pattern, to register itself [3][6]. Developers use the extension.NewRegistry and extension.NewExtension functions to define the extension's identity (e.g., "openshift", "payload", "component-name") [3][7]. 3. Discovery and Execution: The OTE framework provides a standardized interface for test discovery and execution [1]. The binary must support commands such as listing available test suites and individual tests, as well as executing them and reporting results in standardized formats like JUnit XML [3][5][2]. 4. Admission and Extraction: Recent developments have introduced infrastructure for the discovery and extraction of non-payload OTE binaries [8]. This includes the TestExtensionAdmission CRD (testextension.redhat.io/v1), which allows the system to manage, classify, and validate permitted versus unpermitted extensions [8]. In summary, there is no unique binary format; OTE extensions are standard executable binaries that adhere to the OTE framework's Go-based API for registration, discovery, and execution [1][2].

Citations:

🏁 Script executed:

#!/bin/bash
# Check how the binary from /usr/bin/ is included in the release payload
# Look at the docker image definition and promotion

# Search for how other extensions are packaged in images
rg -n 'COPY.*usr/bin\|ADD.*usr/bin' --type yaml -g 'ci-operator/config/**' -C 3 | head -40

# Check if there are any dockerfile literals that reference /usr/bin/
rg -n 'dockerfile_literal' --type yaml -g 'ci-operator/config/**' -A 20 | grep -E 'RUN|COPY|ADD|usr/bin' | head -30

# Look for how extensions are actually delivered in the image
rg -n 'openstack-test' --type yaml -g 'ci-operator/config/**' -B 5 -A 5

Repository: openshift/release

Length of output: 50374

Extract the extension binary for delivery instead of packaging as tarball.

The OTE framework expects executable Go binaries, not tarballs. The current build commands create a tarball (openstack-test-test-extension.tar.gz) and copy it to /usr/bin/, but this needs to be the extracted binary instead. Either extract the tarball before copying to /usr/bin/:

binary_build_commands: make tests-ext-build && cd bin && tar -czvf openstack-test-test-extension.tar.gz openstack-test-tests-ext && tar -xzf openstack-test-test-extension.tar.gz -C /usr/bin/

Or copy the binary directly without creating the tarball intermediate:

binary_build_commands: make tests-ext-build && cp bin/openstack-test-tests-ext /usr/bin/
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/openshift/openstack-test/openshift-openstack-test-main.yaml`
around lines 1 - 2, The current binary_build_commands creates a tarball instead
of delivering the executable expected by OTE; update the build step referenced
by binary_build_commands so the actual Go binary (openstack-test-tests-ext) ends
up in /usr/bin/ — either extract the tarball and copy the extracted binary into
/usr/bin/ or skip tarball creation and copy bin/openstack-test-tests-ext
directly to /usr/bin/ after running make tests-ext-build.

build_root:
from_repository: true
use_build_cache: true
Expand Down Expand Up @@ -28,7 +29,7 @@ resources:
memory: 200Mi
tests:
- as: build
commands: openstack-tests --help
commands: ./bin/openstack-test-tests-ext --help
container:
from: bin
- as: verify
Expand All @@ -41,7 +42,7 @@ tests:
env:
CONFIG_TYPE: minimal
test:
- ref: openstack-test-openstack
- ref: openstack-test-openstack-ote
workflow: openshift-e2e-openstack-ipi
- always_run: false
as: e2e-openstack-ccpmso
Expand All @@ -61,13 +62,16 @@ tests:
requests:
cpu: 100m
timeout: 7h0m0s
- ref: openstack-test-openstack
- ref: openstack-test-openstack-ote
workflow: openshift-e2e-openstack-ipi
- always_run: false
as: e2e-openstack-dualstack
optional: true
steps:
cluster_profile: openstack-hwoffload
test:
- ref: openshift-e2e-test
- ref: openstack-test-openstack-ote
workflow: openshift-e2e-openstack-dualstack
- always_run: false
as: e2e-openstack-dualstack-v6primary
Expand All @@ -76,6 +80,9 @@ tests:
cluster_profile: openstack-hwoffload
env:
CONFIG_TYPE: dualstack-v6primary
test:
- ref: openshift-e2e-test
- ref: openstack-test-openstack-ote
workflow: openshift-e2e-openstack-dualstack
zz_generated_metadata:
branch: main
Expand Down
4 changes: 4 additions & 0 deletions ci-operator/step-registry/openstack/test/openstack-ote/OWNERS
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
approvers:
- openstack-approvers
reviewers:
- openstack-reviewers
56 changes: 56 additions & 0 deletions ...rator/step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/usr/bin/env bash

set -Eeuo pipefail

export OS_CLIENT_CONFIG_FILE="${SHARED_DIR}/clouds.yaml"
export PATH=/usr/libexec/origin:$PATH

declare TEST_ARGS=''

# Force the IPv6 endpoint
if [[ "${CONFIG_TYPE}" == *"singlestackv6"* ]]; then
export OS_CLOUD="${OS_CLOUD}-ipv6"
fi

# For disconnected or otherwise unreachable environments, we want to
# have steps use an HTTP(S) proxy to reach the API server. This proxy
# configuration file should export HTTP_PROXY, HTTPS_PROXY, and NO_PROXY
# environment variables, as well as their lowercase equivalents (note
# that libcurl doesn't recognize the uppercase variables).
if test -f "${SHARED_DIR}/proxy-conf.sh"
then
# shellcheck disable=SC1090
source "${SHARED_DIR}/proxy-conf.sh"
fi

# Set up the test provider for OpenStack
if test -n "${HTTP_PROXY:-}" -o -n "${HTTPS_PROXY:-}"; then
export TEST_PROVIDER='{"type":"openstack","disconnected":true}'
else
export TEST_PROVIDER='{"type":"openstack"}'
fi

# In order for openshift-tests to pull external binary images from the
# payload, we need access enabled to the images on the build farm. In
# order to do that, we need to unset the KUBECONFIG so we talk to the
# build farm, not the cluster under test.
echo "Granting access for image pulling from the build farm..."
KUBECONFIG_BAK=$KUBECONFIG
unset KUBECONFIG
oc adm policy add-role-to-group system:image-puller system:unauthenticated --namespace "${NAMESPACE}" || echo "Warning: Failed to grant image puller access, continuing..."
export KUBECONFIG=$KUBECONFIG_BAK

TEST_SUITE="openstack-test/all"

if [[ -n "${OPENSTACK_TEST_SKIPS}" ]]; then
TESTS="$(openshift-tests run --dry-run --provider "${TEST_PROVIDER}" "${TEST_SUITE}")"
echo "${TESTS}" | grep -v "${OPENSTACK_TEST_SKIPS}" >/tmp/tests
echo "Skipping tests:"
echo "${TESTS}" | grep "${OPENSTACK_TEST_SKIPS}" || { exit_code=$?; echo 'Error: no tests were found matching the OPENSTACK_TEST_SKIPS regex:'; echo "$OPENSTACK_TEST_SKIPS"; return $exit_code; }
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

Use exit instead of return in script context.

Line 49 uses return outside of a function. In a script (non-function context), return is invalid and will cause an error. Use exit to terminate the script with an exit code.

🐛 Proposed fix 
-	echo "${TESTS}" | grep "${OPENSTACK_TEST_SKIPS}" || { exit_code=$?; echo 'Error: no tests were found matching the OPENSTACK_TEST_SKIPS regex:'; echo "$OPENSTACK_TEST_SKIPS"; return $exit_code; }
+	echo "${TESTS}" | grep "${OPENSTACK_TEST_SKIPS}" || { exit_code=$?; echo 'Error: no tests were found matching the OPENSTACK_TEST_SKIPS regex:'; echo "$OPENSTACK_TEST_SKIPS"; exit $exit_code; }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
echo "${TESTS}" | grep "${OPENSTACK_TEST_SKIPS}" || { exit_code=$?; echo 'Error: no tests were found matching the OPENSTACK_TEST_SKIPS regex:'; echo "$OPENSTACK_TEST_SKIPS"; return $exit_code; }
echo "${TESTS}" | grep "${OPENSTACK_TEST_SKIPS}" || { exit_code=$?; echo 'Error: no tests were found matching the OPENSTACK_TEST_SKIPS regex:'; echo "$OPENSTACK_TEST_SKIPS"; exit $exit_code; }
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-commands.sh`
at line 49, The script line that pipes TESTS into grep and assigns exit_code
uses "return $exit_code" which is invalid outside a function; change that final
"return $exit_code" to "exit $exit_code" so the script terminates properly when
no tests match OPENSTACK_TEST_SKIPS (the line referencing TESTS,
OPENSTACK_TEST_SKIPS and exit_code).

TEST_ARGS="${TEST_ARGS:-} --file /tmp/tests"
fi

openshift-tests run "${TEST_SUITE}" ${TEST_ARGS:-} \
--provider "${TEST_PROVIDER}" \
--junit-dir "${ARTIFACT_DIR}/junit" \
-o "${ARTIFACT_DIR}/e2e.log"
11 changes: 11 additions & 0 deletions ...step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-ref.metadata.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"path": "openstack/test/openstack-ote/openstack-test-openstack-ote-ref.yaml",
"owners": {
"approvers": [
"openstack-approvers"
],
"reviewers": [
"openstack-reviewers"
]
}
}
33 changes: 33 additions & 0 deletions ci-operator/step-registry/openstack/test/openstack-ote/openstack-test-openstack-ote-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
ref:
as: openstack-test-openstack-ote
from: tests
cli: latest
commands: openstack-test-openstack-ote-commands.sh
resources:
requests:
cpu: "3"
memory: 600Mi
limits:
memory: 10Gi
env:
- name: OS_CLOUD
default: "openstack"
documentation: Name of cloud to use from ${SHARED_DIR}/clouds.yaml file
- name: OPENSTACK_TEST_SKIPS
default: ""
documentation: |
Regular expression (POSIX basic regular expression) of tests to skip.
It is suggested to test the regex to make sure that it matches with the available tests.
Tests can be listed by using 'openshift-tests run --dry-run (...)'. Sometimes, the tests
that are printed in Prow won't exactly match the list returned by openshift-tests.
- name: CONFIG_TYPE
default: ''
documentation: |
The type of config for the environment to deploy.

* 'dualstack' - Configure the install-config to enable dualstack clusters.
* 'singlestackv6' - Configure the install-config to enable single-stack v6 clusters.
documentation: |-
Runs the OpenShift OpenStack-specific tests using the OTE (OpenShift Tests Extension)
framework. Tests are discovered and run through openshift-tests which loads the
openstack-test extension from the release payload.