Skip to content

chore(e2e): adding localization to nightly jobs #4020

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
teknaS47 wants to merge 18 commits into
base: main
Choose a base branch
from
Open

chore(e2e): adding localization to nightly jobs #4020

teknaS47 wants to merge 18 commits into from
+1,429 −62

Conversation

@teknaS47
Copy link
Member

@teknaS47 teknaS47 commented Jan 19, 2026

Description

Adding localization tests for other locales to the ocp-helm-nightly job

Which issue(s) does this PR fix

PR acceptance criteria

Please make sure that the following steps are complete:

  • GitHub Actions are completed and successful
  • Unit Tests are updated and passing
  • E2E Tests are updated and passing
  • Documentation is updated if necessary (requirement for new features)
  • Add a screenshot if the change is UX/UI related

How to test changes / Special notes to the reviewer

@teknaS47
adding localization to nightly jobs
db9ae4a 
Signed-off-by: Sanket Saikia <sanketsaikia13@gmail.com>
@openshift-ci openshift-ci bot requested review from debsmita1 and elsony January 19, 2026 12:58
@rhdh-qodo-merge
Copy link

rhdh-qodo-merge bot commented Jan 19, 2026 •
edited
Loading

PR Reviewer Guide 🔍

(Review updated until commit 0e9a6bc)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

RHIDP-11153 - Partially compliant

Compliant requirements:

  • Integrate translation e2e tests into the existing nightly CI workflow.
  • Acceptance Criteria: Translation e2e tests are executed as part of the existing nightly CI workflow.

Non-compliant requirements:

  • Add translation end-to-end (e2e) tests to CI without impacting regular CI speed.

Requires further human verification:

  • Confirm localization Playwright project (showcase-localization-fr) actually runs in nightly CI and publishes artifacts as expected.
  • Confirm regular (non-nightly) CI jobs are unaffected in runtime and scope.
  • Validate that localization tests are meaningful/stable across environments (including the OSD-GCP skip condition rationale).
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🔒 No security concerns identified
⚡ Recommended focus areas for review

Possible Regression

The run_tests() signature was changed to require a positional url argument and introduces an optional artifacts_dir. Any existing callers that relied on the old default url behavior (or pass fewer args) may now shift parameters and break test execution/artifact publishing. Ensure all invocations of run_tests/check_and_test across .ibm/pipelines/** pass the correct arguments and that artifacts_dir naming doesn’t collide or create unexpected directory structures.

 
run_tests() {
  local release_name=$1
  local namespace=$2
  local playwright_project=$3
  local url=$4
  local artifacts_dir="${5:-${namespace}}"

  CURRENT_DEPLOYMENT=$((CURRENT_DEPLOYMENT + 1))
  save_status_deployment_namespace $CURRENT_DEPLOYMENT "$namespace"
  save_status_failed_to_deploy $CURRENT_DEPLOYMENT false

  BASE_URL="${url}"
  export BASE_URL
  log::info "BASE_URL: ${BASE_URL}"
  log::info "Running Playwright project '${playwright_project}' against namespace '${namespace}'"

  cd "${DIR}/../../e2e-tests"
  local e2e_tests_dir
  e2e_tests_dir=$(pwd)

  yarn install --immutable > /tmp/yarn.install.log.txt 2>&1
  INSTALL_STATUS=$?
  if [ $INSTALL_STATUS -ne 0 ]; then
    log::error "=== YARN INSTALL FAILED ==="
    cat /tmp/yarn.install.log.txt
    exit $INSTALL_STATUS
  else
    log::success "Yarn install completed successfully."
  fi

  yarn playwright install chromium

  Xvfb :99 &
  export DISPLAY=:99

  (
    set -e
    log::info "Using PR container image: ${TAG_NAME}"
    # Run Playwright directly with --project flag instead of using yarn script aliases
    yarn playwright test --project="${playwright_project}"
  ) 2>&1 | tee "/tmp/${LOGFILE}"

  local RESULT=${PIPESTATUS[0]}

  pkill Xvfb || true

  mkdir -p "${ARTIFACT_DIR}/${artifacts_dir}/test-results"
  mkdir -p "${ARTIFACT_DIR}/${artifacts_dir}/attachments/screenshots"
  cp -a "${e2e_tests_dir}/test-results/"* "${ARTIFACT_DIR}/${artifacts_dir}/test-results" || true
  cp -a "${e2e_tests_dir}/${JUNIT_RESULTS}" "${ARTIFACT_DIR}/${artifacts_dir}/${JUNIT_RESULTS}" || true
  if [[ "${CI}" == "true" ]]; then
    cp "${ARTIFACT_DIR}/${artifacts_dir}/${JUNIT_RESULTS}" "${SHARED_DIR}/junit-results-${artifacts_dir}.xml" || true
  fi

  cp -a "${e2e_tests_dir}/screenshots/"* "${ARTIFACT_DIR}/${artifacts_dir}/attachments/screenshots/" || true
  ansi2html < "/tmp/${LOGFILE}" > "/tmp/${LOGFILE}.html"
  cp -a "/tmp/${LOGFILE}.html" "${ARTIFACT_DIR}/${artifacts_dir}" || true
  cp -a "${e2e_tests_dir}/playwright-report/"* "${ARTIFACT_DIR}/${artifacts_dir}" || true
Flaky Locator

setTheme() now uses a translated title string (with replace("{{theme}}", theme)) for clickBtnByTitleIfNotPressed(...) but still searches a button by exact name equal to theme. If the UI text differs per locale, has different casing/spacing, or uses localized theme names, this may become flaky or fail under localization runs. Consider aligning the locator strategy (use the same translated string for the click target, or prefer role-based locators tied to stable labels/testids).

 
async setTheme(theme: "Light" | "Dark" | "Light Dynamic" | "Dark Dynamic") {
  await this.uiHelper.goToPageUrl(
    "/settings",
    t["user-settings"][lang]["settingsLayout.title"],
  );
  await this.uiHelper.clickBtnByTitleIfNotPressed(
    `${t["user-settings"][lang]["themeToggle.select"].replace("{{theme}}", theme)}`,
  );
  const themeButton = this.page.getByRole("button", {
    name: theme,
    exact: true,
  });
📚 Focus areas based on broader codebase context

Locale Env

run_localization_tests() triggers the localization Playwright project but does not set LOCALE=fr (or otherwise ensure the locale is forced) when executing tests. Since tests default to en when LOCALE is not set, validate that the CI job actually runs in French (e.g., export LOCALE=fr for this invocation or confirm the project config sets it). (Ref 1, Ref 2)

 
run_localization_tests() {
  local url="https://${RELEASE_NAME}-developer-hub-${NAME_SPACE}.${K8S_CLUSTER_ROUTER_BASE}"
  log::section "Running localization tests"
  # French (fr) - uses custom artifacts_dir to avoid overwriting main showcase test artifacts
  check_and_test "${RELEASE_NAME}" "${NAME_SPACE}" "${PW_PROJECT_SHOWCASE_LOCALIZATION_FR}" "${url}" "" "" "${PW_PROJECT_SHOWCASE_LOCALIZATION_FR}"
}

Reference reasoning: The existing i18n E2E documentation defines LOCALE as the mechanism to run Playwright tests in a specific language and explicitly states that English is used by default when the variable is not set. This suggests the nightly localization step should explicitly set LOCALE (or an equivalent mechanism) to guarantee the intended locale is applied.

📄 References
  1. redhat-developer/rhdh/docs/e2e-tests/e2e-i18n-guide.md [149-161]
  2. redhat-developer/rhdh/docs/e2e-tests/e2e-i18n-guide.md [206-215]
  3. redhat-developer/rhdh/docs/e2e-tests/CI.md [142-160]
  4. redhat-developer/rhdh/docs/e2e-tests/CI.md [84-87]
  5. redhat-developer/rhdh/docs/e2e-tests/CI.md [88-93]
  6. redhat-developer/rhdh/docs/e2e-tests/CI.md [181-195]
  7. redhat-developer/rhdh/docs/e2e-tests/CI.md [14-43]
  8. redhat-developer/rhdh/docs/e2e-tests/CI.md [50-79]

@rhdh-qodo-merge
Copy link

rhdh-qodo-merge bot commented Jan 19, 2026

PR Type

Enhancement, Tests

Description

  • Add localization tests for French language to OCP nightly jobs

  • Implement run_localization_tests() function with conditional execution

  • Skip localization tests for OSD-GCP environments

  • Update documentation across multiple memory and rules files

  • Fix terminology from "marketplace" to "extensions" in documentation

File Walkthrough
Relevant files
Enhancement
ocp-nightly.sh
Add localization test execution to nightly pipeline           

.ibm/pipelines/jobs/ocp-nightly.sh

  • Add conditional check to skip localization tests for OSD-GCP jobs
  • Implement new run_localization_tests() function
  • Function runs French localization tests against standard deployment
  • Uses existing check_and_test helper with localization project
 +12/-0   
Documentation
add_extension_metadata.md
Fix directory naming in extension metadata docs                   

.claude/memories/add_extension_metadata.md

  • Fix documentation: change "marketplace directory" to "extensions
    directory"
 +1/-1     
ci-e2e-testing.md
Document French localization tests in CI/E2E guide             

.claude/memories/ci-e2e-testing.md

  • Add showcase-localization-fr to list of Playwright test projects
  • Document new Localization Tests section with French language support
  • Add yarn command for running French localization tests
  • Update config map documentation to include localization project
 +13/-0   
add_extension_metadata.mdc
Update extension metadata rules documentation                       

.cursor/rules/add_extension_metadata.mdc

  • Fix directory reference from "marketplace" to "extensions"
  • Update git commit message to reference "extensions" instead of
    "marketplace"
 +2/-2     
ci-e2e-testing.mdc
Add localization tests to cursor rules documentation         

.cursor/rules/ci-e2e-testing.mdc

  • Add showcase-localization-fr to Playwright projects list
  • Document Localization Tests section with French language details
  • Add yarn command for French localization tests
  • Update config map references for localization project
 +13/-0   
add_extension_metadata.md
Sync extension metadata rules documentation                           

.rulesync/rules/add_extension_metadata.md

  • Fix directory naming from "marketplace" to "extensions"
  • Update git commands to use correct extensions directory paths
  • Update commit message to reference "extensions"
 +4/-4     
ci-e2e-testing.md
Sync CI/E2E testing rules documentation                                   

.rulesync/rules/ci-e2e-testing.md

  • Add showcase-localization-fr to test projects list
  • Document Localization Tests section with French language support
  • Add yarn command for running French localization tests
  • Update config map documentation for localization project
 +13/-0   
CI.md
Document localization testing in CI documentation               

docs/e2e-tests/CI.md

  • Add new Localization Tests section documenting French language testing
  • Document supported languages and future expansion plans
  • Explain when tests run and skip conditions for OSD-GCP
  • Reference implementation location in nightly pipeline script
 +12/-0   

@rhdh-qodo-merge
Copy link

rhdh-qodo-merge bot commented Jan 19, 2026 •
edited
Loading

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
High-level
Improve scalability for adding more locales

The script ocp-nightly.sh hardcodes the French localization test. It should be
refactored to iterate over a list of locales to make it more scalable for adding
new languages in the future.

Examples:

.ibm/pipelines/jobs/ocp-nightly.sh [63-68] 
run_localization_tests() {
  local url="https://${RELEASE_NAME}-developer-hub-${NAME_SPACE}.${K8S_CLUSTER_ROUTER_BASE}"
  log::section "Running localization tests"
  # French (fr)
  check_and_test "${RELEASE_NAME}" "${NAME_SPACE}" "${PW_PROJECT_SHOWCASE_LOCALIZATION_FR}" "${url}"
}

Solution Walkthrough:

Before:

# .ibm/pipelines/jobs/ocp-nightly.sh

run_localization_tests() {
  local url="..."
  log::section "Running localization tests"

  # French (fr)
  check_and_test "${RELEASE_NAME}" "${NAME_SPACE}" "${PW_PROJECT_SHOWCASE_LOCALIZATION_FR}" "${url}"
}

After:

# .ibm/pipelines/jobs/ocp-nightly.sh

run_localization_tests() {
  local url="..."
  log::section "Running localization tests"

  declare -A LOCALES_TO_TEST=(
    ["fr"]="${PW_PROJECT_SHOWCASE_LOCALIZATION_FR}"
    # Future locales can be added here
  )

  for locale in "${!LOCALES_TO_TEST[@]}"; do
    log::info "Running localization tests for ${locale}"
    check_and_test "${RELEASE_NAME}" "${NAME_SPACE}" "${LOCALES_TO_TEST[$locale]}" "${url}"
  done
}
Suggestion importance[1-10]: 7

__

Why: The suggestion correctly identifies a design limitation where the French locale test is hardcoded, and proposes a scalable, loop-based alternative that would significantly ease the addition of future languages, which is mentioned as a possibility in the PR's documentation.

Medium
General
Use wildcard matching for skip check

Replace the regex match !~ with the wildcard pattern != for checking the
JOB_NAME variable.

.ibm/pipelines/jobs/ocp-nightly.sh [37-39]

-if [[ ! "${JOB_NAME}" =~ osd-gcp ]]; then
+if [[ "${JOB_NAME}" != *osd-gcp* ]]; then
   run_localization_tests
 fi
  • Apply / Chat
Suggestion importance[1-10]: 4

__

Why: The suggestion correctly proposes using a shell wildcard pattern, which is more idiomatic and readable for this simple substring check, though the existing regex is also correct.

Low
  • Update

@teknaS47
Copy link
Member Author

teknaS47 commented Jan 19, 2026

/test e2e-ocp-helm-nightly

@github-actions
Copy link
Contributor

github-actions bot commented Jan 19, 2026

The image is available at:

/test e2e-ocp-helm

@teknaS47
Copy link
Member Author

teknaS47 commented Jan 20, 2026

/test e2e-ocp-helm-nightly

@github-actions
Copy link
Contributor

github-actions bot commented Jan 20, 2026

The image is available at:

/test e2e-ocp-helm

@invincibleJai
Copy link
Member

invincibleJai commented Jan 20, 2026

/cc @jrichter1 @HusneShabbir @sanketpathak

@teknaS47
Copy link
Member Author

teknaS47 commented Jan 21, 2026

/test e2e-ocp-helm-nightly

@github-actions
Copy link
Contributor

github-actions bot commented Jan 21, 2026

The image is available at:

/test e2e-ocp-helm

@teknaS47
Copy link
Member Author

teknaS47 commented Jan 21, 2026

/test e2e-ocp-helm

zdrapela
zdrapela reviewed Jan 21, 2026
View reviewed changes
Copy link
Member

@zdrapela zdrapela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic to run the tests looks good. We need to have CI fixed to verify everything works smoothly. I've seen failed tests in https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/redhat-developer_rhdh/4020/pull-ci-redhat-developer-rhdh-main-e2e-ocp-helm-nightly/2013461772190617600/artifacts/e2e-ocp-helm-nightly/redhat-developer-rhdh-ocp-helm-nightly/artifacts/showcase/index.html, but I'm not sure if it's caused by your test.

Anyway, I found some issues that are not a problem of this PR, but more of the code base. Please see my inline comments and let me know if you are up for fixing them or if I should rather take a look. I hope I explained myself well, but let me know if something is not clear 😉

@github-actions
Copy link
Contributor

github-actions bot commented Jan 22, 2026

🚫 Image Push Skipped.

The container image push was skipped because the build was skipped (either due to [skip-build] tag or no relevant changes with existing image)

@rhdh-qodo-merge
Copy link

rhdh-qodo-merge bot commented Jan 27, 2026 

ⓘ Your monthly quota for Qodo has expired. Upgrade your plan
ⓘ Paying users. Check that your Qodo account is linked with this Git user account

@github-actions
Copy link
Contributor

github-actions bot commented Jan 27, 2026

The image is available at:

@teknaS47
Copy link
Member Author

teknaS47 commented Jan 27, 2026

/test e2e-ocp-helm-nightly

@rhdh-qodo-merge
Copy link

rhdh-qodo-merge bot commented Jan 27, 2026 

ⓘ Your monthly quota for Qodo has expired. Upgrade your plan
ⓘ Paying users. Check that your Qodo account is linked with this Git user account

@teknaS47
Copy link
Member Author

teknaS47 commented Jan 27, 2026

/test e2e-ocp-helm-nightly

@rhdh-qodo-merge
Copy link

rhdh-qodo-merge bot commented Jan 27, 2026 

ⓘ Your monthly quota for Qodo has expired. Upgrade your plan
ⓘ Paying users. Check that your Qodo account is linked with this Git user account

@zdrapela
Copy link
Member

zdrapela commented Jan 27, 2026

/review
-i

@rhdh-qodo-merge
Copy link

rhdh-qodo-merge bot commented Jan 27, 2026

Persistent review updated to latest commit 0e9a6bc

@github-actions
Copy link
Contributor

github-actions bot commented Jan 27, 2026

The image is available at:

@teknaS47
Copy link
Member Author

teknaS47 commented Jan 28, 2026

/test e2e-ocp-helm-nightly

@rhdh-qodo-merge
Copy link

rhdh-qodo-merge bot commented Jan 28, 2026 

ⓘ Your monthly quota for Qodo has expired. Upgrade your plan
ⓘ Paying users. Check that your Qodo account is linked with this Git user account

@github-actions
Copy link
Contributor

github-actions bot commented Jan 28, 2026

The image is available at:

@openshift-ci
Copy link

openshift-ci bot commented Jan 28, 2026

@teknaS47: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-ocp-helm-nightly 14f2849 link false /test e2e-ocp-helm-nightly

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@teknaS47
Copy link
Member Author

teknaS47 commented Jan 28, 2026

/test e2e-ocp-helm-nightly

@rhdh-qodo-merge
Copy link

rhdh-qodo-merge bot commented Jan 28, 2026 

ⓘ Your monthly quota for Qodo has expired. Upgrade your plan
ⓘ Paying users. Check that your Qodo account is linked with this Git user account

@sonarqubecloud
Copy link

sonarqubecloud bot commented Jan 28, 2026

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@github-actions
Copy link
Contributor

github-actions bot commented Jan 28, 2026

The image is available at:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@debsmita1 debsmita1 Awaiting requested review from debsmita1

@elsony elsony Awaiting requested review from elsony

@HusneShabbir HusneShabbir Awaiting requested review from HusneShabbir

@jrichter1 jrichter1 Awaiting requested review from jrichter1

@sanketpathak sanketpathak Awaiting requested review from sanketpathak

1 more reviewer

@zdrapela zdrapela zdrapela left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Enhancement Review effort 3/5 Tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@teknaS47 @invincibleJai @zdrapela