Support FOSS tests running locally by nettle · Pull Request #149 · Ericsson/codechecker_bazel

nettle · 2026-01-17T02:44:44Z

Why:
We need to support multiple bazel versions (6, 7, 8)
when running FOSS tests locally, not only from CI.
As of now we copy just .bazelversion with fixed old version.

What:

Remove .bazelversion from templates
Copy .bazelversion from root directory
Change CI to create .bazelversion in root directory
Print out versions in CI for FOSS tests
Add more logging (seen only with -vvv)
Fix small logging bug in test_external_repo.py

Addresses:
#148

Why: We need to support multiple bazel versions (6, 7, 8) when running FOSS tests locally, not only from CI. As of now we copy just .bazelversion with fixed old version. What: - Remove .bazelversion from templates - Copy .bazelversion from root directory - Change CI to create .bazelversion in root directory - Print out versions in CI for FOSS tests - Add more logging (seen only with -vvv) - Fix small logging bug in test_external_repo.py

Szelethus

I quite like the change! @furtib? You are the expert on this one.

Szelethus · 2026-01-21T13:11:36Z

test/foss/yaml-cpp/init.sh

+bazelversion="../../../../.bazelversion"
+[ -f $bazelversion ] && cp $bazelversion .
+cat .bazelversion


I'd prefer if this...

OK, I'll address

Szelethus · 2026-01-21T13:12:16Z

test/foss/yaml-cpp/init.sh

+
+# Check
+bazel version
+which bazel


...and this change were reused from test/foss/templates, and not copy-pasted to each test.

Szelethus · 2026-01-21T13:12:45Z

test/unit/external_repository/test_external_repo.py

        _, stdout, _ = cls.run_command("bazel --version")
        cls.BAZEL_VERSION = stdout.split(' ')[2].strip()
-        logging.debug("Using Bazel", cls.BAZEL_VERSION)
+        logging.debug("Using Bazel %s", cls.BAZEL_VERSION)


Nice catch.

Szelethus

On a second thought: The patch title is very different to what the patch does. Have we ever explicitly not supported running foss tests locally? Is this patch fixing an issue with that? This feels like a CI refactor more than anything.

furtib

The title of the PR does not match what it's doing.
This PR makes no changes to how FOSS tests are run locally, but how they're run in CI.
Otherwise, I see no problem with the patch.

furtib · 2026-01-21T14:32:24Z

test/foss/yaml-cpp/init.sh

+bazel version
+which bazel


This does not make sense to me using the current test setup:

python3 -m unittest discover foss -vvv

This script only runs the first time, and its output is completely ignored.

Yes, that's another huge problem!
Once we run FOSS tests they create test-proj directories which are invisible, cannot be updated, and cannot be removed! I will create a bug about this :)

nettle · 2026-01-21T15:37:51Z

On a second thought: The patch title is very different to what the patch does. Have we ever explicitly not supported running foss tests locally? Is this patch fixing an issue with that? This feels like a CI refactor more than anything.

Right, I'll think about better title and summary.
Trying (and still failing!) to run FOSS tests locally I realized that we have multiple issues with that,
so this patch is just one of the steps to make it work properly, but definitely not the last.

furtib · 2026-01-22T12:56:51Z

I still don't see what issues can occur when running the FOSS tests locally.
I have tried to tidy up the process a bit in #165. Maybe that helps with your issue?
Also, do note that older Bazel 6 versions (like 6.2.1) will consider the WORKSPACE file found in the root of the project before even checking for a MODULE.bazel file, regardless of flags passed to Bazel.

nettle · 2026-01-24T16:23:32Z

I still don't see what issues can occur when running the FOSS tests locally. I have tried to tidy up the process a bit in #165. Maybe that helps with your issue? Also, do note that older Bazel 6 versions (like 6.2.1) will consider the WORKSPACE file found in the root of the project before even checking for a MODULE.bazel file, regardless of flags passed to Bazel.

Hi @furtib and @Szelethus,

In general, I see many problems, unfortunately.
But I'm trying to look beyond works-for-me environment.
You should do that too!

Here is the short list of what's on the surface:

Tests cannot be run locally (partially addressed by Rewrite testfiles to have multiversion testing locally #165 and Support FOSS tests running locally #149) - by that I mean the results are not predictable
Tests do not exercise different versions and environments (trying to address by Introduce Mise #132, see also Consider tools for multi-configuration testing #163 and Execution requirements #164)
Tests do have hidden dependencies on environment and "special" undocumented conditions (see Execution requirements #164)
Tests do not provide enough information, logging (partially addressed in Fix fails on older Bazel versions in unit tests #143, see also Print out stdout and stderr when test executed with -vvv #147)
Tests create artifacts (can be addressed by using tempfile, see my comment)
Tests create hidden! artifacts (test-proj) which may lead to unpredictable results
Tests depends on arifacts created externally (.bazelversion), and it is still a problem
Tests requires external cleaning (temporarily we can workaround with rm -rf test-proj, but this not a solution!)
Tests create cached artifacts (.pytest_cache and __pycache__), well, this is a minor problem :)
Tests do not use bazel test ... for testing (actually we can use that, and probably should)

Basically, what is essential:

All tests must be possible to run locally
All tests must produce the same results when running locally and in CI
All tests must produce the same results for different versions of all tools

Your patch #165, I think, is the way forward, but issues should be solved one by one, by smaller patches.
Very simple: small patch = easier to review and merge.
And do not create new problems! :)

Szelethus · 2026-02-03T09:19:43Z

Do I understand correctly that this patch was superseded by #169 and some others mentioned in #148 (comment)?

nettle · 2026-02-04T03:42:29Z

Do I understand correctly that this patch was superseded by #169 and some others mentioned in #148 (comment)?

Right, I guess we can abandon this patch. Shall we?

nettle changed the title ~~Remove .bazelversion from FOSS tests (#148)~~ Remove .bazelversion from FOSS tests Jan 17, 2026

nettle force-pushed the foss-bazel-version branch 3 times, most recently from 5f09aa8 to 85811f9 Compare January 17, 2026 15:18

nettle marked this pull request as draft January 17, 2026 17:37

nettle changed the title ~~Remove .bazelversion from FOSS tests~~ Support FOSS tests running locally Jan 17, 2026

nettle force-pushed the foss-bazel-version branch from 85811f9 to 7f73184 Compare January 17, 2026 17:43

nettle force-pushed the foss-bazel-version branch from 7f73184 to d81938b Compare January 17, 2026 18:54

nettle marked this pull request as ready for review January 17, 2026 19:03

Szelethus requested review from Szelethus and furtib January 21, 2026 13:07

Szelethus added the test ☑️ Adding or refactoring tests label Jan 21, 2026

Szelethus reviewed Jan 21, 2026

View reviewed changes

Szelethus requested changes Jan 21, 2026

View reviewed changes

furtib requested changes Jan 21, 2026

View reviewed changes

furtib mentioned this pull request Jan 28, 2026

Support FOSS tests running locally #148

Open

nettle marked this pull request as draft February 4, 2026 03:43

Conversation

nettle commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Szelethus left a comment

Choose a reason for hiding this comment

Uh oh!

Szelethus Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nettle Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Szelethus Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Szelethus Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Szelethus left a comment

Choose a reason for hiding this comment

Uh oh!

furtib left a comment

Choose a reason for hiding this comment

Uh oh!

furtib Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

nettle Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nettle commented Jan 21, 2026

Uh oh!

furtib commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nettle commented Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Szelethus commented Feb 3, 2026

Uh oh!

nettle commented Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nettle commented Jan 17, 2026 •

edited

Loading

Szelethus Jan 21, 2026 •

edited

Loading

nettle Jan 21, 2026 •

edited

Loading

furtib commented Jan 22, 2026 •

edited

Loading

nettle commented Jan 24, 2026 •

edited

Loading