Fix the pip requirements setup

[NicolasFussberger]

Cleanup the way that python requirements are used:

• Introduce a proper requirments.txt lock file
• Support bazel run //:requirements.update to update the lockfile

[github-actions]

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run --lockfile_mode=error //:license-check

Status: ⚠️ Needs Review

Click to expand output

[License Check Output]
Extracting Bazel installation...
Starting local Bazel server (8.4.2) and connecting to it...
INFO: Invocation ID: 6e3e34bc-38a0-4020-83fe-9b08ba2e0c31
Computing main repo mapping: 
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
    currently loading: 
Loading: 0 packages loaded
    currently loading: 
Loading: 0 packages loaded
    currently loading: 
Analyzing: target //:license-check (1 packages loaded, 0 targets configured)
Analyzing: target //:license-check (1 packages loaded, 0 targets configured)

Analyzing: target //:license-check (29 packages loaded, 10 targets configured)

Analyzing: target //:license-check (84 packages loaded, 10 targets configured)

Analyzing: target //:license-check (133 packages loaded, 513 targets configured)

Analyzing: target //:license-check (144 packages loaded, 3273 targets configured)

Analyzing: target //:license-check (144 packages loaded, 6822 targets configured)

Analyzing: target //:license-check (157 packages loaded, 8082 targets configured)

Analyzing: target //:license-check (157 packages loaded, 8094 targets configured)

Analyzing: target //:license-check (158 packages loaded, 8218 targets configured)

INFO: Analyzed target //:license-check (162 packages loaded, 10232 targets configured).
[7 / 16] Creating runfiles tree bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/score_tooling+/dash/tool/formatters/dash_format_converter.runfiles [for tool]; 0s local
[13 / 16] JavaToolchainCompileClasses external/rules_java+/toolchains/platformclasspath_classes; 0s disk-cache, processwrapper-sandbox
[14 / 16] JavaToolchainCompileBootClasspath external/rules_java+/toolchains/platformclasspath.jar; 0s disk-cache, processwrapper-sandbox
INFO: Found 1 target...
Target //:license.check.license_check up-to-date:
  bazel-bin/license.check.license_check
  bazel-bin/license.check.license_check.jar
INFO: Elapsed time: 22.373s, Critical Path: 2.61s
INFO: 16 processes: 12 internal, 3 processwrapper-sandbox, 1 worker.
INFO: Build completed successfully, 16 total actions
INFO: Running command line: bazel-bin/license.check.license_check ./formatted.txt <args omitted>
usage: org.eclipse.dash.licenses.cli.Main [-batch <int>] [-cd <url>]
       [-confidence <int>] [-ef <url>] [-excludeSources <sources>] [-help] [-lic
       <url>] [-project <shortname>] [-repo <url>] [-review] [-summary <file>]
       [-timeout <seconds>] [-token <token>]

[github-actions]

The created documentation from the pull request is available at: docu-html

[PiotrKorkus]

The current solution was partially correct. The only thing that needs fixing is that compile_pip_requirements generates file requirements.lock that this PR is removing and in your MODULE pip is parsing requirements_lock.txt which is incorrect.

• Keep requirements.lock
• update tests/integration/requirements.txt with current deps
• Run bazel run //tests/integration:requirements.update
• Remove requirements_lock.txt from repo root
• Replace in MODULE file in pip.parse command:

requirements_lock = "//tests/integration:requirements.lock",

[NicolasFussberger]

Thanks for the review @PiotrKorkus . I am not sure I understand your comment. All the parts you mentioned seem to be already available in the lifecycle repo:

MODULE.bazel contains pip.parse of the requirements lockfile

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "score_lifecycle_pip",
    python_version = PYTHON_VERSION,
    requirements_lock = "//:requirements_lock.txt",
)
use_repo(pip, "score_lifecycle_pip")

BUILD contains compile_pip_requirements

compile_pip_requirements(
    name = "requirements",
    src = "requirements.in",
    data = [
        "//scripts/config_mapping:pip_requirements",
        "//tests/integration:pip_requirements",
    ],
    extra_args = [
        "--no-annotate",
    ],
    requirements_txt = "requirements_lock.txt",
    tags = [
        "manual",
    ],
)

requirements.in file assembles two requirements.txt files

# Python dependencies for generating lifecycle configuration files
-r ./scripts/config_mapping/requirements.txt
-r ./tests/integration/requirements.txt

requirements lockfile exists at requirements_lock.txt

The lockfile moved to the repo root because it also covers the file scripts/config_mapping/requirements.txt and not only the tests/integration/requirements.txt
When I do the changes you are suggesting I will have a proper lockfile for tests/integration/requirements.txt, however, the requirements from scripts/config_mapping/requirements.txt are still not part of the lockfile.

Is your comment about suggesting we shall keep independent lockfiles for the two requirements.txt files instead collecting them in a single requirements.in file?

[PiotrKorkus]

Thanks for the review @PiotrKorkus . I am not sure I understand your comment. All the parts you mentioned seem to be already available in the lifecycle repo:

MODULE.bazel contains pip.parse of the requirements lockfile

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "score_lifecycle_pip",
    python_version = PYTHON_VERSION,
    requirements_lock = "//:requirements_lock.txt",
)
use_repo(pip, "score_lifecycle_pip")

BUILD contains compile_pip_requirements

compile_pip_requirements(
    name = "requirements",
    src = "requirements.in",
    data = [
        "//scripts/config_mapping:pip_requirements",
        "//tests/integration:pip_requirements",
    ],
    extra_args = [
        "--no-annotate",
    ],
    requirements_txt = "requirements_lock.txt",
    tags = [
        "manual",
    ],
)

requirements.in file assembles two requirements.txt files

# Python dependencies for generating lifecycle configuration files
-r ./scripts/config_mapping/requirements.txt
-r ./tests/integration/requirements.txt

requirements lockfile exists at requirements_lock.txt

The lockfile moved to the repo root because it also covers the file scripts/config_mapping/requirements.txt and not only the tests/integration/requirements.txt

Are you saying we should keep independent lockfiles for the two requirements.txt files instead collecting them in a single requirements.in file?

Oh, wait a second. Now I understand what is happening here.

I didn't know its possible to extract specific packages from parsed pip. This makes sense to me now.
The solution is correct, sorry for confusion.

[NicolasFussberger]

Thanks for the review @PiotrKorkus . I am not sure I understand your comment. All the parts you mentioned seem to be already available in the lifecycle repo:
MODULE.bazel contains pip.parse of the requirements lockfile

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "score_lifecycle_pip",
    python_version = PYTHON_VERSION,
    requirements_lock = "//:requirements_lock.txt",
)
use_repo(pip, "score_lifecycle_pip")

BUILD contains compile_pip_requirements

compile_pip_requirements(
    name = "requirements",
    src = "requirements.in",
    data = [
        "//scripts/config_mapping:pip_requirements",
        "//tests/integration:pip_requirements",
    ],
    extra_args = [
        "--no-annotate",
    ],
    requirements_txt = "requirements_lock.txt",
    tags = [
        "manual",
    ],
)

requirements.in file assembles two requirements.txt files

# Python dependencies for generating lifecycle configuration files
-r ./scripts/config_mapping/requirements.txt
-r ./tests/integration/requirements.txt

requirements lockfile exists at requirements_lock.txt
The lockfile moved to the repo root because it also covers the file scripts/config_mapping/requirements.txt and not only the tests/integration/requirements.txt
Are you saying we should keep independent lockfiles for the two requirements.txt files instead collecting them in a single requirements.in file?

Oh, wait a second. Now I understand what is happening here.

I didn't know its possible to extract specific packages from parsed pip. This makes sense to me now. The solution is correct, sorry for confusion.

Thank you, no worries! Could you please approve again, I just rebased to latest main.