Automatic dynamic resolution of pip environments

[nikonikolov]

Description

Consider a monorepo with many separate pip environments

• Let py_library A depend on numpy and have an associated pip environment (i.e. requirements.txt.lock file)
• Let py_library B depend on A and a specific version numpy==X and also have its own pip environment.

For B, whether we end up using the version of numpy from A or from B's pip environment depends entirely on the order of deps in the definition of B.

py_library(
    name = "A",
    deps = [pip_A("numpy")],
)

py_library(
    name = "B",
    deps = [
        # If we accidentally swap the order, we are going to use `pip_A("numpy")`
        pip_B("numpy"),
        "//path/to:A",
    ],
)

When we have a big monorepo with multiple bazel subpackages and multiple pip environments this very quickly gets out of hand and can a lot of problems.

Describe the solution you'd like

It would be extremely nice if the environment can be automatically resolved. For example, instead of the above, we instead write

py_library(
    name = "A",
    deps = [pip_dep("numpy")],
)

py_library(
    name = "B",
    deps = [
        pip_dep("numpy"),
        "//path/to:A",
    ],
)

Here pip_dep automatically resolves to the correct environment at build time. For A it resolves to pip_A, for B it resolves to pip_B (this can also be propagated from py_test and py_binary). For this to function correctly the assumption is that pip_B contains all pip_A packages that A requires, but not necessarily the same versions. This can very easily be achieved with proper dependency management across pip environments - either via -r A_requirements.in or by recursive inclusion in pyproject.toml.

The above behavior is theoretically possible today by adding a special macro pip_dep which resolves based on a string_flag and config_setting. However, this is quite far from convenient from an usage perspective, for example:

• The user needs to select and pass the correct flag for any binary or test that he runs
• Simultaneously running the entire suite of tests for the monorepo is not possible since bazel needs to be invoked with a different flag for every subpackage (this also can make things a lot slower)

Ideally the correct environment can be automatically resolved for a given target based on its path in the repo (if we assume all targets under B use pip_B and all targets under A use pip_A, which is IMO a sensible assumption). Then one can enforce the correct environment via transition. On the command line targets can be invoked without any extra configuration arguments (or configuration arguments can overwrite the automatic resolution).

Example implementation https://gist.github.com/nikonikolov/3204c93621c86b6f6d10723f65e7c1b7. Likely very suboptimal and can be improved. I am in no way a bazel expert, I coded it up with the help of Gemini

Describe alternatives you've considered

1. Using a single lock file for the entire repo -> most of the time impossible due to conflicting requirements in different subpackages
2. Make sure that if B depends on A, it always uses the same versions of the pip packages from A. This means user need to use -c A_requirements.txt.lock to compile B_requirements.txt.lock. With multiple subpackages and conflicting environments, one would be constantly running into version conflicts which will have to be manually resolved (if at all possible), which is obviously far from desirable (hence the need for tools such as uv pip compile or pip-compile)

[aignas]

As I have written in https://bazelbuild.slack.com/archives/CA306CEV6/p1772453395194499?thread_ts=1772211485.311029&cid=CA306CEV6. And I see you're the author of the thread, hello there! For anyone not using Slack, here is the gist of what I have proposed there

We have this: https://rules-python.readthedocs.io/en/latest/howto/multi-platform-pypi-deps.html#defining-custom-settings

If you want to have a handful requirements.txt files and switch between them via a config flag, it could work. For that you would need to transition the py_test and py_binary targets to only one requirements.txt file. This is possible via the config_settings attribute on the rules: https://rules-python.readthedocs.io/en/latest/api/rules_python/python/private/py_binary_rule.html#py_binary.config_settings

This means that you need to define the platforms in the root module and add a bunch of config settings to make this work, but as it is right now, it is feasible to implement it on the client side.

Docs to achieve that could be improved though, so if you end up going the rabbit hole, feel free to submit PRs on this. In the docs right now we have an example how to switch between different binary flavours (i.e. different CUDA versions), but the same approach can work for just different environments.

Your config settings would be:

string_flag(
    name = "pyenv",
    build_setting_default = "none",
)

config_setting(
    name = "is_pyenv_foo",
    flag_values = {":pyenv": "foo"},
)

Your py_binary transittiongs would be:

py_binary(
    ...
    config_settings = {
        "//my_config_settings:pyenv": "foo",
    },
)

# which could be wrapped to:

env_py_binary(
    ...
    pyenv = "foo",
)

That's it, you don't need custom rules anywhere as far as I understand, just a few macros there and there.

[nikonikolov]

Thank you! Looked into it more closely and managed to implement it and get it working. There was a good amount of extra code to write and some peculiarities to consider, e.g. I had to build the internal graph of dependencies across environments (requriements.txt.lock files) in order to provide only the suitable select options for each py_binary / py_test, otherwise the build fails. Maybe some of the code can be integrated in rules_python if deemed suitable, but I am in no way a bazel expert to be able to open a PR for this. Leaving the implementation here

https://gist.github.com/nikonikolov/4e9eab4c1567e4ed415d6932c09a4e8f

[aignas]

I don't fully understand why you needed to write the extra code here as I thought it should have worked. What is the error when the build failed without it?