-
Notifications
You must be signed in to change notification settings - Fork 651
Refactor Python Toolchain Configuration to use Bazel Repository Rules #1050
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
bc4240a
d02caea
7cc2b32
50fc13b
2afa2b2
bd871af
3321f5c
9a8ce89
5977eb5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -51,37 +51,7 @@ function inside_docker() { | |
| fi | ||
| } | ||
|
|
||
| function write_legacy_python_repo() { | ||
| mkdir -p third_party/python_legacy | ||
|
|
||
| # empty WORKSPACE | ||
| cat > third_party/python_legacy/WORKSPACE <<'EOF' | ||
| # AUTOGENERATED by configure.sh. | ||
| # This file is intentionally empty. | ||
| EOF | ||
|
|
||
| # simple BUILD that exports defs.bzl | ||
| cat > third_party/python_legacy/BUILD <<'EOF' | ||
| # AUTOGENERATED by configure.sh. | ||
|
|
||
| package(default_visibility = ["//visibility:public"]) | ||
| exports_files(["defs.bzl"]) | ||
| EOF | ||
|
|
||
| # defs.bzl MUST define 'interpreter' as a string, not a function. | ||
| # We also export py_runtime to satisfy older loads. | ||
| cat > third_party/python_legacy/defs.bzl <<EOF | ||
| # AUTOGENERATED by configure.sh. | ||
| load("@bazel_tools//tools/python:toolchain.bzl", "py_runtime_pair") | ||
| # Absolute path to the python interpreter Bazel/TF should use: | ||
| interpreter = "${PYTHON_BIN_PATH}" | ||
| py_runtime = native.py_runtime | ||
| EOF | ||
|
|
||
| echo | ||
| echo "Created third_party/python_legacy." | ||
| echo "Python interpreter = ${PYTHON_BIN_PATH}" | ||
| } | ||
| # Python detection is now handled by the Bazel repository rule in third_party/python_configure.bzl | ||
|
|
||
| # --- parse args ------------------------------------------------------------ | ||
| USER_PY="" | ||
|
|
@@ -211,8 +181,7 @@ write_tf_rc "build --repo_env=TF_NEED_CUDA=${TF_NEED_CUDA}" | |
| # Make sure repo rules and sub-config see legacy Keras (keras 2 instead of Keras 3) | ||
| write_tf_rc "build --repo_env=TF_USE_LEGACY_KERAS=1" | ||
|
|
||
| # --- write third_party/python_legacy/ with interpreter -------------------- | ||
| write_legacy_python_repo | ||
| # Python detection is now handled by the Bazel repository rule. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same comment as above: I suggest dropping this and having this kind of info only live in the PR description. |
||
|
|
||
| # --- write .bazelrc (imports TF config usual flags) ----------------- | ||
| write_bazelrc "# WARNING: this file (.bazelrc) is AUTOGENERATED and overwritten" | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -184,6 +184,17 @@ build: | |
| </pre> | ||
| <!-- common_typos_enable --> | ||
|
|
||
| The `configure.sh` script detects your Python interpreter and sets up a Bazel toolchain. If you need to manually specify a Python interpreter, you can do so by passing the `--python` flag or by setting the `PYTHON_BIN_PATH` environment variable: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for this work! Minor nit: please keep line lengths to the 80 column convention in this project. |
||
|
|
||
| <!-- common_typos_disable --> | ||
| <pre class="devsite-click-to-copy"> | ||
| <code class="devsite-terminal">./configure.sh --python=/path/to/python</code> | ||
| </pre> | ||
| <!-- common_typos_enable --> | ||
|
|
||
| > [!TIP] | ||
| > You can also bypass manual configuration by passing the Python path directly to Bazel using `--repo_env=PYTHON_BIN_PATH=/path/to/python`. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please reformat to 80-columns. |
||
|
|
||
| Now build TensorFlow Quantum: | ||
|
|
||
| <!-- common_typos_disable --> | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # Copyright 2026 The TensorFlow Quantum Authors. All Rights Reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # ============================================================================== | ||
|
|
||
| # Build file intentionally left empty. | ||
| # This file is used to define the Python interpreter path for the TFQ build. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # Copyright 2026 The TensorFlow Quantum Authors | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| """Generated BUILD file for the Python toolchain repository.""" | ||
|
|
||
| load("@bazel_tools//tools/python:toolchain.bzl", "py_runtime_pair") | ||
|
|
||
| package(default_visibility = ["//visibility:public"]) | ||
|
|
||
| # Detected path: %{PYTHON_BIN_PATH}% | ||
| py_runtime( | ||
| name = "py3_runtime", | ||
| interpreter_path = "%{PYTHON_BIN_PATH}%", | ||
| python_version = "PY3", | ||
| ) | ||
|
|
||
| py_runtime_pair( | ||
| name = "py_runtime_pair", | ||
| py3_runtime = ":py3_runtime", | ||
| ) | ||
|
|
||
| toolchain( | ||
| name = "py_toolchain", | ||
| toolchain = ":py_runtime_pair", | ||
| toolchain_type = "@bazel_tools//tools/python:toolchain_type", | ||
| ) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # Copyright 2026 The TensorFlow Quantum Authors. All Rights Reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # Generated by python_configure.bzl | ||
| interpreter = "%{PYTHON_BIN_PATH}%" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| # Copyright 2026 The TensorFlow Quantum Authors | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| """Repository rule for Python detection and toolchain registration.""" | ||
|
|
||
| def _python_configure_impl(repository_ctx): | ||
| python_bin = repository_ctx.os.environ.get("PYTHON_BIN_PATH") or \ | ||
| repository_ctx.which("python3") or \ | ||
| repository_ctx.which("python") | ||
|
|
||
| if not python_bin: | ||
| fail("Python interpreter not found. Please provide it via --repo_env=PYTHON_BIN_PATH=/path/to/python or set the PYTHON_BIN_PATH environment variable.") | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please reformat to 80-columns. |
||
|
|
||
| substitutions = {"%{PYTHON_BIN_PATH}%": str(python_bin).replace("\\", "\\\\")} | ||
|
|
||
| repository_ctx.template( | ||
| "BUILD", | ||
| Label("//third_party/python:BUILD.tpl"), | ||
| substitutions, | ||
| ) | ||
| repository_ctx.template( | ||
| "defs.bzl", | ||
| Label("//third_party/python:defs.bzl.tpl"), | ||
| substitutions, | ||
| ) | ||
|
|
||
| _python_configure = repository_rule( | ||
| implementation = _python_configure_impl, | ||
| environ = [ | ||
| "PYTHON_BIN_PATH", | ||
| "PATH", | ||
| ], | ||
|
ratgr marked this conversation as resolved.
|
||
| ) | ||
|
|
||
| def python_configure(): | ||
| """Configures the Python toolchain for TFQ, TF, and XLA. | ||
|
|
||
| Three identical repositories are created to satisfy the naming expectations | ||
| of various external dependencies: | ||
| - 'local_config_python': Used by TensorFlow Quantum and its internal rules. | ||
| - 'local_execution_config_python': Required by TensorFlow (org_tensorflow) | ||
| and TSL for certain toolchain configurations. | ||
| - 'python': Provided as a generic handle. | ||
|
|
||
| Although redundant, this ensures compatibility across the diverse dependency | ||
| tree without requiring extensive repo_mapping. | ||
| """ | ||
| _python_configure(name = "local_config_python") | ||
| _python_configure(name = "local_execution_config_python") | ||
| _python_configure(name = "python") | ||
|
ratgr marked this conversation as resolved.
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Although I understand the reason for the comment, it's better to leave this kind of note to the PR description. Comments that say things like "now handled by" will go out of date – in the future, no one reading this file will remember it was done differently. So, I suggest simply dropping this comment.