From 9a6e699ad9006b2203d3e599aefe89eecfe223f9 Mon Sep 17 00:00:00 2001 From: mhucka Date: Tue, 9 Dec 2025 05:37:43 +0000 Subject: [PATCH 01/23] Add new scripts for building and cleaning wheels These encapsulate instructions from Michael Broughton about how to build new versions of TensorFlow Quantum. --- release/README.md | 112 ++++++++++++++++++++++++ release/build_distribution.sh | 159 ++++++++++++++++++++++++++++++++++ release/clean_distribution.sh | 100 +++++++++++++++++++++ 3 files changed, 371 insertions(+) create mode 100644 release/README.md create mode 100755 release/build_distribution.sh create mode 100755 release/clean_distribution.sh diff --git a/release/README.md b/release/README.md new file mode 100644 index 000000000..20e082000 --- /dev/null +++ b/release/README.md @@ -0,0 +1,112 @@ +# Tools for building releases of TensorFlow Quantum + +This directory contains configurations and scripts that the TensorFlow Quantum +maintainers use to create Python packages for software releases. The process of +making a TFQ release is complex, and has not been fully automated. The scripts +in this directory help automate some steps and are a way of capturing the +process more precisely, but there are still manual steps involved. + +## Background: how TensorFlow Quantum is linked with TensorFlow + +TFQ is implemented as a Python library that integrates static C++ objects. Those +C++ objects are linked with TensorFlow static objects when both TFQ and +TensorFlow are installed on your system. Unlike a pure Python library, the +result is platform-dependent: the Python code itself remains portable, but the +underlying C++ objects need to be compiled specifically for each target +environment (operating system and CPU architecture). + +TensorFlow does not provide ABI stability guarantees between versions of +TensorFlow. In order to avoid the need for users to compile the TFQ source code +themselves when they want to install TFQ, each release of TFQ must be pinned to +a specific version of TensorFlow. As a consequence, TFQ releases will not work +with any other version of TensorFlow than the one they are pinned to. + +Python wheels for TFQ are produced by compiling them locally with a toolchain +that matches that used by the version of TensorFlow being targeted by a given +version of TFQ. A number of elements affect whether the whole process succeeds +and the resulting wheel is portable to environments other than the specific +computer TFQ is built on, including: + +* The version of Python and the local Python environment +* The version of TensorFlow +* The TensorFlow build container used +* The Crosstool configuration used +* Whether CUDA is being used, and its version +* The dependency requirements implied by Cirq, NumPy, Protobuf, and + other Python packages + +## Procedure + +Building a TensorFlow Quantum release for Linux involves some additional steps +beyond just building TFQ and producing an initial Python wheel. The procedure +uses `auditwheel` to "repair" the resulting wheel; this improves the +compatibility of the wheel so that it can run on a wider range of Linux +distributions, even if those distributions have different versions of system +libraries. + +The steps are: + +1. Git clone the TensorFlow Quantum repo to a directory on your computer. + +1. `cd` into the local clone directory. + +1. Create a Python virtual environment. + +1. Run `pip install -r requirements.txt` + +1. Verify the major.minor version of Python you are using. The rest of these + instructions use 3.11 as an example. + +1. Run `./release/build_distribution.sh -p 3.11` + +1. If the above succeeds, it will leave the wheel in `/tmp/tensorflow_quantum/` + on your system. Take note of the name of the wheel file that + `build_distribution.sh` prints when it finishes. + +1. Run `./release/clean_distribution.sh /tmp/tensorflow_quantum/WHEEL_FILE`, + where `WHEEL_FILE` is the file noted in the previous step. If this works, it + will create a new wheel file in `../wheelhouse`. If an error occurs, it will + hopefully report the problem. If the error is a platform tag mismatch, run + `./release/clean_distribution.sh -s /tmp/tensorflow_quantum/WHEEL_FILE`; + this will run auditwheel's `show` command on the wheel file to indicate what + version of `manylinux` this wheel can be made to run on if you use + `auditwheel` to repair it. With that information, you may be able to edit + the `build_distribution.sh` script to experiment with different values for + the Crosstool and/or the Docker images used. + +1. Test the wheel. Go to a remotely hosted Colab (or any other Linux platform + that is distinctly difference from yours) upload this new generated wheel + file to local storage in the Colab, and test if it works. (E.g., try to run + `https://www.tensorflow.org/quantum/tutorials/hello_many_worlds` with the + new TensorFlow and TFQ wheels to verify things are working smoothly). + +1. Repeat the `build_distribution.sh` and `clean_distribution.sh` steps for + different versions of Python. + +To make TFQ wheels that will work on other systems, it's essential that the +platform tags of the TFQ wheel must match the tags of the current TensorFlow +version you are targeting. (Visit `https://pypi.org/project/tensorflow//#files` to determine what the tags are). + +## Testing the scripts + +There are some basic unit tests in the fils `build_distribution_test.py` and +`clean_distribution_test.py`. They can be run using as follows: + +```shell +# Move to the top-level directory of the repo before running the tests. +cd .. +bazel test //release:build_distribution_test //release:clean_distribution_test +``` + +## More information + +"TensorFlow SIG Build" is a community group dedicated to the TensorFlow build +process. This repository is a showcase of resources, guides, tools, and builds +contributed by the community, for the community. The following resources may be +useful when trying to figure out how to make this all work. + +* The "TF SIG Build Dockerfiles" document: + https://github.com/tensorflow/build/tree/ff4320fee2cf48568ebd2f476d7714438bfa0bee/tf_sig_build_dockerfiles#readme + +* Other info in the SIG Build repo: https://github.com/tensorflow/build diff --git a/release/build_distribution.sh b/release/build_distribution.sh new file mode 100755 index 000000000..0bbfb98fe --- /dev/null +++ b/release/build_distribution.sh @@ -0,0 +1,159 @@ +#!/usr/bin/env bash +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= + +# Summary: build a wheel for TFQ using a TensorFlow SIG Build container. +# Run this script with the option "-h" to get a usage summary. +# +# To ensure binary compatibility with TensorFlow, TFQ distributions are built +# using TensorFlow's SIG Build containers and Crosstool C++ toolchain. This +# script encapsulates the process. The basic steps this script performs are: +# +# 1. Write to a file a small shell script that does the following: +# +# a) pip install TFQ's requirements.txt file +# b) run TFQ's configure.sh script +# c) run Bazel to build build_pip_package +# d) run the resulting build_pip_package +# e) copy the wheel created by build_pip_package to ./wheels +# +# 2. Start Docker with image tensorflow/build:${tf_version}-python${py_version} +# and run the script written in step 1. +# +# 3. Do some basic tests on the wheel using standard Python utilities. +# +# 4. Exit. + +set -eu + +# Find the top of the local TFQ git tree. Do it early in case this fails. +thisdir=$(CDPATH="" cd -- "$(dirname -- "$0")" && pwd -P) +repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2>/dev/null || \ + echo "${thisdir}/..") + +# Default values for variables that can be changed via command line flags. +tf_version="2.16" +py_version=$(python3 --version | cut -d' ' -f2 | cut -d. -f1,2) +cuda_version="12" +cleanup="true" + +usage="Usage: ${0} [OPTIONS] +Build a distribution wheel for TensorFlow Quantum. + +Configuration options: + -c X.Y Use CUDA version X.Y (default: ${cuda_version}) + -p X.Y Use Python version X.Y (default: ${py_version}) + -t X.Y Use TensorFlow version X.Y (default: ${tf_version}) + +General options: + -e Don't run bazel clean at the end (default: do) + -n Dry run: print commands but don't execute them + -h Show this help message and exit" + +dry_run="false" +while getopts "c:ehnp:t:" opt; do + case "${opt}" in + c) cuda_version="${OPTARG}" ;; + e) cleanup="false" ;; + h) echo "${usage}"; exit 0 ;; + n) dry_run="true" ;; + p) py_version="${OPTARG}" ;; + t) tf_version="${OPTARG}" ;; + *) echo "${usage}" >&2; exit 1 ;; + esac +done +shift $((OPTIND -1)) + +# See https://hub.docker.com/r/tensorflow/build/tags for available containers. +docker_image="tensorflow/build:${tf_version}-python${py_version}" + +# This should match what TensorFlow's .bazelrc file uses. +crosstool="@sigbuild-r${tf_version}-clang_config_cuda//crosstool:toolchain" + +# Note: configure.sh is run inside the container, and it creates a .bazelrc +# file that adds other cxxopt flags. They don't need to be repeated here. +BUILD_OPTIONS="--cxxopt=-O3 --cxxopt=-msse2 --cxxopt=-msse3 --cxxopt=-msse4" + +# Create a script to be run by the shell inside the Docker container. +build_script=$(mktemp /tmp/tfq_build.XXXXXX) +trap 'rm -f "${build_script}" || true' EXIT + +# The printf'ed section dividers are to make it easier to search the output. +cat <<'EOF' > "${build_script}" +#!/usr/bin/env bash +set -o errexit +cd /tfq +PREFIX='[DOCKER] ' +exec > >(sed "s/^/${PREFIX} /") +exec 2> >(sed "s/^/${PREFIX} /" >&2) +printf "Build configuration inside Docker container:\n" +printf " Docker image: ${docker_image}\n" +printf " Crosstool: ${crosstool}\n" +printf " TF version: ${tf_version}\n" +printf " Python version: ${py_version}\n" +printf " CUDA version: ${cuda_version}\n" +printf " vCPUs available: $(nproc)\n" +printf "\n:::::::: Configuring Python environment ::::::::\n\n" +python3 -m pip install --upgrade pip --root-user-action ignore +pip install -r requirements.txt --root-user-action ignore +printf "\n:::::::: Configuring TensorFlow Quantum build ::::::::\n\n" +printf "Y\n" | ./configure.sh +printf "\n:::::::: Starting Bazel build ::::::::\n\n" +bazel build ${build_flags} release:build_pip_package +printf "\n:::::::: Creating Python wheel ::::::::\n\n" +bazel-bin/release/build_pip_package /build_output/ +if [ "${cleanup}" == "true" ]; then + printf "\n:::::::: Cleaning up ::::::::\n\n" + bazel clean --async +fi +EOF + +chmod +x "${build_script}" + +# Use 'set --' to build the command in the positional parameters ($1, $2, ...) +set -- docker run -it --rm --network host \ + -w /tfq \ + -v "${repo_dir}":/tfq \ + -v /tmp/tensorflow_quantum:/build_output \ + -v "${build_script}:/tmp/build_script.sh" \ + -e HOST_PERMS="$(id -u):$(id -g)" \ + -e build_flags="--crosstool_top=${crosstool} ${BUILD_OPTIONS}" \ + -e cuda_version="${cuda_version}" \ + -e py_version="${py_version}" \ + -e tf_version="${tf_version}" \ + -e docker_image="${docker_image}" \ + -e crosstool="${crosstool}" \ + -e cleanup="${cleanup}" \ + "${docker_image}" \ + /tmp/build_script.sh + +if [ "${dry_run}" == "true" ]; then + # Loop through the positional parameters and simply print them. + printf "(Dry run) " + printf '%s ' "$@" + echo + echo '(Dry run) check-wheel-contents /tmp/tensorflow_quantum/*.whl' +else + echo "Spinning up a Docker container with ${docker_image} …" + "$@" + + # Do some basic checks on the wheel file. + echo "Doing basic sanity checks on the wheel …" + pip install -qq check-wheel-contents + check-wheel-contents /tmp/tensorflow_quantum/*.whl + + echo "Done. Look for wheel in /tmp/tensorflow_quantum/." + ls -l /tmp/tensorflow_quantum/ +fi diff --git a/release/clean_distribution.sh b/release/clean_distribution.sh new file mode 100755 index 000000000..680ea6bee --- /dev/null +++ b/release/clean_distribution.sh @@ -0,0 +1,100 @@ +#!/usr/bin/env bash +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= + +# Summary: bundle external shared libraries into the final TFQ wheel. +# Run this script with the option "-h" to get a usage summary. +# +# This uses auditwheel to "repair" the wheel: copy external shared libraries +# into the wheel itself and modify the RPATH entries such that these libraries +# will be picked up at runtime. This accomplishes a similar result as if the +# libraries had been statically linked. + +set -eu + +# Find the top of the local TFQ git tree. Do it early in case this fails. +thisdir=$(CDPATH="" cd -- "$(dirname -- "$0")" && pwd -P) +repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2>/dev/null || \ + echo "${thisdir}/..") + +# Default values for variables that can be changed via command line flags. +docker_image="quay.io/pypa/manylinux_2_34_x86_64" +platform="manylinux_2_17_x86_64" +py_version=$(python3 --version | cut -d' ' -f2 | cut -d. -f1,2) +action="repair" +verbose="" + +usage="Usage: ${0} [OPTIONS] /path/to/wheel.whl +Run auditwheel on the given wheel file. Available options: + +Configuration options: + -m IMG Use manylinux Docker image IMG (default: ${docker_image}) + -p X.Y Use Python version X.Y (default: ${py_version}) + -t TAG Pass --plat TAG to auditwheel (default: ${platform}) + +General options: + -h Show this help message and exit + -n Dry run: print commands but don't execute them + -s Run 'auditwheel show', not repair (default: run 'auditwheel repair') + -v Produce verbose output" + +dry_run="false" +while getopts "hm:np:st:v" opt; do + case "${opt}" in + h) echo "${usage}"; exit 0 ;; + m) docker_image="${OPTARG}" ;; + n) dry_run="true" ;; + p) py_version="${OPTARG}" ;; + s) action="show" ;; + t) platform="${OPTARG}" ;; + v) verbose="--verbose" ;; + *) echo "${usage}" >&2; exit 1 ;; + esac +done +shift $((OPTIND -1)) +if [ ! $# -ge 1 ]; then + echo "ERROR: insufficient arguments." + echo "${usage}" >&2 + exit 1 +fi + +wheel_path="$(realpath "${1}")" +wheel_name="$(basename "${1}")" + +args="" +if [ "${action}" = "repair" ]; then + args="${verbose} --exclude libtensorflow_framework.so.2 --plat ${platform}" +fi + +# Use 'set --' to build the command in the positional parameters ($1, $2, ...) +set -- docker run -it --rm --network host \ + -w /tfq \ + -v "${repo_dir}":/tfq \ + -v "${wheel_path}":"/tmp/${wheel_name}" \ + "${docker_image}" \ + bash -c "auditwheel ${action} ${args} -w /tfq/wheelhouse /tmp/${wheel_name}" + +if [ "${dry_run}" = "true" ]; then + # Loop through the positional parameters and simply print them. + printf "(Dry run) " + printf '%s ' "$@" + echo +else + echo "Running 'auditwheel ${action}' in Docker with image ${docker_image}" + "$@" + if [ "${action}" = "repair" ]; then + echo "Done. New wheel file written to ${repo_dir}/wheelhouse" + fi +fi From 03121e745c8af9ab77aff336262b20d828d22dcc Mon Sep 17 00:00:00 2001 From: mhucka Date: Tue, 9 Dec 2025 05:37:59 +0000 Subject: [PATCH 02/23] Add unit tests for build_distribution & clean_distribution --- release/BUILD | 23 +++++++ release/build_distribution_test.py | 103 +++++++++++++++++++++++++++ release/clean_distribution_test.py | 107 +++++++++++++++++++++++++++++ 3 files changed, 233 insertions(+) create mode 100644 release/build_distribution_test.py create mode 100644 release/clean_distribution_test.py diff --git a/release/BUILD b/release/BUILD index f23d8528f..9aa967bc1 100644 --- a/release/BUILD +++ b/release/BUILD @@ -82,3 +82,26 @@ sh_binary( "//tensorflow_quantum/python/optimizers:spsa_minimizer", ], ) + +exports_files([ + "build_distribution.sh", + "clean_distribution.sh", +]) + +py_test( + name = "build_distribution_test", + srcs = ["build_distribution_test.py"], + data = [ + "build_distribution.sh", + ], + python_version = "PY3", +) + +py_test( + name = "clean_distribution_test", + srcs = ["clean_distribution_test.py"], + data = [ + "clean_distribution.sh", + ], + python_version = "PY3", +) diff --git a/release/build_distribution_test.py b/release/build_distribution_test.py new file mode 100644 index 000000000..f9a003c23 --- /dev/null +++ b/release/build_distribution_test.py @@ -0,0 +1,103 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ +"""Tests for build_distribution.sh script.""" + +import os +import subprocess +import unittest + + +class BuildDistributionTest(unittest.TestCase): + """Tests for build_distribution.sh script.""" + + def setUp(self): + # Find the repo root. Default to the current directory if that fails. + try: + self.repo_root = subprocess.check_output( + ["git", "rev-parse", "--show-toplevel"], + universal_newlines=True).strip() + except subprocess.CalledProcessError: + self.repo_root = os.getcwd() + + self.script = os.path.join(self.repo_root, "release", + "build_distribution.sh") + + def test_dry_run(self): + """Test build_distribution.sh script in dry-run mode.""" + cmd = [self.script, "-n", "-c", "11.2", "-p", "3.9", "-t", "2.10"] + result = subprocess.run(cmd, + capture_output=True, + text=True, + check=False) + self.assertEqual(result.returncode, 0, + f"Script failed with stderr: {result.stderr}") + output = result.stdout + + self.assertIn("(Dry run) docker run", output) + # Check that arguments propagated to the docker image tag and env vars + self.assertIn("tensorflow/build:2.10-python3.9", output) + self.assertIn("cuda_version=11.2", output) + self.assertIn("py_version=3.9", output) + self.assertIn("tf_version=2.10", output) + + self.assertIn( + "(Dry run) check-wheel-contents /tmp/tensorflow_quantum/*.whl", + output) + + def test_defaults(self): + """Test build_distribution.sh script defaults.""" + cmd = [self.script, "-n"] + result = subprocess.run(cmd, + capture_output=True, + text=True, + check=False) + self.assertEqual(result.returncode, 0, + f"Script failed with stderr: {result.stderr}") + output = result.stdout + # Check default cuda version (12) and default cleanup (true) + self.assertIn("cuda_version=12", output) + self.assertIn("cleanup=true", output) + + def test_help(self): + """Test build_distribution.sh script help flag.""" + cmd = [self.script, "-h"] + result = subprocess.run(cmd, + capture_output=True, + text=True, + check=False) + self.assertEqual(result.returncode, 0, + f"Script failed with stderr: {result.stderr}") + self.assertIn("Usage:", result.stdout) + self.assertIn("Build a distribution wheel for TensorFlow Quantum.", + result.stdout) + + def test_invalid_option(self): + """Test build_distribution.sh script with invalid option.""" + cmd = [self.script, "-z"] + result = subprocess.run(cmd, + capture_output=True, + text=True, + check=False) + self.assertNotEqual(result.returncode, 0) + self.assertIn("Usage:", result.stdout + result.stderr) + stderr = result.stderr.lower() + self.assertTrue( + "illegal option" in stderr or "invalid option" in stderr, + "Expected 'illegal option' or 'invalid option' in stderr, " + f"got: {stderr}") + + +if __name__ == "__main__": + unittest.main() diff --git a/release/clean_distribution_test.py b/release/clean_distribution_test.py new file mode 100644 index 000000000..ff96a2c83 --- /dev/null +++ b/release/clean_distribution_test.py @@ -0,0 +1,107 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ +"""Tests for clean_distribution.sh script.""" + +import os +import subprocess +import unittest + + +class CleanDistributionTest(unittest.TestCase): + """Tests for clean_distribution.sh script.""" + + def setUp(self): + # Find the repo root + try: + self.repo_root = subprocess.check_output( + ["git", "rev-parse", "--show-toplevel"], + universal_newlines=True).strip() + except subprocess.CalledProcessError: + self.repo_root = os.getcwd() + + self.script = os.path.join(self.repo_root, "release", + "clean_distribution.sh") + + def test_dry_run(self): + """Test clean_distribution.sh script in dry-run mode.""" + wheel_path = "fake.whl" + cmd = [self.script, "-n", "-p", "3.10", wheel_path] + result = subprocess.run(cmd, + capture_output=True, + text=True, + check=False) + self.assertEqual(result.returncode, 0, + f"Script failed with stderr: {result.stderr}") + output = result.stdout + + self.assertIn("(Dry run) docker run", output) + self.assertIn("auditwheel repair", output) + self.assertIn(f"/tmp/{wheel_path}", output) + + def test_show_action(self): + """Test clean_distribution.sh script with -s (show) action.""" + wheel_path = "fake.whl" + cmd = [self.script, "-n", "-s", wheel_path] + result = subprocess.run(cmd, + capture_output=True, + text=True, + check=False) + self.assertEqual(result.returncode, 0, + f"Script failed with stderr: {result.stderr}") + output = result.stdout + + self.assertIn("auditwheel show", output) + + def test_missing_arg(self): + """Test clean_distribution.sh script fails without wheel argument.""" + cmd = [self.script, "-n"] + result = subprocess.run(cmd, + capture_output=True, + text=True, + check=False) + self.assertNotEqual(result.returncode, 0) + self.assertIn("ERROR: insufficient arguments", + result.stdout + result.stderr) + + def test_help(self): + """Test clean_distribution.sh script help flag.""" + cmd = [self.script, "-h"] + result = subprocess.run(cmd, + capture_output=True, + text=True, + check=False) + self.assertEqual(result.returncode, 0, + f"Script failed with stderr: {result.stderr}") + self.assertIn("Usage:", result.stdout) + self.assertIn("Run auditwheel on the given wheel file.", result.stdout) + + def test_invalid_option(self): + """Test clean_distribution.sh script with invalid option.""" + cmd = [self.script, "-z"] + result = subprocess.run(cmd, + capture_output=True, + text=True, + check=False) + self.assertNotEqual(result.returncode, 0) + self.assertIn("Usage:", result.stdout + result.stderr) + stderr = result.stderr.lower() + self.assertTrue( + "illegal option" in stderr or "invalid option" in stderr, + "Expected 'illegal option' or 'invalid option' in stderr, " + f"got: {stderr}") + + +if __name__ == "__main__": + unittest.main() From 7ad112da497671e68ab969a756731a25b543115e Mon Sep 17 00:00:00 2001 From: mhucka Date: Tue, 9 Dec 2025 05:40:10 +0000 Subject: [PATCH 03/23] Remove `set -x` & a print stmt that leads to confusion The final print statement about the directory is misleading when `build_pip_package.sh` is being run inside a Docker container. On balance, I think it's better to remove it. Hopefully the user running the script will remember that they supplied the destination parameter as an argument on the command line to begin with. --- release/build_pip_package.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/release/build_pip_package.sh b/release/build_pip_package.sh index 8bed5b909..9cdfcf9f4 100755 --- a/release/build_pip_package.sh +++ b/release/build_pip_package.sh @@ -1,20 +1,20 @@ #!/bin/bash # Copyright 2020 The TensorFlow Quantum Authors. All Rights Reserved. -# +# # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at -# +# # http://www.apache.org/licenses/LICENSE-2.0 -# +# # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================== + set -e -set -x EXPORT_DIR="bazel-bin/release/build_pip_package.runfiles/__main__" @@ -53,7 +53,7 @@ function main() { cp dist/*.whl "${DEST}" popd rm -rf ${TMPDIR} - echo $(date) : "=== Output wheel file is in: ${DEST}" + echo "$(date) : === Done." } main "$@" From 247faacbb9c34b78f77334d736dab2c6ba346e85 Mon Sep 17 00:00:00 2001 From: mhucka Date: Tue, 9 Dec 2025 05:43:39 +0000 Subject: [PATCH 04/23] Make sure build_pip_package.sh has the tools it needs --- release/build_pip_package.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/release/build_pip_package.sh b/release/build_pip_package.sh index 9cdfcf9f4..603a060f5 100755 --- a/release/build_pip_package.sh +++ b/release/build_pip_package.sh @@ -16,6 +16,9 @@ set -e +# Ensure packaging tools are present in THIS interpreter. +pip install -qq setuptools wheel build --root-user-action ignore + EXPORT_DIR="bazel-bin/release/build_pip_package.runfiles/__main__" function main() { From 1377c8402b6c10994657a610c4cd8cddfdae0f2f Mon Sep 17 00:00:00 2001 From: mhucka Date: Tue, 9 Dec 2025 22:14:11 +0000 Subject: [PATCH 05/23] Use /bin/bash Per Google guidance, executable scripts should use `/bin/bash`. --- release/build_distribution.sh | 4 ++-- release/clean_distribution.sh | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/release/build_distribution.sh b/release/build_distribution.sh index 0bbfb98fe..3e1da6fba 100755 --- a/release/build_distribution.sh +++ b/release/build_distribution.sh @@ -1,4 +1,4 @@ -#!/usr/bin/env bash +#!/bin/bash # Copyright 2025 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); @@ -92,7 +92,7 @@ trap 'rm -f "${build_script}" || true' EXIT # The printf'ed section dividers are to make it easier to search the output. cat <<'EOF' > "${build_script}" -#!/usr/bin/env bash +#!/bin/bash set -o errexit cd /tfq PREFIX='[DOCKER] ' diff --git a/release/clean_distribution.sh b/release/clean_distribution.sh index 680ea6bee..212da0de4 100755 --- a/release/clean_distribution.sh +++ b/release/clean_distribution.sh @@ -1,4 +1,4 @@ -#!/usr/bin/env bash +#!/bin/bash # Copyright 2025 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); From 7e679288a0157cfe55de6b4793d519ee594ae5f7 Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 02:26:32 +0000 Subject: [PATCH 06/23] Remove testing scripts Per review comments by Michael Broughton, removing the testing scripts because they are probably overkill and a maintenance burden. --- release/BUILD | 23 ------- release/README.md | 11 --- release/build_distribution_test.py | 103 --------------------------- release/clean_distribution_test.py | 107 ----------------------------- 4 files changed, 244 deletions(-) delete mode 100644 release/build_distribution_test.py delete mode 100644 release/clean_distribution_test.py diff --git a/release/BUILD b/release/BUILD index 9aa967bc1..f23d8528f 100644 --- a/release/BUILD +++ b/release/BUILD @@ -82,26 +82,3 @@ sh_binary( "//tensorflow_quantum/python/optimizers:spsa_minimizer", ], ) - -exports_files([ - "build_distribution.sh", - "clean_distribution.sh", -]) - -py_test( - name = "build_distribution_test", - srcs = ["build_distribution_test.py"], - data = [ - "build_distribution.sh", - ], - python_version = "PY3", -) - -py_test( - name = "clean_distribution_test", - srcs = ["clean_distribution_test.py"], - data = [ - "clean_distribution.sh", - ], - python_version = "PY3", -) diff --git a/release/README.md b/release/README.md index 20e082000..50c1b3670 100644 --- a/release/README.md +++ b/release/README.md @@ -88,17 +88,6 @@ platform tags of the TFQ wheel must match the tags of the current TensorFlow version you are targeting. (Visit `https://pypi.org/project/tensorflow//#files` to determine what the tags are). -## Testing the scripts - -There are some basic unit tests in the fils `build_distribution_test.py` and -`clean_distribution_test.py`. They can be run using as follows: - -```shell -# Move to the top-level directory of the repo before running the tests. -cd .. -bazel test //release:build_distribution_test //release:clean_distribution_test -``` - ## More information "TensorFlow SIG Build" is a community group dedicated to the TensorFlow build diff --git a/release/build_distribution_test.py b/release/build_distribution_test.py deleted file mode 100644 index f9a003c23..000000000 --- a/release/build_distribution_test.py +++ /dev/null @@ -1,103 +0,0 @@ -# Copyright 2025 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# https://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================ -"""Tests for build_distribution.sh script.""" - -import os -import subprocess -import unittest - - -class BuildDistributionTest(unittest.TestCase): - """Tests for build_distribution.sh script.""" - - def setUp(self): - # Find the repo root. Default to the current directory if that fails. - try: - self.repo_root = subprocess.check_output( - ["git", "rev-parse", "--show-toplevel"], - universal_newlines=True).strip() - except subprocess.CalledProcessError: - self.repo_root = os.getcwd() - - self.script = os.path.join(self.repo_root, "release", - "build_distribution.sh") - - def test_dry_run(self): - """Test build_distribution.sh script in dry-run mode.""" - cmd = [self.script, "-n", "-c", "11.2", "-p", "3.9", "-t", "2.10"] - result = subprocess.run(cmd, - capture_output=True, - text=True, - check=False) - self.assertEqual(result.returncode, 0, - f"Script failed with stderr: {result.stderr}") - output = result.stdout - - self.assertIn("(Dry run) docker run", output) - # Check that arguments propagated to the docker image tag and env vars - self.assertIn("tensorflow/build:2.10-python3.9", output) - self.assertIn("cuda_version=11.2", output) - self.assertIn("py_version=3.9", output) - self.assertIn("tf_version=2.10", output) - - self.assertIn( - "(Dry run) check-wheel-contents /tmp/tensorflow_quantum/*.whl", - output) - - def test_defaults(self): - """Test build_distribution.sh script defaults.""" - cmd = [self.script, "-n"] - result = subprocess.run(cmd, - capture_output=True, - text=True, - check=False) - self.assertEqual(result.returncode, 0, - f"Script failed with stderr: {result.stderr}") - output = result.stdout - # Check default cuda version (12) and default cleanup (true) - self.assertIn("cuda_version=12", output) - self.assertIn("cleanup=true", output) - - def test_help(self): - """Test build_distribution.sh script help flag.""" - cmd = [self.script, "-h"] - result = subprocess.run(cmd, - capture_output=True, - text=True, - check=False) - self.assertEqual(result.returncode, 0, - f"Script failed with stderr: {result.stderr}") - self.assertIn("Usage:", result.stdout) - self.assertIn("Build a distribution wheel for TensorFlow Quantum.", - result.stdout) - - def test_invalid_option(self): - """Test build_distribution.sh script with invalid option.""" - cmd = [self.script, "-z"] - result = subprocess.run(cmd, - capture_output=True, - text=True, - check=False) - self.assertNotEqual(result.returncode, 0) - self.assertIn("Usage:", result.stdout + result.stderr) - stderr = result.stderr.lower() - self.assertTrue( - "illegal option" in stderr or "invalid option" in stderr, - "Expected 'illegal option' or 'invalid option' in stderr, " - f"got: {stderr}") - - -if __name__ == "__main__": - unittest.main() diff --git a/release/clean_distribution_test.py b/release/clean_distribution_test.py deleted file mode 100644 index ff96a2c83..000000000 --- a/release/clean_distribution_test.py +++ /dev/null @@ -1,107 +0,0 @@ -# Copyright 2025 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# https://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================ -"""Tests for clean_distribution.sh script.""" - -import os -import subprocess -import unittest - - -class CleanDistributionTest(unittest.TestCase): - """Tests for clean_distribution.sh script.""" - - def setUp(self): - # Find the repo root - try: - self.repo_root = subprocess.check_output( - ["git", "rev-parse", "--show-toplevel"], - universal_newlines=True).strip() - except subprocess.CalledProcessError: - self.repo_root = os.getcwd() - - self.script = os.path.join(self.repo_root, "release", - "clean_distribution.sh") - - def test_dry_run(self): - """Test clean_distribution.sh script in dry-run mode.""" - wheel_path = "fake.whl" - cmd = [self.script, "-n", "-p", "3.10", wheel_path] - result = subprocess.run(cmd, - capture_output=True, - text=True, - check=False) - self.assertEqual(result.returncode, 0, - f"Script failed with stderr: {result.stderr}") - output = result.stdout - - self.assertIn("(Dry run) docker run", output) - self.assertIn("auditwheel repair", output) - self.assertIn(f"/tmp/{wheel_path}", output) - - def test_show_action(self): - """Test clean_distribution.sh script with -s (show) action.""" - wheel_path = "fake.whl" - cmd = [self.script, "-n", "-s", wheel_path] - result = subprocess.run(cmd, - capture_output=True, - text=True, - check=False) - self.assertEqual(result.returncode, 0, - f"Script failed with stderr: {result.stderr}") - output = result.stdout - - self.assertIn("auditwheel show", output) - - def test_missing_arg(self): - """Test clean_distribution.sh script fails without wheel argument.""" - cmd = [self.script, "-n"] - result = subprocess.run(cmd, - capture_output=True, - text=True, - check=False) - self.assertNotEqual(result.returncode, 0) - self.assertIn("ERROR: insufficient arguments", - result.stdout + result.stderr) - - def test_help(self): - """Test clean_distribution.sh script help flag.""" - cmd = [self.script, "-h"] - result = subprocess.run(cmd, - capture_output=True, - text=True, - check=False) - self.assertEqual(result.returncode, 0, - f"Script failed with stderr: {result.stderr}") - self.assertIn("Usage:", result.stdout) - self.assertIn("Run auditwheel on the given wheel file.", result.stdout) - - def test_invalid_option(self): - """Test clean_distribution.sh script with invalid option.""" - cmd = [self.script, "-z"] - result = subprocess.run(cmd, - capture_output=True, - text=True, - check=False) - self.assertNotEqual(result.returncode, 0) - self.assertIn("Usage:", result.stdout + result.stderr) - stderr = result.stderr.lower() - self.assertTrue( - "illegal option" in stderr or "invalid option" in stderr, - "Expected 'illegal option' or 'invalid option' in stderr, " - f"got: {stderr}") - - -if __name__ == "__main__": - unittest.main() From eb7b20643290b7a6632941cf9a3ec6d2f6d60898 Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 02:32:52 +0000 Subject: [PATCH 07/23] Tweak format of diagnostic messages printed during build This reduces the number of messages and slightly adjust format and spacing to make things more consistent and have better spacing. --- release/build_distribution.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/release/build_distribution.sh b/release/build_distribution.sh index 3e1da6fba..1f3beb8a5 100755 --- a/release/build_distribution.sh +++ b/release/build_distribution.sh @@ -98,17 +98,16 @@ cd /tfq PREFIX='[DOCKER] ' exec > >(sed "s/^/${PREFIX} /") exec 2> >(sed "s/^/${PREFIX} /" >&2) -printf "Build configuration inside Docker container:\n" +printf ":::::::: Build configuration inside Docker container ::::::::\n" printf " Docker image: ${docker_image}\n" printf " Crosstool: ${crosstool}\n" printf " TF version: ${tf_version}\n" printf " Python version: ${py_version}\n" printf " CUDA version: ${cuda_version}\n" printf " vCPUs available: $(nproc)\n" -printf "\n:::::::: Configuring Python environment ::::::::\n\n" +printf "\n\n:::::::: Configuring Python environment ::::::::\n\n" python3 -m pip install --upgrade pip --root-user-action ignore pip install -r requirements.txt --root-user-action ignore -printf "\n:::::::: Configuring TensorFlow Quantum build ::::::::\n\n" printf "Y\n" | ./configure.sh printf "\n:::::::: Starting Bazel build ::::::::\n\n" bazel build ${build_flags} release:build_pip_package From c7839839438fd29b5ad422604519f7050698f412 Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 02:33:54 +0000 Subject: [PATCH 08/23] Handle full Python versions for -p flag Allow Python version numbers of the form `x.y.z` as the value of `-p`, not just two-digit `x.y` numbers. --- release/build_distribution.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release/build_distribution.sh b/release/build_distribution.sh index 1f3beb8a5..02d180846 100755 --- a/release/build_distribution.sh +++ b/release/build_distribution.sh @@ -69,7 +69,7 @@ while getopts "c:ehnp:t:" opt; do e) cleanup="false" ;; h) echo "${usage}"; exit 0 ;; n) dry_run="true" ;; - p) py_version="${OPTARG}" ;; + p) py_version=$(echo "${OPTARG}" | cut -d. -f1,2) ;; t) tf_version="${OPTARG}" ;; *) echo "${usage}" >&2; exit 1 ;; esac From 2d516e9a7431f804729882222d4f863fd72ea058 Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 02:45:23 +0000 Subject: [PATCH 09/23] Use better Bash syntax --- release/clean_distribution.sh | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/release/clean_distribution.sh b/release/clean_distribution.sh index 212da0de4..baf506966 100755 --- a/release/clean_distribution.sh +++ b/release/clean_distribution.sh @@ -22,10 +22,10 @@ # will be picked up at runtime. This accomplishes a similar result as if the # libraries had been statically linked. -set -eu +set -eu -o pipefail # Find the top of the local TFQ git tree. Do it early in case this fails. -thisdir=$(CDPATH="" cd -- "$(dirname -- "$0")" && pwd -P) +thisdir=$(CDPATH="" cd -- "$(dirname -- "${0}")" && pwd -P) repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2>/dev/null || \ echo "${thisdir}/..") @@ -64,8 +64,8 @@ while getopts "hm:np:st:v" opt; do esac done shift $((OPTIND -1)) -if [ ! $# -ge 1 ]; then - echo "ERROR: insufficient arguments." +if (( $# < 1 )); then + echo "ERROR: need at least one argument argument." echo "${usage}" >&2 exit 1 fi @@ -74,7 +74,7 @@ wheel_path="$(realpath "${1}")" wheel_name="$(basename "${1}")" args="" -if [ "${action}" = "repair" ]; then +if [[ "${action}" == "repair" ]]; then args="${verbose} --exclude libtensorflow_framework.so.2 --plat ${platform}" fi @@ -86,7 +86,7 @@ set -- docker run -it --rm --network host \ "${docker_image}" \ bash -c "auditwheel ${action} ${args} -w /tfq/wheelhouse /tmp/${wheel_name}" -if [ "${dry_run}" = "true" ]; then +if [[ "${dry_run}" == "true" ]]; then # Loop through the positional parameters and simply print them. printf "(Dry run) " printf '%s ' "$@" @@ -94,7 +94,7 @@ if [ "${dry_run}" = "true" ]; then else echo "Running 'auditwheel ${action}' in Docker with image ${docker_image}" "$@" - if [ "${action}" = "repair" ]; then + if [[ "${action}" == "repair" ]]; then echo "Done. New wheel file written to ${repo_dir}/wheelhouse" fi fi From b9c38b107dd20e050c09b9d17e6091aeddc11ccd Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 02:46:44 +0000 Subject: [PATCH 10/23] Avoid `realpath` in favor of more portable approach --- release/clean_distribution.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release/clean_distribution.sh b/release/clean_distribution.sh index baf506966..89a202d15 100755 --- a/release/clean_distribution.sh +++ b/release/clean_distribution.sh @@ -70,7 +70,7 @@ if (( $# < 1 )); then exit 1 fi -wheel_path="$(realpath "${1}")" +wheel_path="$(cd "$(dirname "${1}")" && pwd)/$(basename "${1}")" wheel_name="$(basename "${1}")" args="" From d45c9e8f02962b53b70393fa0e6831980a53c0ad Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 02:53:49 +0000 Subject: [PATCH 11/23] Add script for creating releases This runs `build_distribution.sh` and `clean_distribution.sh` to create read-to-distribut wheels for TFQ. --- release/build_release.sh | 145 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 145 insertions(+) create mode 100755 release/build_release.sh diff --git a/release/build_release.sh b/release/build_release.sh new file mode 100755 index 000000000..76a16b9a8 --- /dev/null +++ b/release/build_release.sh @@ -0,0 +1,145 @@ +#!/bin/bash +# Copyright 2025 The TensorFlow Quantum Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= + +set -eu -o pipefail + +# Summary: do all the steps to generate a wheel for a TFQ release. +# +# This sets up a clean pyenv virtualenv for a given Python version, then runs +# release/build_distribution.sh and release/clean_distribution.sh, and finishes +# by printing some info about the wheel. The wheel is left in ./wheelhouse/. +# The TFQ release number is extracted from setup.py. +# +# Note: this uses build_distribution.sh, which builds the TFQ pip package inside +# a Docker container. The TFQ git directory where this script is found is mapped +# directly inside the running Docker environment. The approach makes it easy to +# iterate on changes to TFQ files in the current directory, and avoids a lot of +# frustrating "which copy of such-and-such file did it use?" questions. However, +# it also brings a risk of unexpected impact of left-overs in the current +# directory. To avoid this, it's best to make a copy of your TFQ git repository +# (or git-clone a fresh copy from GitHub) and clean it before proceeding. + +# ~~~~~~~~ Helpers ~~~~~~~~ + +function quit() { + echo "ERROR: $*." >&2 + exit 1 +} + +function have() { + unset -v have + type $1 &> /dev/null +} + +# ~~~~~~~~ Sanity checks ~~~~~~~~ + +# Go to the top of the local TFQ git tree. Do it early in case this fails. +thisdir=$(CDPATH="" cd -- "$(dirname -- "$0")" && pwd -P) +repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2>/dev/null || \ + quit "this script must be run from inside the TFQ git tree") +cd "${repo_dir}" + +py_version="${1}" +if ! [[ "${py_version}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + quit "the first argument must be a full Python version number" +fi + +build_number="" +if (( $# > 1 )); then + build_number="-${2}" +fi + +setup_file="${repo_dir}/release/setup.py" +if [[ -r "${setup_file}" ]]; then + tfq_version=$(grep -m 1 CUR_VERSION "${setup_file}" | cut -f2 -d'"') +else + quit "cannot read ${setup_file}" +fi + +for program in python3 pip pyenv jq; do + if ! have $program; then + quit "cannot run $program -- maybe it is not installed?" + fi +done + +version=${tfq_version}${build_number} + +# ~~~~~~~~ Set up a new virtual environment ~~~~~~~~ + +# Since the build is done inside a Docker container, it is not really necessary +# to create a virtual Python environment for that part of the process. However, +# we do run some Python commands before and after, and we want those to be done +# in an environment we control with the same Python version being targeted for +# the build. It provides additional isolation. + +echo "~~~~ Starting ${0} for TFQ release ${version}" +echo "~~~~ Current directory: $(pwd)" +echo "~~~~ (Re)creating virtual environment 'tfq-build-venv'" +echo + +# Ensure pyenv is activated. +eval "$(pyenv init -)" + +# Deactivate any pyenv we might be inside right now. +pyenv deactivate >& /dev/null || true + +# Ensure we have the requested version of Python. +pyenv install -s ${py_version} + +# (Re)create a pyenv virtual env with an expressive name. +pyenv virtualenv-delete -f tfq-build-venv || true +pyenv virtualenv -v ${py_version} tfq-build-venv +pyenv activate tfq-build-venv + +pip install --upgrade pip + +# ~~~~~~~~ Build & clean the wheel ~~~~~~~~ + +echo +echo "~~~~ Starting build of TFQ ${version}." +echo + +./release/build_distribution.sh -p "${py_version}" + +# The wheel that was just created will be the most recent file. +tmp_wheel_name="$(/bin/ls -t /tmp/tensorflow_quantum | head -n 1)" +tmp_wheel="/tmp/tensorflow_quantum/${tmp_wheel_name}" + +echo +echo "~~~~ Cleaning wheel ${tmp_wheel}" +echo +./release/clean_distribution.sh "${tmp_wheel}" + +# ~~~~~~~~ Check the result ~~~~~~~~ + +echo +echo "~~~~ Inspecting the wheel." + +pip install -qq wheel-inspect + +final_wheel="wheelhouse/$(/bin/ls -t ./wheelhouse | head -n 1)" + +echo +echo "Requires_python value in wheel: " +wheel2json "${final_wheel}" | jq -r '.dist_info.metadata."requires_python"' + +echo +echo "Tags in wheel:" +wheel2json "${final_wheel}" | jq -r '.dist_info.wheel.tag[]' + +echo +echo "~~~~ All done." +echo "${final_wheel}" From 36b39e3d5a5eb02b3c6a19d80f4bf8efda22173a Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 02:54:13 +0000 Subject: [PATCH 12/23] Use better Bash syntax --- release/build_distribution.sh | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/release/build_distribution.sh b/release/build_distribution.sh index 02d180846..c8115c2ab 100755 --- a/release/build_distribution.sh +++ b/release/build_distribution.sh @@ -1,11 +1,11 @@ #!/bin/bash -# Copyright 2025 Google LLC +# Copyright 2025 The TensorFlow Quantum Authors # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # -# https://www.apache.org/licenses/LICENSE-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, @@ -36,11 +36,11 @@ # # 4. Exit. -set -eu +set -eu -o pipefail # Find the top of the local TFQ git tree. Do it early in case this fails. -thisdir=$(CDPATH="" cd -- "$(dirname -- "$0")" && pwd -P) -repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2>/dev/null || \ +thisdir=$(CDPATH="" cd -- "$(dirname -- "${0}")" && pwd -P) +repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2> /dev/null || \ echo "${thisdir}/..") # Default values for variables that can be changed via command line flags. @@ -113,7 +113,7 @@ printf "\n:::::::: Starting Bazel build ::::::::\n\n" bazel build ${build_flags} release:build_pip_package printf "\n:::::::: Creating Python wheel ::::::::\n\n" bazel-bin/release/build_pip_package /build_output/ -if [ "${cleanup}" == "true" ]; then +if [[ "${cleanup}" == "true" ]]; then printf "\n:::::::: Cleaning up ::::::::\n\n" bazel clean --async fi @@ -138,7 +138,7 @@ set -- docker run -it --rm --network host \ "${docker_image}" \ /tmp/build_script.sh -if [ "${dry_run}" == "true" ]; then +if [[ "${dry_run}" == "true" ]]; then # Loop through the positional parameters and simply print them. printf "(Dry run) " printf '%s ' "$@" From 0fb88fb0fe64f1abdfea17e1889cc1e0a15bccea Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 02:59:59 +0000 Subject: [PATCH 13/23] Fix license headers --- release/build_distribution.sh | 2 +- release/build_release.sh | 2 +- release/clean_distribution.sh | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/release/build_distribution.sh b/release/build_distribution.sh index c8115c2ab..ef3c2fa2c 100755 --- a/release/build_distribution.sh +++ b/release/build_distribution.sh @@ -5,7 +5,7 @@ # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # -# https://www.apache.org/licenses/LICENSE-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, diff --git a/release/build_release.sh b/release/build_release.sh index 76a16b9a8..625bf63d2 100755 --- a/release/build_release.sh +++ b/release/build_release.sh @@ -5,7 +5,7 @@ # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # -# https://www.apache.org/licenses/LICENSE-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, diff --git a/release/clean_distribution.sh b/release/clean_distribution.sh index 89a202d15..8aef3ce4b 100755 --- a/release/clean_distribution.sh +++ b/release/clean_distribution.sh @@ -1,11 +1,11 @@ #!/bin/bash -# Copyright 2025 Google LLC +# Copyright 2025 The TensorFlow Quantum Authors # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # -# https://www.apache.org/licenses/LICENSE-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, From 4a155ad3bb3885aa406f7a874e0175f3ddbd795d Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 03:07:11 +0000 Subject: [PATCH 14/23] Remove calls to check-wheel-contents That doesn't really belong here. Let's put it elsewhere. --- release/build_distribution.sh | 7 ------- 1 file changed, 7 deletions(-) diff --git a/release/build_distribution.sh b/release/build_distribution.sh index ef3c2fa2c..a49bb8e5e 100755 --- a/release/build_distribution.sh +++ b/release/build_distribution.sh @@ -142,17 +142,10 @@ if [[ "${dry_run}" == "true" ]]; then # Loop through the positional parameters and simply print them. printf "(Dry run) " printf '%s ' "$@" - echo - echo '(Dry run) check-wheel-contents /tmp/tensorflow_quantum/*.whl' else echo "Spinning up a Docker container with ${docker_image} …" "$@" - # Do some basic checks on the wheel file. - echo "Doing basic sanity checks on the wheel …" - pip install -qq check-wheel-contents - check-wheel-contents /tmp/tensorflow_quantum/*.whl - echo "Done. Look for wheel in /tmp/tensorflow_quantum/." ls -l /tmp/tensorflow_quantum/ fi From e1688a18afd4f12c493b6a9fe8942204864b219b Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 03:08:02 +0000 Subject: [PATCH 15/23] Add a call to check-wheel-contents --- release/build_release.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/release/build_release.sh b/release/build_release.sh index 625bf63d2..609ee8095 100755 --- a/release/build_release.sh +++ b/release/build_release.sh @@ -128,10 +128,14 @@ echo echo echo "~~~~ Inspecting the wheel." -pip install -qq wheel-inspect +pip install -qq wheel-inspect check-wheel-contents final_wheel="wheelhouse/$(/bin/ls -t ./wheelhouse | head -n 1)" +echo "Check wheel contents:" +echo +check-wheel-contents "${final_wheel}" + echo echo "Requires_python value in wheel: " wheel2json "${final_wheel}" | jq -r '.dist_info.metadata."requires_python"' From b5e1fe83a49a561a5ba79cd33cf5e695da6770fa Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 03:08:59 +0000 Subject: [PATCH 16/23] Fix linter warnings --- release/build_release.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/release/build_release.sh b/release/build_release.sh index 609ee8095..fbbfa2de6 100755 --- a/release/build_release.sh +++ b/release/build_release.sh @@ -41,7 +41,7 @@ function quit() { function have() { unset -v have - type $1 &> /dev/null + type "$1" &> /dev/null } # ~~~~~~~~ Sanity checks ~~~~~~~~ @@ -97,11 +97,11 @@ eval "$(pyenv init -)" pyenv deactivate >& /dev/null || true # Ensure we have the requested version of Python. -pyenv install -s ${py_version} +pyenv install -s "${py_version}" # (Re)create a pyenv virtual env with an expressive name. pyenv virtualenv-delete -f tfq-build-venv || true -pyenv virtualenv -v ${py_version} tfq-build-venv +pyenv virtualenv -v "${py_version}" tfq-build-venv pyenv activate tfq-build-venv pip install --upgrade pip From 3d57a53a0c4de299f3c0a482eacbbf87457ecd57 Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 03:16:58 +0000 Subject: [PATCH 17/23] Run `python -m pip`, not `pip`, to install tools Just to play it safe, make sure we get the copy pip that we mean to get, in case there's some discrepancy inside a virtual environment. --- release/build_pip_package.sh | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/release/build_pip_package.sh b/release/build_pip_package.sh index efd7a734d..c4841f9e5 100755 --- a/release/build_pip_package.sh +++ b/release/build_pip_package.sh @@ -16,26 +16,22 @@ set -e -# Ensure packaging tools are present in this interpreter, and don't complain -# if running as root (which happens inside Docker containers). -pip install -qq setuptools wheel build --root-user-action ignore - # Pick the Python that TFQ/TensorFlow used during configure/build. -# Order: explicit env -> python3 (>= 3.10) +# Order: explicit env -> python3 (>= 3.9) PY="${PYTHON_BIN_PATH:-}" if [[ -z "${PY}" ]]; then if ! command -v python3 >/dev/null 2>&1; then - echo "ERROR: python3 not found. Set PYTHON_BIN_PATH to a Python 3.10+ interpreter." >&2 + echo "ERROR: python3 not found. Set PYTHON_BIN_PATH to a Python 3.9+ interpreter." >&2 exit 2 fi - # Require Python >= 3.10 for TFQ. + # Require Python >= 3.9 for TFQ. if ! python3 - <<'PY' import sys -sys.exit(0 if sys.version_info[:2] >= (3, 10) else 1) +sys.exit(0 if sys.version_info[:2] >= (3, 9) else 1) PY then - echo "ERROR: Python 3.10+ required for TensorFlow Quantum; found $(python3 -V 2>&1)." >&2 + echo "ERROR: Python 3.9+ required for TensorFlow Quantum; found $(python3 -V 2>&1)." >&2 exit 2 fi @@ -44,7 +40,7 @@ fi echo "Using Python: ${PY}" # Ensure packaging tools are present in THIS interpreter. -pip install -qq setuptools wheel build --root-user-action ignore +"${PY}" -m pip install -qq setuptools wheel build --root-user-action ignore EXPORT_DIR="bazel-bin/release/build_pip_package.runfiles/__main__" @@ -76,7 +72,7 @@ main() { "${PY}" -m build -v --wheel ${EXTRA_FLAGS} > /dev/null cp dist/*.whl "${DEST}" popd - rm -rf ${TMPDIR} + rm -rf "${TMPDIR}" echo "$(date) : === Done." } From d89051187544e3b2a4cd5329da33734f6cee29a2 Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 04:07:35 +0000 Subject: [PATCH 18/23] Do some cleanup of build_release.sh Changes: * Add a usage string. * Reduce the length of the script a little bit. * Fix a printf syntax error. * Fix some non-Bash-isms. --- release/build_release.sh | 47 +++++++++++++++++++--------------------- 1 file changed, 22 insertions(+), 25 deletions(-) diff --git a/release/build_release.sh b/release/build_release.sh index fbbfa2de6..7b2768e8f 100755 --- a/release/build_release.sh +++ b/release/build_release.sh @@ -32,29 +32,31 @@ set -eu -o pipefail # directory. To avoid this, it's best to make a copy of your TFQ git repository # (or git-clone a fresh copy from GitHub) and clean it before proceeding. -# ~~~~~~~~ Helpers ~~~~~~~~ +usage="Usage: ${0} PYTHON_VERSION [BUILD_NUMBER] +Build a release for TFQ. This runs scripts to build and clean a distribution +for Python version PYTHON_VERSION, which must be given as a full x.y.z version +string. Optionally accepts a build number as a second argument." function quit() { - echo "ERROR: $*." >&2 + printf 'Error: %b\n' "$*" >&2 exit 1 } -function have() { - unset -v have - type "$1" &> /dev/null -} - -# ~~~~~~~~ Sanity checks ~~~~~~~~ - # Go to the top of the local TFQ git tree. Do it early in case this fails. -thisdir=$(CDPATH="" cd -- "$(dirname -- "$0")" && pwd -P) +thisdir=$(CDPATH="" cd -- "$(dirname -- "${0}")" && pwd -P) repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2>/dev/null || \ - quit "this script must be run from inside the TFQ git tree") + quit "This script must be run from inside the TFQ git tree.") cd "${repo_dir}" +# ~~~~~~~~ Parse arguments and do basic sanity checks ~~~~~~~~ + +if (( $# < 1 )); then + quit "Must provide at least one argument, the Python version.\n\n${usage}" +fi + py_version="${1}" if ! [[ "${py_version}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then - quit "the first argument must be a full Python version number" + quit "The first argument must be a Python version number in the form x.y.z." fi build_number="" @@ -66,12 +68,12 @@ setup_file="${repo_dir}/release/setup.py" if [[ -r "${setup_file}" ]]; then tfq_version=$(grep -m 1 CUR_VERSION "${setup_file}" | cut -f2 -d'"') else - quit "cannot read ${setup_file}" + quit "Cannot read ${setup_file}" fi for program in python3 pip pyenv jq; do - if ! have $program; then - quit "cannot run $program -- maybe it is not installed?" + if ! command -v "${program}" > /dev/null 2>&1; then + quit "Cannot run ${program} -- maybe it is not installed?" fi done @@ -88,7 +90,6 @@ version=${tfq_version}${build_number} echo "~~~~ Starting ${0} for TFQ release ${version}" echo "~~~~ Current directory: $(pwd)" echo "~~~~ (Re)creating virtual environment 'tfq-build-venv'" -echo # Ensure pyenv is activated. eval "$(pyenv init -)" @@ -105,13 +106,12 @@ pyenv virtualenv -v "${py_version}" tfq-build-venv pyenv activate tfq-build-venv pip install --upgrade pip +pip install wheel-inspect check-wheel-contents # ~~~~~~~~ Build & clean the wheel ~~~~~~~~ echo echo "~~~~ Starting build of TFQ ${version}." -echo - ./release/build_distribution.sh -p "${py_version}" # The wheel that was just created will be the most recent file. @@ -120,24 +120,21 @@ tmp_wheel="/tmp/tensorflow_quantum/${tmp_wheel_name}" echo echo "~~~~ Cleaning wheel ${tmp_wheel}" -echo ./release/clean_distribution.sh "${tmp_wheel}" # ~~~~~~~~ Check the result ~~~~~~~~ +final_wheel="wheelhouse/$(/bin/ls -t ./wheelhouse | head -n 1)" + echo echo "~~~~ Inspecting the wheel." -pip install -qq wheel-inspect check-wheel-contents - -final_wheel="wheelhouse/$(/bin/ls -t ./wheelhouse | head -n 1)" - -echo "Check wheel contents:" echo +echo "Check wheel contents:" check-wheel-contents "${final_wheel}" echo -echo "Requires_python value in wheel: " +echo "Requires_python value in wheel:" wheel2json "${final_wheel}" | jq -r '.dist_info.metadata."requires_python"' echo From 14e25cdbb3118424e89bf328c6fa31ad874a1781 Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 05:10:36 +0000 Subject: [PATCH 19/23] Fix some trivial nits in the echo statements Just for consistency. --- release/build_release.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/release/build_release.sh b/release/build_release.sh index 7b2768e8f..b4c541a64 100755 --- a/release/build_release.sh +++ b/release/build_release.sh @@ -111,7 +111,7 @@ pip install wheel-inspect check-wheel-contents # ~~~~~~~~ Build & clean the wheel ~~~~~~~~ echo -echo "~~~~ Starting build of TFQ ${version}." +echo "~~~~ Starting build of TFQ ${version}" ./release/build_distribution.sh -p "${py_version}" # The wheel that was just created will be the most recent file. @@ -127,7 +127,7 @@ echo "~~~~ Cleaning wheel ${tmp_wheel}" final_wheel="wheelhouse/$(/bin/ls -t ./wheelhouse | head -n 1)" echo -echo "~~~~ Inspecting the wheel." +echo "~~~~ Inspecting the wheel" echo echo "Check wheel contents:" From 5c0e054826a23ed74adbfb3a0707faf2d047f925 Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 23:05:00 +0000 Subject: [PATCH 20/23] Update README to describe use of new build_release.sh --- release/README.md | 119 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 95 insertions(+), 24 deletions(-) diff --git a/release/README.md b/release/README.md index 50c1b3670..98051f830 100644 --- a/release/README.md +++ b/release/README.md @@ -2,7 +2,7 @@ This directory contains configurations and scripts that the TensorFlow Quantum maintainers use to create Python packages for software releases. The process of -making a TFQ release is complex, and has not been fully automated. The scripts +making a TFQ release is complex and has not been fully automated. The scripts in this directory help automate some steps and are a way of capturing the process more precisely, but there are still manual steps involved. @@ -32,7 +32,7 @@ computer TFQ is built on, including: * The TensorFlow build container used * The Crosstool configuration used * Whether CUDA is being used, and its version -* The dependency requirements implied by Cirq, NumPy, Protobuf, and +* The dependency requirements implied by Cirq, TF-Keras, NumPy, Protobuf, and other Python packages ## Procedure @@ -44,26 +44,104 @@ compatibility of the wheel so that it can run on a wider range of Linux distributions, even if those distributions have different versions of system libraries. -The steps are: +### Preliminary steps -1. Git clone the TensorFlow Quantum repo to a directory on your computer. +1. Make sure you have `pyenv`, `pip`, and `jq` installed on your system. -1. `cd` into the local clone directory. +2. (Optional) Preinstall Python versions 3.9, 3.10, 3.11, and 3.12 into `pyenv` + so that `build_release.sh` can create virtual environments with those Python + versions without having to install the requested version(s) itself. -1. Create a Python virtual environment. +3. Git clone the TensorFlow Quantum repo to a directory on your computer. -1. Run `pip install -r requirements.txt` +4. `cd` into this local clone directory. -1. Verify the major.minor version of Python you are using. The rest of these - instructions use 3.11 as an example. +### Build the release -1. Run `./release/build_distribution.sh -p 3.11` +1. Run `./release/build_release.sh X.Y.Z`, where _X.Y.Z_ is a Python version + for which you want to build a TFQ release. -1. If the above succeeds, it will leave the wheel in `/tmp/tensorflow_quantum/` +2. If the previous step completes successfully, proceed to the next section + below and test the wheel. + +3. Repeat steps 1–2 for other Python versions. + +### Testing the release + +Testing is currently not automated to the degree that building a release is. +Assuming that one of the procedures above was used to create one or more wheels +for a TFQ release, here are the steps for testing each one. + +1. First, perform a quick local test. + + 1. `cd` out of the TFQ source directory. This is a critical step, because + importing TFQ into a Python interpreter when the current directory is + the TFQ source tree will result in baffling errors (usually something + about `pauli_sum_pb2` not found). + + 1. Create a fresh Python virtual environment. + + 1. Run `pip install /path/to/whl/file`, where `/path/to/whl/file` is the + path to the wheel file corresponding to the version of Python you are + running. + + 1. Run the following snippet. If this results in an error, stop and debug + the problem. + + ```python + import tensorflow_quantum as tfq + `print(tfq.__version__) + ``` + + 1. If the previous snippet ran without error, next try running some + more elaborate TFQ example code. + +2. Second, test in Colab. + + 1. Go to a remotely hosted Colab and make a copy of the Hello Many Worlds + [tutorial notebook]( + https://www.tensorflow.org/quantum/tutorials/hello_many_worlds). + + 1. Using the Colab file explorer, upload a TFQ wheel you created matching + the version of Python running in Colab. (At the time of this writing, + this is Python 3.12.) + + 1. When the upload finishes, right-click on the file name in the Colab file + explorer and copy the path to the file in Colab. + + 1. Find the notebook cell that contains the `!pip install` command for + TensorFlow Quantum. **Replace that command** with the following, pasting + in the path that you copied in the previous step: + + ```python + !pip install /here/paste/the/path/to/the/wheel/file + ``` + + 1. Run the notebook step by step. If Colab asks you to restart the session, + do so, and after it finishes restarting, continue with the remaining + cells in the notebook. + + 1. If the notebook executes all the way through without error, + congratulations! If something fails, proceed to debug the problem. + +## Alternative procedure + +As mentioned above, `build_release.sh` relies on other scripts to do the main +work. Those steps can be run manually, and sometimes that's useful to do that +when debugging problems. The steps in this more manual approach are: + +1. Create a Python virtual environment. (The maintainers currently use `pyenv` + but Python's built-in `venv` should work too.) + +2. Run `pip install -r requirements.txt` + +3. Run `./release/build_distribution.sh` + +4. If the above succeeds, it will leave the wheel in `/tmp/tensorflow_quantum/` on your system. Take note of the name of the wheel file that `build_distribution.sh` prints when it finishes. -1. Run `./release/clean_distribution.sh /tmp/tensorflow_quantum/WHEEL_FILE`, +5. Run `./release/clean_distribution.sh /tmp/tensorflow_quantum/WHEEL_FILE`, where `WHEEL_FILE` is the file noted in the previous step. If this works, it will create a new wheel file in `../wheelhouse`. If an error occurs, it will hopefully report the problem. If the error is a platform tag mismatch, run @@ -74,19 +152,12 @@ The steps are: the `build_distribution.sh` script to experiment with different values for the Crosstool and/or the Docker images used. -1. Test the wheel. Go to a remotely hosted Colab (or any other Linux platform - that is distinctly difference from yours) upload this new generated wheel - file to local storage in the Colab, and test if it works. (E.g., try to run - `https://www.tensorflow.org/quantum/tutorials/hello_many_worlds` with the - new TensorFlow and TFQ wheels to verify things are working smoothly). - -1. Repeat the `build_distribution.sh` and `clean_distribution.sh` steps for - different versions of Python. +6. If the previous step succeeded, go to the next section (Testing the + release files) and do preliminary testing on the wheel. -To make TFQ wheels that will work on other systems, it's essential that the -platform tags of the TFQ wheel must match the tags of the current TensorFlow -version you are targeting. (Visit `https://pypi.org/project/tensorflow//#files` to determine what the tags are). +7. If the tests succeed, repeat the `build_distribution.sh` and + `clean_distribution.sh` steps for different versions of Python. If the + preliminary tests fail, proceed to debugging the reason. ## More information From 2fdb8d37d576e1a52e85e467b276160b918c911a Mon Sep 17 00:00:00 2001 From: mhucka Date: Wed, 24 Dec 2025 23:36:29 +0000 Subject: [PATCH 21/23] Shorten some parts & update test for installed programs Even though this script itself does not need docker, it's better for the user if it checks before proceeding so that it doesn't go through creating a pyenv and doing a pip install only to fail when it runs build_distribution.sh. --- release/build_release.sh | 52 ++++++++++++++++------------------------ 1 file changed, 21 insertions(+), 31 deletions(-) diff --git a/release/build_release.sh b/release/build_release.sh index b4c541a64..2309490c6 100755 --- a/release/build_release.sh +++ b/release/build_release.sh @@ -14,8 +14,6 @@ # limitations under the License. # ============================================================================= -set -eu -o pipefail - # Summary: do all the steps to generate a wheel for a TFQ release. # # This sets up a clean pyenv virtualenv for a given Python version, then runs @@ -23,19 +21,19 @@ set -eu -o pipefail # by printing some info about the wheel. The wheel is left in ./wheelhouse/. # The TFQ release number is extracted from setup.py. # -# Note: this uses build_distribution.sh, which builds the TFQ pip package inside -# a Docker container. The TFQ git directory where this script is found is mapped -# directly inside the running Docker environment. The approach makes it easy to -# iterate on changes to TFQ files in the current directory, and avoids a lot of -# frustrating "which copy of such-and-such file did it use?" questions. However, -# it also brings a risk of unexpected impact of left-overs in the current -# directory. To avoid this, it's best to make a copy of your TFQ git repository -# (or git-clone a fresh copy from GitHub) and clean it before proceeding. +# This script uses build_distribution.sh. The latter builds the TFQ pip package +# inside a TensorFlow Docker container, and maps your local TFQ source tree +# directly inside the running Docker environment at mount point /tfq. The +# present does not assume /tfq, but does assume that relative paths work. + +set -eu -o pipefail usage="Usage: ${0} PYTHON_VERSION [BUILD_NUMBER] -Build a release for TFQ. This runs scripts to build and clean a distribution -for Python version PYTHON_VERSION, which must be given as a full x.y.z version -string. Optionally accepts a build number as a second argument." +Build a release for TFQ. + +This runs scripts to build and clean a distribution for Python version +PYTHON_VERSION, which must be given as a full x.y.z version string. +Optionally accepts a build number as a second argument." function quit() { printf 'Error: %b\n' "$*" >&2 @@ -50,9 +48,7 @@ cd "${repo_dir}" # ~~~~~~~~ Parse arguments and do basic sanity checks ~~~~~~~~ -if (( $# < 1 )); then - quit "Must provide at least one argument, the Python version.\n\n${usage}" -fi +(( $# > 0 )) || quit "Must provide at least one argument.\n\n${usage}" py_version="${1}" if ! [[ "${py_version}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then @@ -60,32 +56,26 @@ if ! [[ "${py_version}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then fi build_number="" -if (( $# > 1 )); then - build_number="-${2}" -fi +(( $# > 1 )) && build_number="-${2}" setup_file="${repo_dir}/release/setup.py" -if [[ -r "${setup_file}" ]]; then - tfq_version=$(grep -m 1 CUR_VERSION "${setup_file}" | cut -f2 -d'"') -else - quit "Cannot read ${setup_file}" -fi +[[ -r "${setup_file}" ]] || quit "Cannot read ${setup_file}" +tfq_version=$(grep -m 1 CUR_VERSION "${setup_file}" | cut -f2 -d'"') + +version=${tfq_version}${build_number} -for program in python3 pip pyenv jq; do +for program in docker pyenv jq; do if ! command -v "${program}" > /dev/null 2>&1; then quit "Cannot run ${program} -- maybe it is not installed?" fi done -version=${tfq_version}${build_number} - # ~~~~~~~~ Set up a new virtual environment ~~~~~~~~ # Since the build is done inside a Docker container, it is not really necessary # to create a virtual Python environment for that part of the process. However, -# we do run some Python commands before and after, and we want those to be done -# in an environment we control with the same Python version being targeted for -# the build. It provides additional isolation. +# we run some Python commands before and after, and want those to be done in an +# environment with the same Python version being targeted for the build. echo "~~~~ Starting ${0} for TFQ release ${version}" echo "~~~~ Current directory: $(pwd)" @@ -100,7 +90,7 @@ pyenv deactivate >& /dev/null || true # Ensure we have the requested version of Python. pyenv install -s "${py_version}" -# (Re)create a pyenv virtual env with an expressive name. +# (Re)create a pyenv virtual env. pyenv virtualenv-delete -f tfq-build-venv || true pyenv virtualenv -v "${py_version}" tfq-build-venv pyenv activate tfq-build-venv From 7361bd6db3804f281949103b912601d2c0d1fd0a Mon Sep 17 00:00:00 2001 From: mhucka Date: Fri, 26 Dec 2025 01:16:35 +0000 Subject: [PATCH 22/23] More cleanup Add a quit() function to all the scripts, plus go back to using full if-then statements for better maintainability. --- release/build_distribution.sh | 13 +++++++---- release/build_release.sh | 41 ++++++++++++++++++++--------------- release/clean_distribution.sh | 17 +++++++++------ 3 files changed, 43 insertions(+), 28 deletions(-) diff --git a/release/build_distribution.sh b/release/build_distribution.sh index a49bb8e5e..ef505a03d 100755 --- a/release/build_distribution.sh +++ b/release/build_distribution.sh @@ -38,10 +38,15 @@ set -eu -o pipefail +function quit() { + printf 'Error: %b\n' "$*" >&2 + exit 1 +} + # Find the top of the local TFQ git tree. Do it early in case this fails. -thisdir=$(CDPATH="" cd -- "$(dirname -- "${0}")" && pwd -P) -repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2> /dev/null || \ - echo "${thisdir}/..") +thisdir=$(dirname "${BASH_SOURCE[0]:?}") +repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2> /dev/null) || \ + quit "This script must be run from inside the TFQ git tree." # Default values for variables that can be changed via command line flags. tf_version="2.16" @@ -71,7 +76,7 @@ while getopts "c:ehnp:t:" opt; do n) dry_run="true" ;; p) py_version=$(echo "${OPTARG}" | cut -d. -f1,2) ;; t) tf_version="${OPTARG}" ;; - *) echo "${usage}" >&2; exit 1 ;; + *) quit "${usage}" ;; esac done shift $((OPTIND -1)) diff --git a/release/build_release.sh b/release/build_release.sh index 2309490c6..2a917337f 100755 --- a/release/build_release.sh +++ b/release/build_release.sh @@ -28,27 +28,29 @@ set -eu -o pipefail -usage="Usage: ${0} PYTHON_VERSION [BUILD_NUMBER] -Build a release for TFQ. - -This runs scripts to build and clean a distribution for Python version -PYTHON_VERSION, which must be given as a full x.y.z version string. -Optionally accepts a build number as a second argument." - function quit() { printf 'Error: %b\n' "$*" >&2 exit 1 } # Go to the top of the local TFQ git tree. Do it early in case this fails. -thisdir=$(CDPATH="" cd -- "$(dirname -- "${0}")" && pwd -P) -repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2>/dev/null || \ - quit "This script must be run from inside the TFQ git tree.") +thisdir=$(dirname "${BASH_SOURCE[0]:?}") +repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2> /dev/null) || \ + quit "This script must be run from inside the TFQ git tree." cd "${repo_dir}" +usage="Usage: ${0} PYTHON_VERSION [BUILD_NUMBER] +Build a release for TFQ. + +This runs scripts to build and clean a distribution for Python version +PYTHON_VERSION, which must be given as a full x.y.z version string. +Optionally accepts a build number as a second argument." + # ~~~~~~~~ Parse arguments and do basic sanity checks ~~~~~~~~ -(( $# > 0 )) || quit "Must provide at least one argument.\n\n${usage}" +if (( $# < 1 )); then + quit "Must provide at least one argument.\n\n${usage}" +fi py_version="${1}" if ! [[ "${py_version}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then @@ -56,14 +58,19 @@ if ! [[ "${py_version}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then fi build_number="" -(( $# > 1 )) && build_number="-${2}" +if (( $# > 1 )); then + build_number="-${2}" +fi setup_file="${repo_dir}/release/setup.py" -[[ -r "${setup_file}" ]] || quit "Cannot read ${setup_file}" -tfq_version=$(grep -m 1 CUR_VERSION "${setup_file}" | cut -f2 -d'"') +if ! [[ -r "${setup_file}" ]]; then + quit "Cannot read ${setup_file}" +fi +tfq_cur_version=$(grep -m 1 CUR_VERSION "${setup_file}" | cut -f2 -d'"') -version=${tfq_version}${build_number} +tfq_version=${tfq_cur_version}${build_number} +# Test these upfront to avoid failing mid-way through a long process. for program in docker pyenv jq; do if ! command -v "${program}" > /dev/null 2>&1; then quit "Cannot run ${program} -- maybe it is not installed?" @@ -77,7 +84,7 @@ done # we run some Python commands before and after, and want those to be done in an # environment with the same Python version being targeted for the build. -echo "~~~~ Starting ${0} for TFQ release ${version}" +echo "~~~~ Starting ${0} for TFQ release ${tfq_version}" echo "~~~~ Current directory: $(pwd)" echo "~~~~ (Re)creating virtual environment 'tfq-build-venv'" @@ -101,7 +108,7 @@ pip install wheel-inspect check-wheel-contents # ~~~~~~~~ Build & clean the wheel ~~~~~~~~ echo -echo "~~~~ Starting build of TFQ ${version}" +echo "~~~~ Starting build of TFQ ${tfq_version}" ./release/build_distribution.sh -p "${py_version}" # The wheel that was just created will be the most recent file. diff --git a/release/clean_distribution.sh b/release/clean_distribution.sh index 8aef3ce4b..38600af56 100755 --- a/release/clean_distribution.sh +++ b/release/clean_distribution.sh @@ -24,10 +24,15 @@ set -eu -o pipefail +function quit() { + printf 'Error: %b\n' "$*" >&2 + exit 1 +} + # Find the top of the local TFQ git tree. Do it early in case this fails. -thisdir=$(CDPATH="" cd -- "$(dirname -- "${0}")" && pwd -P) -repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2>/dev/null || \ - echo "${thisdir}/..") +thisdir=$(dirname "${BASH_SOURCE[0]:?}") +repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2> /dev/null) || \ + quit "This script must be run from inside the TFQ git tree." # Default values for variables that can be changed via command line flags. docker_image="quay.io/pypa/manylinux_2_34_x86_64" @@ -60,14 +65,12 @@ while getopts "hm:np:st:v" opt; do s) action="show" ;; t) platform="${OPTARG}" ;; v) verbose="--verbose" ;; - *) echo "${usage}" >&2; exit 1 ;; + *) quit "${usage}" ;; esac done shift $((OPTIND -1)) if (( $# < 1 )); then - echo "ERROR: need at least one argument argument." - echo "${usage}" >&2 - exit 1 + quit "Must provide at least one argument.\n\n${usage}" fi wheel_path="$(cd "$(dirname "${1}")" && pwd)/$(basename "${1}")" From 8657e30034bbe3805eacdf6bb71c43b41fb0ed86 Mon Sep 17 00:00:00 2001 From: mhucka Date: Fri, 26 Dec 2025 04:33:30 +0000 Subject: [PATCH 23/23] Update and make corrections to release/README.md --- release/README.md | 111 +++++++++++++++++++++++----------------------- 1 file changed, 56 insertions(+), 55 deletions(-) diff --git a/release/README.md b/release/README.md index 98051f830..e146fb788 100644 --- a/release/README.md +++ b/release/README.md @@ -21,8 +21,8 @@ themselves when they want to install TFQ, each release of TFQ must be pinned to a specific version of TensorFlow. As a consequence, TFQ releases will not work with any other version of TensorFlow than the one they are pinned to. -Python wheels for TFQ are produced by compiling them locally with a toolchain -that matches that used by the version of TensorFlow being targeted by a given +Python wheels for TFQ are produced by compiling them with a toolchain that +matches that used to build the version of TensorFlow being targeted by a given version of TFQ. A number of elements affect whether the whole process succeeds and the resulting wheel is portable to environments other than the specific computer TFQ is built on, including: @@ -54,7 +54,7 @@ libraries. 3. Git clone the TensorFlow Quantum repo to a directory on your computer. -4. `cd` into this local clone directory. +4. `cd` into your local clone directory in a Bash shell. ### Build the release @@ -64,47 +64,49 @@ libraries. 2. If the previous step completes successfully, proceed to the next section below and test the wheel. -3. Repeat steps 1–2 for other Python versions. +3. Repeat steps 1–2 above for other Python versions. ### Testing the release Testing is currently not automated to the degree that building a release is. -Assuming that one of the procedures above was used to create one or more wheels -for a TFQ release, here are the steps for testing each one. +The following is the current process. 1. First, perform a quick local test. - 1. `cd` out of the TFQ source directory. This is a critical step, because + 1. `cd` out of the TFQ source directory. This is essential, because importing TFQ into a Python interpreter when the current directory is the TFQ source tree will result in baffling errors (usually something about `pauli_sum_pb2` not found). 1. Create a fresh Python virtual environment. - 1. Run `pip install /path/to/whl/file`, where `/path/to/whl/file` is the - path to the wheel file corresponding to the version of Python you are - running. + 1. Run `pip install /path/to/wheel`, where `/path/to/wheel` is the path to + the wheel file corresponding to the version of Python you are running. - 1. Run the following snippet. If this results in an error, stop and debug - the problem. + 1. (Currently required because TFQ requires the legacy version of Keras.) + Set the environment variable `TF_USE_LEGACY_KERAS` to `1` before + loading any TensorFlow or TensorFlow Quantum code. You can set it in + your shell before starting Python; alternatively, you can execute the + following Python code before any `import tensorflow` or `import + tensorflow_quantum` statements: ```python - import tensorflow_quantum as tfq - `print(tfq.__version__) + import os + os.environ["TF_USE_LEGACY_KERAS"] = "1" ``` - 1. If the previous snippet ran without error, next try running some - more elaborate TFQ example code. + 1. Run some example TFQ code. If it runs without problems, go on to the + next step; if it fails, debug the error. 2. Second, test in Colab. - 1. Go to a remotely hosted Colab and make a copy of the Hello Many Worlds - [tutorial notebook]( + 1. Go to a remotely hosted Colab and make a copy of the [Hello Many Worlds + tutorial notebook]( https://www.tensorflow.org/quantum/tutorials/hello_many_worlds). 1. Using the Colab file explorer, upload a TFQ wheel you created matching the version of Python running in Colab. (At the time of this writing, - this is Python 3.12.) + Colab uses Python 3.12.) 1. When the upload finishes, right-click on the file name in the Colab file explorer and copy the path to the file in Colab. @@ -124,49 +126,48 @@ for a TFQ release, here are the steps for testing each one. 1. If the notebook executes all the way through without error, congratulations! If something fails, proceed to debug the problem. -## Alternative procedure +## More information -As mentioned above, `build_release.sh` relies on other scripts to do the main -work. Those steps can be run manually, and sometimes that's useful to do that -when debugging problems. The steps in this more manual approach are: +"TensorFlow SIG Build" is a community group dedicated to the TensorFlow build +process. This repository is a showcase of resources, guides, tools, and builds +contributed by the community, for the community. The following resources may be +useful when trying to figure out how to make this all work. -1. Create a Python virtual environment. (The maintainers currently use `pyenv` - but Python's built-in `venv` should work too.) +* [The "TF SIG Build Dockerfiles" README file]( + https://github.com/tensorflow/build/tree/ff4320fee2cf48568ebd2f476d7714438bfa0bee/tf_sig_build_dockerfiles#readme) -2. Run `pip install -r requirements.txt` +* Other info in the SIG Build repo: https://github.com/tensorflow/build -3. Run `./release/build_distribution.sh` +The script `build_release.sh` relies on other scripts to do the main work. +Those steps can be run manually, and sometimes it is useful to do so when +debugging problems. The steps are: -4. If the above succeeds, it will leave the wheel in `/tmp/tensorflow_quantum/` - on your system. Take note of the name of the wheel file that - `build_distribution.sh` prints when it finishes. +1. `cd` to the top level of your git clone of the TensorFlow Quantum repo. -5. Run `./release/clean_distribution.sh /tmp/tensorflow_quantum/WHEEL_FILE`, - where `WHEEL_FILE` is the file noted in the previous step. If this works, it - will create a new wheel file in `../wheelhouse`. If an error occurs, it will - hopefully report the problem. If the error is a platform tag mismatch, run - `./release/clean_distribution.sh -s /tmp/tensorflow_quantum/WHEEL_FILE`; - this will run auditwheel's `show` command on the wheel file to indicate what - version of `manylinux` this wheel can be made to run on if you use - `auditwheel` to repair it. With that information, you may be able to edit - the `build_distribution.sh` script to experiment with different values for - the Crosstool and/or the Docker images used. - -6. If the previous step succeeded, go to the next section (Testing the - release files) and do preliminary testing on the wheel. - -7. If the tests succeed, repeat the `build_distribution.sh` and - `clean_distribution.sh` steps for different versions of Python. If the - preliminary tests fail, proceed to debugging the reason. +1. Create a Python virtual environment for one of the supported versions of + Python. (The maintainers currently use `pyenv` but Python's built-in + `venv` should work too.) -## More information +1. Run `pip install -r requirements.txt` -"TensorFlow SIG Build" is a community group dedicated to the TensorFlow build -process. This repository is a showcase of resources, guides, tools, and builds -contributed by the community, for the community. The following resources may be -useful when trying to figure out how to make this all work. +1. Run `./release/build_distribution.sh` -* The "TF SIG Build Dockerfiles" document: - https://github.com/tensorflow/build/tree/ff4320fee2cf48568ebd2f476d7714438bfa0bee/tf_sig_build_dockerfiles#readme +1. If the above succeeds, it will leave the wheel in `/tmp/tensorflow_quantum/` + on your system. Take note of the name of the wheel file that + `build_distribution.sh` prints when it finishes. -* Other info in the SIG Build repo: https://github.com/tensorflow/build +1. Run `./release/clean_distribution.sh /tmp/tensorflow_quantum/WHEEL_FILE`, + where `WHEEL_FILE` is the file noted in the previous step. If this works, it + will create a new wheel file in a subdirectory named `wheelhouse`. If an + error occurs, it will hopefully report the problem. If the error is a Python + platform tag mismatch, run `./release/clean_distribution.sh -s + /tmp/tensorflow_quantum/WHEEL_FILE`; this will run `auditwheel show` on the + wheel file to indicate what version of `manylinux` this wheel can be made to + run on if you use `auditwheel` to repair it. With that information, you may + be able to edit the `build_distribution.sh` script to experiment with + different values for the Crosstool and/or the Docker images used. + +1. If the previous step succeeded, go to testing the release. + +1. If the tests succeed, repeat the `build_distribution.sh` and + `clean_distribution.sh` steps for different versions of Python.