This is an automated email from the ASF dual-hosted git repository.
tqchen pushed a commit to branch v0.25.0
in repository https://gitbox.apache.org/repos/asf/tvm.git
The following commit(s) were added to refs/heads/v0.25.0 by this push:
new b3e249b7d7 [CI] Build CUDA sidecar on plain manylinux, install CUDA
manually (#19882)
b3e249b7d7 is described below
commit b3e249b7d75f8f3bc7cbee48188d3c80ae323437
Author: Ruihang Lai <[email protected]>
AuthorDate: Wed Jun 24 16:42:57 2026 -0400
[CI] Build CUDA sidecar on plain manylinux, install CUDA manually (#19882)
#19754 (besides bumping cibuildwheel to 4.1.0) switched the CUDA runtime
sidecar build onto the preinstalled quay.io/manylinux_cuda image and
dropped the in-script CUDA toolkit install. On that image the built
libtvm_runtime_cuda.so silently lost its libcuda.so.1 dependency: the
libcuda.so stub is not where CMake's FindCUDA looks, so
CUDA_CUDA_LIBRARY resolved empty and ldd no longer lists libcuda,
breaking downstream runtime use.
This restores only the sidecar pieces -- the plain manylinux_2_28 image
and the curl/rpm/dnf CUDA toolkit install. The cibw 4.1.0 bump is kept
(confirmed working). The driver stub is back where FindCUDA finds it, so
the sidecar links against libcuda.so.1 again, as ldd confirms. This is a
hotfix; a follow-up can re-adopt the CUDA image with the stub resolved
explicitly plus a post-build linkage check.
---
.github/workflows/publish_wheel.yml | 9 ++++----
.../package/manylinux_build_libtvm_runtime_cuda.sh | 26 ++++++++++++++--------
2 files changed, 21 insertions(+), 14 deletions(-)
diff --git a/.github/workflows/publish_wheel.yml
b/.github/workflows/publish_wheel.yml
index a2fedda9e0..080ceadd2d 100644
--- a/.github/workflows/publish_wheel.yml
+++ b/.github/workflows/publish_wheel.yml
@@ -44,9 +44,8 @@ env:
jobs:
# Build the CUDA runtime sidecar once per arch and upload it as an artifact
that
- # build_wheels bundles. The Linux legs build inside the official
manylinux_2_28
- # CUDA image (CUDA toolkit preinstalled, see pypa/manylinux) -- the same
glibc
- # baseline cibuildwheel targets for the wheel -- so the sidecar stays
+ # build_wheels bundles. The Linux legs build inside a manylinux_2_28 image
-- the
+ # same glibc baseline cibuildwheel targets for the wheel -- so the sidecar
stays
# ABI-compatible with the wheel's libtvm_runtime.so regardless of the exact
tag.
build_cuda_runtime:
name: ${{ matrix.name }}
@@ -58,13 +57,13 @@ jobs:
include:
- name: "CUDA runtime sidecar (Linux x86_64, manylinux_2_28)"
os: ubuntu-latest
- container:
quay.io/manylinux_cuda/manylinux_2_28_x86_64_cuda13_1:latest
+ container: quay.io/pypa/manylinux_2_28_x86_64:latest
arch: x86_64
script: ci/scripts/package/manylinux_build_libtvm_runtime_cuda.sh
lib: build-wheel-cuda/lib/libtvm_runtime_cuda.so
- name: "CUDA runtime sidecar (Linux aarch64, manylinux_2_28)"
os: ubuntu-24.04-arm
- container:
quay.io/manylinux_cuda/manylinux_2_28_aarch64_cuda13_1:latest
+ container: quay.io/pypa/manylinux_2_28_aarch64:latest
arch: aarch64
script: ci/scripts/package/manylinux_build_libtvm_runtime_cuda.sh
lib: build-wheel-cuda/lib/libtvm_runtime_cuda.so
diff --git a/ci/scripts/package/manylinux_build_libtvm_runtime_cuda.sh
b/ci/scripts/package/manylinux_build_libtvm_runtime_cuda.sh
index ea9e563046..3902ca2f82 100755
--- a/ci/scripts/package/manylinux_build_libtvm_runtime_cuda.sh
+++ b/ci/scripts/package/manylinux_build_libtvm_runtime_cuda.sh
@@ -16,11 +16,9 @@
# specific language governing permissions and limitations
# under the License.
#
-# Build libtvm_runtime_cuda.so inside a manylinux CUDA container, run by the
-# build_cuda_runtime CI job. The official quay.io/manylinux_cuda images ship
-# the CUDA toolkit preinstalled under /usr/local/cuda, so no toolkit install
-# is needed here. Builds the sidecar into build-wheel-cuda/lib/ for the wheel
-# build to bundle.
+# Build libtvm_runtime_cuda.so inside a manylinux container, run by the
+# build_cuda_runtime CI job. Installs the pinned CUDA toolkit and builds the
+# sidecar into build-wheel-cuda/lib/ for the wheel build to bundle.
#
# Usage: manylinux_build_libtvm_runtime_cuda.sh
set -euxo pipefail
@@ -29,11 +27,21 @@ repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.."
&& pwd)"
build_dir="${repo_root}/build-wheel-cuda"
parallel="$(getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)"
+# Install the pinned CUDA toolkit into the manylinux_2_28 container. The RHEL8
+# local-repo RPM is compatible with manylinux_2_28 for both x86_64 and aarch64.
+arch="$(uname -m)"
+cuda_rpm="cuda-repo-rhel8-13-0-local-13.0.2_580.95.05-1.${arch}.rpm"
+curl -fsSLo "/tmp/${cuda_rpm}" \
+
"https://developer.download.nvidia.com/compute/cuda/13.0.2/local_installers/${cuda_rpm}";
+rpm -i "/tmp/${cuda_rpm}"
+dnf clean all
+dnf -y --disablerepo=epel install cuda-toolkit-13-0
+rm -f "/tmp/${cuda_rpm}"
+dnf clean all
+
# Build the CUDA runtime sidecar with CUDA on and LLVM off, so it does not need
-# the LLVM prefix; the main CPU wheel links LLVM statically. The manylinux CUDA
-# image already ships cmake and make, and the build uses the default Makefiles
-# generator (no Ninja), so no build tools are installed here. Put the bundled
-# CPython and CUDA toolchain on PATH for the CMake configure and nvcc.
+# the LLVM prefix; the main CPU wheel links LLVM statically. The manylinux
image
+# ships no cmake/ninja, so install the build tools here.
export PATH="/opt/python/cp310-cp310/bin:/usr/local/cuda/bin:${PATH}"
nvcc --version
