This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f88735f8f62 [chore](build) Add paimon-rust third-party build
integration (#68562)
f88735f8f62 is described below
commit f88735f8f62ce3375c4a6874419495622c070ca7
Author: Gabriel <[email protected]>
AuthorDate: Tue Sep 29 12:44:46 2026 +0800
[chore](build) Add paimon-rust third-party build integration (#68562)
### What problem does this PR solve?
Related PR: #66227
Prepare the third-party dependency for the Paimon Rust reader on master
independently of the reader implementation. This extracts all
dependency-related changes from #66227 after simplifying its Rust
archive installation.
- Download Apache paimon-rust v0.4.0-rc1 with a pinned checksum and
build `paimon-c` with HDFS support.
- Generate the C header with pinned cbindgen 0.29.4 and install the
header and archive using the existing third-party copy/strip
conventions.
- Use Rust 1.94.0 by default for both Lance and Paimon. When changing
the toolchain, rebuild both libraries with the same `RUSTUP_TOOLCHAIN`
to avoid conflicting Rust standard libraries.
- Register the dependency in CMake while preserving libm and glibc
compatibility link ordering.
The changes cover dependency registration, building, and installation
checks, plus build-script regression tests. Existing installations
missing either Rust archive or the generated Paimon header follow the
full dependency rebuild path so Lance and Paimon are rebuilt together.
Header generation inherits the selected Cargo executable and offline
mode. Reader code and its functional tests remain in #66227. The Rust
archive identity helper and its dedicated test file are not included.
### Validation
- Bash syntax and `git diff --check` passed.
- `python3 thirdparty/test/paimon-build-test.py`: all 10 tests passed on
both this branch and #66227. Eight tests fail before the fix. Coverage
includes old Linux/macOS installs, missing archives/header,
complete-install reuse, clean forwarding, custom Cargo, and offline
metadata settings. External compiler/header-generator commands are test
doubles; the tests execute the actual build-script logic.
- Six temporary mock-toolchain scenarios passed on this branch, covering
installation, strip behavior, offline flags, default toolchain
selection, explicitly newer toolchains, and rejection of older
toolchains. The same scenarios passed on #66227.
- CMake fragment checks passed with `GLIBC_COMPATIBILITY` enabled and
disabled; compatibility-library ordering was also checked.
- The official source archive MD5 matches the manifest.
- Full Rust/Doris compilation and final binary ABI validation were not
run locally; compilation is requested through CI.
### Release note
None
### Check List (For Author)
- Test: Build-script smoke checks and dependency/link-order validation
described above.
- Behavior changed: Yes. Third-party builds include paimon-rust and both
Rust libraries default to Rust 1.94.0.
- Does this need documentation: No. This prepares an internal build
dependency.
---
be/CMakeLists.txt | 3 +-
be/cmake/thirdparty.cmake | 7 ++
build.sh | 7 +-
thirdparty/build-thirdparty.sh | 161 ++++++++++++++++++++++++++--
thirdparty/test/paimon-build-test.py | 202 +++++++++++++++++++++++++++++++++++
thirdparty/vars.sh | 7 ++
6 files changed, 378 insertions(+), 9 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 7f8b577814d..02da8f5dc2e 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -860,7 +860,8 @@ if (GLIBC_COMPATIBILITY)
glibc-compatibility-explicit
glibc-compatibility
-lm
- lance_c)
+ lance_c
+ paimon_c)
endif()
if (NOT OS_MACOSX)
diff --git a/be/cmake/thirdparty.cmake b/be/cmake/thirdparty.cmake
index dcb7df7343b..723d1f8bded 100644
--- a/be/cmake/thirdparty.cmake
+++ b/be/cmake/thirdparty.cmake
@@ -120,6 +120,13 @@ add_thirdparty(lance_c LIB64 NOTADD)
if (NOT GLIBC_COMPATIBILITY)
list(APPEND COMMON_THIRDPARTY m lance_c)
endif()
+# libpaimon_c.a (built from paimon-rust) brings in Rust compiler_builtins that
+# would otherwise steal libm symbols. Place libm before it so the final linker
+# resolves C math symbols from the system library first.
+add_thirdparty(paimon_c LIB64 NOTADD)
+if (NOT GLIBC_COMPATIBILITY)
+ list(APPEND COMMON_THIRDPARTY m paimon_c)
+endif()
add_thirdparty(brpc LIB64)
add_thirdparty(rocksdb)
add_thirdparty(cyrus-sasl LIBNAME "lib/libsasl2.a")
diff --git a/build.sh b/build.sh
index 55c5bacb919..40ec83d99b2 100755
--- a/build.sh
+++ b/build.sh
@@ -486,7 +486,12 @@ else
LAST_THIRDPARTY_LIB='hadoop_hdfs_3_4/native/libhdfs.a'
fi
-if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ]]; then
+# Old installs can contain the sentinel but predate Paimon. Rebuild the whole
+# dependency set so Lance and Paimon also use the same selected Rust toolchain.
+if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ||
+ ! -f "${DORIS_THIRDPARTY}/installed/lib64/liblance_c.a" ||
+ ! -f "${DORIS_THIRDPARTY}/installed/lib64/libpaimon_c.a" ||
+ ! -f "${DORIS_THIRDPARTY}/installed/include/paimon_rust/paimon.h" ]];
then
echo "Thirdparty libraries need to be build ..."
# need remove all installed pkgs because some lib like lz4 will throw
error if its lib alreay exists
rm -rf "${DORIS_THIRDPARTY}/installed"
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index e878f5398a9..9ac1e01d02d 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -2372,6 +2372,24 @@ build_pugixml() {
}
# lance-c
+# Publish a complete archive with one rename. Copy/strip must not damage an
installed library
+# or expose a partial first installation if either command fails or the build
is interrupted.
+install_rust_archive() {
+ (
+ set -e
+ local archive="$1"
+ local destination="${TP_INSTALL_DIR}/lib64/${archive##*/}"
+ local staged
+ staged="$(mktemp "${destination}.tmp.XXXXXX")"
+ trap 'rm -f "${staged}"' EXIT
+ cp -p "${archive}" "${staged}"
+ if [[ "${STRIP_TP_LIB}" = "ON" && "${KERNEL}" != 'Darwin' ]]; then
+ strip --strip-debug --strip-unneeded "${staged}"
+ fi
+ mv -f "${staged}" "${destination}"
+ )
+}
+
build_lance_c() {
check_if_source_exist "${LANCE_C_SOURCE}"
cd "${TP_SOURCE_DIR}/${LANCE_C_SOURCE}"
@@ -2381,7 +2399,7 @@ build_lance_c() {
local cargo_bin="${LANCE_C_CARGO:-${CARGO:-cargo}}"
if ! command -v "${cargo_bin}" >/dev/null 2>&1; then
- echo "cargo is required to build lance-c. Install Rust 1.91.0 or set
LANCE_C_CARGO."
+ echo "cargo is required to build lance-c. Install Rust 1.94.0 or set
LANCE_C_CARGO."
exit 1
fi
if [[ ! -x "${TP_INSTALL_DIR}/bin/protoc" ]]; then
@@ -2389,14 +2407,20 @@ build_lance_c() {
exit 1
fi
- local required_rust_version="1.91.0"
+ local required_rust_version="1.94.0"
local cargo_env=(
"CARGO_BUILD_JOBS=${PARALLEL}"
"CARGO_TARGET_DIR=${PWD}/${BUILD_DIR}"
"PROTOC=${TP_INSTALL_DIR}/bin/protoc"
)
if command -v rustup >/dev/null 2>&1 && [[ -z "${RUSTUP_TOOLCHAIN}" ]];
then
- if ! rustup toolchain list | grep -Eq '^1\.91\.0([[:space:]-]|$)'; then
+ # The presence check must look for the toolchain the minimum actually
+ # requires, not a literal: with only an older toolchain installed the
+ # stale check would skip the install below and then force
+ # RUSTUP_TOOLCHAIN to a version rustup cannot dispatch, failing the
+ # build before any archive is produced.
+ local required_rust_regex="${required_rust_version//./\\.}"
+ if ! rustup toolchain list | grep -Eq
"^${required_rust_regex}([[:space:]-]|$)"; then
rustup toolchain install "${required_rust_version}" --profile
minimal
fi
cargo_env+=("RUSTUP_TOOLCHAIN=${required_rust_version}")
@@ -2407,7 +2431,7 @@ build_lance_c() {
echo "failed to get cargo version for lance-c. Install Rust
${required_rust_version} or set LANCE_C_CARGO/RUSTUP_TOOLCHAIN."
exit 1
fi
- # Rust 1.91.0 is the minimum supported version. Allow newer toolchains when
+ # Rust 1.94.0 is the minimum supported version. Allow newer toolchains when
# callers explicitly select one or rustup is unavailable on the system.
if ! awk -v required="${required_rust_version}" -v
actual="${cargo_version}" 'BEGIN {
split(required, r, ".");
@@ -2440,11 +2464,132 @@ build_lance_c() {
mkdir -p "${TP_INSTALL_DIR}/include" "${TP_INSTALL_DIR}/lib64"
rm -rf "${TP_INSTALL_DIR}/include/lance"
cp -av include/lance "${TP_INSTALL_DIR}/include/"
- cp -v "${BUILD_DIR}/release/liblance_c.a" "${TP_INSTALL_DIR}/lib64/"
+ install_rust_archive "${BUILD_DIR}/release/liblance_c.a"
+}
+
+build_paimon_rust() {
+ check_if_source_exist "${PAIMON_RUST_SOURCE}"
+ cd "${TP_SOURCE_DIR}/${PAIMON_RUST_SOURCE}"
+
+ rm -rf "${BUILD_DIR}"
+ mkdir -p "${BUILD_DIR}"
+
+ local cargo_bin="${PAIMON_RUST_CARGO:-${CARGO:-cargo}}"
+ if ! command -v "${cargo_bin}" >/dev/null 2>&1; then
+ echo "cargo is required to build paimon-rust. Install Rust 1.94.0 or
set PAIMON_RUST_CARGO."
+ exit 1
+ fi
+
+ local required_rust_version="1.94.0"
+ local cargo_env=(
+ "CARGO_BUILD_JOBS=${PARALLEL}"
+ "CARGO_TARGET_DIR=${PWD}/${BUILD_DIR}"
+ )
+ if command -v rustup >/dev/null 2>&1 && [[ -z "${RUSTUP_TOOLCHAIN}" ]];
then
+ # The presence check must look for the toolchain the minimum actually
+ # requires, not a literal: with only an older toolchain installed the
+ # stale check would skip the install below and then force
+ # RUSTUP_TOOLCHAIN to a version rustup cannot dispatch, failing the
+ # build before any archive is produced.
+ local required_rust_regex="${required_rust_version//./\\.}"
+ if ! rustup toolchain list | grep -Eq
"^${required_rust_regex}([[:space:]-]|$)"; then
+ rustup toolchain install "${required_rust_version}" --profile
minimal
+ fi
+ cargo_env+=("RUSTUP_TOOLCHAIN=${required_rust_version}")
+ fi
- if [[ "${STRIP_TP_LIB}" = "ON" && "${KERNEL}" != 'Darwin' ]]; then
- strip --strip-debug --strip-unneeded
"${TP_INSTALL_DIR}/lib64/liblance_c.a"
+ local cargo_version
+ if ! cargo_version="$(env "${cargo_env[@]}" "${cargo_bin}" --version | awk
'{print $2}')"; then
+ echo "failed to get cargo version for paimon-rust. Install Rust
${required_rust_version} or set PAIMON_RUST_CARGO/RUSTUP_TOOLCHAIN."
+ exit 1
fi
+ # Rust 1.94.0 is the minimum supported version. Allow newer toolchains when
+ # callers explicitly select one or rustup is unavailable on the system.
+ # NOTE: paimon_c and lance_c are both Rust staticlibs linked into the same
+ # BE binary; they must be built with the SAME rustc toolchain so the linker
+ # resolves both crates' std references against a single std copy. Mixing
+ # toolchains makes the precompiled std hashes differ and the linker pulls
+ # both std copies in, colliding on the unmangled `rust_eh_personality`
+ # (duplicate symbol). Rebuild both lance_c and paimon_rust whenever the
+ # toolchain changes, using the same RUSTUP_TOOLCHAIN for both builds.
+ if ! awk -v required="${required_rust_version}" -v
actual="${cargo_version}" 'BEGIN {
+ split(required, r, ".");
+ split(actual, a, ".");
+ for (i = 1; i <= 3; i++) {
+ if ((a[i] + 0) > (r[i] + 0)) {
+ exit 0;
+ }
+ if ((a[i] + 0) < (r[i] + 0)) {
+ exit 1;
+ }
+ }
+ exit 0;
+ }'; then
+ echo "paimon-rust requires Rust/Cargo ${required_rust_version} or
newer, but found ${cargo_version}."
+ echo "Install Rust ${required_rust_version} or set
PAIMON_RUST_CARGO/RUSTUP_TOOLCHAIN."
+ exit 1
+ fi
+
+ if [[ "${KERNEL}" != 'Darwin' ]]; then
+ cargo_env+=("CFLAGS=${CFLAGS:-} -std=gnu17")
+ fi
+
+ local cargo_args=(build --release --locked -p paimon-c --features
paimon/storage-hdfs)
+ # cbindgen invokes cargo metadata itself; command-line flags on the build
+ # and install calls do not propagate to that child process.
+ cargo_env+=("CARGO=${cargo_bin}")
+ if [[ "$(echo "${PAIMON_RUST_CARGO_OFFLINE}" | tr '[:lower:]'
'[:upper:]')" == "ON" ]]; then
+ cargo_args+=(--offline)
+ cargo_env+=("CARGO_NET_OFFLINE=true")
+ fi
+ env "${cargo_env[@]}" "${cargo_bin}" "${cargo_args[@]}"
+
+ # Generate the C header from the Rust extern "C" surface via cbindgen.
+ # cbindgen is a pinned, Doris-controlled input: an unpinned "current"
+ # release would regenerate paimon.h differently between builds. The
+ # pinned version installs under a Doris-controlled --root and its
+ # resolved absolute path is invoked directly (a custom CARGO_HOME does
+ # not necessarily put cargo-installed binaries on PATH). Offline builds
+ # pass --offline to the install command, exactly like the fetch/build
+ # handling above — cargo fails on a missing local crate cache instead
+ # of reaching for the network.
+ local cbindgen_version="0.29.4"
+ local cbindgen_bin="${PAIMON_RUST_CBINDGEN:-}"
+ if [[ -z "${cbindgen_bin}" ]]; then
+ local
cbindgen_root="${TP_SOURCE_DIR}/.doris-cbindgen-${cbindgen_version}"
+ cbindgen_bin="${cbindgen_root}/bin/cbindgen"
+ if [[ ! -x "${cbindgen_bin}" ]]; then
+ local cbindgen_install_args=(install cbindgen
+ --version "${cbindgen_version}" --locked --root
"${cbindgen_root}")
+ if [[ "$(echo "${PAIMON_RUST_CARGO_OFFLINE}" | tr '[:lower:]'
'[:upper:]')" == "ON" ]]; then
+ cbindgen_install_args+=(--offline)
+ fi
+ echo "cbindgen not found; installing pinned ${cbindgen_version}
via cargo install ..."
+ env "${cargo_env[@]}" "${cargo_bin}" "${cbindgen_install_args[@]}"
+ fi
+ elif [[ ! -x "${cbindgen_bin}" ]]; then
+ echo "PAIMON_RUST_CBINDGEN=${cbindgen_bin} is not an executable file."
+ exit 1
+ fi
+ # Write a temporary cbindgen.toml so the generated header carries our
+ # include-guard / cpp-compat settings without touching the upstream tree.
+ local cbindgen_toml="${BUILD_DIR}/cbindgen.toml"
+ mkdir -p "${BUILD_DIR}"
+ cat >"${cbindgen_toml}" <<'EOF'
+language = "C"
+include_guard = "PAIMON_C_H"
+pragma_once = true
+cpp_compat = true
+EOF
+ env "${cargo_env[@]}" "${cbindgen_bin}" bindings/c \
+ --config "${cbindgen_toml}" \
+ --output "${BUILD_DIR}/release/paimon.h"
+
+ mkdir -p "${TP_INSTALL_DIR}/include" "${TP_INSTALL_DIR}/lib64"
+ rm -rf "${TP_INSTALL_DIR}/include/paimon_rust"
+ mkdir -p "${TP_INSTALL_DIR}/include/paimon_rust"
+ cp -v "${BUILD_DIR}/release/paimon.h"
"${TP_INSTALL_DIR}/include/paimon_rust/"
+ install_rust_archive "${BUILD_DIR}/release/libpaimon_c.a"
}
if [[ "${#packages[@]}" -eq 0 ]]; then
@@ -2526,6 +2671,7 @@ if [[ "${#packages[@]}" -eq 0 ]]; then
icu
mecab_ipadic
pugixml
+ paimon_rust
)
if [[ "$(uname -s)" == 'Darwin' ]]; then
read -r -a packages <<<"binutils gettext ${packages[*]}"
@@ -2636,6 +2782,7 @@ cleanup_package_source() {
juicefs) src_var="JUICEFS_SOURCE" ;;
pugixml) src_var="PUGIXML_SOURCE" ;;
lance_c) src_var="LANCE_C_SOURCE" ;;
+ paimon_rust) src_var="PAIMON_RUST_SOURCE" ;;
aws_sdk) src_var="AWS_SDK_SOURCE" ;;
lzma) src_var="LZMA_SOURCE" ;;
xml2) src_var="XML2_SOURCE" ;;
diff --git a/thirdparty/test/paimon-build-test.py
b/thirdparty/test/paimon-build-test.py
new file mode 100644
index 00000000000..3fef1ffe6fc
--- /dev/null
+++ b/thirdparty/test/paimon-build-test.py
@@ -0,0 +1,202 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Run with python3 thirdparty/test/paimon-build-test.py; no Rust downloads
required."""
+
+import os
+from pathlib import Path
+import re
+import subprocess
+import tempfile
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[2]
+
+
+class PaimonBuildTest(unittest.TestCase):
+ def setUp(self):
+ self.temp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.temp.cleanup)
+ self.work = Path(self.temp.name)
+ self.env = dict(os.environ)
+ for key in ("CARGO", "CARGO_NET_OFFLINE", "RUSTUP_TOOLCHAIN"):
+ self.env.pop(key, None)
+
+ def write(self, name, text=""):
+ path = self.work / name
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(text)
+ return path
+
+ def executable(self, name, text):
+ path = self.write(name, "#!/bin/bash\nset -e\n" + text)
+ path.chmod(0o755)
+ return path
+
+ def run_bash(self, script):
+ return subprocess.run(["bash", "-ec", script], env=self.env,
+ cwd=self.work, text=True, capture_output=True)
+
+ def check_install(self, system, missing, clean=0):
+ script = (ROOT / "build.sh").read_text()
+ gate = script[script.index("# build thirdparty libraries if
necessary."):
+ script.index("update_submodule() {")]
+ sentinel = re.search(r"else\n LAST_THIRDPARTY_LIB='([^']+)'",
gate)[1]
+ if system == "Darwin":
+ sentinel = "libbrotlienc.a"
+ files = ["lib/" + sentinel, "lib64/liblance_c.a",
"lib64/libpaimon_c.a",
+ "include/paimon_rust/paimon.h"]
+ for name in files:
+ if name != missing:
+ self.write("installed/" + name)
+ # Execute the real installation gate, replacing only the expensive
build.
+ self.executable("build-thirdparty.sh", 'printf "%s\\n" "$*" >
"$BUILD_LOG"\n')
+ self.env.update(DORIS_THIRDPARTY=str(self.work), TARGET_SYSTEM=system,
+ CLEAN=str(clean), PARALLEL="2",
BUILD_LOG=str(self.work / "build.log"))
+ result = self.run_bash(gate)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual((self.work / "build.log").exists(), missing is not
None)
+ if missing is not None:
+ args = (self.work / "build.log").read_text().strip()
+ self.assertEqual(args, "-j 2" + (" --clean" if clean else ""))
+ # A full rebuild also replaces Lance built with the previous Rust
version.
+ self.assertFalse((self.work /
"installed/lib64/liblance_c.a").exists())
+
+ def test_old_linux_install_rebuilds(self):
+ self.check_install("Linux", "lib64/libpaimon_c.a")
+
+ def test_old_macos_install_rebuilds(self):
+ self.check_install("Darwin", "lib64/libpaimon_c.a")
+
+ def test_missing_header_rebuilds(self):
+ self.check_install("Linux", "include/paimon_rust/paimon.h")
+
+ def test_missing_lance_rebuilds(self):
+ self.check_install("Linux", "lib64/liblance_c.a")
+
+ def test_clean_option_is_preserved(self):
+ self.check_install("Linux", "lib64/libpaimon_c.a", clean=1)
+
+ def test_complete_install_is_reused(self):
+ self.check_install("Linux", None)
+
+ def archive_installer(self):
+ script = (ROOT / "thirdparty/build-thirdparty.sh").read_text()
+ return re.search(r"^install_rust_archive\(\) \{\n.*?^\}", script, re.M
| re.S)[0]
+
+ def check_archive_publication(self, name):
+ source = self.write("build/" + name, "new archive")
+ source.chmod(0o644)
+ destination = self.work / "installed/lib64" / name
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ self.env.update(TP_INSTALL_DIR=str(self.work / "installed"),
+ STRIP_TP_LIB="ON", KERNEL="Linux", ARCHIVE=str(source))
+ failures = {
+ "copy": 'cp() { printf partial > "${@: -1}"; return 1; }',
+ "strip": 'strip() { printf damaged > "${@: -1}"; return 1; }',
+ "publish": 'mv() { return 1; }',
+ "success": '',
+ }
+ for replacement in (False, True):
+ for stage, injection in failures.items():
+ with self.subTest(archive=name, replacement=replacement,
stage=stage):
+ if destination.exists():
+ destination.unlink()
+ if replacement:
+ destination.write_text("old archive")
+ # Execute the production publisher with failures after
partial copy/strip;
+ # neither a first install nor a replacement may expose
those bytes.
+ result = self.run_bash(self.archive_installer() +
+ '\nstrip() { :; }\n' + injection +
+ '\ninstall_rust_archive "$ARCHIVE"')
+ if stage == "success":
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual(destination.read_text(), "new
archive")
+ self.assertEqual(destination.stat().st_mode & 0o777,
0o644)
+ else:
+ self.assertNotEqual(result.returncode, 0)
+ if replacement:
+ self.assertEqual(destination.read_text(), "old
archive")
+ else:
+ self.assertFalse(destination.exists())
+ self.assertEqual(list(destination.parent.glob(name +
".tmp.*")), [])
+
+ def test_atomic_lance_archive_publication(self):
+ self.check_archive_publication("liblance_c.a")
+
+ def test_atomic_paimon_archive_publication(self):
+ self.check_archive_publication("libpaimon_c.a")
+
+ def check_header(self, offline, inherited_offline="", cargo_override=True):
+ script = (ROOT / "thirdparty/build-thirdparty.sh").read_text()
+ function = re.search(r"^build_paimon_rust\(\) \{\n.*?^\}", script,
re.M | re.S)[0]
+ cargo = self.executable("custom/cargo", r'''case "$1" in
+ --version) echo 'cargo 1.94.0' ;;
+ build)
+ mkdir -p "$CARGO_TARGET_DIR/release"
+ touch "$CARGO_TARGET_DIR/release/libpaimon_c.a"
+ ;;
+ metadata) printf '%s' "${CARGO_NET_OFFLINE:-unset}" > "$METADATA_LOG" ;;
+ *) exit 90 ;;
+esac
+''')
+ # PATH's Cargo deliberately fails: metadata must use the selected
executable.
+ self.executable("bin/cargo", "exit 91\n")
+ cbindgen = self.executable("bin/cbindgen", r'''"${CARGO:-cargo}"
metadata
+while [[ "$1" != --output ]]; do shift; done
+printf '/* Generated test header */\n' > "$2"
+''')
+ self.write("src/paimon/Cargo.toml")
+ self.env.update(PATH=str(self.work / "bin") + os.pathsep +
os.environ["PATH"],
+ TP_SOURCE_DIR=str(self.work / "src"),
PAIMON_RUST_SOURCE="paimon",
+ TP_INSTALL_DIR=str(self.work / "installed"),
BUILD_DIR="build",
+ PARALLEL="1", KERNEL="Linux", STRIP_TP_LIB="OFF",
+ RUSTUP_TOOLCHAIN="1.94.0",
PAIMON_RUST_CARGO_OFFLINE=offline,
+ PAIMON_RUST_CBINDGEN=str(cbindgen),
+ METADATA_LOG=str(self.work / "metadata.log"))
+ if cargo_override:
+ self.env["PAIMON_RUST_CARGO"] = str(cargo)
+ else:
+ self.env.pop("PAIMON_RUST_CARGO", None)
+ self.env["CARGO"] = str(cargo)
+ if inherited_offline:
+ self.env["CARGO_NET_OFFLINE"] = inherited_offline
+ result = self.run_bash("check_if_source_exist() { :; }\n" +
self.archive_installer()
+ + "\n" + function + "\nbuild_paimon_rust")
+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+ expected = "true" if offline.upper() == "ON" else inherited_offline or
"unset"
+ self.assertEqual((self.work / "metadata.log").read_text(), expected)
+ self.assertTrue((self.work /
"installed/include/paimon_rust/paimon.h").exists())
+ self.assertTrue((self.work / "installed/lib64/libpaimon_c.a").exists())
+
+ def test_header_uses_selected_cargo(self):
+ self.check_header("OFF")
+
+ def test_header_metadata_is_offline(self):
+ self.check_header("on", cargo_override=False)
+
+ def test_offline_override_wins(self):
+ self.check_header("ON", inherited_offline="false",
cargo_override=False)
+
+ def test_inherited_offline_is_preserved(self):
+ self.check_header("OFF", inherited_offline="true",
cargo_override=False)
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh
index 348d638c85d..f17ed8df35e 100644
--- a/thirdparty/vars.sh
+++ b/thirdparty/vars.sh
@@ -622,6 +622,12 @@ LANCE_C_NAME="lance-c-v0.1.9.tar.gz"
LANCE_C_SOURCE="lance-c-0.1.9"
LANCE_C_MD5SUM="7138ed44e92d4bc91d5b522a6b92ed64"
+# paimon-rust
+PAIMON_RUST_DOWNLOAD="https://github.com/apache/paimon-rust/archive/refs/tags/v0.4.0-rc1.tar.gz";
+PAIMON_RUST_NAME="paimon-rust-v0.4.0-rc1.tar.gz"
+PAIMON_RUST_SOURCE="paimon-rust-0.4.0-rc1"
+PAIMON_RUST_MD5SUM="23cd0634692108debf77a26ef4dc20a7"
+
# all thirdparties which need to be downloaded is set in array TP_ARCHIVES
export TP_ARCHIVES=(
'MECAB_IPADIC'
@@ -710,6 +716,7 @@ export TP_ARCHIVES=(
'JUICEFS'
'PUGIXML'
'LANCE_C'
+ 'PAIMON_RUST'
)
# Only defined on the platforms upstream ships a prebuilt driver for (see
above).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
