This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new dabd51e5aa4 [improvement](lance) Upgrade prewarm dependency and remove
install guard (#68765)
dabd51e5aa4 is described below
commit dabd51e5aa44cb9b16b48c7cf1732aa84a05cf5e
Author: Gabriel <[email protected]>
AuthorDate: Thu Oct 8 14:22:57 2026 +0800
[improvement](lance) Upgrade prewarm dependency and remove install guard
(#68765)
### What problem does this PR solve?
Related PR: #68698
Related issue: #68692 (dependency preparation only)
Port #68698 to master. Upgrade lance-c from
`cd63420bfbe27f6f0a1edcc873b9191af7d52852` to
`98468344bc9d56aa7ed4a4192e844afa62b34ba1`, including the upstream
synchronous index-prewarm C/C++ APIs. Refresh the Foyer patch context
while preserving its implementation payload and verify that patch
application retains the prewarm APIs and shared-session regression.
Remove the Lance-specific installation fingerprint helper, its test
harness, and their build/CI calls. Existing installations are no longer
rejected or rebuilt because their Lance revision fingerprint differs;
unavailable APIs are reported through normal compilation or linking.
Preserve master's unrelated build and CI changes.
This only prepares the dependency; it does not add a SQL prewarm
interface or complete the parent issue.
### Validation
- Verified the archive checksum and unchanged Foyer implementation
payload.
- Passed `lance-prefilter-patch-test.sh`, covering fresh extraction,
cached-source reuse, re-extraction, stale patch markers, invalid-patch
rejection, macOS definitions, and prewarm API/regression preservation.
- Passed `download-thirdparty-fallback-test.sh`,
`juicefs-default-mirror-test.sh`, `download-thirdparty-md5-test.sh`,
`azure-vcpkg-retry-test.sh`, and `adbc-jni-config-test.sh`.
- Passed affected shell syntax checks, workflow YAML parsing, and `git
diff --check`.
No Doris FE/BE source files change. Full compilation and runtime suites
were not run locally; PR CI will validate the build.
### Release note
Prepare the Lance dependency for index prewarm and remove Lance-specific
installation fingerprint checks. No new SQL interface is exposed.
### Check List (For Author)
- Test: Existing dependency/script regression tests and manual
validation described above.
- Behavior changed: Yes; upgrade lance-c and remove Lance revision
fingerprint checks.
- Does this need documentation: No new SQL interface.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
.github/workflows/build-thirdparty.yml | 3 -
build.sh | 25 +---
thirdparty/build-thirdparty.sh | 7 -
thirdparty/lance-install.sh | 38 ------
thirdparty/patches/lance-c-foyer.patch | 22 ++--
thirdparty/test/lance-install-test.sh | 176 --------------------------
thirdparty/test/lance-prefilter-patch-test.sh | 7 +
thirdparty/vars.sh | 10 +-
8 files changed, 27 insertions(+), 261 deletions(-)
diff --git a/.github/workflows/build-thirdparty.yml
b/.github/workflows/build-thirdparty.yml
index b8ac6624bb4..5e25d5fdc87 100644
--- a/.github/workflows/build-thirdparty.yml
+++ b/.github/workflows/build-thirdparty.yml
@@ -82,9 +82,6 @@ jobs:
run: |
thirdparty/test/adbc-jni-config-test.sh
- - name: Test Lance installation
- run: bash thirdparty/test/lance-install-test.sh
-
build_linux:
name: Build Third Party Libraries (Linux)
needs: changes
diff --git a/build.sh b/build.sh
index 917c10005ae..de304892669 100755
--- a/build.sh
+++ b/build.sh
@@ -480,7 +480,6 @@ if [[ "${CLEAN}" -eq 1 && "${BUILD_BE}" -eq 0 &&
"${BUILD_FE}" -eq 0 && ${BUILD_
fi
# build thirdparty libraries if necessary. check last thirdparty lib
installation
-source "${DORIS_HOME}/thirdparty/lance-install.sh"
if [[ "${TARGET_SYSTEM}" == 'Darwin' ]]; then
LAST_THIRDPARTY_LIB='libbrotlienc.a'
else
@@ -496,21 +495,10 @@ if [[ ! -f
"${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ||
! -f "${DORIS_THIRDPARTY}/installed/include/paimon_rust/paimon.h" ||
! -s "${DORIS_THIRDPARTY}/installed/lib64/libpaimon_c.a" ||
! -s "${DORIS_THIRDPARTY}/installed/include/paimon_rust/paimon.h" ||
- -e "${DORIS_THIRDPARTY}/installed/lib64/.paimon-installing" ]] ||
- ! lance_c_install_is_current "${DORIS_HOME}/thirdparty"
"${DORIS_THIRDPARTY}/installed"; then
- # External trees can be partially updated or pinned to another revision.
Preserve
- # the existing prefix unless their build inputs can produce the requested
Lance version.
- for input in build-thirdparty.sh download-thirdparty.sh vars.sh
lance-install.sh patches/lance-c-foyer.patch; do
- if [[ ! -f "${DORIS_THIRDPARTY}/${input}" || ! -r
"${DORIS_THIRDPARTY}/${input}" ]]; then
- echo "Third-party dependencies require a rebuild, but ${input} is
missing or unreadable." >&2
- echo "Refresh the compilation image or set DORIS_THIRDPARTY to a
complete third-party source tree." >&2
- exit 1
- fi
- done
- if ! expected_lance_fingerprint="$(lance_c_install_fingerprint
"${DORIS_HOME}/thirdparty")" ||
- ! rebuild_lance_fingerprint="$(lance_c_install_fingerprint
"${DORIS_THIRDPARTY}")" ||
- [[ "${rebuild_lance_fingerprint}" != "${expected_lance_fingerprint}"
]]; then
- echo "Lance rebuild sources do not match this checkout; installed
dependencies have been preserved." >&2
+ -e "${DORIS_THIRDPARTY}/installed/lib64/.paimon-installing" ]]; then
+ # Compilation images may contain only installed artifacts; never erase
them without a rebuild source.
+ if [[ ! -f "${DORIS_THIRDPARTY}/build-thirdparty.sh" ]]; then
+ echo "Third-party dependencies require a rebuild, but
build-thirdparty.sh is missing." >&2
echo "Refresh the compilation image or set DORIS_THIRDPARTY to a
complete third-party source tree." >&2
exit 1
fi
@@ -523,11 +511,6 @@ if [[ ! -f
"${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ||
else
bash "${DORIS_THIRDPARTY}/build-thirdparty.sh" -j "${PARALLEL}" --clean
fi
- # An external build script can itself be stale. Never link its old output
silently.
- if ! lance_c_install_is_current "${DORIS_HOME}/thirdparty"
"${DORIS_THIRDPARTY}/installed"; then
- echo "Lance dependency revision does not match this checkout. Refresh
the third-party build tree." >&2
- exit 1
- fi
fi
update_submodule() {
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index da3e36b1e3f..52c2d481b20 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -219,8 +219,6 @@ if [[ ! -f "${TP_DIR}/vars.sh" ]]; then
fi
. "${TP_DIR}/vars.sh"
-. "${TP_DIR}/lance-install.sh"
-LANCE_C_INSTALL_FINGERPRINT="$(lance_c_install_fingerprint "${TP_DIR}")"
cd "${TP_DIR}"
@@ -2509,14 +2507,9 @@ build_lance_c() {
env "${cargo_env[@]}" "${cargo_bin}" "${cargo_args[@]}"
mkdir -p "${TP_INSTALL_DIR}/include" "${TP_INSTALL_DIR}/lib64"
- # Invalidate before publishing either file so interrupted installs cannot
reuse
- # a matching marker with a partial header/archive pair.
- rm -f "${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint"
rm -rf "${TP_INSTALL_DIR}/include/lance"
cp -av include/lance "${TP_INSTALL_DIR}/include/"
install_rust_archive "${BUILD_DIR}/release/liblance_c.a"
- printf '%s\n' "${LANCE_C_INSTALL_FINGERPRINT}" >
"${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint.tmp"
- mv "${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint.tmp"
"${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint"
}
build_paimon_rust() {
diff --git a/thirdparty/lance-install.sh b/thirdparty/lance-install.sh
deleted file mode 100644
index 28594d61c79..00000000000
--- a/thirdparty/lance-install.sh
+++ /dev/null
@@ -1,38 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# Keep the check independent of the installed prefix: an external compilation
image
-# can carry old vars.sh alongside an ABI-compatible but behaviorally stale
archive.
-lance_c_install_fingerprint() (
- local definitions="$1"
- local TP_DIR="${definitions}"
- source "${definitions}/vars.sh" || return 1
- local patch_checksum
- patch_checksum="$(cksum < "${definitions}/patches/lance-c-foyer.patch")"
|| return 1
- printf '%s\n' "${LANCE_C_SOURCE}" "${LANCE_C_MD5SUM}" "${patch_checksum}"
-)
-
-lance_c_install_is_current() {
- local definitions="$1" installed="$2" expected
- [[ -s "${installed}/lib64/liblance_c.a" &&
- -s "${installed}/include/lance/lance.h" &&
- -s "${installed}/include/lance/lance.hpp" &&
- -s "${installed}/lib64/.lance-c-fingerprint" ]] || return 1
- expected="$(lance_c_install_fingerprint "${definitions}")" || return 1
- [[ "$(cat "${installed}/lib64/.lance-c-fingerprint")" == "${expected}" ]]
-}
diff --git a/thirdparty/patches/lance-c-foyer.patch
b/thirdparty/patches/lance-c-foyer.patch
index bb2104ab32d..a67e2a04213 100644
--- a/thirdparty/patches/lance-c-foyer.patch
+++ b/thirdparty/patches/lance-c-foyer.patch
@@ -1,5 +1,5 @@
# Foyer data-cache integration for lance-format/lance-c#73.
-# Base: cd63420bfbe27f6f0a1edcc873b9191af7d52852 (lance-format/lance-c main,
including #93)
+# Base: 98468344bc9d56aa7ed4a4192e844afa62b34ba1 (lance-format/lance-c main,
including #94)
# Source:
https://github.com/Gabriel39/lance-c/commit/24c7ca4bcb9422c113b0d3e07e4efe1173b0bc9f
# The unchanged Foyer payload is reapplied to the newer upstream base with git
apply --3way.
diff --git a/Cargo.lock b/Cargo.lock
@@ -198,7 +198,7 @@ index
18654b862440a90aa6bf0dd846e8f1ad6036fd7c..fd134a7151e1df61da201d867a642e9b
prost = "0.14"
snafu = "0.9"
diff --git a/README.md b/README.md
-index
9f0871b17b13ebfd96916947b4bee0dd06f6a468..441719c9f3904058127e55d1cd29ea1afc878935
100644
+index
f13ad015826a8ac5468cfea247707d987cc0864d..56422c22969882cd43edd5e83a4a5c4d0dadd3d6
100644
--- a/README.md
+++ b/README.md
@@ -68,6 +68,7 @@ Based on the [liblance
RFC](https://github.com/lance-format/lance/discussions/60
@@ -242,11 +242,11 @@ index
9f0871b17b13ebfd96916947b4bee0dd06f6a468..441719c9f3904058127e55d1cd29ea1a
+auto ds = lance::Dataset::open_with_session(session,
"s3://bucket/data.lance");
+```
+
- ### Open at a specific version
+ ### Prewarm an index synchronously
- `lance_dataset_open` takes a `version` argument — `0` means the latest, any
+ Prewarm a logical index before serving queries to move cold index reads out of
diff --git a/include/lance/lance.h b/include/lance/lance.h
-index
94772e2a14bf9a8a950dfb2d39112136e27197c0..fe48eb0d9afd1c7157a3bd2230cce7e769120bac
100644
+index
b91c290a40d37edbb49323d2d11efd3b4f980bb8..0997023974a5205a145a991d53da0ee1a2babc8c
100644
--- a/include/lance/lance.h
+++ b/include/lance/lance.h
@@ -214,6 +214,36 @@ typedef struct LanceSessionCacheStats {
@@ -334,7 +334,7 @@ index
94772e2a14bf9a8a950dfb2d39112136e27197c0..fe48eb0d9afd1c7157a3bd2230cce7e7
void lance_dataset_close(LanceDataset* dataset);
diff --git a/include/lance/lance.hpp b/include/lance/lance.hpp
-index
edcd39fc3caba387ab490fa84139f24e9376130a..17b71957d0f5c419fe0b3392ce11faa757ddbfe1
100644
+index
17e93486c079d3453547af160b875310328da9e1..b049cc519ee9784d4bcc1a087eac237e880b5e74
100644
--- a/include/lance/lance.hpp
+++ b/include/lance/lance.hpp
@@ -176,6 +176,13 @@ struct SqlColumn {
@@ -2243,7 +2243,7 @@ index
1971510de4a13ee6f04fc39c159b088e78e21d55..ba51c87a8cebb6897c48fb55509f71da
// SAFETY: `out_dataset` is non-NULL (checked above) and the caller
// guarantees it points to caller-owned, writable storage of size
diff --git a/tests/c_api_test.rs b/tests/c_api_test.rs
-index
d3bea44529a842fe3ae28ee3f90cd8d45fe0b394..98e6901a2cfeb2cc98df6606e26b04788d4218b2
100644
+index
b00072df3f966f58663359ccd370b399650c2023..e44871e92bc7ff9787bf5220b1ea8a4270da9710
100644
--- a/tests/c_api_test.rs
+++ b/tests/c_api_test.rs
@@ -100,10 +100,83 @@ fn create_large_dataset(num_rows: i32) ->
(tempfile::TempDir, String) {
@@ -2502,7 +2502,7 @@ index
d3bea44529a842fe3ae28ee3f90cd8d45fe0b394..98e6901a2cfeb2cc98df6606e26b0478
fn test_dataset_restore_to_current_latest_writes_new_manifest() {
// Restoring to the current latest still writes a new manifest. The
diff --git a/tests/cpp/test_c_api.c b/tests/cpp/test_c_api.c
-index
efad5ed55987976a3f1820220e42fc58bef42638..050543aebc02454cab09f6e0bd38ec10b1d21720
100644
+index
77af1a69b872bd4152a44e642f30190720daa33f..da15749671cb9448c73d3c61546ecab2adcacb15
100644
--- a/tests/cpp/test_c_api.c
+++ b/tests/cpp/test_c_api.c
@@ -171,6 +171,37 @@ static void test_shared_session(const char *uri) {
@@ -2543,7 +2543,7 @@ index
efad5ed55987976a3f1820220e42fc58bef42638..050543aebc02454cab09f6e0bd38ec10
static void test_scan(const char *uri) {
printf(" test_scan... ");
-@@ -1471,6 +1502,7 @@ int main(int argc, char **argv) {
+@@ -1476,6 +1507,7 @@ int main(int argc, char **argv) {
test_batch_nearest(uri);
test_open_and_metadata(uri);
test_shared_session(uri);
@@ -2552,7 +2552,7 @@ index
efad5ed55987976a3f1820220e42fc58bef42638..050543aebc02454cab09f6e0bd38ec10
test_distance_range(uri);
test_scan_with_limit(uri);
diff --git a/tests/cpp/test_cpp_api.cpp b/tests/cpp/test_cpp_api.cpp
-index
e2d541e4430889aba89ab85c01cc7e3eacf97baa..ac1c10bccfcf399bbbfcfe908d3188484aff0add
100644
+index
3bc313ebecbb0de06badd9f8d75f531fc95809c9..298228f7dd68d28452e6436db5c589183361f2de
100644
--- a/tests/cpp/test_cpp_api.cpp
+++ b/tests/cpp/test_cpp_api.cpp
@@ -131,6 +131,28 @@ static void test_shared_session(const std::string& uri) {
@@ -2584,7 +2584,7 @@ index
e2d541e4430889aba89ab85c01cc7e3eacf97baa..ac1c10bccfcf399bbbfcfe908d318848
static void test_dataset_schema(const std::string& uri) {
TEST(test_dataset_schema);
-@@ -1353,6 +1375,7 @@ int main(int argc, char** argv) {
+@@ -1365,6 +1387,7 @@ int main(int argc, char** argv) {
test_dataset_open(uri);
test_shared_session(uri);
diff --git a/thirdparty/test/lance-install-test.sh
b/thirdparty/test/lance-install-test.sh
deleted file mode 100644
index db93c355c0f..00000000000
--- a/thirdparty/test/lance-install-test.sh
+++ /dev/null
@@ -1,176 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-set -euo pipefail
-ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-work="$(mktemp -d)"
-trap 'rm -rf "${work}"' EXIT
-mkdir -p "${work}/repo/thirdparty/patches" "${work}/external"
-cp "${ROOT}/thirdparty/vars.sh" "${work}/repo/thirdparty/"
-cp "${ROOT}/thirdparty/patches/lance-c-foyer.patch"
"${work}/repo/thirdparty/patches/"
-# Extract the real build gate; all destructive operations stay inside this
temporary install.
-echo 'set -eo pipefail' > "${work}/gate.sh"
-sed -n '/^# build thirdparty libraries if necessary/,/^update_submodule()/p'
"${ROOT}/build.sh" \
- | sed '$d' >> "${work}/gate.sh"
-export DORIS_HOME="${work}/repo" DORIS_THIRDPARTY="${work}/external"
-export TARGET_SYSTEM=Linux CLEAN=0 PARALLEL=1
-export TEST_HELPER="${ROOT}/thirdparty/lance-install.sh"
-cp "${TEST_HELPER}" "${work}/repo/thirdparty/"
-cp -r "${DORIS_HOME}/thirdparty/." "${DORIS_THIRDPARTY}/"
-printf '#!/usr/bin/env bash\nexit 0\n' >
"${DORIS_THIRDPARTY}/download-thirdparty.sh"
-cat > "${DORIS_THIRDPARTY}/build-thirdparty.sh" <<'BUILDER'
-set -euo pipefail
-TP_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-source "${TP_DIR}/lance-install.sh"
-fingerprint="$(lance_c_install_fingerprint "${TP_DIR}")"
-echo rebuilt >> "${DORIS_THIRDPARTY}/builds"
-installed="${DORIS_THIRDPARTY}/installed"
-mkdir -p "${installed}/lib/hadoop_hdfs/native"
"${installed}/lib/hadoop_hdfs_3_4/native" "${installed}/lib64" \
- "${installed}/include/lance" "${installed}/include/paimon_rust"
-for file in lib/hadoop_hdfs/native/libhdfs.a
lib/hadoop_hdfs_3_4/native/libhdfs.a \
- lib64/liblance_c.a lib64/libpaimon_c.a \
- include/lance/lance.h include/lance/lance.hpp
include/paimon_rust/paimon.h; do
- echo artifact > "${installed}/${file}"
-done
-printf '%s\n' "${fingerprint}" > "${installed}/lib64/.lance-c-fingerprint"
-BUILDER
-# A complete legacy image has all old sentinels but no Lance revision marker.
-installed="${DORIS_THIRDPARTY}/installed"
-mkdir -p "${installed}/lib/hadoop_hdfs/native"
"${installed}/lib/hadoop_hdfs_3_4/native" "${installed}/lib64" \
- "${installed}/include/lance" "${installed}/include/paimon_rust"
-for file in lib/hadoop_hdfs/native/libhdfs.a
lib/hadoop_hdfs_3_4/native/libhdfs.a \
- lib64/liblance_c.a lib64/libpaimon_c.a \
- include/lance/lance.h include/lance/lance.hpp
include/paimon_rust/paimon.h; do
- echo legacy > "${installed}/${file}"
-done
-bash "${work}/gate.sh"
-[[ -f "${DORIS_THIRDPARTY}/builds" ]] || { echo 'FAIL: reused unversioned
Lance archive'; exit 1; }
-source "${TEST_HELPER}"
-lance_c_install_is_current "${DORIS_HOME}/thirdparty" "${installed}"
-check_builds() {
- [[ "$(wc -l < "${DORIS_THIRDPARTY}/builds")" -eq "$1" ]]
-}
-check_builds 1
-bash "${work}/gate.sh"
-check_builds 1
-echo 'PASS: legacy image rebuilt; matching install reused'
-# Failed preflight must preserve every dependency, not just the Lance archive.
-expect_preserved_install() {
- cp -a "${installed}" "${work}/saved-install"
- cp "${DORIS_THIRDPARTY}/builds" "${work}/saved-builds"
- if bash "${work}/gate.sh" > "${work}/preflight.log" 2>&1; then
- echo 'FAIL: accepted incomplete or mismatched rebuild sources'; exit 1
- fi
- if ! diff -r "${work}/saved-install" "${installed}"; then
- echo 'FAIL: rejected rebuild sources after changing installed
dependencies'; exit 1
- fi
- cmp "${work}/saved-builds" "${DORIS_THIRDPARTY}/builds"
- rm -rf "${work}/saved-install"
-}
-echo stale > "${installed}/lib64/.lance-c-fingerprint"
-for file in lance-install.sh vars.sh patches/lance-c-foyer.patch
download-thirdparty.sh build-thirdparty.sh; do
- mv "${DORIS_THIRDPARTY}/${file}" "${work}/missing-input"
- expect_preserved_install
- mv "${work}/missing-input" "${DORIS_THIRDPARTY}/${file}"
-done
-echo 'PASS: incomplete external sources rejected before changing installed
dependencies'
-echo stale > "${installed}/lib64/.lance-c-fingerprint"
-bash "${work}/gate.sh"
-check_builds 2
-# Pin and patch updates both invalidate installed artifacts, even with
identical ABI.
-sed 's/^LANCE_C_SOURCE=.*/LANCE_C_SOURCE="lance-c-test-revision"/'
"${DORIS_HOME}/thirdparty/vars.sh" \
- > "${work}/new-vars.sh"
-mv "${work}/new-vars.sh" "${DORIS_HOME}/thirdparty/vars.sh"
-expect_preserved_install
-cp "${DORIS_HOME}/thirdparty/vars.sh" "${DORIS_THIRDPARTY}/vars.sh"
-bash "${work}/gate.sh"
-check_builds 3
-echo '# test patch update' >>
"${DORIS_HOME}/thirdparty/patches/lance-c-foyer.patch"
-expect_preserved_install
-cp "${DORIS_HOME}/thirdparty/patches/lance-c-foyer.patch"
"${DORIS_THIRDPARTY}/patches/"
-bash "${work}/gate.sh"
-check_builds 4
-echo 'PASS: mismatched sources rejected; synchronized pin and patch updates
rebuild'
-rm "${installed}/include/lance/lance.h"
-bash "${work}/gate.sh"
-check_builds 5
-: > "${installed}/lib64/liblance_c.a"
-bash "${work}/gate.sh"
-check_builds 6
-echo 'PASS: incomplete header/archive install rebuilt'
-# A legacy external builder may exit successfully without installing the new
revision.
-cp "${DORIS_THIRDPARTY}/build-thirdparty.sh" "${work}/good-builder.sh"
-printf '#!/usr/bin/env bash\nexit 0\n' >
"${DORIS_THIRDPARTY}/build-thirdparty.sh"
-rm "${installed}/lib64/.lance-c-fingerprint"
-if bash "${work}/gate.sh" > "${work}/old-builder.log" 2>&1; then
- echo 'FAIL: accepted output from a stale external builder'; exit 1
-fi
-grep -q 'Lance dependency revision does not match' "${work}/old-builder.log"
-cp "${work}/good-builder.sh" "${DORIS_THIRDPARTY}/build-thirdparty.sh"
-bash "${DORIS_THIRDPARTY}/build-thirdparty.sh"
-echo 'PASS: stale external builder cannot silently satisfy the revision gate'
-# Images without rebuild sources must fail before deleting installed
dependencies.
-rm "${DORIS_THIRDPARTY}/build-thirdparty.sh"
"${installed}/lib64/.lance-c-fingerprint"
-if bash "${work}/gate.sh" > "${work}/missing-source.log" 2>&1; then
- echo 'FAIL: accepted stale compilation image without build sources'; exit 1
-fi
-[[ -s "${installed}/lib64/liblance_c.a" ]]
-echo 'PASS: missing rebuild source fails without deleting installed artifacts'
-# Execute the actual publication function with a tiny Cargo stand-in. A failed
-# archive copy must invalidate the old marker, and only a complete retry may
stamp it.
-sed -n '/^build_lance_c()/,/^}/p' "${ROOT}/thirdparty/build-thirdparty.sh" >
"${work}/publish-function.sh"
-export TP_DIR="${DORIS_HOME}/thirdparty"
-source "${TP_DIR}/vars.sh"
-export LANCE_C_SOURCE TP_SOURCE_DIR TP_INSTALL_DIR
-export LANCE_C_INSTALL_FINGERPRINT="$(lance_c_install_fingerprint "${TP_DIR}")"
-mkdir -p "${TP_SOURCE_DIR}/${LANCE_C_SOURCE}/include/lance"
"${TP_INSTALL_DIR}/bin"
-echo header > "${TP_SOURCE_DIR}/${LANCE_C_SOURCE}/include/lance/lance.h"
-echo header > "${TP_SOURCE_DIR}/${LANCE_C_SOURCE}/include/lance/lance.hpp"
-printf '#!/bin/sh\nexit 0\n' > "${TP_INSTALL_DIR}/bin/protoc"
-chmod +x "${TP_INSTALL_DIR}/bin/protoc"
-cat > "${work}/cargo" <<'CARGO'
-#!/usr/bin/env bash
-set -eu
-if [[ "$1" == --version ]]; then
- echo 'cargo 1.94.0'; exit 0
-fi
-mkdir -p "${CARGO_TARGET_DIR}/release"
-echo archive > "${CARGO_TARGET_DIR}/release/liblance_c.a"
-CARGO
-chmod +x "${work}/cargo"
-export LANCE_C_CARGO="${work}/cargo" RUSTUP_TOOLCHAIN=1.94.0
-export BUILD_DIR=build KERNEL=Linux LANCE_C_CARGO_OFFLINE=OFF
-cat > "${work}/publish.sh" <<'PUBLISH'
-set -eo pipefail
-check_if_source_exist() { :; }
-install_rust_archive() {
- if [[ "${FAIL_INSTALL:-0}" == 1 ]]; then return 1; fi
- cp "$1" "${TP_INSTALL_DIR}/lib64/liblance_c.a"
-}
-source "$1"
-build_lance_c
-PUBLISH
-bash "${work}/publish.sh" "${work}/publish-function.sh" >
"${work}/publish.log" 2>&1
-lance_c_install_is_current "${TP_DIR}" "${TP_INSTALL_DIR}"
-if FAIL_INSTALL=1 bash "${work}/publish.sh" "${work}/publish-function.sh" >>
"${work}/publish.log" 2>&1; then
- echo 'FAIL: expected archive publication failure'; exit 1
-fi
-[[ ! -e "${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint" ]]
-bash "${work}/publish.sh" "${work}/publish-function.sh" >>
"${work}/publish.log" 2>&1
-lance_c_install_is_current "${TP_DIR}" "${TP_INSTALL_DIR}"
-echo 'PASS: successful install stamped; failed publication invalidated; retry
repaired'
diff --git a/thirdparty/test/lance-prefilter-patch-test.sh
b/thirdparty/test/lance-prefilter-patch-test.sh
index e3dabff6c17..ab3587c56d1 100755
--- a/thirdparty/test/lance-prefilter-patch-test.sh
+++ b/thirdparty/test/lance-prefilter-patch-test.sh
@@ -68,6 +68,13 @@ check_sources() {
|| fail "missing retained cache statistics API"
grep -q 'source = "git+https://github.com/lance-format/lance.git'
"${source}/Cargo.lock" \
|| fail "Lance must come from the upstream git dependency"
+ # Refreshing the Foyer patch must preserve the upstream prewarm binding
and tests.
+ grep -q 'lance_dataset_prewarm_index' "${source}/include/lance/lance.h" \
+ || fail "missing upstream C prewarm API"
+ grep -q 'void prewarm_index(' "${source}/include/lance/lance.hpp" \
+ || fail "missing upstream C++ prewarm API"
+ grep -q 'fn test_prewarm_index_vector_segments_reuse_shared_session'
"${source}/tests/c_api_test.rs" \
+ || fail "missing upstream shared-session prewarm regression"
[[ -f "${source}/patched_mark_foyer" ]] || fail "missing Foyer patch
marker"
}
diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh
index 556f6bf0bf3..969a74c4be4 100644
--- a/thirdparty/vars.sh
+++ b/thirdparty/vars.sh
@@ -617,11 +617,11 @@ PUGIXML_SOURCE=pugixml-1.15
PUGIXML_MD5SUM="3b894c29455eb33a40b165c6e2de5895"
# lance-c
-# Complete-segment prefilter fixes are supplied by upstream lance-c, not local
patches.
-LANCE_C_DOWNLOAD="https://codeload.github.com/lance-format/lance-c/tar.gz/cd63420bfbe27f6f0a1edcc873b9191af7d52852";
-LANCE_C_NAME="lance-c-cd63420bfbe27f6f0a1edcc873b9191af7d52852.tar.gz"
-LANCE_C_SOURCE="lance-c-cd63420bfbe27f6f0a1edcc873b9191af7d52852"
-LANCE_C_MD5SUM="37d82907559c4fdb8ed6e5b767b0d309"
+# Includes lance-c #94: synchronous index prewarm through the upstream C/C++
APIs.
+LANCE_C_DOWNLOAD="https://codeload.github.com/lance-format/lance-c/tar.gz/98468344bc9d56aa7ed4a4192e844afa62b34ba1";
+LANCE_C_NAME="lance-c-98468344bc9d56aa7ed4a4192e844afa62b34ba1.tar.gz"
+LANCE_C_SOURCE="lance-c-98468344bc9d56aa7ed4a4192e844afa62b34ba1"
+LANCE_C_MD5SUM="b452bab5d05f5bce5f0bfebe8b1fb831"
# paimon-rust
PAIMON_RUST_DOWNLOAD="https://github.com/apache/paimon-rust/archive/refs/tags/v0.4.0-rc1.tar.gz";
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
