github-actions[bot] commented on code in PR #67862: URL: https://github.com/apache/doris/pull/67862#discussion_r3987898428
########## docs/lance-jni-packaging.md: ########## @@ -0,0 +1,42 @@ +# Lance JNI packaging for Linux x86_64 + +`bash build.sh --fe` automatically replaces the Linux x86_64 JNI entry in +`lance-core-11.0.0.jar` with the glibc 2.17 rebuild. No local Lance checkout, +Rust toolchain, or `LANCE_JNI_SO` variable is required on the Doris build host. + +The artifact is pinned in `docker/thirdparties/lance-jni-helpers.sh` and published +in [apache/doris-thirdparty](https://github.com/apache/doris-thirdparty/releases/tag/lance-jni-11.0.0-glibc2.17-r1). +Both the compressed archive and the extracted library have fixed SHA256 values. + +The archive is cached under: + +```text +thirdparty/installed/lance-jni/liblance_jni-11.0.0-linux-x86_64-glibc2.17-r1.so.gz +``` + +With a custom `DORIS_THIRDPARTY`, the cache is under that directory's +`installed/lance-jni/`. A valid cache is reused without network access. A corrupt +cache is downloaded again. The existing `REPOSITORY_URL` mirror is tried first +when configured, followed by the pinned GitHub Release URL. Downloads use curl; +packaging also needs gzip, zip, unzip, and sha256sum. + +To update an existing FE output without compiling Doris: + +```bash +bash post-build.sh --fe +# Or use a custom output directory: +bash post-build.sh --fe --output /path/to/output +``` + +Only the JAR entry `nativelib/linux-x86-64/liblance_jni.so` is changed. The Maven +cache, Java classes, and other native entries are preserved. Download, checksum, +or JAR version failures stop packaging and leave the original output JAR intact. +ARM64, macOS, and BE-only packaging do not download or replace this library. + +The rebuild retains Lance's Haswell CPU baseline (AVX2/FMA/F16C). Its GLIBC symbol +requirements were checked statically; a full CentOS 7 FE runtime test is still +required. An already running FE must restart to load a newly packaged JNI library. + +When upgrading Lance, update the release URL, filename, version, and both hashes +in the helper together with `lance.version` in `fe/pom.xml`. Offline packaging +tests are available via `python3 thirdparty/test/lance-jni-packaging-test.py`. Review Comment: **[P2] Add and run the packaging test referenced here** This file is absent at the PR head, so the documented command exits with `No such file or directory`. The `Thirdparty Script Test` workflow is also skipped because its path filter does not include `docker/thirdparties/**`, `post-build.sh`, or the other changed packaging paths. Please add the promised offline test (including the stale/multiple-JAR upgrade case) and wire these paths into its CI trigger so this mandatory download/repack step is actually exercised. ########## docker/thirdparties/lance-jni-helpers.sh: ########## @@ -0,0 +1,117 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Keep these pins in sync with lance.version in fe/pom.xml. +LANCE_JNI_VERSION="11.0.0" +LANCE_JNI_ASSET="liblance_jni-11.0.0-linux-x86_64-glibc2.17-r1.so.gz" +LANCE_JNI_URL="https://github.com/apache/doris-thirdparty/releases/download/lance-jni-11.0.0-glibc2.17-r1/${LANCE_JNI_ASSET}"; +LANCE_JNI_ARCHIVE_SHA256="f9dc713269632e26c06ea2ea20637ca16ec9d76772dc20db0bead11e5cd34a6b" +LANCE_JNI_LIBRARY_SHA256="b6540edd3bdcd76b04f96a9e78fe804a2f1bffbd75eb35abe84de9091ba432ac" + +lance_jni_download() ( + set -eo pipefail + local cache_dir="$1" + local archive="${cache_dir}/${LANCE_JNI_ASSET}" + local work_dir url digest + local -a urls=() + + mkdir -p "${cache_dir}" + if [[ -f "${archive}" ]]; then + digest=$(sha256sum "${archive}" | awk '{print $1}') + if [[ "${digest}" == "${LANCE_JNI_ARCHIVE_SHA256}" ]]; then + printf '%s\n' "${archive}" + exit 0 + fi + echo "Lance JNI cache checksum mismatch; downloading again: ${archive}" >&2 + fi + + work_dir=$(mktemp -d "${cache_dir}/.lance-jni-download.XXXXXX") + trap 'rm -rf -- "${work_dir}"' EXIT + if [[ -n "${REPOSITORY_URL:-}" ]]; then + urls+=("${REPOSITORY_URL%/}/${LANCE_JNI_ASSET}") + fi + urls+=("${LANCE_JNI_URL}") + for url in "${urls[@]}"; do + echo "Downloading Lance JNI from ${url}" >&2 + if ! curl -fL --retry 3 --retry-delay 2 --connect-timeout 10 \ + -o "${work_dir}/library.gz" "${url}"; then + continue + fi + digest=$(sha256sum "${work_dir}/library.gz" | awk '{print $1}') + if [[ "${digest}" != "${LANCE_JNI_ARCHIVE_SHA256}" ]]; then + echo "ERROR: Lance JNI archive SHA256 mismatch from ${url}: expected ${LANCE_JNI_ARCHIVE_SHA256}, got ${digest}" >&2 + continue + fi + # Concurrent builds only publish complete, verified cache files. + mv -f "${work_dir}/library.gz" "${archive}" + printf '%s\n' "${archive}" + exit 0 + done + echo "ERROR: failed to download and verify ${LANCE_JNI_ASSET}" >&2 + exit 1 +) + +lance_jni_replace() ( + set -eo pipefail + local output_dir="$1" + local thirdparty_dir="$2" + local target_system="$3" + local target_arch="$4" + if [[ "${target_system}" != "Linux" || "${target_arch}" != "x86_64" ]]; then + exit 0 + fi + + local entry="nativelib/linux-x86-64/liblance_jni.so" + local target_jar="${output_dir}/fe/lib/lance-core-${LANCE_JNI_VERSION}.jar" + local archive work_dir source_hash packaged_hash + local -a lance_jars + shopt -s nullglob + lance_jars=("${output_dir}/fe/lib/"lance-core-*.jar) Review Comment: **[P2] Preserve non-clean FE builds across the version bump** `build.sh --fe` does not run Maven `clean` by default, and `dependency:copy-dependencies` only adds the newly resolved version to the persistent `fe/fe-core/target/lib`; it does not delete `lance-core-9.1.0-beta.3.jar`. After a developer builds the base and then this head, `build.sh` copies both JARs into `output/fe/lib`, so this exact-one check aborts the otherwise normal incremental build. Please purge obsolete Lance JARs while synchronizing the current dependency (or clean that dependency directory) before enforcing the count. ########## docker/thirdparties/lance-jni-helpers.sh: ########## @@ -0,0 +1,117 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Keep these pins in sync with lance.version in fe/pom.xml. +LANCE_JNI_VERSION="11.0.0" +LANCE_JNI_ASSET="liblance_jni-11.0.0-linux-x86_64-glibc2.17-r1.so.gz" +LANCE_JNI_URL="https://github.com/apache/doris-thirdparty/releases/download/lance-jni-11.0.0-glibc2.17-r1/${LANCE_JNI_ASSET}"; +LANCE_JNI_ARCHIVE_SHA256="f9dc713269632e26c06ea2ea20637ca16ec9d76772dc20db0bead11e5cd34a6b" +LANCE_JNI_LIBRARY_SHA256="b6540edd3bdcd76b04f96a9e78fe804a2f1bffbd75eb35abe84de9091ba432ac" + +lance_jni_download() ( + set -eo pipefail + local cache_dir="$1" + local archive="${cache_dir}/${LANCE_JNI_ASSET}" + local work_dir url digest + local -a urls=() + + mkdir -p "${cache_dir}" + if [[ -f "${archive}" ]]; then + digest=$(sha256sum "${archive}" | awk '{print $1}') + if [[ "${digest}" == "${LANCE_JNI_ARCHIVE_SHA256}" ]]; then + printf '%s\n' "${archive}" + exit 0 + fi + echo "Lance JNI cache checksum mismatch; downloading again: ${archive}" >&2 + fi + + work_dir=$(mktemp -d "${cache_dir}/.lance-jni-download.XXXXXX") + trap 'rm -rf -- "${work_dir}"' EXIT + if [[ -n "${REPOSITORY_URL:-}" ]]; then + urls+=("${REPOSITORY_URL%/}/${LANCE_JNI_ASSET}") + fi + urls+=("${LANCE_JNI_URL}") + for url in "${urls[@]}"; do + echo "Downloading Lance JNI from ${url}" >&2 + if ! curl -fL --retry 3 --retry-delay 2 --connect-timeout 10 \ + -o "${work_dir}/library.gz" "${url}"; then + continue + fi + digest=$(sha256sum "${work_dir}/library.gz" | awk '{print $1}') + if [[ "${digest}" != "${LANCE_JNI_ARCHIVE_SHA256}" ]]; then + echo "ERROR: Lance JNI archive SHA256 mismatch from ${url}: expected ${LANCE_JNI_ARCHIVE_SHA256}, got ${digest}" >&2 + continue + fi + # Concurrent builds only publish complete, verified cache files. + mv -f "${work_dir}/library.gz" "${archive}" + printf '%s\n' "${archive}" + exit 0 + done + echo "ERROR: failed to download and verify ${LANCE_JNI_ASSET}" >&2 + exit 1 +) + +lance_jni_replace() ( + set -eo pipefail + local output_dir="$1" + local thirdparty_dir="$2" + local target_system="$3" + local target_arch="$4" + if [[ "${target_system}" != "Linux" || "${target_arch}" != "x86_64" ]]; then + exit 0 + fi + + local entry="nativelib/linux-x86-64/liblance_jni.so" + local target_jar="${output_dir}/fe/lib/lance-core-${LANCE_JNI_VERSION}.jar" + local archive work_dir source_hash packaged_hash + local -a lance_jars + shopt -s nullglob + lance_jars=("${output_dir}/fe/lib/"lance-core-*.jar) + if [[ ${#lance_jars[@]} -ne 1 || "${lance_jars[0]}" != "${target_jar}" ]]; then + echo "ERROR: expected exactly one lance-core-${LANCE_JNI_VERSION}.jar in ${output_dir}/fe/lib" >&2 + exit 1 + fi + if ! unzip -Z1 "${target_jar}" | grep -Fx "${entry}" >/dev/null; then + echo "ERROR: missing JNI entry ${entry} in ${target_jar}" >&2 + exit 1 + fi + + archive=$(lance_jni_download "${thirdparty_dir}/installed/lance-jni") Review Comment: **[P1] Include the JNI binary's third-party license inventory** The Doris third-party release publishes `RUST_THIRD_PARTY_LICENSES.html` alongside this separately rebuilt native library, but this packaging path downloads only the `.so.gz`. The FE output later receives the repository's `dist/LICENSE-dist.txt` and `dist/licenses`, which do not identify Lance Java/JNI 11 (their only Lance Rust inventory is for an older BE `lance-c` component), and `lance-core-11.0.0.jar` itself contains no license/notice entries. The binary distribution would therefore ship the v11 Rust code without its exact third-party notices. Please add the reviewed v11 inventory under `dist/licenses`, reference it from the FE section of `LICENSE-dist.txt`, and propagate any required NOTICE material. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
