morningman opened a new pull request, #66783: URL: https://github.com/apache/doris/pull/66783
## Proposed changes Two commits, both small. ### 1. `HADOOP_LIBS_3_4` → `hadoop-3.4.2.3-for-doris` Picks up [apache/doris-thirdparty#407](https://github.com/apache/doris-thirdparty/pull/407), which makes hadoop's native build work on macOS. Until now `build_hadoop_libs_3_4` was Linux-only (`build-thirdparty.sh` adds it to the package list under `uname -s == Linux`), because the build could not complete on a Mac at all. Two things stopped it, both platform differences rather than defects: - **`find_package(ZLIB REQUIRED)` could not succeed.** `hadoop_set_find_shared_library_version("1")` pins the search to `libz.1.dylib`, which modern macOS keeps only in the dyld shared cache — the SDK carries a `libz.1.tbd` stub instead — so configure aborted with `Could NOT find ZLIB (missing: ZLIB_LIBRARY)`. zlib is never linked into libhadoop; `ZlibCompressor.c` `dlopen()`s it by basename at run time, and `dlopen("libz.1.dylib")` does resolve against the shared cache. The lookup now takes the plain search for headers and names the runtime library outright. - **`libhdfspp.dylib` failed to link**, leaving 22 `gss_*` symbols undefined through a static libsasl2. Linux' `ld` tolerates undefined symbols in a `.so`; macOS' `ld` does not, and the failure took the whole `make` down with it — `libhdfs.a` included. SASL is used by libhdfspp alone, so it is turned off there. Both fixes sit inside `if(APPLE)` / `if(NOT APPLE)` branches in the fork, so the Linux build is byte for byte unchanged and produces the same artifacts. ### 2. macOS thirdparty CI jobs: `openjdk@11` → `openjdk@17` Doris builds on JDK 17, and hadoop 3.4.2 — the only thirdparty package whose build needs a JDK — compiles cleanly against it. Declaring 11 no longer reflects anything we build or test against. The Linux job stays on openjdk-8, which it also pins through `JAVA_HOME`. ## Verification The new tag was built on macOS 26.5 / arm64 (llvm@20, JDK 17), driven through `thirdparty/build-thirdparty.sh hadoop_libs_3_4` with dependencies resolved from a Doris thirdparty prefix — no extra Homebrew packages. Tested with both cmake 3.25.3 (the version this workflow pins) and cmake 4.4. `build-thirdparty.sh hadoop_libs_3_4` exits 0 and installs: ``` installed/lib/hadoop_hdfs_3_4/native/ libhdfs.a 1.4M arm64 libhadoop.dylib 160K arm64 libhdfs.dylib 135K arm64 libhdfspp.dylib 8.9M arm64 installed/include/hadoop_hdfs_3_4/ hdfs.h installed/lib/hadoop_hdfs_3_4/ common/ (5 jars) hdfs/ (10 jars) ``` `libhdfs.a` exports everything BE's `USE_HADOOP_HDFS` / `USE_DORIS_HADOOP_HDFS` paths call, including the fork's `hdfsSetLogger`, plus `hdfsGetLastExceptionRootCause`, `hdfsBuilderSetKerb5Conf`, `hdfsBuilderSetKeyTabFile`, `hdfsUnbufferFile`, `hdfsFileGetReadStatistics`, `hdfsHSync` and `hdfsHFlush`. The tarball MD5 was checked across three separate downloads of the GitHub tag archive and is stable. ## Scope This PR only bumps the source and the declared CI JDK. It deliberately does **not** add `hadoop_libs_3_4` to the macOS branch of `build-thirdparty.sh`'s package list, nor touch `be/CMakeLists.txt`, which still selects libhdfs3 on macOS. Making the BE actually link hadoop's libhdfs there is a separate change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
