This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch hadoop-3.4.2
in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git
The following commit(s) were added to refs/heads/hadoop-3.4.2 by this push:
new 4f4b5fddd08 [fix](build) make the native build work on macOS (#407)
4f4b5fddd08 is described below
commit 4f4b5fddd08bc7b3ad56c2187e27d7774be8a623
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Aug 14 19:56:53 2026 +0800
[fix](build) make the native build work on macOS (#407)
Two things stop `mvn -Pnative` short on macOS, both platform differences
rather than anything wrong with the tree:
- hadoop-common asks find_package(ZLIB REQUIRED) for a versioned dylib
(hadoop_set_find_shared_library_version "1" -> libz.1.dylib). Modern
macOS keeps no such file on disk - libz lives in the dyld shared cache
and the SDK carries only a libz.1.tbd stub - so the REQUIRED lookup
fails and configure aborts. zlib is never linked into libhadoop: only
its headers are used, and ZlibCompressor.c dlopen()s it by basename at
run time, where dlopen("libz.1.dylib") does resolve against the shared
cache. So let the plain search supply the headers and name the runtime
library outright.
- libhdfspp's shared library links a static libsasl2 whose GSSAPI plugin
leaves gss_* undefined. Linux' ld tolerates undefined symbols in a .so;
macOS' ld does not, so libhdfspp.dylib fails to link and takes the whole
make down with it, libhdfs.a included. SASL is used by libhdfspp alone,
so turn it off through the escape hatch libhdfspp/CMakeLists.txt already
documents. Skipping the libhdfspp subdirectory outright is not an
option - libhdfs pulls $<TARGET_OBJECTS:x_platform_obj> out of that tree.
Both hunks are guarded by if(APPLE); the Linux path is byte for byte
unchanged.
Verified on macOS 26.5 / arm64 (llvm@20, JDK 17, cmake 3.25.3 and 4.4):
build.sh completes and hadoop-dist/target/hadoop-libhdfs-3.4.2 carries an
arm64 native/libhdfs.a exporting the full hdfs* API, alongside
libhadoop.dylib, libhdfs.dylib and libhdfspp.dylib.
Co-authored-by: morningman <[email protected]>
Co-authored-by: Claude Fable 5 <[email protected]>
---
hadoop-common-project/hadoop-common/src/CMakeLists.txt | 16 ++++++++++++++--
.../hadoop-hdfs-native-client/src/CMakeLists.txt | 9 +++++++++
2 files changed, 23 insertions(+), 2 deletions(-)
diff --git a/hadoop-common-project/hadoop-common/src/CMakeLists.txt
b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
index a93082091ec..2b3318c053d 100644
--- a/hadoop-common-project/hadoop-common/src/CMakeLists.txt
+++ b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
@@ -45,10 +45,22 @@ include(HadoopJNI)
# Require zlib.
set(STORED_CMAKE_FIND_LIBRARY_SUFFIXES ${CMAKE_FIND_LIBRARY_SUFFIXES})
-hadoop_set_find_shared_library_version("1")
+if(NOT APPLE)
+ hadoop_set_find_shared_library_version("1")
+endif()
find_package(ZLIB REQUIRED)
set(CMAKE_FIND_LIBRARY_SUFFIXES ${STORED_CMAKE_FIND_LIBRARY_SUFFIXES})
-get_filename_component(HADOOP_ZLIB_LIBRARY ${ZLIB_LIBRARIES} NAME)
+if(APPLE)
+ # macOS keeps no libz.1.dylib on disk - it lives in the dyld shared cache,
and the SDK
+ # carries only a libz.1.tbd stub - so searching for a versioned dylib
above finds
+ # nothing and the REQUIRED find_package aborts the build. zlib is never
linked into
+ # libhadoop (only its headers are used); ZlibCompressor.c dlopen()s it by
basename at
+ # run time, and dlopen("libz.1.dylib") does resolve against the shared
cache. So let
+ # the plain search supply the headers and name the runtime library
outright.
+ set(HADOOP_ZLIB_LIBRARY "libz.1.dylib")
+else()
+ get_filename_component(HADOOP_ZLIB_LIBRARY ${ZLIB_LIBRARIES} NAME)
+endif()
# Look for bzip2.
set(STORED_CMAKE_FIND_LIBRARY_SUFFIXES ${CMAKE_FIND_LIBRARY_SUFFIXES})
diff --git a/hadoop-hdfs-project/hadoop-hdfs-native-client/src/CMakeLists.txt
b/hadoop-hdfs-project/hadoop-hdfs-native-client/src/CMakeLists.txt
index 06eb916bcaf..18718890bb9 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-native-client/src/CMakeLists.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs-native-client/src/CMakeLists.txt
@@ -203,6 +203,15 @@ check_cxx_source_compiles(
return 0;
}"
THREAD_LOCAL_SUPPORTED)
+if (APPLE)
+ # libhdfspp's shared library links the static libsasl2 found under the
Doris thirdparty
+ # prefix, whose GSSAPI plugin leaves gss_* undefined. Linux' ld tolerates
undefined
+ # symbols in a .so; macOS' ld does not, so the dylib fails to link and
takes the whole
+ # make down with it - libhdfs.a included. SASL is used by libhdfspp alone,
which Doris
+ # never links (be/CMakeLists.txt sets BUILD_LIBHDFSPP OFF), so turn it off
through the
+ # escape hatch libhdfspp/CMakeLists.txt documents.
+ set(NO_SASL 1)
+endif()
if (THREAD_LOCAL_SUPPORTED)
add_subdirectory(main/native/libhdfspp)
else()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
