luofeng1994 commented on issue #10360: URL: https://github.com/apache/incubator-gluten/issues/10360#issuecomment-3182473015
> [@luofeng1994](https://github.com/luofeng1994) You need to enable hdfs in your compile command. Could you help try following command? > > `./dev/builddeps-veloxbe.sh --build_arrow=OFF --run_setup_script=OFF --enable_vcpkg=ON --enable_hdfs=ON` I tried this command, but is strange. When I ran this command, the build process ended after packaging the gluten jar. I got the new gluten jar at package/target. `./dev/builddeps-veloxbe.sh --build_arrow=OFF --run_setup_script=OFF --enable_vcpkg=ON` But when I ran with the parameter --enable_hdfs=ON, the build process ended after build the libgluten.so libvelox.so, without any error. The gluten jar at package/target not updated. `./dev/builddeps-veloxbe.sh --build_arrow=OFF --run_setup_script=OFF --enable_vcpkg=ON --enable_hdfs=ON` the log ended like this ` + build_gluten_cpp + echo 'Start to build Gluten CPP' Start to build Gluten CPP + cd /root/luof/incubator-gluten/dev/../cpp + rm -rf build + mkdir build + cd build + GLUTEN_CMAKE_OPTIONS='-DBUILD_VELOX_BACKEND=ON -DCMAKE_BUILD_TYPE=Release -DVELOX_HOME=/root/luof/incubator-gluten/dev/../ep/build-velox/build/velox_ep -DBUILD_TESTS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_BENCHMARKS=OFF -DENABLE_JEMALLOC_STATS=OFF -DENABLE_HBM=OFF -DENABLE_QAT=OFF -DENABLE_IAA=OFF -DENABLE_GCS=OFF -DENABLE_S3=OFF -DENABLE_HDFS=ON -DENABLE_ABFS=OFF -DENABLE_GPU=OFF -DENABLE_ENHANCED_FEATURES=OFF' + '[' Linux == Darwin ']' + cmake -DBUILD_VELOX_BACKEND=ON -DCMAKE_BUILD_TYPE=Release -DVELOX_HOME=/root/luof/incubator-gluten/dev/../ep/build-velox/build/velox_ep -DBUILD_TESTS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_BENCHMARKS=OFF -DENABLE_JEMALLOC_STATS=OFF -DENABLE_HBM=OFF -DENABLE_QAT=OFF -DENABLE_IAA=OFF -DENABLE_GCS=OFF -DENABLE_S3=OFF -DENABLE_HDFS=ON -DENABLE_ABFS=OFF -DENABLE_GPU=OFF -DENABLE_ENHANCED_FEATURES=OFF .. -- Building using CMake version: 3.28.3 -- The C compiler identification is GNU 11.1.0 -- The CXX compiler identification is GNU 11.1.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Add definition NDEBUG -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Check if compiler accepts -pthread -- Check if compiler accepts -pthread - yes -- Found Threads: TRUE -- Found JNI: /usr/jdk64/jdk1.8.0_461/include found components: AWT JVM -- Found glog: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libglog.a -- Found PkgConfig: /usr/bin/pkg-config (found version "1.7.3") Core module final CMAKE_CXX_FLAGS= -Wno-class-memaccess -Wno-error=unused-but-set-variable -Wall -Wno-sign-compare -Wno-comment -Werror -Wno-error=parentheses -Wno-error=unused-function -Wno-error=unused-variable -Wno-strict-aliasing -Wno-ignored-qualifiers -Wno-deprecated-declarations -Wno-attributes -march=armv8-a+crc+crypto -- Found Boost: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/include (found version "1.84.0") -- Set Gluten Proto Directory in /root/luof/incubator-gluten/gluten-core/src/main/resources/org/apache/gluten/proto -- Set Substrait Proto Directory in /root/luof/incubator-gluten/gluten-substrait/src/main/resources/substrait/proto -- Found Arrow library: /usr/local/lib64/libarrow.a -- Found Arrow library: /usr/local/lib64/libarrow_bundled_dependencies.a -- Found Protobuf: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libprotobuf.a;-pthread (found version "3.21.12") -- Found Protobuf: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libprotobuf.a Velox module final CMAKE_CXX_FLAGS= -Wno-class-memaccess -Wno-error=unused-but-set-variable -Wall -Wno-sign-compare -Wno-comment -Werror -Wno-error=parentheses -Wno-error=unused-function -Wno-error=unused-variable -Wno-strict-aliasing -Wno-ignored-qualifiers -Wno-deprecated-declarations -Wno-attributes -march=armv8-a+crc+crypto -- Set Gluten Proto Directory in /root/luof/incubator-gluten/backends-velox/src/main/resources/org/apache/gluten/proto -- Found Protobuf: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libprotobuf.a -- Found libevent include directory: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/include -- Found libevent component: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libevent_core.a -- Found libevent component: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libevent_extra.a -- Found libevent component: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libevent_pthreads.a -- Found libevent 2.2.0 in /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon -- Found ZLIB: optimized;/root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libz.a;debug;/root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/debug/lib/libz.a (found version "1.2.13") -- Found OpenSSL: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libcrypto.a (found version "3.0.8") -- Found Boost: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/include (found version "1.84.0") found components: context filesystem program_options regex system thread chrono atomic -- Found folly: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon -- RE2 Library Can Be Found in re2::re2 -- Found stemmer: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libstemmer.a -- Found the following ICU libraries: -- i18n (required): /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libicui18n.a -- uc (required): /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libicuuc.a -- data (required): /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libicudata.a -- Found ICU: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/include (found version "72.1") -- Configuring done (1.4s) -- Generating done (0.2s) -- Build files have been written to: /root/luof/incubator-gluten/cpp/build + make -j 2 [ 1%] Running Gluten Velox PROTO compiler [ 2%] Running Substrait PROTO compiler [ 2%] Built target velox_jni_proto [ 3%] Running Gluten PROTO compiler [ 3%] Built target jni_proto [ 4%] Building CXX object core/CMakeFiles/gluten.dir/proto/substrait/algebra.pb.cc.o [ 5%] Building CXX object core/CMakeFiles/gluten.dir/proto/substrait/capabilities.pb.cc.o [ 6%] Building CXX object core/CMakeFiles/gluten.dir/proto/substrait/extended_expression.pb.cc.o …… [ 97%] Building CXX object velox/CMakeFiles/velox.dir/utils/VeloxWholeStageDumper.cc.o [ 98%] Linking CXX shared library ../releases/libvelox.so [100%] Built target velox [root@slave2 incubator-gluten]# ` How can I complete the gluten packaging after adding --enable_hdfs=ON ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
