Hello, We recently migrated our C++ Arrow code from 0.16 to 3.0.0. The code works fine on Ubuntu, but we get a segmentation fault in CentOS while reading Arrow Record Batch files. We can successfully read the files from Python or Ubuntu so the files and the writer are fine.
We use Record Batch Stream Reader/Writer to read/write data to files. Sometimes we use GZIP to compress the streams. The migration to 3.0.0 was pretty straight forward with minimal changes to the code https://github.com/Paradigm4/bridge/commit/03e896e84230ddb41bfef68cde5ed9b21192a0e9 We have an extensive test suite and all is good on Ubuntu. On CentOS the write works OK but we get a segmentation fault during reading from C++. We can successfully read the files using PyArrow. Moreover, the files written by CentOS can be successfully read from C++ in Ubuntu. Here is the backtrace I got form gdb when the segmentation fault occurred: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f548c7fb700 (LWP 2649)] 0x00007f545c003340 in ?? () (gdb) bt #0 0x00007f545c003340 in ?? () #1 0x00007f54903377ce in arrow::ipc::ArrayLoader::GetBuffer(int, std::shared_ptr<arrow::Buffer>*) () from /lib64/libarrow.so.300 #2 0x00007f549034006c in arrow::Status arrow::VisitTypeInline<arrow::ipc::ArrayLoader>(arrow::DataType const&, arrow::ipc::ArrayLoader*) () from /lib64/libarrow.so.300 #3 0x00007f5490340db4 in arrow::ipc::ArrayLoader::Load(arrow::Field const*, arrow::ArrayData*) () from /lib64/libarrow.so.300 #4 0x00007f5490318b5b in arrow::ipc::LoadRecordBatchSubset(org::apache::arrow::flatbuf::RecordBatch const*, std::shared_ptr<arrow::Schema> const&, std::vector<bool, std::allocator<bool> > const*, arrow::ipc::DictionaryMemo const*, arrow::ipc::IpcReadOptions const&, arrow::ipc::MetadataVersion, arrow::Compression::type, arrow::io::RandomAccessFile*) () from /lib64/libarrow.so.300 #5 0x00007f549031952a in arrow::ipc::LoadRecordBatch(org::apache::arrow::flatbuf::RecordBatch const*, std::shared_ptr<arrow::Schema> const&, std::vector<bool, std::allocator<bool> > const&, arrow::ipc::DictionaryMemo const*, arrow::ipc::IpcReadOptions const&, arrow::ipc::MetadataVersion, arrow::Compression::type, arrow::io::RandomAccessFile*) () from /lib64/libarrow.so.300 #6 0x00007f54903197ce in arrow::ipc::ReadRecordBatchInternal(arrow::Buffer const&, std::shared_ptr<arrow::Schema> const&, std::vector<bool, std::allocator<bool> > const&, arrow::ipc::DictionaryMemo const*, arrow::ipc::IpcReadOptions const&, arrow::io::RandomAccessFile*) () from /lib64/libarrow.so.300 #7 0x00007f5490345d9c in arrow::ipc::RecordBatchStreamReaderImpl::ReadNext(std::shared_ptr<arrow::RecordBatch>*) () from /lib64/libarrow.so.300 #8 0x00007f549109b479 in scidb::ArrowReader::readObject (this=this@entry=0x7f548c7f7d80, name="index/0", reuse=reuse@entry=true, arrowBatch=std::shared_ptr (empty) 0x0) at XIndex.cpp:104 #9 0x00007f549109cb0a in scidb::XIndex::load (this=this@entry=0x7f545c003ab0, driver=std::shared_ptr (count 3, weak 0) 0x7f545c003e70, query=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<scidb::Query, std::allocator<scidb::Query>, (__gnu_cxx::_Lock_policy)2>' warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<scidb::Query, std::allocator<scidb::Query>, (__gnu_cxx::_Lock_policy)2>' std::shared_ptr (count 7, weak 7) 0x7f546c005330) at XIndex.cpp:286 I also tried Arrow 4.0.0. The code compiled just fine and the behavior was the same, with the same backtrace. The code where the segmentation fault occurs is trying to read a GZIP compressed Record Batch Stream. The file is 144 bytes and has only one column with three int64 values. > file 0 0: gzip compressed data, from Unix > stat 0 File: ‘0’ Size: 144 Blocks: 8 IO Block: 4096 regular file Device: 10302h/66306d Inode: 33715444 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1001/ scidb) Gid: ( 1001/ scidb) Context: unconfined_u:object_r:user_tmp_t:s0 Access: 2021-06-08 04:42:28.653548604 +0000 Modify: 2021-06-08 04:14:14.638927052 +0000 Change: 2021-06-08 04:40:50.221279208 +0000 Birth: - In [29]: s = pyarrow.input_stream('/tmp/bridge/foo/index/0', compression='gzip') In [30]: b = pyarrow.RecordBatchStreamReader(s) In [31]: t = b.read_all() In [32]: t.columns Out[32]: [<pyarrow.lib.ChunkedArray object at 0x7fefb5a552b0> [ [ 0, 5, 10 ] ]] I removed the GZIP compression in both the writer and the reader but the issue persists. So I don't think it is because of the compression. Here is the ldd on the library file which contains the reader and writers that use the Arrow library. It is built on a CentOS 7 with the g++ 4.9.2 compiler. > ldd libbridge.so linux-vdso.so.1 => (0x00007fffe4f10000) libarrow.so.300 => /lib64/libarrow.so.300 (0x00007f8a38908000) libaws-cpp-sdk-s3.so => /opt/aws/lib64/libaws-cpp-sdk-s3.so (0x00007f8a384b3000) libm.so.6 => /lib64/libm.so.6 (0x00007f8a381b1000) librt.so.1 => /lib64/librt.so.1 (0x00007f8a37fa9000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f8a37da5000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f8a37a9e000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8a37888000) libc.so.6 => /lib64/libc.so.6 (0x00007f8a374ba000) libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f8a37057000) libssl.so.10 => /lib64/libssl.so.10 (0x00007f8a36de5000) libbrotlienc.so.1 => /lib64/libbrotlienc.so.1 (0x00007f8a36b58000) libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f8a3694b000) libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f8a3672b000) libutf8proc.so.1 => /lib64/libutf8proc.so.1 (0x00007f8a3647b000) libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f8a3626b000) liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f8a3605c000) libsnappy.so.1 => /lib64/libsnappy.so.1 (0x00007f8a35e56000) libz.so.1 => /lib64/libz.so.1 (0x00007f8a35c40000) libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f8a3593a000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a3571e000) /lib64/ld-linux-x86-64.so.2 (0x00007f8a39b67000) libaws-cpp-sdk-core.so => /opt/aws/lib64/libaws-cpp-sdk-core.so (0x00007f8a35413000) libaws-c-event-stream.so.0unstable => /opt/aws/lib64/libaws-c-event-stream.so.0unstable (0x00007f8a3520b000) libaws-c-common.so.0unstable => /opt/aws/lib64/libaws-c-common.so.0unstable (0x00007f8a34fd9000) libaws-checksums.so => /opt/aws/lib64/libaws-checksums.so (0x00007f8a34dce000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f8a34b81000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f8a34898000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f8a34694000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f8a34461000) libcurl.so.4 => /opt/curl/lib/libcurl.so.4 (0x00007f8a341ea000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f8a33fda000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f8a33dd6000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f8a33bbc000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8a33995000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f8a33733000) > /opt/rh/devtoolset-3/root/usr/bin/g++ --version g++ (GCC) 4.9.2 20150212 (Red Hat 4.9.2-6) Do all of these ring any bells? Thank you! Rares
