Yes, the pre-built binaries are the official RPM packages.
I recompilled 4.0.1 with the default gcc-g++ from CentOS 7 and Debug flag.
The segmentation fault occurred. See below for the backtrace.
Please note that the SciDB database as well as the Plug-in where the Arrow
library is used are compiled with g++ from devtoolset-3. Maybe this problem
is due to the different versions of the g++ compiler being used...
Also note that the code path that writes Arrow files work fine, it is just
the path that reads the files that breaks.
> g++ --version
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fae877fe700 (LWP 16783)]
0x00007fae8eb1e000 in ?? ()
(gdb) bt
#0 0x00007fae8eb1e000 in ?? ()
#1 0x00007fae906bd4d0 in arrow::ipc::ArrayLoader::ReadBuffer
(this=0x7fae877fa090, offset=0, length=24, out=0x7fae5c004010) at
/apache-arrow-4.0.1/cpp/src/arrow/ipc/reader.cc:163
#2 0x00007fae906bd7b8 in arrow::ipc::ArrayLoader::GetBuffer
(this=0x7fae877fa090, buffer_index=1, out=0x7fae5c004010) at
/apache-arrow-4.0.1/cpp/src/arrow/ipc/reader.cc:199
#3 0x00007fae906cbfa7 in
arrow::ipc::ArrayLoader::LoadPrimitive<arrow::Int64Type>
(this=0x7fae877fa090, type_id=arrow::Type::INT64) at
/apache-arrow-4.0.1/cpp/src/arrow/ipc/reader.cc:241
#4 0x00007fae906c72c7 in arrow::ipc::ArrayLoader::Visit<arrow::Int64Type>
(this=0x7fae877fa090, type=...) at
/apache-arrow-4.0.1/cpp/src/arrow/ipc/reader.cc:300
#5 0x00007fae906c2bbc in arrow::VisitTypeInline<arrow::ipc::ArrayLoader>
(type=..., visitor=0x7fae877fa090) at
/apache-arrow-4.0.1/cpp/src/arrow/visitor_inline.h:89
#6 0x00007fae906bd545 in arrow::ipc::ArrayLoader::LoadType
(this=0x7fae877fa090, type=...) at
/apache-arrow-4.0.1/cpp/src/arrow/ipc/reader.cc:166
#7 0x00007fae906bd5f0 in arrow::ipc::ArrayLoader::Load
(this=0x7fae877fa090, field=0x7fae5c004e38, out=0x7fae5c003f88) at
/apache-arrow-4.0.1/cpp/src/arrow/ipc/reader.cc:176
#8 0x00007fae906b1a92 in arrow::ipc::LoadRecordBatchSubset
(metadata=0x7fae8ea140f4, schema=std::shared_ptr (count 2, weak 0)
0x7fae5c004ea8, inclusion_mask=0x0, context=..., file=0x7fae5c003e50)
at /apache-arrow-4.0.1/cpp/src/arrow/ipc/reader.cc:481
#9 0x00007fae906b24e7 in arrow::ipc::LoadRecordBatch
(metadata=0x7fae8ea140f4, schema=std::shared_ptr (count 2, weak 0)
0x7fae5c004ea8, inclusion_mask=std::vector<bool> of length 0, capacity 0,
context=..., file=0x7fae5c003e50)
at /apache-arrow-4.0.1/cpp/src/arrow/ipc/reader.cc:532
#10 0x00007fae906b35f3 in arrow::ipc::ReadRecordBatchInternal
(metadata=..., schema=std::shared_ptr (count 2, weak 0) 0x7fae5c004ea8,
inclusion_mask=std::vector<bool> of length 0, capacity 0, context=...,
file=0x7fae5c003e50)
at /apache-arrow-4.0.1/cpp/src/arrow/ipc/reader.cc:630
#11 0x00007fae906bee31 in arrow::ipc::RecordBatchStreamReaderImpl::ReadNext
(this=0x7fae5c007508, batch=0x7fae877face0) at
/apache-arrow-4.0.1/cpp/src/arrow/ipc/reader.cc:837
#12 0x00007fae912b7349 in scidb::ArrowReader::readObject
(this=this@entry=0x7fae877fad80,
name="index/0", reuse=reuse@entry=true, arrowBatch=std::shared_ptr (empty)
0x0) at XIndex.cpp:104
#13 0x00007fae912b89da in scidb::XIndex::load (this=this@entry=0x7fae5c000c00,
driver=std::shared_ptr (count 3, weak 0) 0x7fae5c003d50, query=warning:
RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<scidb::Query,
std::allocator<scidb::Query>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class
'std::_Sp_counted_ptr_inplace<scidb::Query, std::allocator<scidb::Query>,
(__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 7, weak 7) 0x7fae680022d0) at XIndex.cpp:284
The plug-in code (i.e., XIndex.cpp) is from here
https://github.com/Paradigm4/bridge/tree/arrow3
Thanks!
Rares
On Wed, Jun 9, 2021 at 9:53 PM Sutou Kouhei <k...@clear-code.com> wrote:
> Hi,
>
> > Then I went back to the pre-built binaries for 3.0.0 and 4.0.0 from JFrog
> > and the issue reappeared. I can only infer that it has to do with the way
> > the pre-built binaries are generated...
>
> The pre-built binaries are the official RPM packages, right?
>
> They are built with the default gcc-g++ package not g++ from
> devtoolset-3. This may be related. Could you try building
> your program with the default gcc-g++ package?
>
>
> Thanks,
> --
> kou
>
> In <calq9kxaxnyayqohuj3n0cknrbp6wbtxvj2pog7hcb0icy2r...@mail.gmail.com>
> "Re: C++ Segmentation Fault RecordBatchReader::ReadNext in CentOS only"
> on Wed, 9 Jun 2021 21:39:04 -0700,
> Rares Vernica <rvern...@gmail.com> wrote:
>
> > I got the apache-arrow-4.0.1 source and compiled it with the Debug flag.
> No
> > segmentation fault occurred. I then removed the Debug flag and still no
> > segmentation fault. I then tried the 4.0.0 source. Still no issues.
> > Finally, I tried the 3.0.0 source and still no issues.
> >
> > Then I went back to the pre-built binaries for 3.0.0 and 4.0.0 from JFrog
> > and the issue reappeared. I can only infer that it has to do with the way
> > the pre-built binaries are generated...
> >
> > Here is how I compiled the Arrow sources on my CentOS 7.
> >
> > release$ cmake3 -DARROW_WITH_ZLIB=ON
> > -DCMAKE_C_COMPILER=/opt/rh/devtoolset-3/root/usr/bin/gcc
> > -DCMAKE_CXX_COMPILER=/opt/rh/devtoolset-3/root/usr/bin/g++ ..
> >
> > Thanks,
> > Rares
> >
> > On Tue, Jun 8, 2021 at 5:37 PM Sutou Kouhei <k...@clear-code.com> wrote:
> >
> >> Hi,
> >>
> >> Could you try building Apache Arrow C++ with
> >> -DCMAKE_BUILD_TYPE=Debug and get backtrace again? It will
> >> show the source location on segmentation fault.
> >>
> >> Thanks,
> >> --
> >> kou
> >>
> >> In <calq9kxa8sh07shuckhka9fuzu2n87tbydlp--aahgcwkfwo...@mail.gmail.com>
> >> "C++ Segmentation Fault RecordBatchReader::ReadNext in CentOS only" on
> >> Tue, 8 Jun 2021 12:01:27 -0700,
> >> Rares Vernica <rvern...@gmail.com> wrote:
> >>
> >> > Hello,
> >> >
> >> > We recently migrated our C++ Arrow code from 0.16 to 3.0.0. The code
> >> works
> >> > fine on Ubuntu, but we get a segmentation fault in CentOS while
> reading
> >> > Arrow Record Batch files. We can successfully read the files from
> Python
> >> or
> >> > Ubuntu so the files and the writer are fine.
> >> >
> >> > We use Record Batch Stream Reader/Writer to read/write data to files.
> >> > Sometimes we use GZIP to compress the streams. The migration to 3.0.0
> was
> >> > pretty straight forward with minimal changes to the code
> >> >
> >>
> https://github.com/Paradigm4/bridge/commit/03e896e84230ddb41bfef68cde5ed9b21192a0e9
> >> > We have an extensive test suite and all is good on Ubuntu. On CentOS
> the
> >> > write works OK but we get a segmentation fault during reading from
> C++.
> >> We
> >> > can successfully read the files using PyArrow. Moreover, the files
> >> written
> >> > by CentOS can be successfully read from C++ in Ubuntu.
> >> >
> >> > Here is the backtrace I got form gdb when the segmentation fault
> >> occurred:
> >> >
> >> > Program received signal SIGSEGV, Segmentation fault.
> >> > [Switching to Thread 0x7f548c7fb700 (LWP 2649)]
> >> > 0x00007f545c003340 in ?? ()
> >> > (gdb) bt
> >> > #0 0x00007f545c003340 in ?? ()
> >> > #1 0x00007f54903377ce in arrow::ipc::ArrayLoader::GetBuffer(int,
> >> > std::shared_ptr<arrow::Buffer>*) () from /lib64/libarrow.so.300
> >> > #2 0x00007f549034006c in arrow::Status
> >> > arrow::VisitTypeInline<arrow::ipc::ArrayLoader>(arrow::DataType
> const&,
> >> > arrow::ipc::ArrayLoader*) () from /lib64/libarrow.so.300
> >> > #3 0x00007f5490340db4 in arrow::ipc::ArrayLoader::Load(arrow::Field
> >> > const*, arrow::ArrayData*) () from /lib64/libarrow.so.300
> >> > #4 0x00007f5490318b5b in
> >> >
> >>
> arrow::ipc::LoadRecordBatchSubset(org::apache::arrow::flatbuf::RecordBatch
> >> > const*, std::shared_ptr<arrow::Schema> const&, std::vector<bool,
> >> > std::allocator<bool> > const*, arrow::ipc::DictionaryMemo const*,
> >> > arrow::ipc::IpcReadOptions const&, arrow::ipc::MetadataVersion,
> >> > arrow::Compression::type, arrow::io::RandomAccessFile*) () from
> >> > /lib64/libarrow.so.300
> >> > #5 0x00007f549031952a in
> >> > arrow::ipc::LoadRecordBatch(org::apache::arrow::flatbuf::RecordBatch
> >> > const*, std::shared_ptr<arrow::Schema> const&, std::vector<bool,
> >> > std::allocator<bool> > const&, arrow::ipc::DictionaryMemo const*,
> >> > arrow::ipc::IpcReadOptions const&, arrow::ipc::MetadataVersion,
> >> > arrow::Compression::type, arrow::io::RandomAccessFile*) () from
> >> > /lib64/libarrow.so.300
> >> > #6 0x00007f54903197ce in
> >> arrow::ipc::ReadRecordBatchInternal(arrow::Buffer
> >> > const&, std::shared_ptr<arrow::Schema> const&, std::vector<bool,
> >> > std::allocator<bool> > const&, arrow::ipc::DictionaryMemo const*,
> >> > arrow::ipc::IpcReadOptions const&, arrow::io::RandomAccessFile*) ()
> from
> >> > /lib64/libarrow.so.300
> >> > #7 0x00007f5490345d9c in
> >> >
> >>
> arrow::ipc::RecordBatchStreamReaderImpl::ReadNext(std::shared_ptr<arrow::RecordBatch>*)
> >> > () from /lib64/libarrow.so.300
> >> > #8 0x00007f549109b479 in scidb::ArrowReader::readObject
> >> > (this=this@entry=0x7f548c7f7d80,
> >> > name="index/0", reuse=reuse@entry=true, arrowBatch=std::shared_ptr
> >> (empty)
> >> > 0x0) at XIndex.cpp:104
> >> > #9 0x00007f549109cb0a in scidb::XIndex::load (this=this@entry
> >> =0x7f545c003ab0,
> >> > driver=std::shared_ptr (count 3, weak 0) 0x7f545c003e70,
> query=warning:
> >> > RTTI symbol not found for class
> >> 'std::_Sp_counted_ptr_inplace<scidb::Query,
> >> > std::allocator<scidb::Query>, (__gnu_cxx::_Lock_policy)2>'
> >> > warning: RTTI symbol not found for class
> >> > 'std::_Sp_counted_ptr_inplace<scidb::Query,
> std::allocator<scidb::Query>,
> >> > (__gnu_cxx::_Lock_policy)2>'
> >> > std::shared_ptr (count 7, weak 7) 0x7f546c005330) at XIndex.cpp:286
> >> >
> >> > I also tried Arrow 4.0.0. The code compiled just fine and the behavior
> >> was
> >> > the same, with the same backtrace.
> >> >
> >> > The code where the segmentation fault occurs is trying to read a GZIP
> >> > compressed Record Batch Stream. The file is 144 bytes and has only one
> >> > column with three int64 values.
> >> >
> >> >> file 0
> >> > 0: gzip compressed data, from Unix
> >> >
> >> >> stat 0
> >> > File: ‘0’
> >> > Size: 144 Blocks: 8 IO Block: 4096 regular file
> >> > Device: 10302h/66306d Inode: 33715444 Links: 1
> >> > Access: (0644/-rw-r--r--) Uid: ( 1001/ scidb) Gid: ( 1001/
> scidb)
> >> > Context: unconfined_u:object_r:user_tmp_t:s0
> >> > Access: 2021-06-08 04:42:28.653548604 +0000
> >> > Modify: 2021-06-08 04:14:14.638927052 +0000
> >> > Change: 2021-06-08 04:40:50.221279208 +0000
> >> > Birth: -
> >> >
> >> > In [29]: s = pyarrow.input_stream('/tmp/bridge/foo/index/0',
> >> > compression='gzip')
> >> > In [30]: b = pyarrow.RecordBatchStreamReader(s)
> >> > In [31]: t = b.read_all()
> >> > In [32]: t.columns
> >> > Out[32]:
> >> > [<pyarrow.lib.ChunkedArray object at 0x7fefb5a552b0>
> >> > [
> >> > [
> >> > 0,
> >> > 5,
> >> > 10
> >> > ]
> >> > ]]
> >> >
> >> > I removed the GZIP compression in both the writer and the reader but
> the
> >> > issue persists. So I don't think it is because of the compression.
> >> >
> >> > Here is the ldd on the library file which contains the reader and
> writers
> >> > that use the Arrow library. It is built on a CentOS 7 with the g++
> 4.9.2
> >> > compiler.
> >> >
> >> >> ldd libbridge.so
> >> > linux-vdso.so.1 => (0x00007fffe4f10000)
> >> > libarrow.so.300 => /lib64/libarrow.so.300 (0x00007f8a38908000)
> >> > libaws-cpp-sdk-s3.so => /opt/aws/lib64/libaws-cpp-sdk-s3.so
> >> > (0x00007f8a384b3000)
> >> > libm.so.6 => /lib64/libm.so.6 (0x00007f8a381b1000)
> >> > librt.so.1 => /lib64/librt.so.1 (0x00007f8a37fa9000)
> >> > libdl.so.2 => /lib64/libdl.so.2 (0x00007f8a37da5000)
> >> > libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f8a37a9e000)
> >> > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8a37888000)
> >> > libc.so.6 => /lib64/libc.so.6 (0x00007f8a374ba000)
> >> > libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f8a37057000)
> >> > libssl.so.10 => /lib64/libssl.so.10 (0x00007f8a36de5000)
> >> > libbrotlienc.so.1 => /lib64/libbrotlienc.so.1 (0x00007f8a36b58000)
> >> > libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f8a3694b000)
> >> > libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1
> (0x00007f8a3672b000)
> >> > libutf8proc.so.1 => /lib64/libutf8proc.so.1 (0x00007f8a3647b000)
> >> > libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f8a3626b000)
> >> > liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f8a3605c000)
> >> > libsnappy.so.1 => /lib64/libsnappy.so.1 (0x00007f8a35e56000)
> >> > libz.so.1 => /lib64/libz.so.1 (0x00007f8a35c40000)
> >> > libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f8a3593a000)
> >> > libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a3571e000)
> >> > /lib64/ld-linux-x86-64.so.2 (0x00007f8a39b67000)
> >> > libaws-cpp-sdk-core.so => /opt/aws/lib64/libaws-cpp-sdk-core.so
> >> > (0x00007f8a35413000)
> >> > libaws-c-event-stream.so.0unstable =>
> >> > /opt/aws/lib64/libaws-c-event-stream.so.0unstable (0x00007f8a3520b000)
> >> > libaws-c-common.so.0unstable =>
> >> /opt/aws/lib64/libaws-c-common.so.0unstable
> >> > (0x00007f8a34fd9000)
> >> > libaws-checksums.so => /opt/aws/lib64/libaws-checksums.so
> >> > (0x00007f8a34dce000)
> >> > libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f8a34b81000)
> >> > libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f8a34898000)
> >> > libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f8a34694000)
> >> > libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f8a34461000)
> >> > libcurl.so.4 => /opt/curl/lib/libcurl.so.4 (0x00007f8a341ea000)
> >> > libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f8a33fda000)
> >> > libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f8a33dd6000)
> >> > libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f8a33bbc000)
> >> > libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8a33995000)
> >> > libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f8a33733000)
> >> >
> >> >> /opt/rh/devtoolset-3/root/usr/bin/g++ --version
> >> > g++ (GCC) 4.9.2 20150212 (Red Hat 4.9.2-6)
> >> >
> >> > Do all of these ring any bells?
> >> >
> >> > Thank you!
> >> > Rares
> >>
>
