[
https://issues.apache.org/jira/browse/ARROW-17984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17628471#comment-17628471
]
Ziheng Wang commented on ARROW-17984:
-------------------------------------
```
Thread 42 (Thread 0x7fd1d77fe700 (LWP 10252)):
#0 futex_abstimed_wait (private=0, abstime=0x0, clockid=0, expected=2,
futex_word=<optimized out>) at ../sysdeps/nptl/futex-internal.h:284
#1 __pthread_rwlock_wrlock_full (abstime=0x0, clockid=0, rwlock=0x2fd02f0) at
pthread_rwlock_common.c:830
#2 __GI___pthread_rwlock_wrlock (rwlock=0x2fd02f0) at
pthread_rwlock_wrlock.c:27
#3 0x00007fd4a586ee59 in CRYPTO_THREAD_write_lock () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#4 0x00007fd4a586436f in ossl_namemap_add_names () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#5 0x00007fd4a58512c0 in construct_evp_method () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#6 0x00007fd4a5863e15 in ossl_method_construct_this () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#7 0x00007fd4a5863c8e in algorithm_do_this () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#8 0x00007fd4a586e1db in ossl_provider_doall_activated () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#9 0x00007fd4a5863d4f in ossl_algorithm_do_all () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#10 0x00007fd4a5864015 in ossl_method_construct () from
/home/ubuntu/.local/lib---Type <RET> for--Type <RET> for --Type------Ty--Type
<RET> for mo--Type ----Type <RET> for more, q to quit, c to continue without
paging--
python3.8/site-packages/pyarrow/libarrow.so.900
#11 0x00007fd4a5851876 in evp_generic_fetch () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#12 0x00007fd4a584fd1a in EVP_CIPHER_fetch () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#13 0x00007fd4a5c6ad22 in ssl_evp_cipher_fetch () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#14 0x00007fd4a5c605b3 in ssl_load_ciphers () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#15 0x00007fd4a5c6c1b1 in SSL_CTX_new_ex () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#16 0x00007fd4a5c4e727 in ossl_connect_step1 () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#17 0x00007fd4a5c51c7d in ossl_connect_common () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#18 0x00007fd4a5c1a348 in Curl_ssl_connect_nonblocking () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#19 0x00007fd4a5c32766 in Curl_http_connect () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#20 0x00007fd4a5bfc2eb in multi_runsingle () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#21 0x00007fd4a5bfd76b in curl_multi_perform () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#22 0x00007fd4a5bec833 in easy_transfer () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#23 0x00007fd4a5becde9 in curl_easy_perform () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#24 0x00007fd4a5737984 in
Aws::Http::CurlHttpClient::MakeRequest(std::shared_ptr<Aws::Http::HttpRequest>
const&, Aws::Utils::RateLimits::RateLimiterInterface*,
Aws::Utils::RateLimits::RateLimiterInterface*) const () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#25 0x00007fd4a56fa581 in
Aws::Client::AWSClient::AttemptOneRequest(std::shared_ptr<Aws::Http::HttpRequest>
const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char
const*) const () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#26 0x00007fd4a56fb3d0 in
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&,
Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char
const*, char const*) const () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#27 0x00007fd4a57072f1 in Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI
const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char
const*, char const*, char const*) const () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#28 0x00007fd4a5219dd9 in arrow::fs::ResolveS3BucketRegion(std::string const&)
() from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#29 0x00007fd4a521b6a3 in arrow::fs::S3Options::FromUri(arrow::internal::Uri
const&, std::string*) () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#30 0x00007fd4a4528c4f in arrow::fs::(anonymous
namespace)::FileSystemFromUriReal(arrow::internal::Uri const&, std::string
const&, arrow::io::IOContext const&, std::string*) () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#31 0x00007fd4a45298c0 in arrow::fs::FileSystemFromUri(std::string const&,
arrow::io::IOContext const&, std::string*) () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#32 0x00007fd4a4529aea in arrow::fs::FileSystemFromUriOrPath(std::string
const&, arrow::io::IOContext const&, std::string*) () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#33 0x00007fd4a4529cfd in arrow::fs::FileSystemFromUriOrPath(std::string
const&, std::string*) () from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#34 0x00007fd488170b7d in
__pyx_pw_7pyarrow_3_fs_10FileSystem_3from_uri(_object*, _object*, _object*) ()
from
/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/_fs.cpython-38-x86_64-linux-gnu.so
#35 0x00000000005f6929 in PyCFunction_Call ()
#36 0x00000000005f74f6 in _PyObject_MakeTpCall ()
#37 0x0000000000571164 in _PyEval_EvalFrameDefault ()
#38 0x0000000000569dba in _PyEval_EvalCodeWithName ()
#39 0x00000000005f6eb3 in _PyFunction_Vectorcall ()
#40 0x000000000059ca27 in ?? ()
#41 0x00000000005f745f in _PyObject_MakeTpCall ()
#42 0x000000000057164c in _PyEval_EvalFrameDefault ()
#43 0x0000000000569dba in _PyEval_EvalCodeWithName ()
#44 0x00000000005f6eb3 in _PyFunction_Vectorcall ()
#45 0x000000000056cc1f in _PyEval_EvalFrameDefault ()
#46 0x0000000000569dba in _PyEval_EvalCodeWithName ()
#47 0x00000000005f6eb3 in _PyFunction_Vectorcall ()
#48 0x00000000005f6082 in PyObject_Call ()
#49 0x000000000056d2d5 in _PyEval_EvalFrameDefault ()
#50 0x00000000005f6cd6 in _PyFunction_Vectorcall ()
#51 0x000000000056bbfa in _PyEval_EvalFrameDefault ()
```
> pq.read_table doesn't seem to be thread safe
> --------------------------------------------
>
> Key: ARROW-17984
> URL: https://issues.apache.org/jira/browse/ARROW-17984
> Project: Apache Arrow
> Issue Type: Bug
> Components: Parquet
> Affects Versions: 9.0.0
> Reporter: Ziheng Wang
> Priority: Major
> Attachments: _usr_bin_python3.8.1000.crash
>
>
> Before PR: [https://github.com/apache/arrow/pull/13799] gets merged in
> master, I am using multithreading to improve read bandwidth from S3. Even
> after that PR gets merged, I probably will still try to use multithreading to
> some extent.
> However pq.read_table from S3 doesn't seem to be thread safe. Seems like it
> uses the new dataset reader under the hood. I cannot provide a reproduction,
> not a stable one anyway. But this is roughly the script I have been using
> ~~~
> def get_next_batch(self, mapper_id, pos=None):
> def download(file):
> return pq.read_table("s3://" + self.bucket + "/" + file,
> columns=self.columns, filters=self.filters)
>
> executor = concurrent.futures.ThreadPoolExecutor(max_workers=self.workers)
> futures= \{executor.submit(download, file): file for file in my_files}
> for future inconcurrent.futures.as_completed(futures):
> yield future.result()
> ~~~
> The errors all have to do with malloc segfaults which makes me suspect the
> connection object is being reused across different pq.read_table invocations
> in different threads
> ```
> (InputReaderNode pid=25001, ip=172.31.60.29) malloc_consolidate(): invalid
> chunk size
> (InputReaderNode pid=25001, ip=172.31.60.29) *** SIGABRT received at
> time=1665464922 on cpu 9 ***
> (InputReaderNode pid=25001, ip=172.31.60.29) PC: @ 0x7f9a480a803b
> (unknown) raise
> (InputReaderNode pid=25001, ip=172.31.60.29) @ 0x7f9a480a80c0
> 4160 (unknown)
> (InputReaderNode pid=25001, ip=172.31.60.29) @ 0x7f9a480fa32c
> (unknown) (unknown)
> ```
> Note, this multithreaded code is running inside a Ray actor process, but that
> shouldn't be a problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
