kaushal9696 opened a new issue, #50908:
URL: https://github.com/apache/arrow/issues/50908
**Components:** Python, C++
## Environment
- pyarrow 25.0.0 (PyPI wheel, bundled mimalloc), pandas + pyarrow
dataset/parquet path
- Python 3.14.4, CPython, Linux x86_64 (Ubuntu-family, kernel 7.0.0),
uvicorn/FastAPI service
- Not OOM (VmRSS 161 MB at crash, process peak 130 MB), not disk.
## Summary
If the **first import of pyarrow and Arrow's first allocation happen on a
short-lived worker thread that subsequently exits**, a later allocation on a
*new* thread crashes with SIGSEGV inside the bundled mimalloc's per-thread heap
initialisation. When the import + first allocation happen on the main thread
(or any thread that stays alive), the same workload is stable under heavy
concurrency.
This bites any FastAPI/uvicorn-style service that lazily imports
pandas/pyarrow inside a request handler: anyio's worker threads have a
10-second idle lifetime (`WorkerThread.MAX_IDLE_TIME`), so the import lands on
a thread that is pruned seconds later, and the next burst of traffic on fresh
threads segfaults the process.
## Crash
From a core dump (gdb, `thread apply all bt`; crashing thread was the newest
of 23):
```
#5 mi_thread_init () libarrow.so.2500
#6 _mi_malloc_generic ()
#8 arrow::BaseMemoryPoolImpl<MimallocAllocator>::Allocate(...)
#12 arrow::AllocateEmptyBitmap(...)
#14 parquet::SerializedFile::PreBuffer(...)
#45 pyarrow._dataset.Scanner.to_table
#49 _PyEval_EvalFrameDefault
```
Disassembly at the fault: `mov (%rdx),%rax` then `=> mov 0x18(%rax),%rax`
with **rax == 0** — a NULL dereference inside `mi_thread_init`.
## Isolation matrix (3 runs per configuration, deterministic)
| pyarrow import | first Arrow allocation | outcome |
|---|---|---|
| throwaway thread (exits) | same throwaway thread | **SEGV 3/3** |
| main thread | throwaway thread | survived 3/3 |
| throwaway thread | main thread | survived 3/3 |
| main thread | main thread | survived 3/3 |
The trigger is **thread identity at load/first-allocation time, not
concurrency**: with the import on the main thread, 8 concurrent fresh threads
performed 6,000 parquet reads with no fault.
## Reproduction shape
```python
import threading
def load_and_read():
import pyarrow.dataset as ds # first pyarrow import in the
process
ds.dataset("some.parquet").to_table() # first Arrow allocation
t = threading.Thread(target=load_and_read)
t.start(); t.join() # thread exits — heap/TLS torn down
# later: allocations from NEW short-lived threads (e.g. anyio worker threads
# serving an ASGI app) crash in mi_thread_init.
```
End-to-end reproduction (our service shape): uvicorn app whose handler
lazily imports pandas and reads parquet; hammer with 21 concurrent GETs per
burst, 12 s idle between bursts (idle > anyio's 10 s worker lifetime so each
burst gets fresh threads). Pre-workaround the process died with signal 11 on
the third burst, reproducibly. Post-workaround: 10 bursts, 210 requests, all
200.
## Hypothesis
mimalloc's process-wide initialisation appears to bind state to the thread
that performed it; when that thread exits, its TLS/heap teardown leaves a
structure NULL that `mi_thread_init` later dereferences on a new thread. (We
can share the full backtrace and the isolation script on request.)
## Workarounds (verified)
- Import pyarrow and perform one real allocation on the **main thread**
before any worker-thread use (we ship this as a startup warm-up).
- `ARROW_DEFAULT_MEMORY_POOL=system` (set before import) also sidesteps it,
at the cost of changing the allocator globally.
---
*Reporter context: hit in production behind FastAPI/uvicorn; root-caused
from an apport core dump; happy to provide the isolation script, full `thread
apply all bt`, and library build info.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
