New submission from Ilya Valmianski <ivalm...@gmail.com>:
Tried pickling a dictionary with multiple pandas tables and python primitive types. Pandas tables are large so full object size is ~200GB but system should not be OOM (crashed with ~300 GB system memory available). Reproduced on two machines running RHEL 7.5. Tried using Python 3.6 and 3.7. Tried pickle and dill. All python versions were installed as environments via Anaconda. All segfault in pickle. Here is a dump: Fatal Python error: Segmentation fault Current thread 0x00007f724af23740 (most recent call first): File "/apps/anaconda3/lib/python3.6/pickle.py", line 496 in save File "/apps/anaconda3/lib/python3.6/pickle.py", line 805 in _batch_appends File "/apps/anaconda3/lib/python3.6/pickle.py", line 781 in save_list File "/apps/anaconda3/lib/python3.6/pickle.py", line 476 in save File "/apps/anaconda3/lib/python3.6/pickle.py", line 751 in save_tuple File "/apps/anaconda3/lib/python3.6/pickle.py", line 476 in save File "/apps/anaconda3/lib/python3.6/pickle.py", line 634 in save_reduce File "/apps/anaconda3/lib/python3.6/pickle.py", line 521 in save File "/apps/anaconda3/lib/python3.6/pickle.py", line 847 in _batch_setitems File "/apps/anaconda3/lib/python3.6/pickle.py", line 821 in save_dict File "/apps/anaconda3/lib/python3.6/site-packages/dill/_dill.py", line 893 in save_module_dict File "/apps/anaconda3/lib/python3.6/pickle.py", line 476 in save File "/apps/anaconda3/lib/python3.6/pickle.py", line 634 in save_reduce File "/apps/anaconda3/lib/python3.6/pickle.py", line 521 in save File "/apps/anaconda3/lib/python3.6/pickle.py", line 847 in _batch_setitems File "/apps/anaconda3/lib/python3.6/pickle.py", line 821 in save_dict File "/apps/anaconda3/lib/python3.6/site-packages/dill/_dill.py", line 893 in save_module_dict File "/apps/anaconda3/lib/python3.6/pickle.py", line 476 in save File "/apps/anaconda3/lib/python3.6/pickle.py", line 409 in dump File "/apps/anaconda3/lib/python3.6/site-packages/dill/_dill.py", line 286 in dump File "precompute_control_patients.py", line 220 in main File "/apps/anaconda3/lib/python3.6/site-packages/absl/app.py", line 251 in _run_main File "/apps/anaconda3/lib/python3.6/site-packages/absl/app.py", line 300 in run File "precompute_control_patients.py", line 227 in <module> ---------- messages: 351213 nosy: Ilya Valmianski priority: normal severity: normal status: open title: Segfault when pickling dictionary with large pandas dataframes type: crash versions: Python 3.6, Python 3.7 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue38039> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
