[issue37434] Segfault in typeobject.c at _PyObject_GC_UNTRACK(type)
New submission from Tim Lusk : I'm working on porting an application from Python 3.6.8 to 3.7.3 and am running into this segfault. I'm not able to pin point exactly what Python code is causing this as it's a very large enterprise application and I can't seem to pinpoint exactly where this is coming from. The application consistently crashes on start every time with this stack trace. Program received signal SIGSEGV, Segmentation fault. 0x77922ae5 in type_dealloc (type=0x7fffed185480) at /usr/src/debug/Python-3.7.3/Objects/typeobject.c:3283 3283_PyObject_GC_UNTRACK(type); (gdb) bt #0 0x77922ae5 in type_dealloc (type=0x7fffed185480) at /usr/src/debug/Python-3.7.3/Objects/typeobject.c:3283 #1 0x778ee721 in free_keys_object (keys=0x7fffee258bc0) at /usr/src/debug/Python-3.7.3/Objects/dictobject.c:559 #2 0x778f20af in dict_dealloc (mp=0x7fffee209c60) at /usr/src/debug/Python-3.7.3/Objects/dictobject.c:1913 #3 0x77904d15 in module_clear (m=0x7fffee211048) at /usr/src/debug/Python-3.7.3/Objects/moduleobject.c:749 #4 0x77a3aeda in delete_garbage (collectable=0x7fff7420, old=0x77dd75f0 <_PyRuntime+400>) at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:769 #5 0x77a3b41d in collect (generation=2, n_collected=0x7fff7488, n_uncollectable=0x7fff7490, nofail=0) at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:924 #6 0x77a3b845 in collect_with_callback (generation=2) at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:1036 #7 0x77a3bcef in gc_collect_impl (module=0x7615d9f8, generation=2) at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:1138 #8 0x77a3ba13 in gc_collect (module=0x7615d9f8, args=0xcc6cd0, nargs=0, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Modules/clinic/gcmodule.c.h:100 #9 0x778b48e5 in _PyMethodDef_RawFastCallKeywords (method=0x77d8b800 , self=0x7615d9f8, args=0xcc6cd0, nargs=0, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Objects/call.c:656 #10 0x778b4bab in _PyCFunction_FastCallKeywords (func=0x7615ecf0, args=0xcc6cd0, nargs=0, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Objects/call.c:730 #11 0x779cbf2c in call_function (pp_stack=0x7fff78b8, oparg=0, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:4568 #12 0x779c6344 in _PyEval_EvalFrameDefault (f=0xcc6b08, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:3093 #13 0x779b8067 in PyEval_EvalFrameEx (f=0xcc6b08, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:547 #14 0x778b3b1a in function_code_fastcall (co=0x7620bc00, args=0x764fef90, nargs=1, globals=0x764e6120) at /usr/src/debug/Python-3.7.3/Objects/call.c:283 #15 0x778b4096 in _PyFunction_FastCallKeywords (func=0x7fffedb63488, stack=0x764fef88, nargs=1, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Objects/call.c:408 #16 0x779cc357 in call_function (pp_stack=0x7fff9140, oparg=1, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:4616 #17 0x779c64c9 in _PyEval_EvalFrameDefault (f=0x764fee08, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:3124 #18 0x779b8067 in PyEval_EvalFrameEx (f=0x764fee08, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:547 #19 0x779ca14f in _PyEval_EvalCodeWithName (_co=0x7620bc90, globals=0x764e6120, locals=0x764e6120, args=0x0, argcount=0, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:3930 #20 0x779ca347 in PyEval_EvalCodeEx (_co=0x7620bc90, globals=0x764e6120, locals=0x764e6120, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:3959 #21 0x779b7ff1 in PyEval_EvalCode (co=0x7620bc90, globals=0x764e6120, locals=0x764e6120) at /usr/src/debug/Python-3.7.3/Python/ceval.c:524 #22 0x779b2963 in builtin_exec_impl (module=0x77e61c28, source=0x7620bc90, globals=0x764e6120, locals=0x764e6120) at /usr/src/debug/Python-3.7.3/Python/bltinmodule.c:1079 #23 0x779aff3b in builtin_exec (module=0x77e61c28, args=0x6c5b58, nargs=2) at /usr/src/debug/Python-3.7.3/Python/clinic/bltinmodule.c.h:283 #24 0x778b48bd in _PyMethodDef_RawFastCallKeywords (method=0x77d7bae0 , self=0x77e61c28, args=0x6c5b58, nargs=2, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Objects/call.c:651 #25 0x778b4bab in _PyCFunction_FastCallKeywords (func=0x76566d80, args=0x6c5b58, nargs=2, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Objects/call.c:730 #26 0x779cbf2c in call_function (pp_stack=0x7fffada0, oparg=2, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:4568
[issue37434] Segfault in typeobject.c at _PyObject_GC_UNTRACK(type)
Tim Lusk added the comment:
Alright I was finally able to reproduce the issue with a small code snip and
capture the stack in gdb. It looks like the issue has something to do with an
exception thrown while importing a class (and possibly only when the class is a
c_ext, but I haven't confirmed that)
[root@933fdf38d838 ~]# python3.7 -m venv crash-test
[root@933fdf38d838 ~]# crash-test/bin/pip install protobuf
Looking in indexes: https://files.pythonhosted.org/simple
Collecting protobuf
Downloading
https://files.pythonhosted.org/packages/ce/86/9f6123c4c6f481862f286dbe13aa2e97bdedd7662f5fc3033c1a41f32f88/protobuf-3.8.0-cp37-cp37m-manylinux1_x86_64.whl
(1.2MB)
100% || 1.2MB 2.3MB/s
Collecting six>=1.9 (from protobuf)
Downloading
https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in
./crash-test/lib/python3.7/site-packages (from protobuf) (40.8.0)
Installing collected packages: six, protobuf
Successfully installed protobuf-3.8.0 six-1.12.0
[root@933fdf38d838 ~]# scl enable devtoolset-8 "gdb ./crash-test/bin/python"
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-2.el6
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./crash-test/bin/python...Reading symbols from
/usr/lib/debug/usr/bin/python3.7m.debug...
warning: Skipping deprecated .gdb_index section in
/usr/lib/debug/usr/bin/python3.7m.debug.
Do "set use-deprecated-index-sections on" before the file is read
to use the section anyway.
done.
done.
(gdb) run
Starting program: /root/crash-test/bin/python
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.212.el6.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Python 3.7.3 (default, Jun 27 2019, 15:27:42)
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> import gc
>>>
>>> gc.disable()
>>>
>>> for i in range(100):
... warnings.filterwarnings('error')
... try:
... from google.protobuf.pyext import _message
... except Exception as ex:
... print(i)
...
... gc.collect()
...
0
1
2
3
4
5
Program received signal SIGSEGV, Segmentation fault.
0x77922ae5 in type_dealloc (type=0x72277480) at
/usr/src/debug/Python-3.7.3/Objects/typeobject.c:3283
3283_PyObject_GC_UNTRACK(type);
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.5-7.el6_0.x86_64 expat-2.0.1-13.el6_8.x86_64
libgcc-4.4.7-23.el6.x86_64 libstdc++-4.4.7-23.el6.x86_64
ncurses-libs-5.7-4.20090207.el6.x86_64
nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64
openresty-openssl-1.0.2s-1.el6.x86_64 readline-6.0-4.el6.x86_64
xz-libs-4.999.9-0.5.beta.20091007git.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x77922ae5 in type_dealloc (type=0x72277480) at
/usr/src/debug/Python-3.7.3/Objects/typeobject.c:3283
#1 0x778ee721 in free_keys_object (keys=0x764e3bc0) at
/usr/src/debug/Python-3.7.3/Objects/dictobject.c:559
#2 0x778f20af in dict_dealloc (mp=0x747a1708) at
/usr/src/debug/Python-3.7.3/Objects/dictobject.c:1913
#3 0x77904d15 in module_clear (m=0x722f3ea8) at
/usr/src/debug/Python-3.7.3/Objects/moduleobject.c:749
#4 0x77a3aeda in delete_garbage (collectable=0x7fffc250,
old=0x77dd75f0 <_PyRuntime+400>)
at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:769
#5 0x77a3b41d in collect (generation=2, n_collected=0x7fffc2b8,
n_uncollectable=0x7fffc2c0, nofail=0)
at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:924
#6 0x77a3b845 in collect_with_callback (generation=2) at
/usr/src/debug/Python-3.7.3/Modules/gcmodule.c:1036
#7 0x77a3bcef in gc_collect_impl (module=0x761f6138, generation=2)
at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:1138
#8 0x77a3ba13 in gc_collect (module=0x761f6138,
args=0x764e7768, nargs=0, kwname
