[issue39672] SIGSEGV crash on shutdown with shelve & c pickle
New submission from zd nex :
Hello,
so I was transferring some our old code from Python2.7 to new and find that new
version seems to crash quite a lot. After some finding (good thing
faulthandler) I think I tracked it down to to Shelve.__del__ method > going to
C Pickle module (not python one). Here it is crash itself. Attached zip has 3
file. When shelve.close is used it does not seem to crash every time.
$python3.8 -X faulthandler ce_test_2.py
start
end
Fatal Python error: Segmentation fault
Current thread 0x7fb22e299740 (most recent call first):
File "/usr/lib/python3.8/shelve.py", line 124 in __setitem__
File "/usr/lib/python3.8/shelve.py", line 168 in sync
File "/usr/lib/python3.8/shelve.py", line 144 in close
File "/usr/lib/python3.8/shelve.py", line 162 in __del__
Neoprávněný přístup do paměti (SIGSEGV)
Code for crash is here:
import shelve
import material
data = shelve.open("test3", flag="c",writeback=True)
def test_shelve(data):
for k,v in data.items():
pass
print("start")
test_shelve(data)
#data.close() #fixes SIGSEGV at shutdown
#actually problem is in c pickle module; when Python pickle module is used it
works
print("end")
#after this it is crash
Code just loads module and shelve and opens file. Then in another function it
cycles through data and that creates crash in C pickle module at shutdown.
Weird thing is that when cycle through data is not in function it does not
crash. Also crash can be avoided when C Pickle is traded for Python Pickle.
In REPL it is quite similar just list on shelve.items() and exit makes Python
crash.
Python 3.8.1 (default, Dec 22 2019, 08:15:39)
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import shelve
>>> import material
>>> data = shelve.open("test3", flag="c",writeback=True)
>>> list(data.items())
[('H1615', Material(name='Třešeň Romana', code='H1615', vars=0))]
>>> exit()
Fatal Python error: Segmentation fault
Current thread 0x7f14a2546740 (most recent call first):
File "/usr/lib/python3.8/shelve.py", line 124 in __setitem__
File "/usr/lib/python3.8/shelve.py", line 168 in sync
File "/usr/lib/python3.8/shelve.py", line 144 in close
File "/usr/lib/python3.8/shelve.py", line 162 in __del__
Neoprávněný přístup do paměti (SIGSEGV)
Hopefully you can fix this.
--
components: Library (Lib)
files: test_crash_shelve.zip
messages: 362186
nosy: zd nex
priority: normal
severity: normal
status: open
title: SIGSEGV crash on shutdown with shelve & c pickle
type: crash
versions: Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file48899/test_crash_shelve.zip
___
Python tracker
<https://bugs.python.org/issue39672>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39672] SIGSEGV crash on shutdown with shelve & c pickle
zd nex added the comment: So I was trying to figure out what is crash it self and it looks to me that it is related to import. Do you know how I can properly debug this crash? -- ___ Python tracker <https://bugs.python.org/issue39672> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39672] SIGSEGV crash on shutdown with shelve & c pickle
Change by zd nex : -- versions: +Python 3.9 ___ Python tracker <https://bugs.python.org/issue39672> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39672] SIGSEGV crash on shutdown with shelve & c pickle
zd nex added the comment: Hello, well and in 3.8 it does not crash for you? Is there some devel build of 3.9 for ubuntu which I can try? I have tested it on 3.7,3.8 and 3.6 and it crashed always when close was not present or when list was called in another function. -- ___ Python tracker <https://bugs.python.org/issue39672> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39672] SIGSEGV crash on shutdown with shelve & c pickle
zd nex added the comment: Ok I will try pdb -- ___ Python tracker <https://bugs.python.org/issue39672> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39672] SIGSEGV crash on shutdown with shelve & c pickle
zd nex added the comment: Okay I have tried to run it with breakpoint() but it does not crash on 3.8 -- ___ Python tracker <https://bugs.python.org/issue39672> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39672] SIGSEGV crash on shutdown with shelve & c pickle
zd nex added the comment: Okay I have managed to crash it when exit() was called and also I am attaching output with -v python3.8 -X faulthandler ce_test_2.py start end --Return-- > /home/fractal/workspace/test_py_crash/ce_test_2.py(19)()->None -> breakpoint() (Pdb) exit() Traceback (most recent call last): File "ce_test_2.py", line 19, in breakpoint() File "/usr/lib/python3.8/bdb.py", line 92, in trace_dispatch return self.dispatch_return(frame, arg) File "/usr/lib/python3.8/bdb.py", line 154, in dispatch_return if self.quitting: raise BdbQuit bdb.BdbQuit Fatal Python error: Segmentation fault Current thread 0x7f976975f740 (most recent call first): File "/usr/lib/python3.8/shelve.py", line 124 in __setitem__ File "/usr/lib/python3.8/shelve.py", line 168 in sync File "/usr/lib/python3.8/shelve.py", line 144 in close File "/usr/lib/python3.8/shelve.py", line 162 in __del__ Neoprávněný přístup do paměti (SIGSEGV) $ # destroy string # cleanup[2] removing threading # cleanup[2] removing atexit # cleanup[2] removing logging # cleanup[2] removing material # cleanup[2] removing _dbm # cleanup[2] removing dbm.ndbm # cleanup[2] removing dbm # destroy dbm # cleanup[2] removing _gdbm # cleanup[2] removing dbm.gnu # cleanup[2] removing _ast # cleanup[2] removing ast # cleanup[2] removing dbm.dumb # destroy _ast Fatal Python error: Segmentation fault Current thread 0x7fd5477cb740 (most recent call first): File "/usr/lib/python3.8/shelve.py", line 124 in __setitem__ File "/usr/lib/python3.8/shelve.py", line 168 in sync File "/usr/lib/python3.8/shelve.py", line 144 in close File "/usr/lib/python3.8/shelve.py", line 162 in __del__ Neoprávněný přístup do paměti (SIGSEGV) -- Added file: https://bugs.python.org/file48907/crash_pdb.txt ___ Python tracker <https://bugs.python.org/issue39672> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39672] SIGSEGV crash on shutdown with shelve & c pickle
zd nex added the comment: So I was trying it again in Python 3.6.9 and 3.8.1 directly in REPL. And it behaves same. I have tried it on two different linux boxes (both 64bit) where I have diffrent versions. In both of them it crashes in same way .. destroy _ast and then it crashes and faulthandler again shows shelve (pickle) So I am attaching new crash reports directly from REPL where I just call list(data.items()) and then exit() Btw it seems to me that when PDB is active crash does not occurs until exit() is called. -- Added file: https://bugs.python.org/file48912/crash.txt ___ Python tracker <https://bugs.python.org/issue39672> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39672] SIGSEGV crash on shutdown with shelve & c pickle
zd nex added the comment: Hi, i was looking on failing tests on attached pull request and it seems to me that it intentionally should create new entry for saving. Maybe that save should actually happen, but it should be fixed in different way? From my small tests I was thinking that problem was actually in pickle, but maybe it is connected to shelve itself. -- ___ Python tracker <https://bugs.python.org/issue39672> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39956] Numeric Literals vs string "1_1" on input int() or float() or literal_eval
New submission from zd nex :
So currently if python code contains 1_1 it is handled as number 11. When user
uses int("1_1") it also creates 11 and when ast.literal_eval is used it is also
created instead of string. How can user get SyntaxError input on int or
literal_eval with obviously wrong input (some keyboards have . next to _) like
int(input()) in REPL? In python2.7 this was checked, but now even string is
handled as number. Is there some reason?
I understand reasoning behind PEP515, that int(1_1) can create 11, but why
int("1_1") creates it also? Previously users used literal_eval for safe check
of values, but now user can put 1_1 and it is transferred as number. Is there
some plan to be able control behavior of these functions? I was now with some
students, which used python2.7 and they find it also confusing. Most funny
thing is that when they do same thing in JavaScript parseInt("1_1") they get 1,
in old python this was error and now we give them 11.
I would suggest that it would be possible to strictly check strings, as it was
in old Python2.7. This way user would be able to use _ in code to arrange
numbers, but it would also allow checks on wrong inputs of users which were
meant something else, for example if you use it in try/except in console.
------
messages: 364112
nosy: zd nex
priority: normal
severity: normal
status: open
title: Numeric Literals vs string "1_1" on input int() or float() or
literal_eval
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8, Python 3.9
___
Python tracker
<https://bugs.python.org/issue39956>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39672] SIGSEGV crash on shutdown with shelve & c pickle
zd nex added the comment:
Hello,
so I was trying to figure out where actually is problem is. As I do not think
it is in shelve itself. So I have made different method for __setitem__ on
shelve and I have found that it is actually in pickle.dump >
Here is code which I have used
def __setitem__(self, key, value):
if self.writeback:
self.cache[key] = value
f = BytesIO()
print("set")
p = pickle.Pickler(f, self._protocol)
try:
print("before dumps - > crash",value)
p.dump(value)
print("after dump > will not be printed on crash")
except Exception as e:
print("error set",e)
print("after dump",key)
self.dict[key.encode(self.keyencoding)] = f.getvalue()
print("saved")
When in this code user changes p.dump to another method cpython crash does not
happen. Can you please try to see if it is like that?
--
components: +Interpreter Core, ctypes
___
Python tracker
<https://bugs.python.org/issue39672>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39672] SIGSEGV crash on shutdown with shelve & c pickle
zd nex added the comment: So I want to properly debug this? How I can debug that call dump() for pickle? It does not seem to be possible. I guess I need to make some custom build? -- ___ Python tracker <https://bugs.python.org/issue39672> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39672] Segmentation fault on shutdown with shelve & c pickle
Change by zd nex : -- title: SIGSEGV crash on shutdown with shelve & c pickle -> Segmentation fault on shutdown with shelve & c pickle ___ Python tracker <https://bugs.python.org/issue39672> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39672] Segmentation fault on shutdown with shelve & c pickle
zd nex added the comment: Hello, ok but it seems to me that this segfault happens always (it is not random)? So I guess that there should be way how fix C pickle, no? Or something else should be done with __del__ method of shelve. Because in Python2 this was normally working. It seems to me that when user just reads data of shelve and then exit happens this can happen. Why Python2 normally worked? -- ___ Python tracker <https://bugs.python.org/issue39672> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
