[issue22676] _pickle's whichmodule() is slow

Changes by Antoine Pitrou : -- resolution: -> fixed stage: commit review -> resolved status: open -> closed ___ Python tracker ___ __

[issue22676] _pickle's whichmodule() is slow

Roundup Robot added the comment: New changeset 3e3bec66409c by Antoine Pitrou in branch 'default': Fix uninitialized variable after #22676. https://hg.python.org/cpython/rev/3e3bec66409c -- ___ Python tracker _

[issue22676] _pickle's whichmodule() is slow

STINNER Victor added the comment: This issue may have introduce a regression, please see: Issue #22971: test_pickle: "Fatal Python error: Cannot recover from stack overflow." on FreeBSD buildbot -- ___ Python tracker

[issue22676] _pickle's whichmodule() is slow

Steve Dower added the comment: That patch looks good to me (better than my fix would have been :) ) -- ___ Python tracker ___ ___ Pyth

[issue22676] _pickle's whichmodule() is slow

Antoine Pitrou added the comment: Here is a patch. -- Added file: http://bugs.python.org/file37310/whichmodule_error.patch ___ Python tracker ___

[issue22676] _pickle's whichmodule() is slow

STINNER Victor added the comment: On Windows with Visual Studio, I got a compiler warning. In whichmodule(), get_dotted_path() is called with module whereas module is not initialiazed: dotted_path = get_dotted_path(module, global_name, allow_qualname); -- _

[issue22676] _pickle's whichmodule() is slow

Zachary Ware added the comment: This is still causing a somewhat serious warning on Windows, see [1] for example. The condition warned about affects every platform. It took me some time to make sense of that function, but I think I finally understand what's going on. I think Steve's suggesti

[issue22676] _pickle's whichmodule() is slow

Steve Dower added the comment: (Looks like I was outsmarted by the tracker when trying to put line numbers in my links... is there a way to do that?) -- ___ Python tracker ___ _

[issue22676] _pickle's whichmodule() is slow

Steve Dower added the comment: The fix for this now uses module without initializing it, which could lead to a crash if get_dotted_path() tries to raise an exception (it puts module into the error string with %R). See Modules/_pickle.c#l1656 and Modules/_pickle.c#l1538. I think the fix is to i

[issue22676] _pickle's whichmodule() is slow

Antoine Pitrou added the comment: Now applied. As Georg said, though, the definitive fix is to add a __module__ attribute to your global objects. -- resolution: -> fixed stage: needs patch -> resolved status: open -> closed ___ Python tracker

[issue22676] _pickle's whichmodule() is slow

Roundup Robot added the comment: New changeset e5ad1f27fb54 by Antoine Pitrou in branch 'default': Issue #22676: Make the pickling of global objects which don't have a __module__ attribute less slow. https://hg.python.org/cpython/rev/e5ad1f27fb54 -- nosy: +python-dev __

[issue22676] _pickle's whichmodule() is slow

Georg Brandl added the comment: I didn't have numpy anyway, I tested on a function with __module__ set to None manually. But I see the same speedup for Ellipsis. -- ___ Python tracker

[issue22676] _pickle's whichmodule() is slow

Antoine Pitrou added the comment: Victor, see https://github.com/numpy/numpy/issues/4952 -- ___ Python tracker ___ ___ Python-bugs-lis

[issue22676] _pickle's whichmodule() is slow

Antoine Pitrou added the comment: Attached patch produces a 8x speedup on Ellipsis. -- Added file: http://bugs.python.org/file36986/whichmodule.patch ___ Python tracker ___ _

[issue22676] _pickle's whichmodule() is slow

STINNER Victor added the comment: whichmodule_speedup.diff: I proposed once a more generic solution which is much more complex to implement. Lazy formatting of Exception message: in most cases, the message is not used. Replace AttributeError(message) with AttributeError(attr=name), only format

[issue22676] _pickle's whichmodule() is slow

STINNER Victor added the comment: > For example we can take numpy where some ufuncs (i.e. function-like objects > implemented in C) don't have a __module__. Oh. I was not aware of that. Is there a way to fix numpy to set the __module__ attribute? Maybe we should warn users that serialiaing obj

[issue22676] _pickle's whichmodule() is slow

Antoine Pitrou added the comment: So, a proper way to reproduce it is through Ellipsis (which does go through whichmodule()): $ PYTHONHASHSEED=0 python3.4 -m timeit -s "import pickle; d=Ellipsis" "pickle.dumps(d)" 1000 loops, best of 3: 201 usec per loop -- __

[issue22676] _pickle's whichmodule() is slow

Antoine Pitrou added the comment: Actually, numpy.add takes a different path. Sorry. -- ___ Python tracker ___ ___ Python-bugs-list ma

[issue22676] _pickle's whichmodule() is slow

Georg Brandl added the comment: Attached patch addresses the repeated __repr__ calling and gives a 4x speedup here. -- keywords: +patch Added file: http://bugs.python.org/file36985/whichmodule_speedup.diff ___ Python tracker

[issue22676] _pickle's whichmodule() is slow

Antoine Pitrou added the comment: Note the problem is easily reproduce. For example we can take numpy where some ufuncs (i.e. function-like objects implemented in C) don't have a __module__. $ PYTHONHASHSEED=0 python3.4 -m timeit -s "import pickle, numpy; d=numpy.add" "pickle.dumps(d)" 1000 lo

[issue22676] _pickle's whichmodule() is slow

Antoine Pitrou added the comment: There's no doubt that whichmodule() has grown more complex since 3.2. We probably cannot eliminate the O() component, but could at least make it faster. -- stage: -> needs patch title: Creating the string representation of a module is slower -> _pickl