Re: [Python-Dev] "make test" routinely fails to terminate
> me> On the 3.7 branch, "make test" routinely fails to terminate. > Antoine> Can you try to rebuild Python? Use "make distclean" if that helps. > Thanks, Antoine. That solved the termination problem. I still have problems > with test_asyncio failing, but I can live with that for now. Final follow-up. I finally got myself a workable, updateable 3.7 branch in my fork. It looks like the asyncio issues are alsy resolved on both 3.7 and master. Skip ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Add __reversed__ methods for dict
Hi, since dict keys are sorted by their insertion order since Python 3.6 and that it’s part of Python specs since 3.7 a proposal has been made in bpo-33462 to add the __reversed__ method to dict and dict views. Concerns have been raised in the comments that this feature may add too much bloat in the core interpreter and be harmful for other Python implementations. Given the different issues this change creates, I see three possibilities: 1. Accept the proposal has it is for dict and dict views, this would add about 300 lines and three new types in dictobject.c 2. Accept the proposal only for dict, this would add about 80 lines and one new type in dictobject.c while still being useful for some use cases 3. Drop the proposal as the whole, while having some use, reversed(dict(a=1, b=2)) may not be very common and could be done using OrderedDict instead. What’s your stance on the issue ? Best regards, Rémi Lapeyre ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add __reversed__ methods for dict
Please go find some real world code that would benefit from this. Don't make up examples, just show some code in a repository (public if possible, but private is okay, as long as you can quote small amounts of code from it) where te existence of reverse iteration over a dict would have been helpful. On Thu, May 24, 2018 at 5:55 AM, Rémi Lapeyre wrote: > > Hi, > > since dict keys are sorted by their insertion order since Python 3.6 and > that > it’s part of Python specs since 3.7 a proposal has been made in bpo-33462 > to > add the __reversed__ method to dict and dict views. > > Concerns have been raised in the comments that this feature may add too > much > bloat in the core interpreter and be harmful for other Python > implementations. > > Given the different issues this change creates, I see three possibilities: > > 1. Accept the proposal has it is for dict and dict views, this would add > about > 300 lines and three new types in dictobject.c > > 2. Accept the proposal only for dict, this would add about 80 lines and one > new type in dictobject.c while still being useful for some use cases > > 3. Drop the proposal as the whole, while having some use, > reversed(dict(a=1, b=2)) > may not be very common and could be done using OrderedDict instead. > > What’s your stance on the issue ? > > Best regards, > Rémi Lapeyre > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add __reversed__ methods for dict
(Also this probably belongs in python-ideas, unless there's already a bugs.python.org issue for it -- but you didn't mention that so I assume it's just an idea? How did you reach the line count estimates?) On Fri, May 25, 2018 at 8:46 AM, Guido van Rossum wrote: > Please go find some real world code that would benefit from this. Don't > make up examples, just show some code in a repository (public if possible, > but private is okay, as long as you can quote small amounts of code from > it) where te existence of reverse iteration over a dict would have been > helpful. > > On Thu, May 24, 2018 at 5:55 AM, Rémi Lapeyre > wrote: > >> >> Hi, >> >> since dict keys are sorted by their insertion order since Python 3.6 and >> that >> it’s part of Python specs since 3.7 a proposal has been made in bpo-33462 >> to >> add the __reversed__ method to dict and dict views. >> >> Concerns have been raised in the comments that this feature may add too >> much >> bloat in the core interpreter and be harmful for other Python >> implementations. >> >> Given the different issues this change creates, I see three possibilities: >> >> 1. Accept the proposal has it is for dict and dict views, this would add >> about >> 300 lines and three new types in dictobject.c >> >> 2. Accept the proposal only for dict, this would add about 80 lines and >> one >> new type in dictobject.c while still being useful for some use cases >> >> 3. Drop the proposal as the whole, while having some use, >> reversed(dict(a=1, b=2)) >> may not be very common and could be done using OrderedDict instead. >> >> What’s your stance on the issue ? >> >> Best regards, >> Rémi Lapeyre >> ___ >> Python-Dev mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% >> 40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add __reversed__ methods for dict
It looks like an optimization, since you can already do something like reversed(list(d)). Do you have benchmark numbers to see the benefit of your change? Even if reversed(list(d)) is slow, I'm not sure that it's worth it to optimize it, since it's a rare usecase. Victor 2018-05-24 14:55 GMT+02:00 Rémi Lapeyre : > > Hi, > > since dict keys are sorted by their insertion order since Python 3.6 and that > it’s part of Python specs since 3.7 a proposal has been made in bpo-33462 to > add the __reversed__ method to dict and dict views. > > Concerns have been raised in the comments that this feature may add too much > bloat in the core interpreter and be harmful for other Python implementations. > > Given the different issues this change creates, I see three possibilities: > > 1. Accept the proposal has it is for dict and dict views, this would add about > 300 lines and three new types in dictobject.c > > 2. Accept the proposal only for dict, this would add about 80 lines and one > new type in dictobject.c while still being useful for some use cases > > 3. Drop the proposal as the whole, while having some use, reversed(dict(a=1, > b=2)) > may not be very common and could be done using OrderedDict instead. > > What’s your stance on the issue ? > > Best regards, > Rémi Lapeyre > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add __reversed__ methods for dict
INADA Naoki asked Rémi Lapeyre in https://bugs.python.org/issue33462 to start a discussion on python-dev. Victor 2018-05-25 17:48 GMT+02:00 Guido van Rossum : > (Also this probably belongs in python-ideas, unless there's already a > bugs.python.org issue for it -- but you didn't mention that so I assume it's > just an idea? How did you reach the line count estimates?) > > On Fri, May 25, 2018 at 8:46 AM, Guido van Rossum wrote: >> >> Please go find some real world code that would benefit from this. Don't >> make up examples, just show some code in a repository (public if possible, >> but private is okay, as long as you can quote small amounts of code from it) >> where te existence of reverse iteration over a dict would have been helpful. >> >> On Thu, May 24, 2018 at 5:55 AM, Rémi Lapeyre >> wrote: >>> >>> >>> Hi, >>> >>> since dict keys are sorted by their insertion order since Python 3.6 and >>> that >>> it’s part of Python specs since 3.7 a proposal has been made in bpo-33462 >>> to >>> add the __reversed__ method to dict and dict views. >>> >>> Concerns have been raised in the comments that this feature may add too >>> much >>> bloat in the core interpreter and be harmful for other Python >>> implementations. >>> >>> Given the different issues this change creates, I see three >>> possibilities: >>> >>> 1. Accept the proposal has it is for dict and dict views, this would add >>> about >>> 300 lines and three new types in dictobject.c >>> >>> 2. Accept the proposal only for dict, this would add about 80 lines and >>> one >>> new type in dictobject.c while still being useful for some use cases >>> >>> 3. Drop the proposal as the whole, while having some use, >>> reversed(dict(a=1, b=2)) >>> may not be very common and could be done using OrderedDict instead. >>> >>> What’s your stance on the issue ? >>> >>> Best regards, >>> Rémi Lapeyre >>> ___ >>> Python-Dev mailing list >>> [email protected] >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) > > > > > -- > --Guido van Rossum (python.org/~guido) > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2018-05-18 - 2018-05-25)
Python tracker at https://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.
Issues counts and deltas:
open6699 (+13)
closed 38700 (+63)
total 45399 (+76)
Open issues with patches: 2638
Issues opened (54)
==
#27485: urllib.splitport -- is it official or not?
https://bugs.python.org/issue27485 reopened by serhiy.storchaka
#27535: Ignored ResourceWarning warnings leak memory in warnings regis
https://bugs.python.org/issue27535 reopened by vstinner
#0: Better error handling in PyImport_Cleanup()
https://bugs.python.org/issue0 reopened by vstinner
#33573: statistics.median does not work with ordinal scale
https://bugs.python.org/issue33573 opened by W deW
#33575: Python relies on C undefined behavior float-cast-overflow
https://bugs.python.org/issue33575 opened by gregory.p.smith
#33576: Make exception wrapping less intrusive for __set_name__ calls
https://bugs.python.org/issue33576 opened by ncoghlan
#33578: cjkcodecs missing getstate and setstate implementations
https://bugs.python.org/issue33578 opened by libcthorne
#33579: calendar.timegm not always an inverse of time.gmtime
https://bugs.python.org/issue33579 opened by eitan.adler
#33581: Document "optional components that are commonly included in Py
https://bugs.python.org/issue33581 opened by Antony.Lee
#33582: formatargspec deprecated but does nto emit DeprecationWarning.
https://bugs.python.org/issue33582 opened by mbussonn
#33586: 2.7.15 missing release notes on download page
https://bugs.python.org/issue33586 opened by ericvw
#33587: inspect.getsource performs unnecessary filesystem stat call wh
https://bugs.python.org/issue33587 opened by Pankaj Pandey
#33590: sched.enter priority has no impact on execution
https://bugs.python.org/issue33590 opened by sahilmn
#33591: ctypes does not support fspath protocol
https://bugs.python.org/issue33591 opened by mrh1997
#33592: Document contextvars C API
https://bugs.python.org/issue33592 opened by Elvis.Pranskevichus
#33594: add deprecation since 3.5 for a few methods of inspect.
https://bugs.python.org/issue33594 opened by mbussonn
#33595: FIx references to lambda "arguments"
https://bugs.python.org/issue33595 opened by adelfino
#33597: Compact PyGC_Head
https://bugs.python.org/issue33597 opened by inada.naoki
#33598: ActiveState Recipes links in docs, and the apparent closure of
https://bugs.python.org/issue33598 opened by tritium
#33600: [EASY DOC] Python 2: document that platform.linux_distribution
https://bugs.python.org/issue33600 opened by vstinner
#33601: [EASY DOC] Py_UTF8Mode is not documented
https://bugs.python.org/issue33601 opened by serhiy.storchaka
#33602: Remove set and queue references from Data Types
https://bugs.python.org/issue33602 opened by adelfino
#33603: Subprocess Thread handles grow with each call and aren't relea
https://bugs.python.org/issue33603 opened by GranPrego
#33604: HMAC default to MD5 marked as to be removed in 3.6
https://bugs.python.org/issue33604 opened by mbussonn
#33605: Detect accessing event loop from a different thread outside of
https://bugs.python.org/issue33605 opened by hniksic
#33606: Improve logging performance when logger disabled
https://bugs.python.org/issue33606 opened by vinay.sajip
#33607: [subinterpreters] Explicitly track object ownership (and alloc
https://bugs.python.org/issue33607 opened by eric.snow
#33608: [subinterpreters] Add a cross-interpreter-safe mechanism to in
https://bugs.python.org/issue33608 opened by eric.snow
#33609: Document that dicts preserve insertion order
https://bugs.python.org/issue33609 opened by yselivanov
#33610: IDLE: Make multiple improvements to CodeContext
https://bugs.python.org/issue33610 opened by terry.reedy
#33613: test_multiprocessing_fork: test_semaphore_tracker_sigint() fai
https://bugs.python.org/issue33613 opened by vstinner
#33614: Compilation of Python fails on AMD64 Windows8.1 Refleaks 3.x
https://bugs.python.org/issue33614 opened by vstinner
#33615: test__xxsubinterpreters crashed on x86 Gentoo Refleaks 3.x
https://bugs.python.org/issue33615 opened by vstinner
#33616: typing.NoReturn is undocumented
https://bugs.python.org/issue33616 opened by srittau
#33617: subprocess.Popen etc do not accept os.PathLike in passed seque
https://bugs.python.org/issue33617 opened by altendky
#33618: Support TLS 1.3
https://bugs.python.org/issue33618 opened by christian.heimes
#33623: Fix possible SIGSGV when asyncio.Future is created in __del__
https://bugs.python.org/issue33623 opened by yselivanov
#33624: Implement subclass hooks for asyncio abstract classes
https://bugs.python.org/issue33624 opened by asvetlov
#33625: Release GIL for grp.getgr{nam,gid} and pwd.getpw{nam,uid}
https://bugs.python.org/issue33625 opened by wg
#33627: test-complex of test_numeric_tower.test_complex() crashes
Re: [Python-Dev] Add __reversed__ methods for dict
It's worth nothing that OrderedDict already supports reversed(). The argument could go both ways: 1. dict is similar to OrderedDict nowadays, so it should support reversed() too; 2. you can use OrderedDict to signal explicitly that you care about ordering; no need to add anything to dict. Regards Antoine. On Thu, 24 May 2018 14:55:32 +0200 Rémi Lapeyre wrote: > > Hi, > > since dict keys are sorted by their insertion order since Python 3.6 and that > it’s part of Python specs since 3.7 a proposal has been made in bpo-33462 to > add the __reversed__ method to dict and dict views. > > Concerns have been raised in the comments that this feature may add too much > bloat in the core interpreter and be harmful for other Python implementations. > > Given the different issues this change creates, I see three possibilities: > > 1. Accept the proposal has it is for dict and dict views, this would add about > 300 lines and three new types in dictobject.c > > 2. Accept the proposal only for dict, this would add about 80 lines and one > new type in dictobject.c while still being useful for some use cases > > 3. Drop the proposal as the whole, while having some use, reversed(dict(a=1, > b=2)) > may not be very common and could be done using OrderedDict instead. > > What’s your stance on the issue ? > > Best regards, > Rémi Lapeyre > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available
I tried this implementation to add no-copy pickling for large numpy arrays and seems to work as expected (for a simple contiguous array). I took some notes on the numpy tracker to advertise this PEP to the numpy developers: https://github.com/numpy/numpy/issues/11161 -- Olivier ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add __reversed__ methods for dict
> On May 25, 2018, at 9:32 AM, Antoine Pitrou wrote: > > It's worth nothing that OrderedDict already supports reversed(). > The argument could go both ways: > > 1. dict is similar to OrderedDict nowadays, so it should support > reversed() too; > > 2. you can use OrderedDict to signal explicitly that you care about > ordering; no need to add anything to dict. Those are both valid sentiments :-) My thought is that guaranteed insertion order for regular dicts is brand new, so it will take a while for the notion settle in and become part of everyday thinking about dicts. Once that happens, it is probably inevitable that use cases will emerge and that __reversed__ will get added at some point. The implementation seems straightforward and it isn't much of a conceptual leap to expect that a finite ordered collection would be reversible. Given that dicts now track insertion order, it seems reasonable to want to know the most recent insertions (i.e. looping over the most recently added tasks in a task dict). Other possible use cases will likely correspond to how we use the Unix tail command. If those use cases arise, it would be nice for __reversed__ to already be supported so that people won't be tempted to implement an ugly workaround using popitem() calls followed by reinsertions. Raymond . ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available
> On May 24, 2018, at 10:57 AM, Antoine Pitrou wrote: > > While PEP 574 (pickle protocol 5 with out-of-band data) is still in > draft status, I've made available an implementation in branch "pickle5" > in my GitHub fork of CPython: > https://github.com/pitrou/cpython/tree/pickle5 > > Also I've published an experimental backport on PyPI, for Python 3.6 > and 3.7. This should help people play with the new API and features > without having to compile Python: > https://pypi.org/project/pickle5/ > > Any feedback is welcome. Thanks for doing this. Hope it isn't too late, but I would like to suggest that protocol 5 support fast compression by default. We normally pickle objects so that they can be transported (saved to a file or sent over a socket). Transport costs (reading and writing a file or socket) are generally proportional to size, so compression is likely to be a net win (much as it was for header compression in HTTP/2). The PEP lists compression as a possible a refinement only for large objects, but I expect is will be a win for most pickles to compress them in their entirety. Raymond ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add __reversed__ methods for dict
OK, +1 On Fri, May 25, 2018 at 10:26 AM, Raymond Hettinger < [email protected]> wrote: > > > > On May 25, 2018, at 9:32 AM, Antoine Pitrou wrote: > > > > It's worth nothing that OrderedDict already supports reversed(). > > The argument could go both ways: > > > > 1. dict is similar to OrderedDict nowadays, so it should support > > reversed() too; > > > > 2. you can use OrderedDict to signal explicitly that you care about > > ordering; no need to add anything to dict. > > Those are both valid sentiments :-) > > My thought is that guaranteed insertion order for regular dicts is brand > new, so it will take a while for the notion settle in and become part of > everyday thinking about dicts. Once that happens, it is probably > inevitable that use cases will emerge and that __reversed__ will get added > at some point. The implementation seems straightforward and it isn't much > of a conceptual leap to expect that a finite ordered collection would be > reversible. > > Given that dicts now track insertion order, it seems reasonable to want to > know the most recent insertions (i.e. looping over the most recently added > tasks in a task dict). Other possible use cases will likely correspond to > how we use the Unix tail command. > > If those use cases arise, it would be nice for __reversed__ to already be > supported so that people won't be tempted to implement an ugly workaround > using popitem() calls followed by reinsertions. > > > Raymond > > . > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available
On Fri, 25 May 2018 10:36:08 -0700 Raymond Hettinger wrote: > > On May 24, 2018, at 10:57 AM, Antoine Pitrou wrote: > > > > While PEP 574 (pickle protocol 5 with out-of-band data) is still in > > draft status, I've made available an implementation in branch "pickle5" > > in my GitHub fork of CPython: > > https://github.com/pitrou/cpython/tree/pickle5 > > > > Also I've published an experimental backport on PyPI, for Python 3.6 > > and 3.7. This should help people play with the new API and features > > without having to compile Python: > > https://pypi.org/project/pickle5/ > > > > Any feedback is welcome. > > Thanks for doing this. > > Hope it isn't too late, but I would like to suggest that protocol 5 support > fast compression by default. We normally pickle objects so that they can be > transported (saved to a file or sent over a socket). Transport costs (reading > and writing a file or socket) are generally proportional to size, so > compression is likely to be a net win (much as it was for header compression > in HTTP/2). > > The PEP lists compression as a possible a refinement only for large objects, > but I expect is will be a win for most pickles to compress them in their > entirety. It's not too late (the PEP is still a draft, and there's a lot of time before 3.8), but I wonder what would be the benefit of making it a part of the pickle specification, rather than compressing independently. Whether and how to compress is generally a compromise between transmission (or storage) speed and computation speed. Also, there are specialized compressors for higher efficiency (for example, Blosc has datatype-specific compression for Numpy arrays). Such knowledge can be embodied in domain-specific libraries such as Dask/distributed, but it cannot really be incorporated in pickle itself. Do you have something specific in mind? Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available
On 25.05.2018 20:36, Raymond Hettinger wrote: On May 24, 2018, at 10:57 AM, Antoine Pitrou wrote: While PEP 574 (pickle protocol 5 with out-of-band data) is still in draft status, I've made available an implementation in branch "pickle5" in my GitHub fork of CPython: https://github.com/pitrou/cpython/tree/pickle5 Also I've published an experimental backport on PyPI, for Python 3.6 and 3.7. This should help people play with the new API and features without having to compile Python: https://pypi.org/project/pickle5/ Any feedback is welcome. Thanks for doing this. Hope it isn't too late, but I would like to suggest that protocol 5 support fast compression by default. We normally pickle objects so that they can be transported (saved to a file or sent over a socket). Transport costs (reading and writing a file or socket) are generally proportional to size, so compression is likely to be a net win (much as it was for header compression in HTTP/2). The PEP lists compression as a possible a refinement only for large objects, but I expect is will be a win for most pickles to compress them in their entirety. I would advise against that. Pickle format is unreadable as it is, compression will make it literally impossible to diagnose problems. Python supports transparent compression, e.g. with the 'zlib' codec. Raymond ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available
On 2018-05-25, Antoine Pitrou wrote: > Do you have something specific in mind? I think compressed by default is a good idea. My quick proposal: - Use fast compression like lz4 or zlib with Z_BEST_SPEED - Add a 'compress' keyword argument with a default of None. For protocol 5, None means to compress. Providing 'compress' != None for older protocols will raise an error. The compression overhead will be small compared to the pickle/unpickle costs. If someone wants to apply their own (e.g. better) compression, they can set compress=False. An alternative idea is to have two different protocol formats. E.g. 5 and 6. One is "pickle 5" with compression, one without compression. I don't like that as much since it breaks the idea that higher protocol numbers are "better". Regards, Neil ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available
On Fri, 25 May 2018 14:50:57 -0600 Neil Schemenauer wrote: > On 2018-05-25, Antoine Pitrou wrote: > > Do you have something specific in mind? > > I think compressed by default is a good idea. My quick proposal: > > - Use fast compression like lz4 or zlib with Z_BEST_SPEED > > - Add a 'compress' keyword argument with a default of None. For > protocol 5, None means to compress. Providing 'compress' != None > for older protocols will raise an error. The question is what purpose does it serve for pickle to do it rather than for the user to compress the pickle themselves. You're basically saving one line of code. Am I missing some other advantage? (also note that it requires us to ship the lz4 library with Python, or another modern compression library such as zstd; zlib's performance characteristics are outdated) Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available
On 2018-05-25, Antoine Pitrou wrote: > The question is what purpose does it serve for pickle to do it rather > than for the user to compress the pickle themselves. You're basically > saving one line of code. It's one line of code everywhere pickling or unpicking happens. And you probably need to import a compression module, so at least two lines. Then maybe you need to figure out if the pickle is compressed and what kind of compression is used. So, add a few more lines. It seems logical to me that users of pickle want it to be fast and produce small pickles. Compressing by default seems the right choice, even though it complicates the implementation. Ivan brings up a valid point that compressed pickles are harder to debug. However, I think that's much less important than being small. > it requires us to ship the lz4 library with Python Yeah, that's not so great. I think zlib with Z_BEST_SPEED would be fine. However, some people might worry it is too slow or doesn't compress enough. Having lz4 as a battery included seems like a good idea anyhow. I understand that it is pretty well established as a useful compression method. Obviously requiring a new C library to be included expands the effort of implementation a lot. This discussion can easily lead into bikeshedding (e.g. relative merits of different compression schemes). Since I'm not volunteering to implement anything, I will stop responding at this point. ;-) Regards, Neil ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add __reversed__ methods for dict
I am also in agreement. On Fri, May 25, 2018, 13:49 Guido van Rossum wrote: > OK, +1 > > On Fri, May 25, 2018 at 10:26 AM, Raymond Hettinger < > [email protected]> wrote: > >> >> >> > On May 25, 2018, at 9:32 AM, Antoine Pitrou >> wrote: >> > >> > It's worth nothing that OrderedDict already supports reversed(). >> > The argument could go both ways: >> > >> > 1. dict is similar to OrderedDict nowadays, so it should support >> > reversed() too; >> > >> > 2. you can use OrderedDict to signal explicitly that you care about >> > ordering; no need to add anything to dict. >> >> Those are both valid sentiments :-) >> >> My thought is that guaranteed insertion order for regular dicts is brand >> new, so it will take a while for the notion settle in and become part of >> everyday thinking about dicts. Once that happens, it is probably >> inevitable that use cases will emerge and that __reversed__ will get added >> at some point. The implementation seems straightforward and it isn't much >> of a conceptual leap to expect that a finite ordered collection would be >> reversible. >> >> Given that dicts now track insertion order, it seems reasonable to want >> to know the most recent insertions (i.e. looping over the most recently >> added tasks in a task dict). Other possible use cases will likely >> correspond to how we use the Unix tail command. >> >> If those use cases arise, it would be nice for __reversed__ to already be >> supported so that people won't be tempted to implement an ugly workaround >> using popitem() calls followed by reinsertions. >> >> >> Raymond >> >> . >> >> ___ >> Python-Dev mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ramseydsilva%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available
On Fri, May 25, 2018 at 3:35 PM, Neil Schemenauer wrote: > This discussion can easily lead into bikeshedding (e.g. relative > merits of different compression schemes). Since I'm not > volunteering to implement anything, I will stop responding at this > point. ;-) I think the bikeshedding -- or more to the point, the fact that there's a wide variety of options for compressing pickles, and none of them are appropriate in all circumstances -- means that this is something that should remain a separate layer. Even super-fast algorithms like lz4 are inefficient when you're transmitting pickles between two processes on the same system – they still add extra memory copies. And that's a very common use case. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
