Re: [Python-Dev] PEP 488: elimination of PYO files
On Mar 07, 2015, at 12:30 PM, Scott Dial wrote: >As a packager, this PEP is a bit silent on it's expectations about what >will happen with (for instance) Debian and Fedora packages for Python. >My familiarity is with Fedora, and on that platform, we ship .pyc and >.pyo files (using -O for the .pyo). Is it your expectation that such >platforms will still distribute -O only? Or also -OO? In my world, all >of the __pycache__ directories are owned by root. On Debuntu, we don't create pyo files by default, although IIRC it's an option. The problem has been the collision between -O and -OO, so this PEP may open the door to us generating both optimization levels automatically. It's not something that's very high on my todo list, but if there's enough interest it probably wouldn't be difficult to add. Cheers, -Barry ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
For the record here: +1 on the PEP from me (the comments I made on import-sig have already incorporated into this version of the PEP) On 8 March 2015 at 08:03, Brett Cannon wrote: > > > On Sat, Mar 7, 2015 at 12:39 PM Scott Dial > wrote: >> >> On 2015-03-06 11:34 AM, Brett Cannon wrote: >> > This PEP proposes eliminating the concept of PYO files from Python. >> > To continue the support of the separation of bytecode files based on >> > their optimization level, this PEP proposes extending the PYC file >> > name to include the optimization level in bytecode repository >> > directory (i.e., the ``__pycache__`` directory). >> >> As a packager, this PEP is a bit silent on it's expectations about what >> will happen with (for instance) Debian and Fedora packages for Python. >> My familiarity is with Fedora, and on that platform, we ship .pyc and >> .pyo files (using -O for the .pyo). Is it your expectation that such >> platforms will still distribute -O only? Or also -OO? In my world, all >> of the __pycache__ directories are owned by root. > > > I assume they will generate all .pyc files at all levels, but I don't if > it's my place to dictate such a thing since bytecode files are an > optimization to Python itself and do not influence how people interact with > the interpreter like with PEP 394 (The "python" Command on Unix-Like > Systems). = Fedora near term = Nothing significant will change for RPMs with policy compliant (https://fedoraproject.org/wiki/Packaging:Python#Byte_compiling) spec files, even after switching to Python 3.5. The only different will be that the current .pyo files will be replaced with files using the new PEP 488 naming scheme. Folks using custom spec files with their own pattern matches that assume "pyo" may need to adjust them (e.g. to grab everything in __pycache__, as recommended in the packaging guidelines) = Fedora long term = After the switch to Python 3.5 (which, given the release schedule, I currently expect will be in Fedora 24 early next year), the RPM Python build macros *might* be changed to generate -OO files in addition to -O ones, making it easier to run Python applications in -OO mode to reduce memory usage. The PEP doesn't mandate such a change, but it does enable it. This would have the downside of making every Python package in the distro slightly larger (all 15k+ of them), so there'd need to be a clear and compelling pay-off to justify that cost. While folks may be tempted to say "disk space is cheap", VM and container provisioning latency are performance critical in a number of use cases, so making the Fedora Cloud and Atomic Host images even slightly larger isn't likely to win us any friends, so actually making such a change would require compelling benchmarks demonstrating reductions in runtime memory usage that justify the larger image size, as well as confirming that the change doesn't place undue pressure on the DVD ISO image contents. This isn't really the right list for more in-depth discussion of what Fedora *might* do though - any discussion will be entirely theoretical until after Python 3.5 comes out, and even then the appropriate list would be [email protected] rather than here. Regards, Nick. P.S. For those that aren't already aware, I became a voting member of Fedora's Environments & Stacks Working Group several months ago and now work closely with Slavek Kabrda (the Fedora Python maintainer, amongst other things) on Python's integration into Fedora and its derivatives. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
On 8 March 2015 at 12:13, Victor Stinner wrote: > Hi, > > FYI I commited the implementation of os.scandir() written by Ben Hoyt. > I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the > final patch today). Thanks to everyone that worked on getting this PEP through to Final status! Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
On 8 March 2015 at 02:13, Victor Stinner wrote: > FYI I commited the implementation of os.scandir() written by Ben Hoyt. > I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the > final patch today). Yay! Great news. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support
On 26 February 2015 at 21:48, Paul Moore wrote: > On 26 February 2015 at 21:34, Guido van Rossum wrote: >> Accepted! >> >> Thanks for your patience, Paul, and thanks everyone for their feedback. >> >> I know there are still a few small edits to the PEP, but those don't affect >> my acceptance. Congrats! > > Excellent, thanks to everyone for the helpful comments throughout. I'm still looking for someone who has the time to commit the final patch on http://bugs.python.org/issue23491. Python 3.5a2 is due out today (at least according to the PEP) so I've probably missed that, but it would be nice if it could be in for alpha 3 at least. Thanks, Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support
On Sun, Mar 8, 2015, 08:40 Paul Moore wrote: On 26 February 2015 at 21:48, Paul Moore wrote: > On 26 February 2015 at 21:34, Guido van Rossum wrote: >> Accepted! >> >> Thanks for your patience, Paul, and thanks everyone for their feedback. >> >> I know there are still a few small edits to the PEP, but those don't affect >> my acceptance. Congrats! > > Excellent, thanks to everyone for the helpful comments throughout. I'm still looking for someone who has the time to commit the final patch on http://bugs.python.org/issue23491. Python 3.5a2 is due out today (at least according to the PEP) so I've probably missed that, but it would be nice if it could be in for alpha 3 at least. If no one gets to it I can submit on Friday. -brett Thanks, Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 448 review
Anyone have time to do a code review?
http://bugs.python.org/issue2292
On Mon, Mar 2, 2015 at 4:54 PM, Neil Girdhar wrote:
> It's from five days ago. I asked Joshua to take a look at something, but
> I guess he is busy.
>
> Best,
>
> Neil
>
> —
>
> The latest file there is from Feb 26, while your message that the patch
> was ready for review is from today -- so is the
> patch from five days ago the most recent?
>
> --
> ~Ethan~
>
> On Mon, Mar 2, 2015 at 3:18 PM, Neil Girdhar
> wrote:
>
>> http://bugs.python.org/issue2292
>>
>> On Mon, Mar 2, 2015 at 3:17 PM, Victor Stinner
>> wrote:
>>
>>> Where is the patch?
>>>
>>> Victor
>>>
>>> Le lundi 2 mars 2015, Neil Girdhar a écrit :
>>>
>>> Hi everyone,
The patch is ready for review now, and I should have time this week to
make changes and respond to comments.
Best,
Neil
On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum
wrote:
> I'm back, I've re-read the PEP, and I've re-read the long thread with
> "(no subject)".
>
> I think Georg Brandl nailed it:
>
> """
>
>
>
>
>
>
>
>
> *I like the "sequence and dict flattening" part of the PEP, mostly
> because itis consistent and should be easy to understand, but the
> comprehension syntaxenhancements seem to be bad for readability and
> "comprehending" what the codedoes.The call syntax part is a mixed bag on
> the one hand it is nice to be consistent with the extended possibilities
> in
> literals (flattening), but on the other hand there would be small but
> annoying inconsistencies anyways (e.g. the duplicate kwarg case above).*
> """
>
> Greg Ewing followed up explaining that the inconsistency between dict
> flattening and call syntax is inherent in the pre-existing different rules
> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while
> f(a=1,
> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1,
> **{'a': 1}) is a TypeError.)
>
> For me, allowing f(*a, *b) and f(**d, **e) and all the other
> combinations for function calls proposed by the PEP is an easy +1 -- it's
> a
> straightforward extension of the existing pattern, and anybody who knows
> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d,
> **e) means shouldn't be hard either. Understanding the edge case for
> duplicate keys with f(**d, **e) is a little harder, but the error messages
> are pretty clear, and it is not a new edge case.
>
> The sequence and dict flattening syntax proposals are also clean and
> logical -- we already have *-unpacking on the receiving side, so allowing
> *x in tuple expressions reads pretty naturally (and the similarity with *a
> in argument lists certainly helps). From here, having [a, *x, b, *y] is
> also natural, and then the extension to other displays is natural: {a, *x,
> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me.
>
> So that leaves comprehensions. IIRC, during the development of the
> patch we realized that f(*x for x in xs) is sufficiently ambiguous that we
> decided to disallow it -- note that f(x for x in xs) is already somewhat
> of
> a special case because an argument can only be a "bare" generator
> expression if it is the only argument. The same reasoning doesn't apply
> (in
> that form) to list, set and dict comprehensions -- while f(x for x in xs)
> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the
> same as [(x for x in xs)] (that's a list of one element, and the element
> is
> a generator expression).
>
> The basic premise of this part of the proposal is that if you have a
> few iterables, the new proposal (without comprehensions) lets you create a
> list or generator expression that iterates over all of them, essentially
> flattening them:
>
> >>> xs = [1, 2, 3]
> >>> ys = ['abc', 'def']
> >>> zs = [99]
> >>> [*xs, *ys, *zs]
> [1, 2, 3, 'abc', 'def', 99]
> >>>
>
> But now suppose you have a list of iterables:
>
> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
> >>> [*xss[0], *xss[1], *xss[2]]
> [1, 2, 3, 'abc', 'def', 99]
> >>>
>
> Wouldn't it be nice if you could write the latter using a
> comprehension?
>
> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
> >>> [*xs for xs in xss]
> [1, 2, 3, 'abc', 'def', 99]
> >>>
>
> This is somewhat seductive, and the following is even nicer: the *xs
> position may be an expression, e.g.:
>
> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
> >>> [*xs[:2] for xs in xss]
> [1, 2, 'abc', 'def', 99]
> >>>
>
> On the other hand, I had to explore the possibilities here by
>
Re: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
On 8 March 2015 at 13:31, Ben Hoyt wrote: > Thanks for committing this, Victor! And fixing the d_type issue on funky > platforms. > > Others: if you want to benchmark this, the simplest way is to use my > os.walk() benchmark.py test program here: > https://github.com/benhoyt/scandir -- it compares the built-in os.walk() > implemented with os.listdir() with a version of walk() implemented with > os.scandir(). I see huge gains on Windows (12-50x) and modest gains on my > Linux VM (3-5x). > > Note that the actual CPython version of os.walk() doesn't yet use > os.scandir(). I intend to open a separate issue for that shortly (or Victor > can). But that part should be fairly straight-forward, as I already have a > version available in my GitHub project. > I think it would be a good idea to report the type of drive/mount along with the results. I imagine that there might be significant differences between solid state drives, hard drives and network mounts. Tim Delaney ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Tunning binary insertion sort algorithm in Timsort.
We can optimize the TimSort algorithm by optimizing its binary insertion sort. The current version of binary insertion sort use this idea: Use binary search to find a final position in sorted list for a new element X. Then insert X to that location. I suggest another idea: Use binary search to find a final postion in sorted list for a new element X. Before insert X to that location, compare X with its next element. For the next element, we already know if it is lower or bigger than X, so we can reduce the search area to the left side or on the right side of X in the sorted list. I have applied my idea on java.util. ComparableTimSort.sort() and testing. The execute time is reduced by 2%-6% with array of random integer. Here is detail about algorithm and testing: https://github.com/nhapq/Optimize_binary_insertion_sort Sincerely. phqnha ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] boxing and unboxing data types
When data is passed from Python to a native library (such as in an O/S call),
how does the unboxing of data types occur?
For a specific instance, os.open allows an integer whose various bits express
desired behavior as `flags` -- if flags is
1, for example, the file is open write-only.
If I pass an int-like object to os.open, __int__ is called; if I pass a thin
wrapper over int to os.open, __int__ is not
called.
So the real question: anywhere in Python where an int is expected (for
lower-level API work), but not directly
received, should __int__ (or __index__) be called? and failure to do so is a
bug?
Here's my simple test code:
class Wrap:
def __init__(self, value):
self.value = value
def __int__(self):
print('__int__')
return self.value
def __index__(self):
print('__index__')
return self.value
class Thin(int):
def __int__(self):
print('__int__')
return super().__int__()
def __index__(self):
print('__index__')
return super().__index__()
two = Wrap(2)
[0, 1, 2][two]
# __index__
# 2
import struct
struct.pack('i', two)
# __index__
# b'\x02\x00\x00\x00'
t = Thin(1)
huh = os.open('blah.txt', t)
# Traceback (most recent call last):
# File "", line 1, in
# FileNotFoundError: [Errno 2] No such file or directory: 'blah.txt'
--
~Ethan~
signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] boxing and unboxing data types
On Sun, Mar 08, 2015 at 08:31:30PM -0700, Ethan Furman wrote:
> When data is passed from Python to a native library (such as in an O/S
> call), how does the unboxing of data types occur?
[...]
> So the real question: anywhere in Python where an int is expected (for
> lower-level API work), but not directly received, should __int__ (or
> __index__) be called? and failure to do so is a bug?
I think the answer is in the docs:
https://docs.python.org/3/reference/datamodel.html#object.__int__
Immediately below that __index__ is described, with this note:
In order to have a coherent integer type class, when
__index__() is defined __int__() should also be defined,
and both should return the same value.
The PEP adding __index__ is also useful:
https://www.python.org/dev/peps/pep-0357/
My summary is as follows:
__int__ is used as the special method for int(), and it should coerce
the object to an integer. This may be lossy e.g. int(2.999) --> 2 or may
involve a conversion from a non-numeric type to integer e.g. int("2").
__index__ is used when the object in question actually represents an
integer of some kind, e.g. a fixed-with integer. Conversion should be
lossless and conceptually may be thought of a way of telling Python
"this value actually is an int, even though it doesn't inherit from int"
(for some definition of "is an int").
There's no built-in way of calling __index__ that I know of (no
equivalent to int(obj)), but slicing at the very least will call it,
e.g. seq[a:] will call type(a).__index__.
If you define __index__ for your class, you should also define __int__
and have the two return the same value. I would expect that an IntFlags
object should inherit from int, and if that is not possible, practical
or desirable for some reason, then it should define __index__ and
__int__.
Failure to call __index__ is not necessarily a bug. I think it is
allowed for functions to insist on an actual int, as slicing did for
many years, but it is an obvious enhancement to allow such functions to
accept arbitrary int-like objects.
Does that answer your questions?
--
Steve
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] boxing and unboxing data types
On Mar 8, 2015 9:13 PM, "Steven D'Aprano" wrote: > > There's no built-in way of calling __index__ that I know of (no > equivalent to int(obj)), There's operator.index(obj), at least. > but slicing at the very least will call it, > e.g. seq[a:] will call type(a).__index__. -n ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] boxing and unboxing data types
On 03/08/2015 09:12 PM, Steven D'Aprano wrote: > Does that answer your questions? No, unfortunately. You correctly guessed my question is motivated by the IntFlag discussion. I guess it could boil down to: if IntEnum was not based on 'int', but instead had the __int__ and __index__ methods (plus all the other __xxx__ methods that int has), would it still be a drop-in replacement for actual ints? Even when being used to talk to non-Python libs? -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] boxing and unboxing data types
On 09.03.15 06:33, Ethan Furman wrote:
I guess it could boil down to: if IntEnum was not based on 'int', but instead
had the __int__ and __index__ methods
(plus all the other __xxx__ methods that int has), would it still be a drop-in
replacement for actual ints? Even when
being used to talk to non-Python libs?
If you don't call isinstance(x, int) (PyLong_Check* in C).
Most conversions from Python to C implicitly call __index__ or __int__,
but unfortunately not all.
>>> float(Thin(42))
42.0
>>> float(Wrap(42))
Traceback (most recent call last):
File "", line 1, in
TypeError: float() argument must be a string or a number, not 'Wrap'
>>> '%*s' % (Thin(5), 'x')
'x'
>>> '%*s' % (Wrap(5), 'x')
Traceback (most recent call last):
File "", line 1, in
TypeError: * wants int
>>> OSError(Thin(2), 'No such file or directory')
FileNotFoundError(2, 'No such file or directory')
>>> OSError(Wrap(2), 'No such file or directory')
OSError(<__main__.Wrap object at 0xb6fe81ac>, 'No such file or directory')
>>> re.match('(x)', 'x').group(Thin(1))
'x'
>>> re.match('(x)', 'x').group(Wrap(1))
Traceback (most recent call last):
File "", line 1, in
IndexError: no such group
And to be ideal drop-in replacement IntEnum should override such methods
as __eq__ and __hash__ (so it could be used as mapping key). If all
methods should be overridden to quack as int, why not take an int?
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] boxing and unboxing data types
On 03/08/2015 11:07 PM, Serhiy Storchaka wrote: > If you don't call isinstance(x, int) (PyLong_Check* in C). > > Most conversions from Python to C implicitly call __index__ or __int__, but > unfortunately not all. [snip examples] Thanks, Serhiy, that's what I was looking for. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
On 3/8/2015 8:17 PM, nha pham wrote: We can optimize the TimSort algorithm by optimizing its binary insertion sort. The current version of binary insertion sort use this idea: Use binary search to find a final position in sorted list for a new element X. Then insert X to that location. I suggest another idea: Use binary search to find a final postion in sorted list for a new element X. Before insert X to that location, compare X with its next element. For the next element, we already know if it is lower or bigger than X, so we can reduce the search area to the left side or on the right side of X in the sorted list. I have applied my idea on java.util. ComparableTimSort.sort() and testing. The execute time is reduced by 2%-6% with array of random integer. Here is detail about algorithm and testing: https://github.com/nhapq/Optimize_binary_insertion_sort Apply the idea to timsort in python, and try the different cases discussed in Tim's documentation. If it still looks good, open an issue with the patch and put tim.peters as nosy. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
