Re: [Python-Dev] bpo-30806 netrc.__repr__() is broken for writing to file (GH-2491)
30.09.17 10:10, INADA Naoki пише: https://github.com/python/cpython/commit/b24cd055ecb3eea9a15405a6ca72dafc739e6531 commit: b24cd055ecb3eea9a15405a6ca72dafc739e6531 branch: master author: James Sexton committer: INADA Naoki date: 2017-09-30T16:10:31+09:00 summary: bpo-30806 netrc.__repr__() is broken for writing to file (GH-2491) netrc file format doesn't support quotes and escapes. See https://linux.die.net/man/5/netrc The commit message looks confusing to me. Is netrc.__repr__() is broken now? Or this change makes netrc file format supporting quotes and escapes now? Please read the following thread: https://mail.python.org/pipermail/python-dev/2011-May/111303.html. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)
On Sep 29, 2017 18:21, "Guido van Rossum" wrote: PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and Elvis. This is getting really off-topic, but I do have updates to add to PEP 555 if there is interest in that. IMO, 555 is better and most likely faster than 550, but on the other hand, the issues with PEP 550 are most likely not going to be a problem for me personally. -- Koos ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)
Your PEP is currently incomplete. If you don't finish it, it is not even a contender. But TBH it's not my favorite anyway, so you could also just withdraw it. On Oct 1, 2017 9:13 AM, "Koos Zevenhoven" wrote: > On Sep 29, 2017 18:21, "Guido van Rossum" wrote: > > > PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and > Elvis. > > > This is getting really off-topic, but I do have updates to add to PEP 555 > if there is interest in that. IMO, 555 is better and most likely faster > than 550, but on the other hand, the issues with PEP 550 are most likely > not going to be a problem for me personally. > > -- Koos > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)
On Oct 1, 2017 19:26, "Guido van Rossum" wrote: Your PEP is currently incomplete. If you don't finish it, it is not even a contender. But TBH it's not my favorite anyway, so you could also just withdraw it. I can withdraw it if you ask me to, but I don't want to withdraw it without any reason. I haven't changed my mind about the big picture. OTOH, PEP 521 is elegant and could be used to implement PEP 555, but 521 is almost certainly less performant and has some problems regarding context manager wrappers that use composition instead of inheritance. -- Koos On Oct 1, 2017 9:13 AM, "Koos Zevenhoven" wrote: > On Sep 29, 2017 18:21, "Guido van Rossum" wrote: > > > PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and > Elvis. > > > This is getting really off-topic, but I do have updates to add to PEP 555 > if there is interest in that. IMO, 555 is better and most likely faster > than 550, but on the other hand, the issues with PEP 550 are most likely > not going to be a problem for me personally. > > -- Koos > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Investigating time for `import requests`
See also https://github.com/requests/requests/issues/4315 I tried new `-X importtime` option to `import requests`. Full output is here: https://gist.github.com/methane/96d58a29e57e5be97769897462ee1c7e Currently, it took about 110ms. And major parts are from Python stdlib. Followings are root of slow stdlib subtrees. import time: self [us] | cumulative | imported package import time: 1374 | 14038 | logging import time: 2636 | 4255 | socket import time: 2902 | 11004 | ssl import time: 1162 | 16694 | http.client import time: 656 | 5331 | cgi import time: 7338 | 7867 | http.cookiejar import time: 2930 | 2930 | http.cookies *1. logging* logging is slow because it is imported in early stage. It imports many common, relatively slow packages. (collections, functools, enum, re). Especially, traceback module is slow because linecache. import time: 1419 | 5016 | tokenize import time: 200 | 5910 | linecache import time: 347 | 8869 | traceback I think it's worth enough to import linecache lazily. *2. socket* import time: 807 | 1221 | selectors import time: 2636 | 4255 | socket socket imports selectors for socket.send_file(). And selectors module use ABC. That's why selectors is bit slow. And socket module creates four enums. That's why import socket took more than 2.5ms excluding subimports. *3. ssl* import time: 2007 | 2007 | ipaddress import time: 2386 | 2386 | textwrap import time: 2723 | 2723 | _ssl ... import time: 306 |988 | base64 import time: 2902 | 11004 | ssl I already created pull request about removing textwrap dependency from ssl. https://github.com/python/cpython/pull/3849 ipaddress and _ssl module are bit slow too. But I don't know we can improve them or not. ssl itself took 2.9 ms. It's because ssl has six enums. *4. http.client* import time: 1376 | 2448 | email.header ... import time: 1469 | 7791 | email.utils import time: 408 | 10646 | email._policybase import time: 939 | 12210 | email.feedparser import time: 322 | 12720 | email.parser ... import time: 599 | 1361 | email.message import time: 1162 | 16694 | http.client email.parser has very large import tree. But I don't know how to break the tree. *5. cgi* import time: 1083 | 1083 | html.entities import time: 560 | 1643 | html ... import time: 656 | 2609 | shutil import time: 424 | 3033 | tempfile import time: 656 | 5331 | cgi cgi module uses tempfile to save uploaded file. But requests imports cgi just for `cgi.parse_header()`. tempfile is not used. Maybe, it's worth enough to import it lazily. FYI, cgi depends on very slow email.parser too. But this tree doesn't contain it because http.client is imported before cgi. Even though it's not problem for requests, it may affects to real CGI application. Of course, startup time is very important for CGI applications too. *6. http.cookiejar and http.cookies* It's slow because it has many `re.compile()` *Ideas* There are some places to break large import tree by "import in function" hack. ABC is slow, and it's used widely without almost no real need. (Who need selectors is ABC?) We can't remove ABC dependency because of backward compatibility. But I hope ABC is implemented in C by Python 3.7. Enum is slow, maybe slower than most people think. I don't know why exactly, but I suspect that it's because namespace dict implemented in Python. Anyway, I think we can have C implementation of IntEnum and IntFlag, like namedtpule vs PyStructSequence. It doesn't need to 100% compatible with current enum. Especially, no need for using metaclass. Another major slowness comes from compiling regular expression. I think we can increase cache size of `re.compile` and use ondemand cached compiling (e.g. `re.match()`), instead of "compile at import time" in many modules. PEP 562 -- Module __getattr__ helps a lot too. It make possible to split collection module and strings module. (strings module is used often for constants like strings.ascii_letters, but strings.Template cause import time re.compile()) Regards, -- Inada Naoki ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Investigating time for `import requests`
On Sun, Oct 1, 2017 at 7:04 PM, INADA Naoki wrote: > 4. http.client > > import time: 1376 | 2448 | email.header > ... > import time: 1469 | 7791 | email.utils > import time: 408 | 10646 | email._policybase > import time: 939 | 12210 | email.feedparser > import time: 322 | 12720 | email.parser > ... > import time: 599 | 1361 | email.message > import time: 1162 | 16694 | http.client > > email.parser has very large import tree. > But I don't know how to break the tree. There is some work to get urllib3/requests to stop using http.client, though it's not clear if/when it will actually happen: https://github.com/shazow/urllib3/pull/1068 > Another major slowness comes from compiling regular expression. > I think we can increase cache size of `re.compile` and use ondemand cached > compiling (e.g. `re.match()`), > instead of "compile at import time" in many modules. In principle re.compile() itself could be made lazy -- return a regular exception object that just holds the string, and then compiles and caches it the first time it's used. Might be tricky to do in a backwards compatibility way if it moves detection of invalid regexes from compile time to use time, but it could be an opt-in flag. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Investigating time for `import requests`
On 10/1/2017 7:34 PM, Nathaniel Smith wrote: Another major slowness comes from compiling regular expression. I think we can increase cache size of `re.compile` and use ondemand cached compiling (e.g. `re.match()`), instead of "compile at import time" in many modules. In principle re.compile() itself could be made lazy -- return a regular exception object that just holds the string, and then compiles and caches it the first time it's used. Might be tricky to do in a backwards compatibility way if it moves detection of invalid regexes from compile time to use time, but it could be an opt-in flag. Would be interesting to know how many of the in-module, compile time re.compile calls use dynamic values, versus string constants. Seems like string constant parameters to re.compile calls could be moved to on-first-use compiling without significant backwards incompatibility impact if there is an adequate test suite... and if there isn't an adequate test suite, should we care about the deferred detection? ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)
On Sun, Oct 1, 2017 at 1:52 PM, Koos Zevenhoven wrote: > On Oct 1, 2017 19:26, "Guido van Rossum" wrote: > > Your PEP is currently incomplete. If you don't finish it, it is not even a > contender. But TBH it's not my favorite anyway, so you could also just > withdraw it. > > > I can withdraw it if you ask me to, but I don't want to withdraw it > without any reason. I haven't changed my mind about the big picture. OTOH, > PEP 521 is elegant and could be used to implement PEP 555, but 521 is > almost certainly less performant and has some problems regarding context > manager wrappers that use composition instead of inheritance. > It is my understanding that PEP 521 (which proposes to add optional __suspend__ and __resume__ methods to the context manager protocol, to be called whenever a frame is suspended or resumed inside a `with` block) is no longer a contender because it would be way too slow. I haven't read it recently or thought about it, so I don't know what the second issue you mention is about (though it's presumably about the `yield` in a context manager implemented using a generator decorated with `@contextlib.contextmanager`). So it's really between PEP 550 and PEP 555. And there are currently too many parts of PEP 555 that are left to the imagination of the reader. So, again, I ask you to put up or shut up. It's your choice. If you don't want to do the work completing the PEP you might as well withdraw (once I am satisfied with Yury's PEP I will just accept it when there's no contender). If you do complete it I will probably still choose PEP 550 -- but at the moment the choice would be between something I understand completely and something that's too poorly specified to be able to reason about it. --Guido > -- Koos > > > > On Oct 1, 2017 9:13 AM, "Koos Zevenhoven" wrote: > >> On Sep 29, 2017 18:21, "Guido van Rossum" wrote: >> >> >> PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and >> Elvis. >> >> >> This is getting really off-topic, but I do have updates to add to PEP 555 >> if there is interest in that. IMO, 555 is better and most likely faster >> than 550, but on the other hand, the issues with PEP 550 are most likely >> not going to be a problem for me personally. >> >> -- Koos >> > > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)
One more thing. I would really appreciate it if you properly wrapped lines in your PEP around column 72 instead of using a single line per paragraph. This is the standard convention, see the template in PEP 12. On Sun, Oct 1, 2017 at 8:42 PM, Guido van Rossum wrote: > On Sun, Oct 1, 2017 at 1:52 PM, Koos Zevenhoven wrote: > >> On Oct 1, 2017 19:26, "Guido van Rossum" wrote: >> >> Your PEP is currently incomplete. If you don't finish it, it is not even >> a contender. But TBH it's not my favorite anyway, so you could also just >> withdraw it. >> >> >> I can withdraw it if you ask me to, but I don't want to withdraw it >> without any reason. I haven't changed my mind about the big picture. OTOH, >> PEP 521 is elegant and could be used to implement PEP 555, but 521 is >> almost certainly less performant and has some problems regarding context >> manager wrappers that use composition instead of inheritance. >> > > It is my understanding that PEP 521 (which proposes to add optional > __suspend__ and __resume__ methods to the context manager protocol, to be > called whenever a frame is suspended or resumed inside a `with` block) is > no longer a contender because it would be way too slow. I haven't read it > recently or thought about it, so I don't know what the second issue you > mention is about (though it's presumably about the `yield` in a context > manager implemented using a generator decorated with > `@contextlib.contextmanager`). > > So it's really between PEP 550 and PEP 555. And there are currently too > many parts of PEP 555 that are left to the imagination of the reader. So, > again, I ask you to put up or shut up. It's your choice. If you don't want > to do the work completing the PEP you might as well withdraw (once I am > satisfied with Yury's PEP I will just accept it when there's no contender). > If you do complete it I will probably still choose PEP 550 -- but at the > moment the choice would be between something I understand completely and > something that's too poorly specified to be able to reason about it. > > --Guido > > >> -- Koos >> >> >> >> On Oct 1, 2017 9:13 AM, "Koos Zevenhoven" wrote: >> >>> On Sep 29, 2017 18:21, "Guido van Rossum" wrote: >>> >>> >>> PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and >>> Elvis. >>> >>> >>> This is getting really off-topic, but I do have updates to add to PEP >>> 555 if there is interest in that. IMO, 555 is better and most likely faster >>> than 550, but on the other hand, the issues with PEP 550 are most likely >>> not going to be a problem for me personally. >>> >>> -- Koos >>> >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 553
Hope you don't mind me CC'ing python-dev.
On Sun, Oct 1, 2017 at 9:38 AM, Barry Warsaw wrote:
> You seem to be in PEP review mode :)
>
> What do you think about 553? Still want to wait, or do you think it’s
> missing anything? So far, all the feedback has been positive, and I think
> we can basically just close the open issues and both the PEP and
> implementation are ready to go.
>
> Happy to do more work on it if you feel it’s necessary.
> -B
>
I'm basically in agreement. Some quick notes:
- There's a comma instead of a period at the end of the 4th bullet in the
Rationale: "Breaking the idiom up into two lines further complicates the
use of the debugger," . Also I don't understand how this complicates use
(even though I also always type it as one line). And the lint warning is
actually useful when you accidentally leave this in code you send to CI
(assuming that runs a linter, as it should). TBH the biggest argument (to
me) is that I simply don't know *how* I would enter some IDE's debugger
programmatically. I think it should also be pointed out that even if an IDE
has a way to specify conditional breakpoints, the UI may be such that it's
easier to just add the check to the code -- and then the breakpoint()
option is much more attractive than having to look up how it's done in your
particular IDE (especially since this is not all that common).
- There's no rationale for the *args, **kwds part of the breakpoint()
signature. (I vaguely recall someone on the mailing list asking for it but
it seemed far-fetched at best.)
- The explanation of the relationship between sys.breakpoint() and
sys.__breakpointhook__ was unclear to me -- I had to go to the docs for
__displayhook__ (
https://docs.python.org/3/library/sys.html#sys.__displayhook__) to
understand that sys.__breakpointhook__ is simply initialized to the same
function as sys.breakpointhook, and the idea is that if you screw up you
can restore sys.breakpointhook from the other rather than having to restart
your process. Is that in fact it? The text "``sys.__breakpointhook__`` then
stashes the default value of ``sys.breakpointhook()`` to make it easy to
reset" seems to imply some action that would happen... when? how?
- Some pseudo-code would be nice. It seems that it's like this:
# in builtins
def breakpoint(*args, **kwds):
import sys
return sys.breakpointhook(*args, **kwds)
# in sys
def breakpointhook(*args, **kwds):
import os
hook = os.getenv('PYTHONBREAKPOINT')
if hook == '0':
return None
if not hook:
import pdb
return pdb.set_trace(*args, **kwds)
if '.' not in hook:
import builtins
mod = builtins
funcname = hook
else:
modname, funcname = hook.rsplit('.', 1)
__import__(modname)
import sys
mod = sys.modules[modname]
func = getattr(mod, funcname)
return func(*args, **kwds)
__breakpointhook__ = breakpointhook
Except that the error handling should be a bit better. (In particular the
PEP specifies a try/except around most of the code in sys.breakpointhook()
that issues a RuntimeWarning and returns None.)
- Not sure what the PEP's language around evaluation of PYTHONBREAKPOINT
means for the above pseudo code. I *think* the PEP author's opinion is that
the above pseudo-code is fine. Since programs can mutate their own
environment, I think something like `os.environ['PYTHONBREAKPOINT'] =
'foo.bar.baz'; breakpoint()` should result in foo.bar.baz() being imported
and called, right?
- I'm not quite sure what sort of fast-tracking for PYTHONBREAKPOINT=0 you
had in mind beyond putting it first in the code above.
- Did you get confirmation from other debuggers? E.g. does it work for
IDLE, Wing IDE, PyCharm, and VS 2015?
- I'm not sure what the point would be of making a call to breakpoint() a
special opcode (it seems a lot of work for something of this nature). ISTM
that if some IDE modifies bytecode it can do whatever it well please
without a PEP.
- I don't see the point of calling `pdb.pm()` at breakpoint time. But it
could be done using the PEP with `import pdb; sys.breakpointhook = pdb.pm`
right? So this hardly deserves an open issue.
- I haven't read the actual implementation in the PR. A PEP should not
depend on the actual proposed implementation for disambiguation of its
specification (hence my proposal to add pseudo-code to the PEP).
That's what I have!
--
--Guido van Rossum (python.org/~guido )
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Investigating time for `import requests`
> On Oct 1, 2017, at 7:34 PM, Nathaniel Smith wrote: > > In principle re.compile() itself could be made lazy -- return a > regular exception object that just holds the string, and then compiles > and caches it the first time it's used. Might be tricky to do in a > backwards compatibility way if it moves detection of invalid regexes > from compile time to use time, but it could be an opt-in flag. ISTM that someone writing ``re.compile(pattern)`` is explicitly saying they want the regex to be pre-compiled. For cache on first-use, we already have a way to do that with ``re.search(pattern, some string)`` which compiles and then caches. What would be more interesting would be to have a way to save the compiled regex in a pyc file so that it can be restored on load rather than recomputed. Also, we should remind ourselves that making more and more things lazy is a false optimization unless those things never get used. Otherwise, all we're doing is ending the timing before all the relevant work is done. If the lazy object does get used, we've made the actual total execution time worse (because of the overhead of the lazy evaluation logic). Raymond ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 553
On 10/2/2017 12:44 AM, Guido van Rossum wrote:
- There's no rationale for the *args, **kwds part of the breakpoint()
signature. (I vaguely recall someone on the mailing list asking for it
but it seemed far-fetched at best.)
If IDLE's event-driven GUI debugger were rewritten to run in the user
process, people wanting to debug a tkinter program should be able to
pass in their root, with its mainloop, rather than having the debugger
create its own, as it normally would. Something else could come up.
- The explanation of the relationship between sys.breakpoint() and
sys.__breakpointhook__ was unclear to me -- I had to go to the docs for
__displayhook__
(https://docs.python.org/3/library/sys.html#sys.__displayhook__) to
understand that sys.__breakpointhook__ is simply initialized to the same
function as sys.breakpointhook, and the idea is that if you screw up you
can restore sys.breakpointhook from the other rather than having to
restart your process. Is that in fact it? The text
"``sys.__breakpointhook__`` then stashes the default value of
``sys.breakpointhook()`` to make it easy to reset" seems to imply some
action that would happen... when? how?
- Some pseudo-code would be nice. It seems that it's like this:
This will be helpful to anyone implementing their own breakpointhook.
# in builtins
def breakpoint(*args, **kwds):
import sys
return sys.breakpointhook(*args, **kwds)
# in sys
def breakpointhook(*args, **kwds):
import os
hook = os.getenv('PYTHONBREAKPOINT')
if hook == '0':
return None
if not hook:
import pdb
return pdb.set_trace(*args, **kwds)
if '.' not in hook:
import builtins
mod = builtins
funcname = hook
else:
modname, funcname = hook.rsplit('.', 1)
__import__(modname)
import sys
mod = sys.modules[modname]
func = getattr(mod, funcname)
return func(*args, **kwds)
__breakpointhook__ = breakpointhook
Except that the error handling should be a bit better. (In particular
the PEP specifies a try/except around most of the code in
sys.breakpointhook() that issues a RuntimeWarning and returns None.)
- Not sure what the PEP's language around evaluation of PYTHONBREAKPOINT
means for the above pseudo code. I *think* the PEP author's opinion is
that the above pseudo-code is fine. Since programs can mutate their own
environment, I think something like `os.environ['PYTHONBREAKPOINT'] =
'foo.bar.baz'; breakpoint()` should result in foo.bar.baz() being
imported and called, right?
- I'm not quite sure what sort of fast-tracking for PYTHONBREAKPOINT=0
you had in mind beyond putting it first in the code above.
- Did you get confirmation from other debuggers? E.g. does it work for
IDLE, Wing IDE, PyCharm, and VS 2015?
- I'm not sure what the point would be of making a call to breakpoint()
a special opcode (it seems a lot of work for something of this nature).
ISTM that if some IDE modifies bytecode it can do whatever it well
please without a PEP.
- I don't see the point of calling `pdb.pm ()` at
breakpoint time. But it could be done using the PEP with `import pdb;
sys.breakpointhook = pdb.pm ` right? So this hardly
deserves an open issue.
- I haven't read the actual implementation in the PR. A PEP should not
depend on the actual proposed implementation for disambiguation of its
specification (hence my proposal to add pseudo-code to the PEP).
--
Terry Jan Reedy
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
