Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs
A progmatic workaround for yourself: https://addons.mozilla.org/de/firefox/addon/py3direct/ https://chrome.google.com/webstore/detail/py3redirect/codfjigcljdnlklcaopdciclmmdandig Bar Harel schrieb am Do., 7. Sep. 2017 um 07:52 Uhr: > A bit radical but do you believe we can contact Google to alter the search > results? > It's for the benefit of the user after all, and many just switch to python > 3 in the version picker anyway. > > On Thu, Sep 7, 2017, 4:18 AM Guido van Rossum wrote: > >> That's a good idea. Could you suggest it to >> https://github.com/python/pythondotorg/ ? (There is actually a version >> switcher on every page, but it's rather polite. :-) >> >> On Wed, Sep 6, 2017 at 3:52 PM, Ryan Gonzalez wrote: >> >>> Right now, many Google searches for Python modules return the Python 2 >>> documentation. IMO since 2 will be reaching EOL in around 3 years, it >>> would be nice to have a giant red box at the top with a link to the >>> Python 3 documentation. >>> >>> SFML already does something like this: >>> https://www.sfml-dev.org/tutorials/2.3/ >>> >>> (I mean, it would be even nicer to have a "jump to latest version" for >>> *all* not-new Python versions, though I figured just 2 would be a lot >>> easier.) >>> >>> -- >>> Ryan (ライアン) >>> Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else >>> http://refi64.com/ >>> ___ >>> Python-ideas mailing list >>> [email protected] >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> ___ >> Python-ideas mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > ___ > Python-ideas mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] if as
Hi! I’ve been having this idea for a few years and I thought I finally see if what others think of it. I have no experience in language design and I don’t know if this is something I’ve picked up in some other language. I also do not know what the ramifications of implementing this idea would be. I just keep thinking about it :) I quite often write code like the following in python: result = computation() if result: do_something_with_computation(result) More often than not this snippet evolves from something like this: if computation(): … That is, I use the truthiness of the value at first. As the code grows I refactor to actually do something with the result. What I would love to see is the following syntax instead, which to me is much cleaner: if computation() as result: do_something_with_result(result) Basically the result variable would be the result of the if condition’s expression and it would be available the same way it would be if we used my initial snippet (much the same way that the result of with expressions also stays around outside the with-block). Any feedback is appreciated :) Cheers, Denis ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] if as
On 7 September 2017 at 11:43, Denis Krienbühl wrote: > What I would love to see is the following syntax instead, which to me is much > cleaner: > >if computation() as result: >do_something_with_result(result) Hi - thanks for your suggestion! This has actually come up quite a lot in the past. Here's a couple of links to threads you might want to read (it's not surprising if you missed these, it's not that easy to come up with a good search term for this topic). https://mail.python.org/pipermail/python-ideas/2012-January/013461.html https://mail.python.org/pipermail/python-ideas/2009-March/003423.html (This thread includes a note by Guido that he intentionally left out this functionality) In summary, it's a reasonably commonly suggested idea, but there's not enough benefit to warrant adding it to the language. Paul ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] if as
> On 7 Sep 2017, at 13:11, Paul Moore wrote: > > On 7 September 2017 at 11:43, Denis Krienbühl wrote: >> What I would love to see is the following syntax instead, which to me is >> much cleaner: >> >> if computation() as result: >> do_something_with_result(result) > > Hi - thanks for your suggestion! This has actually come up quite a lot > in the past. Here's a couple of links to threads you might want to > read (it's not surprising if you missed these, it's not that easy to > come up with a good search term for this topic). > > https://mail.python.org/pipermail/python-ideas/2012-January/013461.html > https://mail.python.org/pipermail/python-ideas/2009-March/003423.html > (This thread includes a note by Guido that he intentionally left out > this functionality) > > In summary, it's a reasonably commonly suggested idea, but there's not > enough benefit to warrant adding it to the language. > > Paul I see, thank you for digging those threads up. I’ll read them to learn a thing or two :) ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Extension of python/json syntax to support explicitly sets and ordered dict.
Hi all,
few days ago I thought about a way to rapresent sets and
ordered dicts using a json compatible syntax, these are my conclusions:
A set could be defined as { item1, item2, item3[...] }
with {,} as an empty set
An ordered dict could be defined as [ item1: value1, item2: value2 ... ]
with [:] ase an empty odered dict
It could be used inside python code or to serialize python structures in
a json-like format (pyson maybe ?).
What do you think about ?
Kind regards, Matteo.
--
email: [email protected], [email protected]
web: www.alternativeoutput.it irc: #[email protected]
linkedin: http://lnkd.in/SPQG87
___
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Extension of python/json syntax to support explicitly sets and ordered dict.
On Thu, Sep 07, 2017 at 01:36:31PM +0200, Matteo Nastasi wrote:
> A set could be defined as { item1, item2, item3[...] }
Guido's time machine strikes again. Python 3:
py> s = {1, 2, 3}
py> type(s)
> with {,} as an empty set
That looks like a typo. I don't think that having a literal for empty
sets is important enough that we need worry about the lack.
> An ordered dict could be defined as [ item1: value1, item2: value2 ... ]
> with [:] ase an empty odered dict
I think that's been proposed before.
I don't hate that suggestion, but I don't think its very useful either.
We already have a de facto "Ordered Mapping" literal that can be passed
to the OrderedDict constructor:
OrderedDict(
[(key1, value1), (key2, value2), (key3, value3), ...]
)
Its not quite as compact, but it isn't too awful. And if you really
don't like it, try:
OrderedDict(zip(keys, values))
--
Steven
___
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Extension of python/json syntax to support explicitly sets and ordered dict.
I was thinking about an extended json too, not just python syntax.
What do you think about it ?
Regards and thank you for your time.
Matteo.
On Thu, Sep 07, 2017 at 09:54:15PM +1000, Steven D'Aprano wrote:
> On Thu, Sep 07, 2017 at 01:36:31PM +0200, Matteo Nastasi wrote:
>
> > A set could be defined as { item1, item2, item3[...] }
>
> Guido's time machine strikes again. Python 3:
>
> py> s = {1, 2, 3}
> py> type(s)
>
>
>
> > with {,} as an empty set
>
> That looks like a typo. I don't think that having a literal for empty
> sets is important enough that we need worry about the lack.
>
>
> > An ordered dict could be defined as [ item1: value1, item2: value2 ... ]
> > with [:] ase an empty odered dict
>
> I think that's been proposed before.
>
> I don't hate that suggestion, but I don't think its very useful either.
> We already have a de facto "Ordered Mapping" literal that can be passed
> to the OrderedDict constructor:
>
> OrderedDict(
> [(key1, value1), (key2, value2), (key3, value3), ...]
> )
>
> Its not quite as compact, but it isn't too awful. And if you really
> don't like it, try:
>
> OrderedDict(zip(keys, values))
>
>
>
> --
> Steven
> ___
> Python-ideas mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
email: [email protected], [email protected]
web: www.alternativeoutput.it irc: #[email protected]
linkedin: http://lnkd.in/SPQG87
___
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Extension of python/json syntax to support explicitly sets and ordered dict.
On Thu, Sep 7, 2017 at 9:59 PM, Matteo Nastasi wrote: > I was thinking about an extended json too, not just python syntax. > > What do you think about it ? > > Regards and thank you for your time. Extend JSON? No thank you. Down the path of extending simple standards in proprietary ways lies a madness that I do not wish on my best friends, much less my worst enemies. ChrisA ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] if as
> Sent: Thursday, September 07, 2017 at 6:43 AM > From: "Denis Krienbühl" > To: [email protected] > Subject: [Python-ideas] if as > > Hi! > > I’ve been having this idea for a few years and I thought I finally see if > what others think of it. I have no experience in language design and I don’t > know if this is something I’ve picked up in some other language. I also do > not know what the ramifications of implementing this idea would be. I just > keep thinking about it :) > > I quite often write code like the following in python: > > result = computation() >if result: > do_something_with_computation(result) > > More often than not this snippet evolves from something like this: > > if computation(): > … > > That is, I use the truthiness of the value at first. As the code grows I > refactor to actually do something with the result. > > What I would love to see is the following syntax instead, which to me is much > cleaner: > >if computation() as result: >do_something_with_result(result) > > Basically the result variable would be the result of the if condition’s > expression and it would be available the same way it would be if we used my > initial snippet (much the same way that the result of with expressions also > stays around outside the with-block). > > Any feedback is appreciated :) I also often wonder why we are left doing an assignment and test. You have two options: 1. assign to a variable then test and use 2. repeat the function call I would offer that 'with' [sh|c]ould be used: with test() as x: handle_truthy(x) else: handle_falsey() # do we provide x here too? Because None vs False? ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs
A sitemap.xml may be helpful for indicating search relevance to search crawlers? https://www.google.com/search?q=sphinx+sitemap On Thursday, September 7, 2017, Bar Harel wrote: > A bit radical but do you believe we can contact Google to alter the search > results? > It's for the benefit of the user after all, and many just switch to python > 3 in the version picker anyway. > > On Thu, Sep 7, 2017, 4:18 AM Guido van Rossum > wrote: > >> That's a good idea. Could you suggest it to https://github.com/python/ >> pythondotorg/ ? (There is actually a version switcher on every page, but >> it's rather polite. :-) >> >> On Wed, Sep 6, 2017 at 3:52 PM, Ryan Gonzalez > > wrote: >> >>> Right now, many Google searches for Python modules return the Python 2 >>> documentation. IMO since 2 will be reaching EOL in around 3 years, it >>> would be nice to have a giant red box at the top with a link to the >>> Python 3 documentation. >>> >>> SFML already does something like this: https://www.sfml-dev.org/ >>> tutorials/2.3/ >>> >>> (I mean, it would be even nicer to have a "jump to latest version" for >>> *all* not-new Python versions, though I figured just 2 would be a lot >>> easier.) >>> >>> -- >>> Ryan (ライアン) >>> Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else >>> http://refi64.com/ >>> ___ >>> Python-ideas mailing list >>> [email protected] >>> >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> ___ >> Python-ideas mailing list >> [email protected] >> >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs
On Thu, Sep 07, 2017 at 05:51:31AM +, Bar Harel wrote: > A bit radical but do you believe we can contact Google to alter the search > results? You should try DuckDuckGo, it recognises "python" searches and quotes from, and links to, the Python 3 documentation at the top of the page before the search results. E.g.: https://duckduckgo.com/html/?q=python%20functools Both Bing and Yahoo also gives the Python 2 docs first: https://www.bing.com/search?q=python+functools https://au.search.yahoo.com/search?p=python+functools -- Steve ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] if as
On Thu, Sep 07, 2017 at 04:36:40PM +0200, Jason H wrote: > I also often wonder why we are left doing an assignment and test. You have > two options: > 1. assign to a variable then test and use > 2. repeat the function call Personally, I don't see what's wrong with the "assign then test" idiom. x = something() if x: do_stuff() > I would offer that 'with' [sh|c]ould be used: > with test() as x: >handle_truthy(x) > else: >handle_falsey() # do we provide x here too? Because None vs False? This would cause confusing errors and mysterious behaviour, depending on whether the test() object was a context manager or not. Which should take priority? If you see: with spam() as x: do_stuff is that a context manager with block (like "with open(...) as f") or your boolean if test in disguise? Having "with" sometimes be a disguised "if" and sometimes a regular "with" will make it really, really hard to reason about code. -- Steve ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs
We just need someone with SEO experience to fix this for us. On Thu, Sep 7, 2017 at 9:13 AM, Steven D'Aprano wrote: > On Thu, Sep 07, 2017 at 05:51:31AM +, Bar Harel wrote: > > A bit radical but do you believe we can contact Google to alter the > search > > results? > > You should try DuckDuckGo, it recognises "python" searches and quotes > from, and links to, the Python 3 documentation at the top of the page > before the search results. E.g.: > > https://duckduckgo.com/html/?q=python%20functools > > Both Bing and Yahoo also gives the Python 2 docs first: > > https://www.bing.com/search?q=python+functools > > https://au.search.yahoo.com/search?p=python+functools > > -- > Steve > ___ > Python-ideas mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] lazy import via __future__ or compiler analysis
This is a half baked idea that perhaps could work. Maybe call it 2-stage module load instead of lazy. Introduce a lazy module import process that modules can opt-in to. The opt-in would either be with a __future__ statement or the compiler would statically analyze the module and determine if it is safe. E.g. if the module has no module level statements besides imports. .pyc files get some other bits of information: A) whether the module has opted for lazy import (IS_LAZY) B) the modules imported by the module (i.e. top-level imports, IMPORT_LIST) Make __import__ understand this data and do lazy loading for modules that want it. Sub-modules that have import side-effects will still execute as normal and the side effects will happen when the parent module is imported.. This would consist of a recursive process, something like: def load_module(name): if not IS_LAZY(name): import as usual else: create lazy version of module 'name' for subname in IMPORT_LIST(name): load_module(subname) An additional idea from Barry W, if a module wants lazy loading but wants to do some init when the module is "woken up", define a __init__ top-level function. Python would call that function when attributes of the module are first actually used. My plan was to implement this with a Python __import__ implementation. I would unmarshal the .pyc, compute IS_LAZY and IMPORT_LIST at import time. So, not gaining a lot of speedup. It would prove if the idea works in terms of not causing application crashes, etc. I could try running it with bigger apps and see how many modules are flagged for lazy loading. ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
Hi all, As part of the multi-core work I'm proposing the addition of the "interpreters" module to the stdlib. This will expose the existing subinterpreters C-API to Python code. I've purposefully kept the API simple. Please let me know what you think. -eric https://www.python.org/dev/peps/pep-0554/ https://github.com/python/peps/blob/master/pep-0554.rst https://github.com/python/cpython/pull/1748 https://github.com/python/cpython/pull/1802 https://github.com/ericsnowcurrently/cpython/tree/high-level-interpreters-module ** PEP: 554 Title: Multiple Interpreters in the Stdlib Author: Eric Snow Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2017-09-05 Python-Version: 3.7 Post-History: Abstract This proposal introduces the stdlib "interpreters" module. It exposes the basic functionality of subinterpreters that exists in the C-API. Rationale = Running code in multiple interpreters provides a useful level of isolation within the same process. This can be leveraged in number of ways. Furthermore, subinterpreters provide a well-defined framework in which such isolation may extended. CPython has supported subinterpreters, with increasing levels of support, since version 1.5. While the feature has the potential to be a powerful tool, subinterpreters have suffered from neglect because they are not available directly from Python. Exposing the existing functionality in the stdlib will help reverse the situation. Proposal The "interpreters" module will be added to the stdlib. It will provide a high-level interface to subinterpreters and wrap the low-level "_interpreters" module. The proposed API is inspired by the threading module. The module provides the following functions: enumerate(): Return a list of all existing interpreters. get_current(): Return the currently running interpreter. get_main(): Return the main interpreter. create(): Initialize a new Python interpreter and return it. The interpreter will be created in the current thread and will remain idle until something is run in it. The module also provides the following class: Interpreter(id): id: The interpreter's ID (read-only). is_running(): Return whether or not the interpreter is currently running. destroy(): Finalize and destroy the interpreter. run(code): Run the provided Python code in the interpreter, in the current OS thread. Supported code: source text. Copyright = This document has been placed in the public domain. ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
On 7 September 2017 at 19:26, Eric Snow wrote:
> As part of the multi-core work I'm proposing the addition of the
> "interpreters" module to the stdlib. This will expose the existing
> subinterpreters C-API to Python code. I've purposefully kept the API
> simple. Please let me know what you think.
Looks good. I agree with the idea of keeping the interface simple in
the first instance - we can easily add extra functionality later, but
removing stuff (or worse still, finding that stuff we thought was OK
but had missed corner cases of was broken) is much harder.
>run(code):
>
> Run the provided Python code in the interpreter, in the current
> OS thread. Supported code: source text.
The only quibble I have is that I'd prefer it if we had a
run(callable, *args, **kwargs) method. Either instead of, or as well
as, the run(string) one here.
Is there any reason why passing a callable and args is unsafe, and/or
difficult? Naively, I'd assume that
interp.call('f(a)')
would be precisely as safe as
interp.call(f, a)
Am I missing something? Name visibility or scoping issues come to mind
as possible complications I'm not seeing. At the least, if we don't
want a callable-and-args form yet, a note in the PEP explaining why
it's been omitted would be worthwhile.
Paul
___
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Extension of python/json syntax to support explicitly sets and ordered dict.
IIRC in CPython 3.6 and PyPy dicts are ordered based on insertion anyway;
although it's an implementation-specific detail, realistically it removes
some of the use cases for ordered dictionary literals.
--
Ryan (ライアン)
Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone
elsehttp://refi64.com
On Sep 7, 2017 at 6:40 AM, >
wrote:
Hi all,
few days ago I thought about a way to rapresent sets and
ordered dicts using a json compatible syntax, these are my conclusions:
A set could be defined as { item1, item2, item3[...] }
with {,} as an empty set
An ordered dict could be defined as [ item1: value1, item2: value2 ... ]
with [:] ase an empty odered dict
It could be used inside python code or to serialize python structures in
a json-like format (pyson maybe ?).
What do you think about ?
Kind regards, Matteo.
--
email: [email protected], [email protected]
web: www.alternativeoutput.it irc: #[email protected]
linkedin: http://lnkd.in/SPQG87
___
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
On Thu, Sep 7, 2017 at 11:52 AM, Paul Moore wrote:
> The only quibble I have is that I'd prefer it if we had a
> run(callable, *args, **kwargs) method. Either instead of, or as well
> as, the run(string) one here.
>
> Is there any reason why passing a callable and args is unsafe, and/or
> difficult? Naively, I'd assume that
>
> interp.call('f(a)')
>
> would be precisely as safe as
>
> interp.call(f, a)
The problem for now is with sharing objects between interpreters. The
simplest safe approach currently is to restrict execution to source
strings. Then there are no complications. Interpreter.call() makes
sense but I'd like to wait until we get feel for how subinterpreters
get used and until we address some of the issues with object passing.
FWIW, here are what I see as the next steps for subinterpreters in the stdlib:
1. add a basic queue class for passing objects between interpreters
* only support strings at first (though Nick pointed out we could
fall back to pickle or marshal for unsupported objects)
2. implement CSP on top of subinterpreters
3. expand the queue's supported types
4. add something like Interpreter.call()
I didn't include such a queue in this proposal because I wanted to
keep it as focused as possible. I'll add a note to the PEP about
this.
>
> Am I missing something? Name visibility or scoping issues come to mind
> as possible complications I'm not seeing. At the least, if we don't
> want a callable-and-args form yet, a note in the PEP explaining why
> it's been omitted would be worthwhile.
I'll add a note to the PEP. Thanks for pointing this out. :)
-eric
___
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs
Guido van Rossum wrote: > We just need someone with SEO experience to fix this for us. I'm not an SEO expert, but I think a possible approach would be using (or partly abusing) the element on the documentation pages: https://support.google.com/webmasters/answer/139066 Python's documentation is already using these canonical links, just in an incomplete way: The Python documentation for 2.7 and 3.5 and later has a pointing to the generic documentation URL of the same major Python version, e.g. in https://docs.python.org/3/library/functools.html you can find this in the page source: https://docs.python.org/3/library/functools.html"; /> This is why if you search for "python 3 $module" in Google, you'll never see a direct link to the 3.5 or 3.6 versions of the documentation (because Google merges them with the generic docs.python.org/3/), but you still results for versions 3.2, 3.3 etc. of the documentation (because the lack the canonical links). A very good step would be to also add this canoncial link to the documentation versions 3.0-3.4, this will make docs.python.org/3.3/ etc. vanish from Google and probably rank the generic docs.python.org/3/ higher than now. And now to the abuse part (I'm honestly not sure how well this would actually work): If you would add such a canonical link in docs.python.org/2/ pointing to docs.python.org/3/ (at least where the same module exists in both versions), you would eventually hide /2/ from the search results. If you don't want to be so extreme (people still want Python 2 documentation if they search for "python 2 $module") you could remove the canonical link just from the specific docs.python.org/2.7/ which will then be ranked much lower than docs.python.org/3/ in the search results, but at least still show up. Sebastian ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] lazy import via __future__ or compiler analysis
Neil Schemenauer wrote:
> Introduce a lazy module import process that modules can opt-in to.
> The opt-in would either be with a __future__ statement or the
> compiler would statically analyze the module and determine if it is
> safe. E.g. if the module has no module level statements besides
> imports.
and `def` and `class` of course.
There are a few other things that might end up marking a module as
"industrious" (my thesaurus's antonym for "lazy"). There will likely be
assignments of module global such as:
MY_CONST = 'something'
and it may even be a little more complicated:
COLORS = dict(
red=1,
blue=2,
green=3,
)
REVERSE = {value: key for key, value in COLORS.items()}
A naive evaluation of such a module might not notice them as lazy, but I
think they could still be treated as such.
Function and class decorators might also be false positives. E.g.
@public
def my_public_function():
pass
or even
@mungify
class Munged:
pass
Maybe that's just the cost of doing business, and if they clear the lazy
flag, so be it. But it feels like doing so will leave quite a bit of
lazy loading opportunity left on the table. And I'm not sure you can
solve all of those by moving things to a module level __init__().
Cheers,
-Barry
___
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] if as
Sadly it’s hard to create a context manager that skips its body like this: with unpack(computation()) as result: do_something_with_result(result) You can do it with some hackery like described here: https://stackoverflow.com/a/12594789/247482 class unpack: def __init__(self, pred): self.pred = pred def __enter__(self): if self.pred: return self.pred # else skip the with block’s body sys.settrace(lambda *args, **kw: None) frame = inspect.currentframe(1) frame.f_trace = self.trace def trace(self, frame, event, arg): raise def __exit__(self, type, value, traceback): return True # suppress the exception Steven D'Aprano schrieb am Do., 7. Sep. 2017 um 18:26 Uhr: > On Thu, Sep 07, 2017 at 04:36:40PM +0200, Jason H wrote: > > > I also often wonder why we are left doing an assignment and test. You > have two options: > > 1. assign to a variable then test and use > > 2. repeat the function call > > Personally, I don't see what's wrong with the "assign then test" idiom. > > x = something() > if x: > do_stuff() > > > > I would offer that 'with' [sh|c]ould be used: > > with test() as x: > >handle_truthy(x) > > else: > >handle_falsey() # do we provide x here too? Because None vs False? > > > This would cause confusing errors and mysterious behaviour, depending on > whether the test() object was a context manager or not. Which should > take priority? If you see: > > with spam() as x: >do_stuff > > is that a context manager with block (like "with open(...) as f") or > your boolean if test in disguise? > > Having "with" sometimes be a disguised "if" and sometimes a regular > "with" will make it really, really hard to reason about code. > > > -- > Steve > ___ > Python-ideas mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs
Sebastian Krause wrote: > This is why if you search for "python 3 $module" in Google, you'll > never see a direct link to the 3.5 or 3.6 versions of the > documentation (because Google merges them with the generic > docs.python.org/3/), but you still results for versions 3.2, 3.3 > etc. of the documentation (because the lack the canonical links). > > A very good step would be to also add this canoncial link to the > documentation versions 3.0-3.4, this will make docs.python.org/3.3/ > etc. vanish from Google and probably rank the generic > docs.python.org/3/ higher than now. Here is Nick Coghlan's bpo issue which added these canonical links: https://bugs.python.org/issue26355 - looks like applying this to the older doc versions never happened in the end. ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] if as
sorry, it’s a bit more difficult. this works: https://gist.github.com/flying-sheep/86dfcc1bdd71a33fa3483b83e254084c Philipp A. schrieb am Do., 7. Sep. 2017 um 21:18 Uhr: > Sadly it’s hard to create a context manager that skips its body like this: > > with unpack(computation()) as result: > do_something_with_result(result) > > You can do it with some hackery like described here: > https://stackoverflow.com/a/12594789/247482 > > class unpack: > def __init__(self, pred): > self.pred = pred > > def __enter__(self): > if self.pred: > return self.pred > # else skip the with block’s body > sys.settrace(lambda *args, **kw: None) > frame = inspect.currentframe(1) > frame.f_trace = self.trace > > def trace(self, frame, event, arg): > raise > > def __exit__(self, type, value, traceback): > return True # suppress the exception > > Steven D'Aprano schrieb am Do., 7. Sep. 2017 um > 18:26 Uhr: > >> On Thu, Sep 07, 2017 at 04:36:40PM +0200, Jason H wrote: >> >> > I also often wonder why we are left doing an assignment and test. You >> have two options: >> > 1. assign to a variable then test and use >> > 2. repeat the function call >> >> Personally, I don't see what's wrong with the "assign then test" idiom. >> >> x = something() >> if x: >> do_stuff() >> >> >> > I would offer that 'with' [sh|c]ould be used: >> > with test() as x: >> >handle_truthy(x) >> > else: >> >handle_falsey() # do we provide x here too? Because None vs False? >> >> >> This would cause confusing errors and mysterious behaviour, depending on >> whether the test() object was a context manager or not. Which should >> take priority? If you see: >> >> with spam() as x: >>do_stuff >> >> is that a context manager with block (like "with open(...) as f") or >> your boolean if test in disguise? >> >> Having "with" sometimes be a disguised "if" and sometimes a regular >> "with" will make it really, really hard to reason about code. >> >> >> -- >> Steve >> ___ >> Python-ideas mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
On 7 September 2017 at 20:14, Eric Snow wrote:
> On Thu, Sep 7, 2017 at 11:52 AM, Paul Moore wrote:
>> Is there any reason why passing a callable and args is unsafe, and/or
>> difficult? Naively, I'd assume that
>>
>> interp.call('f(a)')
>>
>> would be precisely as safe as
>>
>> interp.call(f, a)
>
> The problem for now is with sharing objects between interpreters. The
> simplest safe approach currently is to restrict execution to source
> strings. Then there are no complications. Interpreter.call() makes
> sense but I'd like to wait until we get feel for how subinterpreters
> get used and until we address some of the issues with object passing.
Ah, OK. so if I create a new interpreter, none of the classes,
functions, or objects defined in my calling code will exist within the
target interpreter? That makes sense, but I'd missed that nuance from
the description. Again, this is probably worth noting in the PEP.
And for the record, based on that one fact, I'm perfectly OK with the
initial API being string-only.
> FWIW, here are what I see as the next steps for subinterpreters in the stdlib:
>
> 1. add a basic queue class for passing objects between interpreters
> * only support strings at first (though Nick pointed out we could
> fall back to pickle or marshal for unsupported objects)
> 2. implement CSP on top of subinterpreters
> 3. expand the queue's supported types
> 4. add something like Interpreter.call()
>
> I didn't include such a queue in this proposal because I wanted to
> keep it as focused as possible. I'll add a note to the PEP about
> this.
This all sounds very reasonable. Thanks for the clarification.
Paul
___
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] if as
On Fri, Sep 8, 2017 at 5:40 AM, Philipp A. wrote:
> sorry, it’s a bit more difficult. this works:
> https://gist.github.com/flying-sheep/86dfcc1bdd71a33fa3483b83e254084c
If this can be made to work - even hackily - can it be added into
contextlib.contextmanager (or technically its helper class)?
Currently, its __enter__ looks like this:
def __enter__(self):
try:
return next(self.gen)
except StopIteration:
raise RuntimeError("generator didn't yield") from None
If that raise were replaced with the hackiness of skipping the body,
we could wrap everything up nicely:
@contextlib.contextmanager
def iff(thing):
if thing: yield thing
# otherwise don't yield
for i in range(1, 11):
with iff(random.randrange(3)) as val:
print(i, val) # won't print any zeroes
ChrisA
___
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
Eric Snow wrote: > 1. add a basic queue class for passing objects between interpreters > * only support strings at first (though Nick pointed out we could > fall back to pickle or marshal for unsupported objects) > 2. implement CSP on top of subinterpreters > 3. expand the queue's supported types > 4. add something like Interpreter.call() How is the GIL situation with subinterpreters these days, is the long-term goal still "solving multi-core Python", i.e. using multiple CPU cores from within the same process? Or is it mainly used for isolation? Sebastian ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] lazy import via __future__ or compiler analysis
Barry Warsaw wrote: > There are a few other things that might end up marking a module as > "industrious" (my thesaurus's antonym for "lazy"). Good points. The analysis can be simple at first and then we can enhance it to be smarter about what is okay and still lazy load. We may evolve it over time too, making things that are not strictly safe still not trigger the "industrious" load lazy anyhow. Another idea is to introduce __lazy__ or some such in the global namespace of the module, if present, e.g. __lazy__ = True then the analysis doesn't do anything except return True. The module has explicitly stated that side-effects in the top-level code are okay to be done in a lazy fashion. Perhaps with a little bit of smarts in the analsis and a little sprinkling of __lazy__ flags, we can get a big chunk of modules to lazy load. ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
On Thu, Sep 7, 2017 at 12:44 PM, Paul Moore wrote: > Ah, OK. so if I create a new interpreter, none of the classes, > functions, or objects defined in my calling code will exist within the > target interpreter? That makes sense, but I'd missed that nuance from > the description. Again, this is probably worth noting in the PEP. I'll make sure the PEP is more clear about this. > > And for the record, based on that one fact, I'm perfectly OK with the > initial API being string-only. Great! :) -eric ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
On Thu, Sep 7, 2017 at 1:14 PM, Sebastian Krause wrote: > How is the GIL situation with subinterpreters these days, is the > long-term goal still "solving multi-core Python", i.e. using > multiple CPU cores from within the same process? Or is it mainly > used for isolation? The GIL is still process-global. The goal is indeed to change this to support actual multi-core parallelism. However, the benefits of interpreter isolation are certainly a win otherwise. :) -eric ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
On Thu, Sep 7, 2017 at 11:26 AM, Eric Snow wrote: > Hi all, > > As part of the multi-core work I'm proposing the addition of the > "interpreters" module to the stdlib. This will expose the existing > subinterpreters C-API to Python code. I've purposefully kept the API > simple. Please let me know what you think. My concern about this is the same as it was last time -- the work looks neat, but right now, almost no-one uses subinterpreters (basically it's Jep and mod_wsgi and that's it?), and therefore many packages get away with ignoring subinterpreters. Numpy is the one I'm most familiar with: when we get subinterpreter bugs we close them wontfix, because supporting subinterpreters properly would require non-trivial auditing, add overhead for non-subinterpreter use cases, and benefit a tiny tiny fraction of our users. If we add a friendly python-level API like this, then we're committing to this being a part of Python for the long term and encouraging people to use it, which puts pressure on downstream packages to do that work... but it's still not clear whether any benefits will actually materialize. I've actually argued with the PyPy devs to try to convince them to add subinterpreter support as part of their experiments with GIL-removal, because I think the semantics would genuinely be nicer to work with than raw threads, but they're convinced that it's impossible to make this work. Or more precisely, they think you could make it work in theory, but that it would be impossible to make it meaningfully more efficient than using multiple processes. I want them to be wrong, but I have to admit I can't see a way to make it work either... If this is being justified by the multicore use case, and specifically by the theory that having two interpreters in the same process will allow for more efficient communication than two interpreters in two different processes, then... why should we believe that that's actually possible? I want your project to succeed, but if it's going to fail then it seems better if it fails before we commit to exposing new APIs. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs
On 7 September 2017 at 12:24, Sebastian Krause wrote: > Sebastian Krause wrote: >> This is why if you search for "python 3 $module" in Google, you'll >> never see a direct link to the 3.5 or 3.6 versions of the >> documentation (because Google merges them with the generic >> docs.python.org/3/), but you still results for versions 3.2, 3.3 >> etc. of the documentation (because the lack the canonical links). >> >> A very good step would be to also add this canoncial link to the >> documentation versions 3.0-3.4, this will make docs.python.org/3.3/ >> etc. vanish from Google and probably rank the generic >> docs.python.org/3/ higher than now. > > Here is Nick Coghlan's bpo issue which added these canonical links: > https://bugs.python.org/issue26355 - looks like applying this to the > older doc versions never happened in the end. Right, as adding those will need to be handled through the web server and/or by manually regenerating the docs with the additional HTML headers - making the changes to the CPython repo won't achieve anything, since the docs for those branches aren't automatically regenerated anymore. Another big task that could be undertaken is to start re-routing unqualified deep links to Python 3 - the reason we still haven't done that is because the tree layout is actually different (as per the six.moves module), so it isn't a trivial rewrite rule the way the current redirection into the Python 2 docs is. Given such a mapping, it would also be possible to add the corresponding canonical URL entries to the Python 2.7 documentation to merge their search ranking in to the corresponding Python 3 pages. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
On 7 September 2017 at 15:48, Nathaniel Smith wrote: > I've actually argued with the PyPy devs to try to convince them to add > subinterpreter support as part of their experiments with GIL-removal, > because I think the semantics would genuinely be nicer to work with > than raw threads, but they're convinced that it's impossible to make > this work. Or more precisely, they think you could make it work in > theory, but that it would be impossible to make it meaningfully more > efficient than using multiple processes. I want them to be wrong, but > I have to admit I can't see a way to make it work either... The gist of the idea is that with subinterpreters, your starting point is multiprocessing-style isolation (i.e. you have to use pickle to transfer data between subinterpreters), but you're actually running in a shared-memory threading context from the operating system's perspective, so you don't need to rely on mmap to share memory over a non-streaming interface. It's also definitely the case that to make this viable, we'd need to provide fast subinterpreter friendly alternatives to C globals for use by extension modules (otherwise adding subinterpreter compatibility will be excessively painful), and PEP 550 is likely to be helpful there. Personally, I think it would make sense to add the module under PEP 411 provisional status, and make it's continued existence as a public API contingent on actually delivering on the "lower overhead multi-core support than multiprocessing" goal (even if it only delivers on that front on Windows, where process creation is more expensive and there's no fork() equivalent). However, I'd also be entirely happy with our adding it as a private "_subinterpreters" API for testing & experimentation purposes (see https://bugs.python.org/issue30439 ), and reconsidering introducing it as a public API after there's more concrete evidence as to what can actually be achieved based on it. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
On Thu, Sep 7, 2017 at 4:23 PM, Nick Coghlan wrote: > On 7 September 2017 at 15:48, Nathaniel Smith wrote: >> I've actually argued with the PyPy devs to try to convince them to add >> subinterpreter support as part of their experiments with GIL-removal, >> because I think the semantics would genuinely be nicer to work with >> than raw threads, but they're convinced that it's impossible to make >> this work. Or more precisely, they think you could make it work in >> theory, but that it would be impossible to make it meaningfully more >> efficient than using multiple processes. I want them to be wrong, but >> I have to admit I can't see a way to make it work either... > > The gist of the idea is that with subinterpreters, your starting point > is multiprocessing-style isolation (i.e. you have to use pickle to > transfer data between subinterpreters), but you're actually running in > a shared-memory threading context from the operating system's > perspective, so you don't need to rely on mmap to share memory over a > non-streaming interface. The challenge is that streaming bytes between processes is actually really fast -- you don't really need mmap for that. (Maybe this was important for X11 back in the 1980s, but a lot has changed since then :-).) And if you want to use pickle and multiprocessing to send, say, a single big numpy array between processes, that's also really fast, because it's basically just a few memcpy's. The slow case is passing complicated objects between processes, and it's slow because pickle has to walk the object graph to serialize it, and walking the object graph is slow. Copying object graphs between subinterpreters has the same problem. So the only case I can see where I'd expect subinterpreters to make communication dramatically more efficient is if you have a "deeply immutable" type: one where not only are its instances immutable, but all objects reachable from those instances are also guaranteed to be immutable. So like, a tuple except that when you instantiate it it validates that all of its elements are also marked as deeply immutable, and errors out if not. Then when you go to send this between subinterpreters, you can tell by checking the type of the root object that the whole graph is immutable, so you don't need to walk it yourself. However, it seems impossible to support user-defined deeply-immutable types in Python: types and functions are themselves mutable and hold tons of references to other potentially mutable objects via __mro__, __globals__, __weakrefs__, etc. etc., so even if a user-defined instance can be made logically immutable it's still going to hold references to mutable things. So the one case where subinterpreters win is if you have a really big and complicated set of nested pseudo-tuples of ints and strings and you're bottlenecked on passing it between interpreters. Maybe frozendicts too. Is that enough to justify the whole endeavor? It seems dubious to me. I guess the other case where subprocesses lose to "real" threads is startup time on Windows. But starting a subinterpreter is also much more expensive than starting a thread, once you take into account the cost of loading the application's modules into the new interpreter. In both cases you end up needing some kind of process/subinterpreter pool or cache to amortize that cost. Obviously I'm committing the cardinal sin of trying to guess about performance based on theory instead of measurement, so maybe I'm wrong. Or maybe there's some deviously clever trick I'm missing. I hope so -- a really useful subinterpreter multi-core store would be awesome. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Extension of python/json syntax to support explicitly sets and ordered dict.
a json-like format (pyson maybe ?). I gonna pyson is a fine idea. But don't call it extended JSON ;-) For me, the point would be to capture Python' s richer data types. But would you need an OrderedDict? As pointed out, in recent cPython ( and pypy) dicts are ordered by default, but it's not part of the language spec) And you can use ast.literal_eval as a pyson parser, so it's almost ready to use :-) -CHB What do you think about ? Kind regards, Matteo. -- email: [email protected], [email protected] web: www.alternativeoutput.it irc: #[email protected] linkedin: http://lnkd.in/SPQG87 ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
On Thu, Sep 7, 2017 at 5:15 PM Nathaniel Smith wrote: > On Thu, Sep 7, 2017 at 4:23 PM, Nick Coghlan wrote: > > The gist of the idea is that with subinterpreters, your starting point > > is multiprocessing-style isolation (i.e. you have to use pickle to > > transfer data between subinterpreters), but you're actually running in > > a shared-memory threading context from the operating system's > > perspective, so you don't need to rely on mmap to share memory over a > > non-streaming interface. > > The challenge is that streaming bytes between processes is actually > really fast -- you don't really need mmap for that. (Maybe this was > important for X11 back in the 1980s, but a lot has changed since then > :-).) And if you want to use pickle and multiprocessing to send, say, > a single big numpy array between processes, that's also really fast, > because it's basically just a few memcpy's. The slow case is passing > complicated objects between processes, and it's slow because pickle > has to walk the object graph to serialize it, and walking the object > graph is slow. Copying object graphs between subinterpreters has the > same problem. > This doesn't match up with my (somewhat limited) experience. For example, in this table of bandwidth estimates from Matthew Rocklin (CCed), IPC is about 10x slower than a memory copy: http://matthewrocklin.com/blog/work/2015/12/29/data-bandwidth This makes a considerable difference when building a system do to parallel data analytics in Python (e.g., on NumPy arrays), which is exactly what Matthew has been working on for the past few years. I'm sure there are other ways to avoid this expensive IPC without using sub-interpreters, e.g., by using a tool like Plasma ( http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/). But I'm skeptical of your assessment that the current multiprocessing approach is fast enough. ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
Those numbers were for common use in Python tools and reflected my anecdotal experience at the time with normal Python tools. I'm sure that there are mechanisms to achieve faster speeds than what I experienced. That being said, here is a small example. In [1]: import multiprocessing In [2]: data = b'0' * 1 # 100 MB In [3]: from toolz import identity In [4]: pool = multiprocessing.Pool() In [5]: %time _ = pool.apply_async(identity, (data,)).get() CPU times: user 76 ms, sys: 64 ms, total: 140 ms Wall time: 252 ms This is about 400MB/s for a roundtrip On Thu, Sep 7, 2017 at 9:00 PM, Stephan Hoyer wrote: > On Thu, Sep 7, 2017 at 5:15 PM Nathaniel Smith wrote: > >> On Thu, Sep 7, 2017 at 4:23 PM, Nick Coghlan wrote: >> > The gist of the idea is that with subinterpreters, your starting point >> > is multiprocessing-style isolation (i.e. you have to use pickle to >> > transfer data between subinterpreters), but you're actually running in >> > a shared-memory threading context from the operating system's >> > perspective, so you don't need to rely on mmap to share memory over a >> > non-streaming interface. >> >> The challenge is that streaming bytes between processes is actually >> really fast -- you don't really need mmap for that. (Maybe this was >> important for X11 back in the 1980s, but a lot has changed since then >> :-).) And if you want to use pickle and multiprocessing to send, say, >> a single big numpy array between processes, that's also really fast, >> because it's basically just a few memcpy's. The slow case is passing >> complicated objects between processes, and it's slow because pickle >> has to walk the object graph to serialize it, and walking the object >> graph is slow. Copying object graphs between subinterpreters has the >> same problem. >> > > This doesn't match up with my (somewhat limited) experience. For example, > in this table of bandwidth estimates from Matthew Rocklin (CCed), IPC is > about 10x slower than a memory copy: > http://matthewrocklin.com/blog/work/2015/12/29/data-bandwidth > > This makes a considerable difference when building a system do to parallel > data analytics in Python (e.g., on NumPy arrays), which is exactly what > Matthew has been working on for the past few years. > > I'm sure there are other ways to avoid this expensive IPC without using > sub-interpreters, e.g., by using a tool like Plasma ( > http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/). > But I'm skeptical of your assessment that the current multiprocessing > approach is fast enough. > ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Lazy creation of module level functions and classes
This is an idea that come out of the lazy loading modules idea. Larry Hastings mentioned what a good improvement this was for PHP. I think it would help a lot of Python too. Very many functions and classes are not actually needed but are instantiated anyhow. Back of napkin idea: Write AST transformer tool, change top-level functions and classes to be like properties (can use __class__ I guess) Transform is something like: # old code def inc(x): return x + 1 # transformed code def __make_inc(code=): obj = eval(code) _ModuleClass.inc = obj # only do eval once return obj inc = property(__make_inc) Totally seat of pants idea but I can't think of a reason why it shouldn't work. It seems much more powerful than lazying loading modules. In the lazy module case, you load the whole module if any part is touched. Many modules only have a small fraction of their functions and classes actually used. If this transformer idea works, the standard Python compiler could be changed to do the above stuff, no transformer needed. ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
First of all, thanks for the feedback and encouragement! Responses in-line below. -eric On Thu, Sep 7, 2017 at 3:48 PM, Nathaniel Smith wrote: > My concern about this is the same as it was last time -- the work > looks neat, but right now, almost no-one uses subinterpreters > (basically it's Jep and mod_wsgi and that's it?), and therefore many > packages get away with ignoring subinterpreters. My concern is that this is a chicken-and-egg problem. The situation won't improve until subinterpreters are more readily available. > Numpy is the one I'm > most familiar with: when we get subinterpreter bugs we close them > wontfix, because supporting subinterpreters properly would require > non-trivial auditing, add overhead for non-subinterpreter use cases, > and benefit a tiny tiny fraction of our users. The main problem of which I'm aware is C globals in libraries and extension modules. PEPs 489 and 3121 are meant to help but I know that there is at least one major situation which is still a blocker for multi-interpreter-safe module state. Other than C globals, is there some other issue? > If we add a friendly python-level API like this, then we're committing > to this being a part of Python for the long term and encouraging > people to use it, which puts pressure on downstream packages to do > that work... but it's still not clear whether any benefits will > actually materialize. I'm fine with Nick's idea about making this a "provisional" module. Would that be enough to ease your concern here? > I've actually argued with the PyPy devs to try to convince them to add > subinterpreter support as part of their experiments with GIL-removal, > because I think the semantics would genuinely be nicer to work with > than raw threads, but they're convinced that it's impossible to make > this work. Or more precisely, they think you could make it work in > theory, but that it would be impossible to make it meaningfully more > efficient than using multiple processes. I want them to be wrong, but > I have to admit I can't see a way to make it work either... Yikes! Given the people involved I don't find that to be a good sign. Nevertheless, I still consider my ultimate goals to be tractable and will press forward. At each step thus far, the effort has led to improvements that extend beyond subinterpreters and multi-core. I see that trend continuing for the entirety of the project. Even if my final goal is not realized, the result will still be significantly net positive...and I still think it will still work out. :) > If this is being justified by the multicore use case, and specifically > by the theory that having two interpreters in the same process will > allow for more efficient communication than two interpreters in two > different processes, then... why should we believe that that's > actually possible? I want your project to succeed, but if it's going > to fail then it seems better if it fails before we commit to exposing > new APIs. The project is partly about performance. However, it's also particularly about offering a alternative concurrency model with an implementation that can run in multiple threads simultaneously in the same process. On Thu, Sep 7, 2017 at 5:15 PM, Nathaniel Smith wrote: > The slow case is passing > complicated objects between processes, and it's slow because pickle > has to walk the object graph to serialize it, and walking the object > graph is slow. Copying object graphs between subinterpreters has the > same problem. The initial goal is to support passing only strings between interpreters. Later efforts will involve investigating approaches to efficiently and safely passing other objects. > So the only case I can see where I'd expect subinterpreters to make > communication dramatically more efficient is if you have a "deeply > immutable" type > [snip] > However, it seems impossible to support user-defined deeply-immutable > types in Python: > [snip] I agree that it is currently not an option. That is part of the exercise. There are a number of possible solutions to explore once we get to that point. However, this PEP isn't about that. I'm confident enough about the possibilities that I'm comfortable with moving forward here. > I guess the other case where subprocesses lose to "real" threads is > startup time on Windows. But starting a subinterpreter is also much > more expensive than starting a thread, once you take into account the > cost of loading the application's modules into the new interpreter. In > both cases you end up needing some kind of process/subinterpreter pool > or cache to amortize that cost. Interpreter startup costs (and optimization strategies) are another aspect of the project which deserve attention. However, we'll worry about that after the core functionality has been achieved. > Obviously I'm committing the cardinal sin of trying to guess about > performance based on theory instead of measurement, so maybe I'm > wrong. Or maybe ther
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
On Thu, Sep 7, 2017 at 12:44 PM, Paul Moore wrote: > On 7 September 2017 at 20:14, Eric Snow wrote: >> I didn't include such a queue in this proposal because I wanted to >> keep it as focused as possible. I'll add a note to the PEP about >> this. > > This all sounds very reasonable. Thanks for the clarification. Hmm. Now I'm starting to think some form of basic queue would be important enough to include in the PEP. I'll see if that feeling holds tomorrow. -eric ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
On Thu, Sep 7, 2017 at 6:14 PM, Matthew Rocklin wrote: > Those numbers were for common use in Python tools and reflected my anecdotal > experience at the time with normal Python tools. I'm sure that there are > mechanisms to achieve faster speeds than what I experienced. That being > said, here is a small example. > > > In [1]: import multiprocessing > In [2]: data = b'0' * 1 # 100 MB > In [3]: from toolz import identity > In [4]: pool = multiprocessing.Pool() > In [5]: %time _ = pool.apply_async(identity, (data,)).get() > CPU times: user 76 ms, sys: 64 ms, total: 140 ms > Wall time: 252 ms > > This is about 400MB/s for a roundtrip Awesome, thanks for bringing numbers into my wooly-headed theorizing :-). On my laptop I actually get a worse result from your benchmark: 531 ms for 100 MB == ~200 MB/s round-trip, or 400 MB/s one-way. So yeah, transferring data between processes with multiprocessing is slow. This is odd, though, because on the same machine, using socat to send 1 GiB between processes using a unix domain socket runs at 2 GB/s: # terminal 1 ~$ rm -f /tmp/unix.sock && socat -u -b32768 UNIX-LISTEN:/tmp/unix.sock "SYSTEM:pv -W > /dev/null" 1.00GiB 0:00:00 [1.89GiB/s] [<=> ] # terminal 2 ~$ socat -u -b32768 "SYSTEM:dd if=/dev/zero bs=1M count=1024" UNIX:/tmp/unix.sock 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.529814 s, 2.0 GB/s (Notice that the pv output is in GiB/s and the dd output is in GB/s. 1.89 GiB/s = 2.03 GB/s, so they actually agree.) On my system, Python allocates + copies memory at 2.2 GB/s, so bulk byte-level IPC is within 10% of within-process bulk copying: # same 100 MB bytestring as above In [7]: bytearray_data = bytearray(data) In [8]: %timeit bytearray_data.copy() 45.3 ms ± 540 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) In [9]: 0.100 / 0.0453 # GB / seconds Out[9]: 2.207505518763797 I don't know why multiprocessing is so slow -- maybe there's a good reason, maybe not. But the reason isn't that IPC is intrinsically slow, and subinterpreters aren't going to automatically be 5x faster because they can use memcpy. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
On Thu, Sep 7, 2017 at 8:11 PM, Eric Snow wrote: > First of all, thanks for the feedback and encouragement! Responses > in-line below. I hope it's helpful! More responses in-line as well. > On Thu, Sep 7, 2017 at 3:48 PM, Nathaniel Smith wrote: >> My concern about this is the same as it was last time -- the work >> looks neat, but right now, almost no-one uses subinterpreters >> (basically it's Jep and mod_wsgi and that's it?), and therefore many >> packages get away with ignoring subinterpreters. > > My concern is that this is a chicken-and-egg problem. The situation > won't improve until subinterpreters are more readily available. Okay, but you're assuming that "more libraries work well with subinterpreters" is in fact an improvement. I'm asking you to convince me of that :-). Are there people saying "oh, if only subinterpreters had a Python API and less weird interactions with C extensions, I could do "? So far they haven't exactly taken the world by storm... >> Numpy is the one I'm >> most familiar with: when we get subinterpreter bugs we close them >> wontfix, because supporting subinterpreters properly would require >> non-trivial auditing, add overhead for non-subinterpreter use cases, >> and benefit a tiny tiny fraction of our users. > > The main problem of which I'm aware is C globals in libraries and > extension modules. PEPs 489 and 3121 are meant to help but I know > that there is at least one major situation which is still a blocker > for multi-interpreter-safe module state. Other than C globals, is > there some other issue? That's the main one I'm aware of, yeah, though I haven't looked into it closely. >> If we add a friendly python-level API like this, then we're committing >> to this being a part of Python for the long term and encouraging >> people to use it, which puts pressure on downstream packages to do >> that work... but it's still not clear whether any benefits will >> actually materialize. > > I'm fine with Nick's idea about making this a "provisional" module. > Would that be enough to ease your concern here? Potentially, yeah -- basically I'm fine with anything that doesn't end up looking like python-dev telling everyone "subinterpreters are the future! go forth and yell at any devs who don't support them!". What do you think the criteria for graduating to non-provisional status should be, in this case? [snip] >> So the only case I can see where I'd expect subinterpreters to make >> communication dramatically more efficient is if you have a "deeply >> immutable" type >> [snip] >> However, it seems impossible to support user-defined deeply-immutable >> types in Python: >> [snip] > > I agree that it is currently not an option. That is part of the > exercise. There are a number of possible solutions to explore once we > get to that point. However, this PEP isn't about that. I'm confident > enough about the possibilities that I'm comfortable with moving > forward here. I guess I would be much more confident in the possibilities here if you could give: - some hand-wavy sketch for how subinterpreter A could call a function that as originally defined in subinterpreter B without the GIL, which seems like a precondition for sharing user-defined classes - some hand-wavy sketch for how refcounting will work for objects shared between multiple subinterpreters without the GIL, without majorly impacting single-thread performance (I actually forgot about this problem in my last email, because PyPy has already solved this part!) These are the two problems where I find it most difficult to have faith. [snip] >> I hope so -- a really useful subinterpreter multi-core stor[y] would be >> awesome. > > Agreed! Thanks for the encouragement. :) Thanks for attempting such an ambitious project :-). -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Hexadecimal floating literals
The support of hexadecimal floating literals (like 0xC.68p+2) is
included in just released C++17 standard. Seems this becomes a mainstream.
In Python float.hex() returns hexadecimal string representation. Is it a
time to add more support of hexadecimal floating literals? Accept them
in float constructor and in Python parser? And maybe add support of
hexadecimal formatting ('%x' and '{:x}')?
___
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
