Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-16 Thread Christian Tismer 
Not trying to argue about Mercurial, but if a major amount
of startup time is spent in site.py:

I think in cases like hg command line scripts there is no need
to import site just for hg scripts.

Maybe that would improve things if those startup scripts avoid importing
site? Or do they, already?

cheers - chris


On 16.04.14 04:35, Guido van Rossum wrote:
> Well, that's the part that does "import site". Anything that speeds up
> the code in Lib/site.py might help. :-)
>
>
> On Tue, Apr 15, 2014 at 5:23 PM, Terry Reedy  > wrote:
>
> On 4/15/2014 5:26 PM, Brett Cannon wrote:
>
> To finish my timing work I decided to see
> where Py_InitializeEx_Private() spends its time. The following
> is a
> breakdown measured in microseconds running using -E:
>
> INIT:
> setlocale: 11
> envvar: 2
> random init: 2
> interp creation: 15
> thread creation: 6
> GIL: 10
> _Py_ReadyTypes(): 930
> more types: 44
> builtins: 141
> exceptions: 287
> sys: 258
> _PyImport_Init: 15
> import hooks: 4
> _PyWarnings_Init(): 15
> ENTERING import_init():
>PyImport_ImportFrozenModule(_frozen_importlib): 1186
>interp->importlib: 6
>PyInit_imp(): 15
>_imp: 3
>importlib._install(): 876
>_PyImportZip_Init(): 130
> _PyFaulthandler_Init(): 13
> time: 3
> ENTERING initfsencoding():
>codec lookup: 2179
> signals: 120
> tracemalloc: 7
> __main__: 13
> initstdio(): 1568
> warnings module: 4
> initsite(): 9552
>
>
> It looks like initsite takes half the time. Can it be sped up?
>
>
> -- 
> Terry Jan Reedy
>
>
> ___
> Python-Dev mailing list
> [email protected] 
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
>
> -- 
> --Guido van Rossum (python.org/~guido )
>
>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com


-- 
Christian Tismer :^)   
Software Consulting  : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-16 Thread Christian Heimes 
On 16.04.2014 04:35, Guido van Rossum wrote:
> Well, that's the part that does "import site". Anything that speeds up
> the code in Lib/site.py might help. :-)

Antoine, Victor and me have implemented a couple of speed ups for
"import site" already. I removed several unnecessary imports by
rearranging some code. AFAIK imports on OSX haven't been optimized yet.

On Linux I get:

$ ./python -c "import sys; print(len(sys.modules))"
34
$ ./python -S -c "import sys; print(len(sys.modules))"
22

Brett, are you an OSX developer? :)

Christian

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-16 Thread Brett Cannon 
On Apr 16, 2014 3:47 AM, "Christian Heimes"  wrote:
>
> On 16.04.2014 04:35, Guido van Rossum wrote:
> > Well, that's the part that does "import site". Anything that speeds up
> > the code in Lib/site.py might help. :-)
>
> Antoine, Victor and me have implemented a couple of speed ups for
> "import site" already. I removed several unnecessary imports by
> rearranging some code. AFAIK imports on OSX haven't been optimized yet.
>
> On Linux I get:
>
> $ ./python -c "import sys; print(len(sys.modules))"
> 34
> $ ./python -S -c "import sys; print(len(sys.modules))"
> 22
>
> Brett, are you an OSX developer? :)

Yep, my laptop is OS X.

-Brett

>
> Christian
>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/brett%40python.org
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-16 Thread Terry Reedy 

On 4/16/2014 3:46 AM, Christian Heimes wrote:

On 16.04.2014 04:35, Guido van Rossum wrote:

Well, that's the part that does "import site". Anything that speeds up
the code in Lib/site.py might help. :-)


Antoine, Victor and me have implemented a couple of speed ups for
"import site" already. I removed several unnecessary imports by
rearranging some code. AFAIK imports on OSX haven't been optimized yet.

On Linux I get:

$ ./python -c "import sys; print(len(sys.modules))"
34
$ ./python -S -c "import sys; print(len(sys.modules))"
22


With 3.4.0 installed on Windows, or 3.5 repository, 39 and 25

--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-16 Thread Antoine Pitrou 
On Wed, 16 Apr 2014 09:39:34 +0200
Christian Tismer  wrote:
> 
> I think in cases like hg command line scripts there is no need
> to import site just for hg scripts.

If you don't import site you won't be able to import Mercurial in most
cases.
Also, talking about "scripts" is a bit silly here - Mercurial is a
full-blown applications with many modules and tens of thousands of
lines of code.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (2.7): do not generate pipe names in the temporary dir

2014-04-16 Thread Antoine Pitrou 
On Mon, 14 Apr 2014 18:24:44 +0200 (CEST)
benjamin.peterson  wrote:
> http://hg.python.org/cpython/rev/ea677e26dbeb
> changeset:   90269:ea677e26dbeb
> branch:  2.7
> parent:  90261:c095ff9b0e84
> user:Benjamin Peterson 
> date:Mon Apr 14 12:24:37 2014 -0400
> summary:
>   do not generate pipe names in the temporary dir
> 
> files:
>   Lib/multiprocessing/connection.py |  2 +-
>   1 files changed, 1 insertions(+), 1 deletions(-)
> 
> 
> diff --git a/Lib/multiprocessing/connection.py 
> b/Lib/multiprocessing/connection.py
> --- a/Lib/multiprocessing/connection.py
> +++ b/Lib/multiprocessing/connection.py
> @@ -90,7 +90,7 @@
>  return tempfile.mktemp(prefix='listener-', dir=get_temp_dir())
>  elif family == 'AF_PIPE':
>  return tempfile.mktemp(prefix=r'\\.\pipe\pyc-%d-%d-' %
> -   (os.getpid(), _mmap_counter.next()))
> +   (os.getpid(), _mmap_counter.next()), dir="")

This is some Windows-specific code, right? Did you witness problems
with this? Your commit message doesn't mention an issue number.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [numpy wishlist] PyMem_*Calloc

2014-04-16 Thread Julian Taylor 
Hi,
In NumPy what we want is the tracing, not the exchangeable allocators.
I don't think it is a good idea for the core of a whole stack of
C-extension based modules to replace the default allocator or allowing
other modules to replace the allocator NumPy uses.

I think it would be more useful if Python provides functions to
register memory allocations and frees and the tracemalloc module
registers handlers for these register functions.
If no trace allocation tracer is registered the functions just return
immediately.
That way the tracemalloc can be used with arbitrary allocators as long
as they register their allocations with Python.

For example a hugepage allocator, which you would not want to use that
as the default allocator for all python objects, but you may still
want to trace its usage:

my_hugetlb_alloc(size)
p = mmap('hugepagefs', ..., MAP_HUGETLB);
PyMem_Register_Alloc(p, size, __func__, __line__);
return p

my_hugetlb_free(p);
PyMem_Register_Free(p, __func__, __line__);
munmap(p, ...);

normally the registers are nops, but if tracemalloc did register
tracers the memory is tracked, e.g. tracemodule does this on start():
tracercontext.register_alloc = trace_alloc
tracercontext.register_free = trace_free
tracercontext.data = mycontext
PyMem_SetTracer(&tracercontext)

Regards,
Julian Taylor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (2.7): do not generate pipe names in the temporary dir

2014-04-16 Thread Benjamin Peterson 
On Wed, Apr 16, 2014, at 7:39, Antoine Pitrou wrote:
> On Mon, 14 Apr 2014 18:24:44 +0200 (CEST)
> benjamin.peterson  wrote:
> > http://hg.python.org/cpython/rev/ea677e26dbeb
> > changeset:   90269:ea677e26dbeb
> > branch:  2.7
> > parent:  90261:c095ff9b0e84
> > user:Benjamin Peterson 
> > date:Mon Apr 14 12:24:37 2014 -0400
> > summary:
> >   do not generate pipe names in the temporary dir
> > 
> > files:
> >   Lib/multiprocessing/connection.py |  2 +-
> >   1 files changed, 1 insertions(+), 1 deletions(-)
> > 
> > 
> > diff --git a/Lib/multiprocessing/connection.py 
> > b/Lib/multiprocessing/connection.py
> > --- a/Lib/multiprocessing/connection.py
> > +++ b/Lib/multiprocessing/connection.py
> > @@ -90,7 +90,7 @@
> >  return tempfile.mktemp(prefix='listener-', dir=get_temp_dir())
> >  elif family == 'AF_PIPE':
> >  return tempfile.mktemp(prefix=r'\\.\pipe\pyc-%d-%d-' %
> > -   (os.getpid(), _mmap_counter.next()))
> > +   (os.getpid(), _mmap_counter.next()), dir="")
> 
> This is some Windows-specific code, right? Did you witness problems
> with this? Your commit message doesn't mention an issue number.

http://buildbot.python.org/all/builders/x86%20Windows7%202.7/builds/2549/steps/test/logs/stdio

I'd be interested to know how that ever worked.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (2.7): do not generate pipe names in the temporary dir

2014-04-16 Thread Antoine Pitrou 
On Wed, 16 Apr 2014 08:40:33 -0700
Benjamin Peterson  wrote:

> On Wed, Apr 16, 2014, at 7:39, Antoine Pitrou wrote:
> > On Mon, 14 Apr 2014 18:24:44 +0200 (CEST)
> > benjamin.peterson  wrote:
> > > http://hg.python.org/cpython/rev/ea677e26dbeb
> > > changeset:   90269:ea677e26dbeb
> > > branch:  2.7
> > > parent:  90261:c095ff9b0e84
> > > user:Benjamin Peterson 
> > > date:Mon Apr 14 12:24:37 2014 -0400
> > > summary:
> > >   do not generate pipe names in the temporary dir
> > > 
> > > files:
> > >   Lib/multiprocessing/connection.py |  2 +-
> > >   1 files changed, 1 insertions(+), 1 deletions(-)
> > > 
> > > 
> > > diff --git a/Lib/multiprocessing/connection.py 
> > > b/Lib/multiprocessing/connection.py
> > > --- a/Lib/multiprocessing/connection.py
> > > +++ b/Lib/multiprocessing/connection.py
> > > @@ -90,7 +90,7 @@
> > >  return tempfile.mktemp(prefix='listener-', dir=get_temp_dir())
> > >  elif family == 'AF_PIPE':
> > >  return tempfile.mktemp(prefix=r'\\.\pipe\pyc-%d-%d-' %
> > > -   (os.getpid(), _mmap_counter.next()))
> > > +   (os.getpid(), _mmap_counter.next()), 
> > > dir="")
> > 
> > This is some Windows-specific code, right? Did you witness problems
> > with this? Your commit message doesn't mention an issue number.
> 
> http://buildbot.python.org/all/builders/x86%20Windows7%202.7/builds/2549/steps/test/logs/stdio
> 
> I'd be interested to know how that ever worked.

I don't know, but perhaps opening an issue would allow a Windows expert
(or Richard) to chime in.

Regards

Antoine.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-16 Thread Martin v. Löwis 
Am 14.04.14 23:51, schrieb Brett Cannon:
> It was realized during PyCon that since we are freezing importlib we
> could now consider freezing all the modules to cut out having to stat or
> read them from disk.
[...]
> Thoughts?

They still get read from disk, except that it is the operating system
that does the reading. So what you really save is the access to many
tiny files; something that can also be achieved with the zipfile import.
So I wonder how your all-frozen binary compares to a standard binary
with a python35.zip.

If it is comparable, I'd rather extend on that route, i.e. promote
putting the standard library into a zip file in the default
installation, and also find a way where (say) /usr/bin/hg could
conveniently specify a zip file that will contain the Mercurial
byte code. For example, we could support a -Z option for the interpreter
which would allow to append a zip file to a script that gets put on
sys.path.

Regards,
Martin

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [numpy wishlist] PyMem_*Calloc

2014-04-16 Thread Victor Stinner 
Hi,

2014-04-16 7:51 GMT-04:00 Julian Taylor :
> In NumPy what we want is the tracing, not the exchangeable allocators.

Did you read the PEP 445? Using the new malloc API, in fact you can
have both: install new allocators and set up hooks on allocators.
http://legacy.python.org/dev/peps/pep-0445/

The PEP 445 has been implemented in Python 3.4, we don't plan to
rewrite it. So it's probably better to try to understand how it was
designed and why we chose this design.

See the talk I just have at Pycon Montreal for more information on how
tracemalloc works. Slides:
https://raw.githubusercontent.com/haypo/conf/master/2014-Pycon-Montreal/tracemalloc.pdf

Video:
http://pyvideo.org/video/2698/track-memory-leaks-in-python

> my_hugetlb_alloc(size)
> p = mmap('hugepagefs', ..., MAP_HUGETLB);
> PyMem_Register_Alloc(p, size, __func__, __line__);
> return p
>
> my_hugetlb_free(p);
> PyMem_Register_Free(p, __func__, __line__);
> munmap(p, ...);

This is exactly how tracemalloc works. The advantage of the PEP 445 is
that you have a null overhead when tracemalloc is disabled. There is
no need to check if a trace function is present or not.

You can chain multiple hooks.

See also the calloc issue which was written for NumPy:
http://bugs.python.org/issue21233

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-16 Thread Terry Reedy 

On 4/16/2014 12:25 PM, "Martin v. Löwis" wrote:

Am 14.04.14 23:51, schrieb Brett Cannon:

It was realized during PyCon that since we are freezing importlib we
could now consider freezing all the modules to cut out having to stat or
read them from disk.

[...]

Thoughts?


They still get read from disk, except that it is the operating system
that does the reading. So what you really save is the access to many
tiny files; something that can also be achieved with the zipfile import.
So I wonder how your all-frozen binary compares to a standard binary
with a python35.zip.

If it is comparable, I'd rather extend on that route, i.e. promote
putting the standard library into a zip file in the default
installation, and also find a way where (say) /usr/bin/hg could
conveniently specify a zip file that will contain the Mercurial
byte code. For example, we could support a -Z option for the interpreter
which would allow to append a zip file to a script that gets put on
sys.path.


This could be useful for Idle also, as its startup is noticeably 
sluggish and could definitely stand to be zippier. About 50 Idle modules 
are imported in the user process and, I presume, at least as many in the 
Idle process.


PS. In the user process sys.modules, there are numerous null entries 
like these:

>>> sys.modules['idlelib.os']
>>> sys.modules['idlelib.tokenize']
>>> sys.modules['idlelib.io']
>>> 

Does anyone know the most likely reason?

--
Terry Jan Reedy


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-16 Thread Dr. Brett Cannon 
Is this Python 2 or 3? In Python 2 it means an attempt to perform a
relative import failed but an absolute in succeeded, e.g. from idlelib you
imported os, so import tried idlelib.is and then os. You should definitely
consider using a future import to guarantee absolute imports.

On Wednesday, April 16, 2014 2:57:35 PM, Terry Reedy 
wrote:

> On 4/16/2014 12:25 PM, "Martin v. Löwis" wrote:
> > Am 14.04.14 23:51, schrieb Brett Cannon:
> >> It was realized during PyCon that since we are freezing importlib we
> >> could now consider freezing all the modules to cut out having to stat or
> >> read them from disk.
> > [...]
> >> Thoughts?
> >
> > They still get read from disk, except that it is the operating system
> > that does the reading. So what you really save is the access to many
> > tiny files; something that can also be achieved with the zipfile import.
> > So I wonder how your all-frozen binary compares to a standard binary
> > with a python35.zip.
> >
> > If it is comparable, I'd rather extend on that route, i.e. promote
> > putting the standard library into a zip file in the default
> > installation, and also find a way where (say) /usr/bin/hg could
> > conveniently specify a zip file that will contain the Mercurial
> > byte code. For example, we could support a -Z option for the interpreter
> > which would allow to append a zip file to a script that gets put on
> > sys.path.
>
> This could be useful for Idle also, as its startup is noticeably
> sluggish and could definitely stand to be zippier. About 50 Idle modules
> are imported in the user process and, I presume, at least as many in the
> Idle process.
>
> PS. In the user process sys.modules, there are numerous null entries
> like these:
>  >>> sys.modules['idlelib.os']
>  >>> sys.modules['idlelib.tokenize']
>  >>> sys.modules['idlelib.io']
>  >>> 
>
> Does anyone know the most likely reason?
>
> --
> Terry Jan Reedy
>
>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> brett%40python.org
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-16 Thread Christian Tismer 
On 16/04/14 16:35, Antoine Pitrou wrote:
> On Wed, 16 Apr 2014 09:39:34 +0200
> Christian Tismer  wrote:
>>
>> I think in cases like hg command line scripts there is no need
>> to import site just for hg scripts.
> 
> If you don't import site you won't be able to import Mercurial in most
> cases.


Oh is that so?
I thought about hg as a set of command-line scripts with a defined
interface that should IMO not be dependent from any site-specific
settings.
Instead, it should be configured at install time, and after that
be completely independent from site settings.


> Also, talking about "scripts" is a bit silly here - Mercurial is a
> full-blown applications with many modules and tens of thousands of
> lines of code.


Thanks for regarding my comments as silly.

In my sense it would be silly if that full-blown app would suffer from
or relate to site at all, tho. Apps are meant to be self-contained.

regards - Chris

-- 
Christian Tismer :^)   
Software Consulting  : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-16 Thread Nick Coghlan 
On 16 April 2014 12:25, "Martin v. Löwis"  wrote:
> Am 14.04.14 23:51, schrieb Brett Cannon:
>> It was realized during PyCon that since we are freezing importlib we
>> could now consider freezing all the modules to cut out having to stat or
>> read them from disk.
> [...]
>> Thoughts?
>
> They still get read from disk, except that it is the operating system
> that does the reading. So what you really save is the access to many
> tiny files; something that can also be achieved with the zipfile import.
> So I wonder how your all-frozen binary compares to a standard binary
> with a python35.zip.
>
> If it is comparable, I'd rather extend on that route, i.e. promote
> putting the standard library into a zip file in the default
> installation, and also find a way where (say) /usr/bin/hg could
> conveniently specify a zip file that will contain the Mercurial
> byte code. For example, we could support a -Z option for the interpreter
> which would allow to append a zip file to a script that gets put on
> sys.path.

Has anyone tried running mercurial as a zipfile with __main__.py and a
prepended shebang line rather than as a collection of independent
files?

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-16 Thread Nick Coghlan 
On 16 April 2014 16:33, Christian Tismer  wrote:
>
> In my sense it would be silly if that full-blown app would suffer from
> or relate to site at all, tho. Apps are meant to be self-contained.

For better or for worse, Python applications like Mercurial are
currently considered an integrated part of Linux distros and hence
subject to the usual de-duplication rules (like not embedding their
own Python interpreter, and relying on shared system libraries when
appropriate). And yes, this does cause problems (but not integrating
them causes major security update management problems).

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Language Summit notes

2014-04-16 Thread Taavi Burns 
I was the guy at the back taking pictures. I also took notes! :)


Language Summit
---

Intros
~~

Everyone starts by doing a 15 second intro. People are good and keep it
short, and we make it through all ~40 people. People start introduce
themselves saying "module X is my fault". Guido just says, "It's my
fault."

Release Management
~~

Larry Hastings talks about release versioning! How to use branches for
the 3.5 RC process. When RCs start, should default be locked down,
should it become 3.5.1, or should it become 3.6? Or something else?
After a bit of discussion, people asked to move the decision to email.

PyPy


PyPy update: 7 iterations of stm! (compared to "The JIT took 5?") STM is
25-50% slower on a single-threaded workload, but shows nearly linear
speedups with more cores on non-conflicting workloads. Brett Cannon: "So
it's only HOW much faster than CPython?" It's not sure if there will be
multiple PyPy releases or everyone takes the hit and STM is default. STM
fundraiser II is active now.

PyPy oving to 2.7.6 stdlib soon. A bunch of effort has gone to
supporting CFFI porting in the community which aids porting of packages
to PyPy. Something about Python 3 support that I didn't quite catch.
Alex Gaynor says that most people who look to PyPy are looking for
Python 2 right now.

IronPython
~~

IronPython is active again! New contributors! Python 3 support! What can
the PSF do to help IronPython? IronPython should ask for support if it
would help.

Jython
~~

Nobody from the team was present, but they sent a report which Michael
Foord talked through with the group. Jython moving to Java 7 minimum
runtime. Added buffer protocol (memoryview etc), required for Python 3
support. PyPA tooling? CFFI backend is coming! EuroPython sprint in
July. Looking forward to Python 3 because of string types vs JVM. But
it's slow (work?).

Packaging
~

pip will suddenly need to become more user-friendly, now that it's
bundled with 3.4. Instead of being used by people who need it, it's
going to be presented to people who don't know what it is. Nick Coghlan:
"PyPI's security model: it didn't have one." Richard Jones says the new
PyPI code is nearly ready. It has some rough bits (OAuth, OpenID) but
they're rough anyway. Replacing old crufty code with nice code; then
moving forward with shiny new features. Oh, and wheels!

Want to be at the point where Windows and OSX users can download the
binary distro from python.org, and ``pip install django`` and have it
work. Numpy is apparently still a sore point; conda would still be a
better way for most people. Largely two kinds of packages: those that
are self-contained (may include c-extensions though), and those that
depend on system libraries (e.g. link to libmysql). Packaging has to
make both of those work, and if it's made easier to make distro packages
(i.e. RedHat packages, Debian packages, etc), it should be easier to
build conda packages. Nick doesn't want to cross the line where python
packaging (PyPI) does the distro side of things. Currently, "we do c
extensions, but not linking to C libraries". Also, "metadata 2" has been
put off by ~12 months, but is becoming the last thing standing before
dropping setup.py and having wheels for all? Bits of metdata 2 have been
pushed out as extensions or "decide later" bits, which has been
instructive in terms of "how far can we get".

Build chain in stdlib: bad idea, release schedules are different. Need a
build system that supports all the verisons in the wild. Setuptools is
now version-independent distutils.

Oh, and Apple screwed up python, building it with a compiler that's not
in xcode.

``pkg_resources`` has a dependency on ``setuptools`` which is WRONG,
needs to be the other way around. But ``pkg_resources`` does what it
does very well.

Technically pip itself doesn't ship with 3.4, but rather ensurepip. This
is so that new versions of pip can be brought in in point releases, and
for security updates. Windows and OSX installers run ensurepip.
``virtualenv`` installs pip by default at this point, too.

Someone asks, "Can we do that for SSL?" Everyone laughs.

Disussion about packaging continues. Glyph asks if the PSF could fund a
usability study on installing Python. People generally seem to think
it's a good idea.

Go to `packaging.python.org `__!
Documentation! Quite a few sections are already filled in: please
contribute!

There is no "one installer" that has everything you need for 2.7 right
now.

Nick: "C development is stupidly complicated."

Nature of the old guard of Python programmers: you were already a C
developer, and just wanted Python to help make some things easier. Nick
got into Python for DSP testing. The demographics have changed. How do
we change the docs and ecosystem to avoid the assumption that Python
programmers already know how to program in C? "We have the wrong people
maintaining the docs."

"A distro is

[Python-Dev] New mailing list for workflow/workflow infrastructure discussion/tasks

2014-04-16 Thread R. David Murray 
Based on a number of conversations at PyCon, we've created a new mailing list:

   https://mail.python.org/mailman/listinfo/core-workflow 

The purpose of this list is to facilitate the conversations and coordinate
the work that needs to happen to improve our development workflow.
Nick's PEP is one piece of this conversation, but there are many other
aspects to it as well.

Here is the list description:

This is a place to discuss and work on improvements to the CPython core
development workflow and the infrastructure that supports that workflow.
This includes changes to the roundup interface and functionality, rietveld,
mercurial, buildbots, and any other infrastructure we may add.  It also
includes discussing how we use these tools, and most importantly how we use
these tools to integrate the community beyond the core developers into the
workflow that gets patches committed to the python repository.  This means
that it also includes discussions of the process of bringing in new
contributors, including how we use the core-mentorship list, as well as how
we organize ticket triage, and how we make use of external resources such
as openhatch.  Discussions of documentation and how we organize and
maintain the documentation are also appropriate.

Anyone who is interested helping with this, or who wants to keep up with the
evolution of our thoughts on these topics, are invited to sign up to the
mailing list.

We will of course check in with python-dev and/or python-committers
as appropriate.

--David

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-16 Thread Terry Reedy 

> On Wednesday, April 16, 2014 2:57:35 PM, Terry Reedy  > wrote:

> PS. In the user process sys.modules, there are numerous null
> entries like these:
>   >>> sys.modules['idlelib.os']
>   >>> sys.modules['idlelib.tokenize'__]
>   >>> sys.modules['idlelib.io ']
>   >>> 

On 4/16/2014 3:10 PM, Dr. Brett Cannon wrote:

Is this Python 2 or 3?


Py 2. I should have said so. The entries do not appear in py3.


In Python 2 it means an attempt to perform a relative import failed but
an absolute in succeeded, e.g. from idlelib you imported os, so import

> tried idlelib.is  and then  os.

*I* have not done anything. For tokenize, for instance, the existing 
code just does what I though were absolute imports, in 2 files.

  import tokenize

Perhaps the extra entries have something to do with the fact that these 
startup imports are invisible to user code, just like those done by the 
interpreter itself on startup. 2.7 uses spawnv (and 3.4 uses subprocces) 
to run something like one of the following.

  python -c "__import__('idlelib.run').run.main(False)"
  python -c "__import__('run').main(False)"

run.py has several normal lines with
  import 
  from idlelib import 
and ditto for some of the imported idlelib modules.

> You should definitely consider using a future import to guarantee

absolute imports.


--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New mailing list for workflow/workflow infrastructure discussion/tasks

2014-04-16 Thread Ned Deily 
In article <[email protected]>,
 "R. David Murray"  wrote:

> https://mail.python.org/mailman/listinfo/core-workflow

FYI, I've submitted a request for it to be mirrored at 
gmane.comp.python.devel.core-workflow

-- 
 Ned Deily,
 [email protected]

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 466: Network security enhancements for Python 2.7.7

2014-04-16 Thread Nick Coghlan 
I've reworded the PEP to make it clear it is now just about
backporting a specific set of enhancements to 2.7.7, as well as
switching to updating to new OpenSSL feature releases in the binary
installers.

The idea of an open ended backport policy is now listed as a rejected
variant. I believe that change addresses Guido's main remaining
concern, so I think this version is ready for pronouncement.

Regards,
Nick.


PEP: 466
Title: Network Security Enhancements for Python 2.7.7
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan ,
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 23-Mar-2014
Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014, 16-Apr-2014


Abstract


Most CPython tracker issues are classified as errors in behaviour or
proposed enhancements. Most patches to fix behavioural errors are
applied to all active maintenance branches.  Enhancement patches are
restricted to the default branch that becomes the next Python version.

This cadence works reasonably well during Python's normal 18-24 month
feature release cycle, which is still applicable to the Python 3 series.
However, the age of the standard library in Python 2 has now reached a point
where it is sufficiently far behind the state of the art in network security
protocols for it to be causing real problems in use cases where upgrading to
Python 3 in the near term may not be feasible.

In recognition of the additional practical considerations that have arisen
during the 4+ year maintenance cycle for Python 2.7, this PEP allows a
critical set of network security related features to be backported from
Python 3.4 to the upcoming Python 2.7.7 maintenance release.

While this PEP does not make any changes to the core development team's
handling of security-fix-only branches that are no longer in active
maintenance, it *does* recommend that commercial redistributors providing
extended support periods for the Python standard library either backport
these features to their supported versions, or else explicitly disclaim
support for the use of older versions in roles that involve connecting
directly to the public internet.


New security related features in Python 2.7.7
=

Under this proposal, the following features will be backported from Python
3.4 to the upcoming Python 2.7.7 maintenance release:

* in the ``os`` module:

  * persistent file descriptor for ``os.urandom()``.

* in the ``hmac`` module:

  * constant time comparison function (``hmac.compare_digest()``).

* in the ``hashlib`` module:

  * password hashing function (``hashlib.pbkdf2_hmac()``).
  * details of hash algorithm availability (``hashlib.algorithms_guaranteed``
and ``hashlib.algorithms_available``).

* in the ``ssl`` module:

  * this module is almost entirely synchronised with its Python 3
counterpart, bringing TLSv1.x settings, SSLContext manipulation, Server
Name Indication, access to platform certificate stores, standard
library support for peer hostname validation and more to the Python 2
series.
  * the only ``ssl`` module features *not* backported under this policy are
the ``ssl.RAND_*`` functions that provide access to OpenSSL's random
number generation capabilities - use ``os.urandom()`` instead.

As a general change in maintenance policy, permission is also granted to
upgrade to newer feature releases of OpenSSL when preparing the binary
installers for new maintenance releases of Python 2.7.

This PEP does NOT propose a general exception for backporting new features
to Python 2.7 - every new feature proposed for backporting will still need
to be justified independently. In particular, it will need to be explained
why relying on and independently updated backport on the Python Package Index
instead is not an acceptable solution.


Backwards compatibility considerations
==

As in the Python 3 series, the backported ``ssl.create_default_context()``
API is granted a backwards compatibility exemption that permits the
protocol, options, cipher and other settings of the created SSL context to
be updated in maintenance releases to use higher default security settings.
This allows them to appropriately balance compatibility and security at the
time of the maintenance release, rather than at the time of the original
feature release.

This PEP does *not* grant any other exemptions to the usual backwards
compatibility policy for maintenance releases. Instead, by explicitly
encouraging the use of feature based checks, it is designed to make it easier
to write more secure cross-version compatible Python software, while still
limiting the risk of breaking currently working software when upgrading to
a new Python 2.7 maintenance release.

In all cases where this proposal allows new features to be backported to
the Python 2.7 release series, it is possible to write cross-version
compatible

Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-16 Thread Antoine Pitrou 
On Wed, 16 Apr 2014 22:33:49 +0200
Christian Tismer  wrote:
> On 16/04/14 16:35, Antoine Pitrou wrote:
> > On Wed, 16 Apr 2014 09:39:34 +0200
> > Christian Tismer  wrote:
> >>
> >> I think in cases like hg command line scripts there is no need
> >> to import site just for hg scripts.
> > 
> > If you don't import site you won't be able to import Mercurial in most
> > cases.
> 
> 
> Oh is that so?
> I thought about hg as a set of command-line scripts with a defined
> interface that should IMO not be dependent from any site-specific
> settings.

Well perhaps you should take a look. Mercurial is actually a package
(in the Python sense) named "mercurial" with a bunch of modules inside
it. When it gets installed, it's typically installed into
site-packages, which means site.py is necessary for the install
location to crop up into sys.path.

Regards

Antoine.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Language Summit notes

2014-04-16 Thread Antoine Pitrou 

Hi Taavi,

Thanks for the report!

> Disussion about packaging continues. Glyph asks if the PSF could fund a
> usability study on installing Python. People generally seem to think
> it's a good idea.

What does this mean exactly? Under OS X and Linux, Python is typically
installed by default. And under Windows, it's a simple installer that
even non-Windows users like me have no problem executing. So what is
the problem this is trying to solve?

> There is no "one installer" that has everything you need for 2.7 right
> now.

Neither for 3.x, for that record.

> Lunch
> ~
> 
> There was food!

Good to know nobody starved to death :-)

> AP exams are starting to allow Python, but it's 10% of the AP CS exams.

"AP"?
(I thought that was me, but it sounds unlikely :-))

> Selena says that some American parents are worried because they don't
> know if their kids could get jobs [Taavi: loud surprise from audience].

?? That's not exclusively American.

> Estimate of 1 person full time for 1.5 years to get mercurial running on
> Python 3, at half the speed of Python 2.

That sounds exagerated, IMO.

> Guido: Can we get the PSF to issue a press release that 2.7 support will
> continue?

What does "support" mean exactly here?

> -  Glyph will tell us what to do in 3.5.

Thank you, Glyph!

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Language Summit notes

2014-04-16 Thread Benjamin Peterson 
On Wed, Apr 16, 2014, at 15:26, Antoine Pitrou wrote:
> 
> Hi Taavi,
> 
> Thanks for the report!
> 
> > Disussion about packaging continues. Glyph asks if the PSF could fund a
> > usability study on installing Python. People generally seem to think
> > it's a good idea.
> 
> What does this mean exactly? Under OS X and Linux, Python is typically
> installed by default. And under Windows, it's a simple installer that
> even non-Windows users like me have no problem executing. So what is
> the problem this is trying to solve?

The installers might be hard to find on the website or hard to use.

> 
> > There is no "one installer" that has everything you need for 2.7 right
> > now.
> 
> Neither for 3.x, for that record.
> 
> > Lunch
> > ~
> > 
> > There was food!
> 
> Good to know nobody starved to death :-)
> 
> > AP exams are starting to allow Python, but it's 10% of the AP CS exams.
> 
> "AP"?
> (I thought that was me, but it sounds unlikely :-))

https://en.wikipedia.org/wiki/Advanced_Placement
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Language Summit notes

2014-04-16 Thread Donald Stufft 

On Apr 16, 2014, at 6:38 PM, Benjamin Peterson  wrote:

> On Wed, Apr 16, 2014, at 15:26, Antoine Pitrou wrote:
>> 
>> Hi Taavi,
>> 
>> Thanks for the report!
>> 
>>> Disussion about packaging continues. Glyph asks if the PSF could fund a
>>> usability study on installing Python. People generally seem to think
>>> it's a good idea.
>> 
>> What does this mean exactly? Under OS X and Linux, Python is typically
>> installed by default. And under Windows, it's a simple installer that
>> even non-Windows users like me have no problem executing. So what is
>> the problem this is trying to solve?
> 
> The installers might be hard to find on the website or hard to use.
> 
>> 
>>> There is no "one installer" that has everything you need for 2.7 right
>>> now.
>> 
>> Neither for 3.x, for that record.
>> 
>>> Lunch
>>> ~
>>> 
>>> There was food!
>> 
>> Good to know nobody starved to death :-)
>> 
>>> AP exams are starting to allow Python, but it's 10% of the AP CS exams.
>> 
>> "AP"?
>> (I thought that was me, but it sounds unlikely :-))
> 
> https://en.wikipedia.org/wiki/Advanced_Placement
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

Possibly Glyph meant installing a Python *stack*, which likely includes 
setuptools and
pip in order to actually get other things installable. Possibly also a compiler 
setup
for installing C things.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Language Summit notes

2014-04-16 Thread R. David Murray 
On Wed, 16 Apr 2014 15:38:21 -0700, Benjamin Peterson  
wrote:
> On Wed, Apr 16, 2014, at 15:26, Antoine Pitrou wrote:
> > 
> > Hi Taavi,
> > 
> > Thanks for the report!
> > 
> > > Disussion about packaging continues. Glyph asks if the PSF could fund a
> > > usability study on installing Python. People generally seem to think
> > > it's a good idea.
> > 
> > What does this mean exactly? Under OS X and Linux, Python is typically
> > installed by default. And under Windows, it's a simple installer that
> > even non-Windows users like me have no problem executing. So what is
> > the problem this is trying to solve?
> 
> The installers might be hard to find on the website or hard to use.

Exactly.  As Glyph said, the usability study starts from giving the
user the task "install Python on your computer", or "get X working
on your computer" where X is something that is written in Python.
Running the installer is just one step in that process, and not one of
the difficult ones.

Let's give it a whirl:

Googling for 'python' got me python.org as the top hit.  So far so
good.

Clicking on python.org, I looked for a download link.  There wasn't a
button, but there was a header that says 'Download' with a nice download
icon next to it, just below the big blue area that comprises most of
the screen.  I couldn't click on 'download', though.  Nor on the little
download icon.  Then I saw the 'click here' text below it, so I clicked
on that.  That took me to an old-website-style page that was a wall of
text that starts out talking about python 3.0 being released in 2008.
(No, I didn't read the text before the link that explained where I was
going to be sent: I'm looking for a download link.)

Going back to the main page, I noticed that below the 'click here'
there was another link labeled "Python 3.4.0 - Python 2.7.6" (yes,
I know that's really two links, but no that isn't obvious, so someone
clicking on "it" is likely to get one or the other of them more or
less at random, although maybe they'll notice as the link changes
color when their cursor moves over it...).  So clicking on that, I
get another slightly less intimidating wall of text with a lot of "PEP
NNN" lines.  Scrolling down, after a section labeled 'More Resources',
there is a "Download" header...which is again not clickable.  Oh but look,
there's a 'download page' link in the middle of the paragraph after that
"Download" header.

Clicking on *that*, the first couple links on the page are 'Release Notes'
and 'Detailed Release Information'.  Below that we have a Files section.
It starts out with Mac OSX installer, then XZ compressed source tarball,
then gzipped source tarball, then, finally, a Windows installer...followed
by something about a debug information file, a help information file,
another debug information file, and another Windows installer file.
Oh, yeah, that's 64 bit vs 32 bit.  (Are normal windows users
comfortable about picking the right one of those? I assume they are.)

You will note that I missed the 'downloads' link in the menu bar in the
middle of the big blue section on the front page.  If we rewind, and I
hover there, I see a big "Python Source" with buttons for 3.4 and 2.7.
Guess what I'm going to do next if I'm a naive user?  Hmm.  "xxx.xz".
That doesn't look like an installer or executable, so cancel that.
Oh yeah, there's a 'windows' link on the left.  If I click that...I'm
back on that wall-of-peps page.

Googling 'install python' will get you to python.org as the top hit, with
some secondary links that take you to one of the links I found above,
or other less useful ones...but none of the top google links are to a
page that directly contains a link for downloading a windows installer,
not even the one labeled 'Windows'.

There is an 'Installing Python on Windows' link further down the google
results that links to a fairly good page from python-guide.org, whose
first link lets you download the 2.7.6 msi.  I guess that's the 32
bit version.  But it then tells you go to python.org to make sure you
get the latest release, and tells you to click on a link that doesn't
exist any more (the "windows installer" link).

So, yeah.

Usability.

--David

PS: the old website was worse in many ways, and was even more of a wall of
text, but at least once you see the "download now" link in that wall of
text it only takes one click to get to the windows installer (or zero,
if you happen to notice the 'windows installer link in the left hand
navigation bar), and the two windows installers are at the top of the
list on that download page.

PPS: I believe I've heard rumors that the download pages are a work
in progress, so I'm sure they will get *better*...but I think
the above indicates why a usability study isn't a bad idea :)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-ar

Re: [Python-Dev] Language Summit notes

2014-04-16 Thread Terry Reedy 

On 4/16/2014 6:26 PM, Antoine Pitrou wrote:


AP exams are starting to allow Python, but it's 10% of the AP CS exams.


"AP"?
(I thought that was me, but it sounds unlikely :-))


AP = Advanced Placement. US and Canadian high school students who have 
taken advanced (AP) courses equivalent to American college freshman 
courses can take AP exams to demonstrate that they learned and retained 
enough that they should get college credit for the course.

https://en.wikipedia.org/wiki/Advanced_Placement

I believe there is a committee for each subject that sets out a syllabus 
describing the subject that may be tested. The CS exam originally tested 
knowledge of Pascal (1984-1999) and switched to C++ (1999-2003) and then 
Java (2004-date).

https://en.wikipedia.org/wiki/Advanced_Placement_Computer_Science#AP_Computer_Science_A

The report is that Python is creeping in, though I an not sure exactly 
what the report above means. Python replacing Java as the AP CS language 
would be a fairly big deal here.


--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com