Re: [Python-Dev] Computed Goto dispatch for Python 2

[M.-A. Lemburg]

On 28.05.2015 02:17, Parasa, Srinivas Vamsi wrote:
> Hi All,
> 
> This is Vamsi from Server Scripting Languages Optimization team at Intel 
> Corporation.
> 
> Would like to submit a request to enable the computed goto based dispatch in 
> Python 2.x (which happens to be enabled by default in Python 3 given its 
> performance benefits on a wide range of workloads). We talked about this 
> patch with Guido and he encouraged us to submit a request on Python-dev 
> (email conversation with Guido shown at the bottom of this email).

+1.

It's been successful for Python 3, doesn't change semantics and
increases performance.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 28 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Berker Peksağ]

On Thu, May 28, 2015 at 3:17 AM, Parasa, Srinivas Vamsi
 wrote:
> Attached is the computed goto patch (along with instructions to run) for 
> Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin  at 
> http://bugs.python.org/issue4753). We built and tested this patch for Python 
> 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon – Haswell EP 
> CPU with 18 cores, hyper-threading off, turbo off).

Hi Vamsi,

Thank you for your work and your detailed email.

I'm -1 on the idea because:

* Performance improvements are not bug fixes
* The patch doesn't make the migration process from Python 2 to Python 3 easier
* In long term, it would be nice to work on making Python 3 better:
See http://bugs.python.org/issue11549 for an example task.

--Berker
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Nick Coghlan]

On 28 May 2015 at 18:54, Berker Peksağ  wrote:
> On Thu, May 28, 2015 at 3:17 AM, Parasa, Srinivas Vamsi
>  wrote:
>> Attached is the computed goto patch (along with instructions to run) for 
>> Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin  at 
>> http://bugs.python.org/issue4753). We built and tested this patch for Python 
>> 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon – Haswell EP 
>> CPU with 18 cores, hyper-threading off, turbo off).
>
> Hi Vamsi,
>
> Thank you for your work and your detailed email.
>
> I'm -1 on the idea because:
>
> * Performance improvements are not bug fixes

The "nothing except backwards compatible bug fixes in maintenance
releases" rule was adopted for the *benefit of Python users*. When a
new feature release can be reliably expected every 18-24 months, it
makes sense to err heavily on the side of minimising risks to
stability when it comes to making judgement calls on whether or not a
change is appropriate to a maintenance release or not.

Python 2.7 is an unusual case, as even though there *are* newer
feature releases available, the barriers to migration are much higher
than they would otherwise be. Each progressive 3.x release has brought
those barriers down a bit, and 3.5 and the static type checking work
being done through mypy et al will bring them down even further, but
version migration work is still work where end users don't see any
immediate practical benefit - they only see the benefit *after*
they're able to drop Python 2 compatibility, and can start using
Python 3 only features like matrix multiplication and the async/await
syntax.

*Adding* features to Python 2.7 is quite rightly still controversial,
as they add complexity to the compatibility matrix for testing
purposes. Code that runs correctly with the PEP 466 and 476 changes to
SSL handling, may fail on earlier versions.

Internal performance improvements, by contrast, don't hurt end users
at all beyond the stability risks, and in this case, the request to
make the change is being accompanied by the offer to assist with
ongoing maintenance (including engaging an experienced core developer
to help coach Intel contributors through the contribution process).

So when folks ask "What changed?" in relation to this request, what
changed is the fact that it isn't expected to be a one off
contribution, but rather part of a broader effort focused on improving
the performance of both Python 2 and Python 3, including contributions
to ongoing maintenance activities.

> * The patch doesn't make the migration process from Python 2 to Python 3 
> easier
> * In long term, it would be nice to work on making Python 3 better:

Indeed, but we also need help living up to the "Python 2.7 will be
supported to 2020" commitment. Python 2.7 maintenance is *not* a
particularly exciting task, and it's only going to get less
interesting as Python 3 adoption climbs, so we're going to need paid
contributors to start filling the gap as volunteers (quite reasonably)
move on to the more inherently rewarding task of working to move
Python 3 forward.

That's going to be a negotiation process - companies don't typically
contribute paid development time to open source projects out of the
kindness of their hearts, they do it either because they're using the
project themselves, because of deals they've made with individual
contributors around how they spend their time, or because it helps
them influence the direction of upstream development in ways that help
them and their customers. (And sometimes it's a mix of all 3 of those
factors)

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Nick Coghlan]

On 28 May 2015 at 19:47, Nick Coghlan  wrote:
> That's going to be a negotiation process - companies don't typically
> contribute paid development time to open source projects out of the
> kindness of their hearts, they do it either because they're using the
> project themselves, because of deals they've made with individual
> contributors around how they spend their time, or because it helps
> them influence the direction of upstream development in ways that help
> them and their customers. (And sometimes it's a mix of all 3 of those
> factors)

And to be completely transparent about this: this probably won't be
the last of these kinds of discussions we're likely to see.

Various folks (including me) have been negotiating to have their
employers fund paid CPython contribution positions and as we coach
colleagues that take up these roles through the core team's
contribution processes, one of the consequences will be that we will
sometimes advocate for acceptance of changes that we would have
historically rejected as imposing too high a maintenance burden for an
all-volunteer development team to be expected to deal with.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Julian Taylor]

won't this need python compiled with gcc 5.1 to have any effect? Which
compiler version was used for the benchmark?
the issue that negated most computed goto improvements
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=39284) was only closed
very recently (r212172, 9f4ec746affbde1)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Maciej Fijalkowski]

> I'm -1 on the idea because:
>
> * Performance improvements are not bug fixes
> * The patch doesn't make the migration process from Python 2 to Python 3 
> easier

And this is why people have been porting Python applications to Go.
Maybe addressing Python performance and making Python (2 or 3) a
better language/platform would mitigate that.

Cheers,
fijal
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Matthias Klose]

On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote:
> Hi All,
> 
> This is Vamsi from Server Scripting Languages Optimization team at Intel 
> Corporation.
> 
> Would like to submit a request to enable the computed goto based dispatch in 
> Python 2.x (which happens to be enabled by default in Python 3 given its 
> performance benefits on a wide range of workloads). We talked about this 
> patch with Guido and he encouraged us to submit a request on Python-dev 
> (email conversation with Guido shown at the bottom of this email).
> 
> Attached is the computed goto patch (along with instructions to run) for 
> Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin  at 
> http://bugs.python.org/issue4753). We built and tested this patch for Python 
> 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP 
> CPU with 18 cores, hyper-threading off, turbo off).
> 
> Below is a summary of the performance we saw on the "grand unified python 
> benchmarks" suite (available at https://hg.python.org/benchmarks/). We made 3 
> rigorous runs of the following benchmarks. In each rigorous run, a benchmark 
> is run 100 times with and without the computed goto patch. Below we show the 
> average performance boost for the 3 rigorous runs.
> 
> Python 2.7.10 (original) vs Computed Goto performance
> Benchmark

-1

As Gregory pointed out, there are other options to build the interpreter, and we
are missing data how these compare with your patch.

I assume, you tested with the Intel compiler, so it would be good to see results
for other compilers as well (GCC, clang).  Please could you provide the data for
LTO and profile guided optimized builds (maybe combined too)?  I'm happy to work
with you on setting up these builds, but currently don't have the machine
resources to do so myself.

If the benefits show up for these configurations too, then I'm +/-0 on this 
patch.

Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Raymond Hettinger]

> On May 28, 2015, at 1:54 AM, Berker Peksağ  wrote:
> 
> * Performance improvements are not bug fixes

Practicality beats purity here.   
Recognize that a huge number of Python users will remain in the Python2.7 world
for some time.  We have a responsibility to the bulk of our users (my estimate 
is
that adoption rate for Python 3 is under 2%).  The computed goto patch makes
substantial performance improvements.  It is callous to deny the improvement
to 2.7 users.


> * The patch doesn't make the migration process from Python 2 to Python 3 
> easier

Sorry, that is a red-herring (an orthogonal issue).
If you care about 2-to-3 migration, then start
opposing proposals for API changes that increase
the semantic difference between 2 and 3.



Raymond

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Parasa, Srinivas Vamsi]

Hi Matthias and Gregory,

The results shown were run on Python 2.7.10 built using gcc. The goal of our 
team is to make long-term open source contributions with emphasis on 
performance optimization and support for the larger community and hence icc 
wasn't used.

We've experimented with gcc profile-guided optimization (PGO) and LTO a month 
ago. PGO being an independent/orthogonal optimization, it shows improvement for 
both the stock version (i.e. current switch based dispatch) and the 
computed-goto version. We ran PGO optimized Python on the workloads available 
at language benchmarks game 
(http://benchmarksgame.alioth.debian.org/u64/python.php) and found that PGO 
benefits computed-goto version more than the stock version. I haven't run PGO 
optimized Python with the "grand unified python benchmarks" (GUPB) suite 
...please give me a day or two and will get back to you with PGO (and LTO) 
numbers as well. (LTO hasn't shown much benefit so far on the language 
benchmarks game workloads).

Also, in our analysis using CPU performance counters, we found that python 
workloads (in general) have higher CPU front-end issues (mainly I-cache misses) 
and PGO is very helpful in mitigating those issues. We're also investigating 
and working on ways to further reduce those front-end issues and speedup Python 
workloads.

Thanks,
Vamsi

-Original Message-
From: Matthias Klose [mailto:d...@ubuntu.com] 
Sent: Thursday, May 28, 2015 5:01 AM
To: Parasa, Srinivas Vamsi; 'python-dev@python.org'
Subject: Re: [Python-Dev] Computed Goto dispatch for Python 2

On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote:
> Hi All,
> 
> This is Vamsi from Server Scripting Languages Optimization team at Intel 
> Corporation.
> 
> Would like to submit a request to enable the computed goto based dispatch in 
> Python 2.x (which happens to be enabled by default in Python 3 given its 
> performance benefits on a wide range of workloads). We talked about this 
> patch with Guido and he encouraged us to submit a request on Python-dev 
> (email conversation with Guido shown at the bottom of this email).
> 
> Attached is the computed goto patch (along with instructions to run) for 
> Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin  at 
> http://bugs.python.org/issue4753). We built and tested this patch for Python 
> 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP 
> CPU with 18 cores, hyper-threading off, turbo off).
> 
> Below is a summary of the performance we saw on the "grand unified python 
> benchmarks" suite (available at https://hg.python.org/benchmarks/). We made 3 
> rigorous runs of the following benchmarks. In each rigorous run, a benchmark 
> is run 100 times with and without the computed goto patch. Below we show the 
> average performance boost for the 3 rigorous runs.
> 
> Python 2.7.10 (original) vs Computed Goto performance Benchmark

-1

As Gregory pointed out, there are other options to build the interpreter, and 
we are missing data how these compare with your patch.

I assume, you tested with the Intel compiler, so it would be good to see 
results for other compilers as well (GCC, clang).  Please could you provide the 
data for LTO and profile guided optimized builds (maybe combined too)?  I'm 
happy to work with you on setting up these builds, but currently don't have the 
machine resources to do so myself.

If the benefits show up for these configurations too, then I'm +/-0 on this 
patch.

Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Nick Coghlan]

On 28 May 2015 at 23:37, Nick Coghlan  wrote:
> On 28 May 2015 at 22:00, Matthias Klose  wrote:
>> On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote:
>>> Hi All,
>>>
>>> This is Vamsi from Server Scripting Languages Optimization team at Intel 
>>> Corporation.
>>>
>>> Would like to submit a request to enable the computed goto based dispatch 
>>> in Python 2.x (which happens to be enabled by default in Python 3 given its 
>>> performance benefits on a wide range of workloads). We talked about this 
>>> patch with Guido and he encouraged us to submit a request on Python-dev 
>>> (email conversation with Guido shown at the bottom of this email).
>>>
>>> Attached is the computed goto patch (along with instructions to run) for 
>>> Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin  at 
>>> http://bugs.python.org/issue4753). We built and tested this patch for 
>>> Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - 
>>> Haswell EP CPU with 18 cores, hyper-threading off, turbo off).
>>>
>>> Below is a summary of the performance we saw on the "grand unified python 
>>> benchmarks" suite (available at https://hg.python.org/benchmarks/). We made 
>>> 3 rigorous runs of the following benchmarks. In each rigorous run, a 
>>> benchmark is run 100 times with and without the computed goto patch. Below 
>>> we show the average performance boost for the 3 rigorous runs.
>>>
>>> Python 2.7.10 (original) vs Computed Goto performance
>>> Benchmark
>>
>> -1
>>
>> As Gregory pointed out, there are other options to build the interpreter, 
>> and we
>> are missing data how these compare with your patch.
>
> That's shifting the goal posts: suggesting that we should optimise
> Python 2 differently from the way we optimised Python 3 isn't a
> reasonable request to make in the context of a backporting proposal.

Sorry, I misread your email. I thought you were talking about the part
of Greg's email where he suggested different optimisation techniques,
but you were actually referring to the part where he requested more
detail on the compiler used and the number for gcc and clang.

*That* request I agree with.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Nick Coghlan]

On 28 May 2015 at 22:00, Matthias Klose  wrote:
> On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote:
>> Hi All,
>>
>> This is Vamsi from Server Scripting Languages Optimization team at Intel 
>> Corporation.
>>
>> Would like to submit a request to enable the computed goto based dispatch in 
>> Python 2.x (which happens to be enabled by default in Python 3 given its 
>> performance benefits on a wide range of workloads). We talked about this 
>> patch with Guido and he encouraged us to submit a request on Python-dev 
>> (email conversation with Guido shown at the bottom of this email).
>>
>> Attached is the computed goto patch (along with instructions to run) for 
>> Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin  at 
>> http://bugs.python.org/issue4753). We built and tested this patch for Python 
>> 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP 
>> CPU with 18 cores, hyper-threading off, turbo off).
>>
>> Below is a summary of the performance we saw on the "grand unified python 
>> benchmarks" suite (available at https://hg.python.org/benchmarks/). We made 
>> 3 rigorous runs of the following benchmarks. In each rigorous run, a 
>> benchmark is run 100 times with and without the computed goto patch. Below 
>> we show the average performance boost for the 3 rigorous runs.
>>
>> Python 2.7.10 (original) vs Computed Goto performance
>> Benchmark
>
> -1
>
> As Gregory pointed out, there are other options to build the interpreter, and 
> we
> are missing data how these compare with your patch.

That's shifting the goal posts: suggesting that we should optimise
Python 2 differently from the way we optimised Python 3 isn't a
reasonable request to make in the context of a backporting proposal.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Nick Coghlan]

On 28 May 2015 at 21:55, Maciej Fijalkowski  wrote:
>> I'm -1 on the idea because:
>>
>> * Performance improvements are not bug fixes
>> * The patch doesn't make the migration process from Python 2 to Python 3 
>> easier
>
> And this is why people have been porting Python applications to Go.

For folks hitting the kinds of scalability problems that Go is
designed to help with, a few percentage points here and there in
CPython performance aren't going to make a big difference - they'll
need the kinds of speed multipliers that PyPy can offer.

Given that Go can't run Python C extensions any more than PyPy can,
and involves a rewrite in a different programming language to boot,
we'd do well to ponder what Go currently offers that PyPy doesn't. If
we ignore the fashion-driven aspect of "Google wrote it, so it must be
cool" (which we can't do anything about), and if we ignore the
multi-vendor commercial support question (which tends to significantly
lag community adoption for true community driven projects like PyPy),
then one of the big keys in my view is the easy redistributability of
Go binaries.

For Linux based network services (and even Windows these days), Docker
containers offer a potentially compelling way of bundling the PyPy
runtime with Python applications, and Docker, Inc already maintain a
set of PyPy base images at https://registry.hub.docker.com/_/pypy/

Docker's image layering model then means that applications sharing a
PyPy runtime shouldn't need to download the interpreter runtime itself
more than once.

As a result, I personally suspect that better documenting and
promoting the CPython->Docker+PyPy migration option is likely to offer
a more effective alternative to CPython->Go migrations than the more
modest performance improvements we can make to the CPython runtime
itself. (I still think the latter are a good idea, though - there's no
point leaving Python 2.7 slower than it needs to be given the offer of
assistance in maintaining it)

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Usability of the limited API

[Paul Moore]

With Python 3.5 shipping an embeddable copy of the interpreter on
Windows, I thought I'd try out a simple embedded interpreter as an
experiment. I wanted to use the limited API, as I'd rather it were
easy to upgrade the interpreter without recompiling the embedding app.

But the "Very high-level embedding" example in the docs doesn't
compile with the limited API.

#include 

int
main(int argc, char *argv[])
{
wchar_t *program = Py_DecodeLocale(argv[0], NULL);
if (program == NULL) {
fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
exit(1);
}
Py_SetProgramName(program);  /* optional but recommended */
Py_Initialize();
PyRun_SimpleString("from time import time,ctime\n"
   "print('Today is', ctime(time()))\n");
Py_Finalize();
PyMem_RawFree(program);
return 0;
}

The Py_DecodeLocale/Py_SetProgramName/PyMem_RawFree bit can probably
be replaced by a Py_SetProgramName call specifying a static value,
it's not exactly crucial. (Py_DecodeLocale appears to be defined as in
the limited API by the headers, but not exported from python3.dll, by
the way, which implies that something's out of sync).

But PyRun_SimpleString doesn't appear to be exposed in the limited
API, even though https://docs.python.org/3/c-api/veryhigh.html doesn't
mention this, and https://docs.python.org/3/c-api/stable.html says
that functions not part of the stable API will be marked as such.

I dumped out the exported symbols from python3.dll, which is the
simplest way I could think of finding out what is in the limited API
(it's hardly user friendly, but never mind). And frustratingly, none
of the very high level PyRun_XXX APIs are available.

At this point, I think I'll probably just give up and use the full
API, but it does make me question whether the limited API is actually
usable as it stands.

I was hoping to be able to suggest as an application bundling option
that people could write a trivial wrapper script in C to fire up a
Python script, and bundle that along with its dependencies and the
embeddable Python distribution. Looks like that's doable, but only
using the full API, which makes upgrading the bundled Python
interpreter a bit messier. Ah, well, no huge loss :-(

But after this experiment, I do wonder - is the limited API really a
viable option for embedders?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Usability of the limited API

[Nick Coghlan]

On 29 May 2015 at 00:11, Paul Moore  wrote:
> I was hoping to be able to suggest as an application bundling option
> that people could write a trivial wrapper script in C to fire up a
> Python script, and bundle that along with its dependencies and the
> embeddable Python distribution. Looks like that's doable, but only
> using the full API, which makes upgrading the bundled Python
> interpreter a bit messier. Ah, well, no huge loss :-(
>
> But after this experiment, I do wonder - is the limited API really a
> viable option for embedders?

I personally suspect the requirement to preserve source compatibility
with Python 2 has meant that the limited ABI hasn't been exercised
very well to date.

As far as the high level embedding API goes, I expect it's just an
oversight that it's missing from the stable ABI. There are some that
*can't* be exposed (the ones that rely on FILE pointers), but the
others should be OK.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Brett Cannon]

On Thu, May 28, 2015 at 8:47 AM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

>
> > On May 28, 2015, at 1:54 AM, Berker Peksağ 
> wrote:
> >
> > * Performance improvements are not bug fixes
>
> Practicality beats purity here.

Recognize that a huge number of Python users will remain in the Python2.7
> world
> for some time.  We have a responsibility to the bulk of our users (my
> estimate is
> that adoption rate for Python 3 is under 2%).


Where does that number come from? I have not seen numbers less than 5% for
the overall community adoption (and all of them are extremely rough and all
skewed towards Python 2 for various technical reasons).


>   The computed goto patch makes
> substantial performance improvements.  It is callous to deny the
> improvement
> to 2.7 users.
>

But you could argue that "Special cases aren't special enough to break the
rules" and that's what we are proposing here by claiming Python 2.7 is a
special case and thus we should accept a patch that is not a one-liner
change to gain some performance in a bugfix release.


>
>
> > * The patch doesn't make the migration process from Python 2 to Python 3
> easier
>
> Sorry, that is a red-herring (an orthogonal issue).
> If you care about 2-to-3 migration, then start
> opposing proposals for API changes that increase
> the semantic difference between 2 and 3.
>

That's misdirection for Berker's point that the proposal at hand does not
help with getting to people to Python 3 even though what is proposed is an
enhancement and not a bugfix (since we are not fixing a performance
regression). I had someone on Twitter earlier this month point out that
Python 3 was slower than Python 2 on some benchmark and that was a reason
they weren't switching. Doing this is not going to help make that case
(although I think arguing about performance between 2 and 3 is misleading
when I've seen other workloads win out in Python 3).

I'm -0 on this because I would like to stick to our policy of no
enhancements in a bugfix release, but in the end it's Benjamin's call as
2.7 RM as to whether this is appropriate for 2.7.11.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Donald Stufft]

On May 28, 2015 at 10:10:03 AM, Nick Coghlan (ncogh...@gmail.com) wrote:
> On 28 May 2015 at 21:55, Maciej Fijalkowski wrote:
> >> I'm -1 on the idea because:
> >>
> >> * Performance improvements are not bug fixes
> >> * The patch doesn't make the migration process from Python 2 to Python 3 
> >> easier
> >
> > And this is why people have been porting Python applications to Go.
>  
> For folks hitting the kinds of scalability problems that Go is
> designed to help with, a few percentage points here and there in
> CPython performance aren't going to make a big difference - they'll
> need the kinds of speed multipliers that PyPy can offer.
>  
> Given that Go can't run Python C extensions any more than PyPy can,
> and involves a rewrite in a different programming language to boot,
> we'd do well to ponder what Go currently offers that PyPy doesn't. If
> we ignore the fashion-driven aspect of "Google wrote it, so it must be
> cool" (which we can't do anything about), and if we ignore the
> multi-vendor commercial support question (which tends to significantly
> lag community adoption for true community driven projects like PyPy),
> then one of the big keys in my view is the easy redistributability of
> Go binaries.
>  
> For Linux based network services (and even Windows these days), Docker
> containers offer a potentially compelling way of bundling the PyPy
> runtime with Python applications, and Docker, Inc already maintain a
> set of PyPy base images at https://registry.hub.docker.com/_/pypy/
>  
> Docker's image layering model then means that applications sharing a
> PyPy runtime shouldn't need to download the interpreter runtime itself
> more than once.
>  
> As a result, I personally suspect that better documenting and
> promoting the CPython->Docker+PyPy migration option is likely to offer
> a more effective alternative to CPython->Go migrations than the more
> modest performance improvements we can make to the CPython runtime
> itself. (I still think the latter are a good idea, though - there's no
> point leaving Python 2.7 slower than it needs to be given the offer of
> assistance in maintaining it)
>  
> Regards,
> Nick.
>  
> --
> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io  
>  


I think docker is a pretty crummy answer to Go’s static binaries. What I would
love is for Python to get:

* The ability to import .so modules via zipzimport (ideally without a temporary
  directory, but that might require newer APIs from libc and such).
* The ability to create a “static” Python that links everything it needs into
  the binary to do a zipimport of everything else (including the stdlib).
* The ability to execute a zipfile that has been concat onto the end of the
  Python binary.

I think that if we get all of that, we could easily create a single file 
executable
with real, native support from Python by simply compiling Python in that static
mode and then appending a zip file containing the standard library and any other
distributions we need to the end of it.

We’d probably want some more quality of life improvements around accessing 
resources
from within that zip file as well, but that can be done as a library easier than
the above three things can.

---  
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Skip Montanaro]

On Thu, May 28, 2015 at 9:02 AM, Brett Cannon  wrote:
> But you could argue that "Special cases aren't special enough to break the
> rules" and that's what we are proposing here by claiming Python 2.7 is a
> special case and thus we should accept a patch that is not a one-liner
> change to gain some performance in a bugfix release.

One can read anything he wants into the Zen. I could respond with this
retort: "Although practicality beats purity," but I won't. :-)

Skip
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Paul Moore]

On 28 May 2015 at 15:37, Donald Stufft  wrote:
> I think docker is a pretty crummy answer to Go’s static binaries. What I would
> love is for Python to get:
>
> * The ability to import .so modules via zipzimport (ideally without a 
> temporary
>   directory, but that might require newer APIs from libc and such).
> * The ability to create a “static” Python that links everything it needs into
>   the binary to do a zipimport of everything else (including the stdlib).
> * The ability to execute a zipfile that has been concat onto the end of the
>   Python binary.
>
> I think that if we get all of that, we could easily create a single file 
> executable
> with real, native support from Python by simply compiling Python in that 
> static
> mode and then appending a zip file containing the standard library and any 
> other
> distributions we need to the end of it.
>
> We’d probably want some more quality of life improvements around accessing 
> resources
> from within that zip file as well, but that can be done as a library easier 
> than
> the above three things can.

+1. The new embeddable Python distribution for Windows is a great step
forward for this. It's not single-file, but it's easy to produce a
single-directory self-contained application with it. I don't know if
there's anything equivalent for Linux/OSX - maybe it's something we
should look at for them as well (although the whole "static binaries"
concept seems to be fairly frowned on in the Unix world, from what
I've seen).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Usability of the limited API

[Paul Moore]

On 28 May 2015 at 15:28, Steve Dower  wrote:
> I don't have the issue number handy, but it should be near the top of the
> recently modified list.

I recall seeing that issue. I'm fine with that - getting the two in
sync is obviously worth doing (and clearly in hand). I'm personally
not sure whether automating the exposure of symbols is the correct
approach, as I'm not sure people typically even consider the stable
API when adding functions. Is the default (what you get if somebody
just blindly adds a symbol with no thought for the stable API) to
expose it or not? If the default is that it's not exposed, then
automation seems reasonable, otherwise I'm not so sure.

The bigger issue for me is that it looks like the stable API doesn't
include functions that allow you to just run a script/file.

At a minimum, PyRun_SimpleString should be available. I'd also like to
see a variant of PyRun_SimpleFile that let you pass a filename (either
a .py file, or a zipfile, or a directory name - basically what you can
pass to the Python interpreter directly). You can sort of do it via
"import runpy; runpy.run_path(filename)", but you get into all sorts
of fun with requoting filenames, etc.

With the fact that we're distributing an embeddable interpreter for
Windows, I'd like to be able to promote it as a bundling option,
easier to use than things like py2exe/cx_Freeze.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Brett Cannon]

On Thu, May 28, 2015 at 10:47 AM Skip Montanaro 
wrote:

> On Thu, May 28, 2015 at 9:02 AM, Brett Cannon  wrote:
> > But you could argue that "Special cases aren't special enough to break
> the
> > rules" and that's what we are proposing here by claiming Python 2.7 is a
> > special case and thus we should accept a patch that is not a one-liner
> > change to gain some performance in a bugfix release.
>
> One can read anything he wants into the Zen. I could respond with this
> retort: "Although practicality beats purity," but I won't. :-)
>

Because Raymond already said that. =) And you explicitly made the point I
was trying to implicitly make: in this instance there is no clear argument
for or against that is going to make this an obvious decision based on PEP
20.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Usability of the limited API

[Steve Dower]

Paul Moore wrote:
> On 28 May 2015 at 15:28, Steve Dower  wrote:
>> I don't have the issue number handy, but it should be near the top of
>> the recently modified list.
> 
> I recall seeing that issue. I'm fine with that - getting the two in sync is
> obviously worth doing (and clearly in hand). I'm personally not sure whether
> automating the exposure of symbols is the correct approach, as I'm not sure
> people typically even consider the stable API when adding functions. Is the
> default (what you get if somebody just blindly adds a symbol with no thought 
> for
> the stable API) to expose it or not? If the default is that it's not exposed,
> then automation seems reasonable, otherwise I'm not so sure.

Now I'm at my desk, the issue is http://bugs.python.org/issue23903

I believe new symbols are considered stable by default, so perhaps we actually 
want a test that will generate a C file that imports everything "stable" and 
will break the buildbots if someone adds something new without explicitly 
adding it to the list of stable functions? That sounds like the only way to 
make the extra overhead of two lists work. I guess we could also invert all the 
logic in the header files so that symbols are unstable by default.

> The bigger issue for me is that it looks like the stable API doesn't include
> functions that allow you to just run a script/file.

I think a combination of Py_CompileString and PyEval_EvalCode is what you want 
here, though I can't think of any good reason not to have a stable helper 
method (or even a macro?) for this.

> At a minimum, PyRun_SimpleString should be available. I'd also like to see a
> variant of PyRun_SimpleFile that let you pass a filename (either a .py file, 
> or
> a zipfile, or a directory name - basically what you can pass to the Python
> interpreter directly). You can sort of do it via "import runpy;
> runpy.run_path(filename)", but you get into all sorts of fun with requoting
> filenames, etc.
> 
> With the fact that we're distributing an embeddable interpreter for Windows, 
> I'd
> like to be able to promote it as a bundling option, easier to use than things
> like py2exe/cx_Freeze.

Agreed. This is the point of the zip file :)

Cheers,
Steve

> Paul

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Donald Stufft]

On May 28, 2015 at 10:55:08 AM, Steve Dower (steve.do...@microsoft.com) wrote:
> Donald Stufft wrote:
> > I think docker is a pretty crummy answer to Go’s static binaries. What I 
> > would
> > love is for Python to get:
> >
> > * The ability to import .so modules via zipzimport (ideally without a 
> > temporary
> > directory, but that might require newer APIs from libc and such).
> > * The ability to create a “static” Python that links everything it needs 
> > into
> > the binary to do a zipimport of everything else (including the stdlib).
> > * The ability to execute a zipfile that has been concat onto the end of the
> > Python binary.
> >
> > I think that if we get all of that, we could easily create a single file
> > executable with real, native support from Python by simply compiling Python 
> > in
> > that static mode and then appending a zip file containing the standard 
> > library
> > and any other distributions we need to the end of it.
>  
> And it would look like a 20MB+ file just for a simple 1KB Python script...
>  
> For Windows at least, I'd prefer to have some app-style installer generation 
> (e.g. http://pynsist.readthedocs.org/en/latest/)  
> which, combined with the embeddable Python distro (new for 3.5.0b1 in case 
> anyone missed  
> it), can simply extract everything into an install directory and run it from 
> there. None  
> of the items on the list above are needed for or would help with this.
>  
> (Some other Windows-specific advantages of the latter - installers get 
> special compatibility  
> treatment when the OS does stuff to break them, modifying official Python 
> binaries breaks  
> the signatures, signed executables are fully scanned before running (slow if 
> the file  
> is big), IT departments know how to deal with installers and users know how 
> to deal with  
> installed binaries, and probably more.)
>  
> Alright everyone, back on topic now unless you want to rename the thread :)
>  
> Cheers,
> Steve
>  
> 

Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python
script + whatever other pieces you need. It would be entirely possible to only
include the parts of the standard library you actually need. There's no rule
that if your single file executable doesn't use xmlrpc that you have to include
xmlrpc just for purities sake.

This isn't something that can't be done today using something like PyInstaller,
it's just super janky and finnicky because it's being hacked in after the fact
by PyInstaller and because it's not an officially supported thing a lot of
projects simply don't support it. A CLI I worked on that uses PyInstaller is
5MB. It's certainly a trade off but it's not nearly as big of a trade off as
you say.

---  
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Mark Lawrence]

On 28/05/2015 15:47, Skip Montanaro wrote:

On Thu, May 28, 2015 at 9:02 AM, Brett Cannon  wrote:

But you could argue that "Special cases aren't special enough to break the
rules" and that's what we are proposing here by claiming Python 2.7 is a
special case and thus we should accept a patch that is not a one-liner
change to gain some performance in a bugfix release.


One can read anything he wants into the Zen. I could respond with this
retort: "Although practicality beats purity," but I won't. :-)

Skip



That's good, otherwise you'd just be repeating what Raymond said further 
up this subthread two hours and one minute before you didn't say it :)


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Donald Stufft]

On May 28, 2015 at 11:30:37 AM, Steve Dower (steve.do...@microsoft.com) wrote:
> Donald Stufft wrote:
> > Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB 
> > Python script + whatever  
> other pieces you need.
>  
> For contrast, here are the things you need on Windows to be able to get to an 
> interactive  
> prompt (I don't know how other platforms get this down to 4KB...):
>  
> * python.exe (or some equivalent launcher) 39KB
> * python35.dll 3,788KB
> * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not 
> redistributable  
> so doesn't count here)
> * 26 files in Lib 343KB
>  
> This gets you to ">>>", and basically everything after that is going to fail 
> for some reason.  
> That's an unavoidable 4,257KB.
>  
> The rest of the stdlib adds another ~16MB once you exclude the test suite, so 
> a fully functioning  
> Python is not cheap. (Using compressed .pyc's in a zip file can make a big 
> difference here  
> though, assuming you're willing to trade CPU for HDD.)
>  
> Cheers,
> Steve
>  
>  

You don’t need a "fully functioning Python" for a single file binary, you only
need enough to actually run your application. For example, if you're making
an application that can download files over HTTP, you don't need to include
parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email,
mailcap, mailbox, imaplib, nntplib, etc.

Of course deciding which pieces you include in the zip file you're appending
to the end of Python is up to whatever tool builds this executable which
doesn't need to be part of Python itself. If Python itself gained the ability
to operate in that manner than third party tools could handle trying to do the
optimizations where it only includes the things it actually needs in the stdlib
and excludes things it doesn't. The key thing here is that since you're doing
a single file binary, you don't need to have a Python which is suitable to
execute random Python code, you only need one that is suitable to execute this
particular code so you can specialize what that includes.

---  
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Barry Warsaw]

On May 28, 2015, at 11:39 AM, Donald Stufft wrote:

>You don’t need a "fully functioning Python" for a single file binary, you
>only need enough to actually run your application. For example, if you're
>making an application that can download files over HTTP, you don't need to
>include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite,
>csv, email, mailcap, mailbox, imaplib, nntplib, etc.

There are actually two related but different use cases to "single file
executables".

The first is nicely solved by tools like pex, where you don't need to include
a fully functional Python at the head of the zip file because the environment
you're deploying it into will have enough Python to make the zip work.  This
can certainly result in smaller zip files.  This is the approach I took with
Snappy Ubuntu Core support for Python 3, based on the current situation that
the atomic upgrade client is written in Python 3.  If that changes and Python
3 is removed from the image, then this approach won't work.

pex (and others) does a great job at this, so unless there are things better
refactored into upstream Python, I don't think we need to do much here.

The second use case is as you describe: put a complete functional Python
environment at the head of the zip file so you don't need anything in the
target deployment environment.  "Complete" can easily mean the entire stdlib,
and although that would usually be more bloated than you normally need, hey,
it's just some extra unused bits so who cares? .  I think this would be
an excellent starting point which can be optimized to trim unnecessary bits
later, maybe by third party tools.

>Of course deciding which pieces you include in the zip file you're appending
>to the end of Python is up to whatever tool builds this executable which
>doesn't need to be part of Python itself. If Python itself gained the ability
>to operate in that manner than third party tools could handle trying to do
>the optimizations where it only includes the things it actually needs in the
>stdlib and excludes things it doesn't. The key thing here is that since
>you're doing a single file binary, you don't need to have a Python which is
>suitable to execute random Python code, you only need one that is suitable to
>execute this particular code so you can specialize what that includes.

I'd love to see Python itself gain such a tool, but if it had the critical
pieces to execute in this way, that would enable a common approach to
supporting this in third party tools, on a variety of platforms.

I do think single-file executables are an important piece to Python's
long-term competitiveness.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Guido van Rossum]

Wow. Such thread. :-)

This patch could save companies like Dropbox a lot of money. We run a ton
of Python code in large datacenters, and while we are slow in moving to
Python 3, we're good at updating to the latest 2.7.

The patch is forward and backward compatible.I'm strongly in favor.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Barry Warsaw]

Go seems to be popular where I work.  It is replacing Python in a number of
places, although Python (and especially Python 3) is still a very important
part of our language toolbox.

There are several reasons why Go is gaining popularity.  Single-file
executables is definitely a reason; it makes deployment very easy, even if it
increases the maintenance burden (e.g. without shared libraries, you have
multiple copies of things so when a security fix is required for one of those
things you have to recompile the world).

Start up times and memory footprint are also factors.  Probably not much to be
done about the latter, but perhaps PEP 432 can lead to improvements in the
former.  (Hey Nick, I'm guessing you'll want to bump that one back to 3.6.)

Certainly better support for multi-cores comes up a lot.  It should be a SMoE
to just get rid of the GIL once and for all .

One thing I've seen more than once is that new development happens in Python
until the problem is understood, then the code is ported to Go.  Python's
short path from idea to working code, along with its ability to quickly morph
as requirements and understanding changes, its batteries included philosophy,
and its "fits-your-brain" consistency are its biggest strengths!

On May 28, 2015, at 10:37 AM, Donald Stufft wrote:

>I think docker is a pretty crummy answer to Go’s static binaries. What I would
>love is for Python to get:
>
>* The ability to import .so modules via zipzimport (ideally without a
>temporary   directory, but that might require newer APIs from libc and such).

+1 - Thomas Wouters mentioned at the language summit some work being done on
glibc to add dlopen_from_memory() (sp?) which would allow for loading .so
files directly from a zip.  Not sure what the status is of that, but it would
be a great addition.

>* The ability to create a “static” Python that links everything it needs into
>the binary to do a zipimport of everything else (including the stdlib).

+1

>*The ability to execute a zipfile that has been concat onto the end of the  
>Python binary.

+1

>I think that if we get all of that, we could easily create a single file
>executable with real, native support from Python by simply compiling Python
>in that static mode and then appending a zip file containing the standard
>library and any other distributions we need to the end of it.
>
>We’d probably want some more quality of life improvements around accessing
>resources from within that zip file as well, but that can be done as a
>library easier than the above three things can.

E.g. you really should be using the pkg_resources APIs for loading resources
from your packages, otherwise you're gonna have problems with zip
executables.  We've talked before about adopting some of these APIs into
Python's stdlib.  pkgutil is a start, and the higher level APIs from
pkg_resources should probably go there.

Cheers,
-Barry

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Steve Dower]

Donald Stufft wrote:
> I think docker is a pretty crummy answer to Go’s static binaries. What I would
> love is for Python to get:
> 
> * The ability to import .so modules via zipzimport (ideally without a 
> temporary
> directory, but that might require newer APIs from libc and such).
> * The ability to create a “static” Python that links everything it needs into
> the binary to do a zipimport of everything else (including the stdlib).
> * The ability to execute a zipfile that has been concat onto the end of the
> Python binary.
> 
> I think that if we get all of that, we could easily create a single file
> executable with real, native support from Python by simply compiling Python in
> that static mode and then appending a zip file containing the standard library
> and any other distributions we need to the end of it.

And it would look like a 20MB+ file just for a simple 1KB Python script...

For Windows at least, I'd prefer to have some app-style installer generation 
(e.g. http://pynsist.readthedocs.org/en/latest/) which, combined with the 
embeddable Python distro (new for 3.5.0b1 in case anyone missed it), can simply 
extract everything into an install directory and run it from there. None of the 
items on the list above are needed for or would help with this.

(Some other Windows-specific advantages of the latter - installers get special 
compatibility treatment when the OS does stuff to break them, modifying 
official Python binaries breaks the signatures, signed executables are fully 
scanned before running (slow if the file is big), IT departments know how to 
deal with installers and users know how to deal with installed binaries, and 
probably more.)

Alright everyone, back on topic now unless you want to rename the thread :)

Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Chris Barker]

On Thu, May 28, 2015 at 9:23 AM, Chris Barker  wrote:

> Barry Warsaw wrote:
> >> I do think single-file executables are an important piece to Python's 
> >> long-term
> competitiveness.
>
> Really? It seems to me that desktop development is dying. What are the
> critical use-cases for a single file executable?
>

oops, sorry -- I see this was addressed in another thread. Though I guess I
still don't see why "single file" is critical, over "single thing to
install" -- like a OS-X app bundle that can just be dragged into the
Applications folder.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Paul Moore]

On 28 May 2015 at 16:58, Barry Warsaw  wrote:
> On May 28, 2015, at 11:39 AM, Donald Stufft wrote:
>
>>You don’t need a "fully functioning Python" for a single file binary, you
>>only need enough to actually run your application. For example, if you're
>>making an application that can download files over HTTP, you don't need to
>>include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite,
>>csv, email, mailcap, mailbox, imaplib, nntplib, etc.
>
> There are actually two related but different use cases to "single file
> executables".
>
> The first is nicely solved by tools like pex, where you don't need to include
> a fully functional Python at the head of the zip file because the environment
> you're deploying it into will have enough Python to make the zip work.  This
> can certainly result in smaller zip files.  This is the approach I took with
> Snappy Ubuntu Core support for Python 3, based on the current situation that
> the atomic upgrade client is written in Python 3.  If that changes and Python
> 3 is removed from the image, then this approach won't work.
>
> pex (and others) does a great job at this, so unless there are things better
> refactored into upstream Python, I don't think we need to do much here.

One problem with pex is that it doesn't appear to work on Windows (I
just gave it a try, and got errors because it relies on symlinks).

IMO, any solution to "distributing Python applications" that is
intended to compete with the idea that "go produces nice single-file
executables" needs to be cross-platform. At the moment, zipapp (and in
general, the core support for running applications from a zip file)
handles this for the case where you're allowed to assume an already
installed Python interpreter. The proviso here, as Donald pointed out,
is that it doesn't handle C extensions.

The biggest problem with 3rd-party solutions is that they don't always
support the full range of platforms that Python supports. That's fine
for a 3rd party tool, but if we want to have a response to people
asking how to bundle their application written in Python, we need a
better answer than "if you're on Windows, use py2exe, or if you're on
Unix use pex, or maybe..."

Python has core support for the equivalent of Java's jar format in
zipapp. It's not well promoted (and doesn't support C extensions) but
it's a pretty viable option for a lot of situations.

> The second use case is as you describe: put a complete functional Python
> environment at the head of the zip file so you don't need anything in the
> target deployment environment.  "Complete" can easily mean the entire stdlib,
> and although that would usually be more bloated than you normally need, hey,
> it's just some extra unused bits so who cares? .  I think this would be
> an excellent starting point which can be optimized to trim unnecessary bits
> later, maybe by third party tools.

Tools like py2exe and cx_Freeze do this, and are pretty commonly used
on Windows. An obvious example of use is Mercurial. If you're looking
at this scenario, a good place to start would probably be
understanding why cx_Freeze isn't more commonly used on Unix (AFAIK,
it supports Unix, but I've only ever really heard of it being used on
Windows).

I suspect "single file executables" just aren't viewed as a desirable
solution on Unix. Although Donald referred to a 4K binary, which
probably means just a stub exe that depends on system-installed .so
files, likely including Python (I'm just guessing here). It's easy to
do something similar on Windows, but it's *not* what most Windows
users think of when you say a "single file executable for a Python
program" (because there's no system package manager doing dependencies
for you).

Again, platform-specific answers are one thing, and are relatively
common, but having a good cross-platform answer at the language level
(a section on docs.python.org "How to ship your Python program") is
much harder.

>>Of course deciding which pieces you include in the zip file you're appending
>>to the end of Python is up to whatever tool builds this executable which
>>doesn't need to be part of Python itself. If Python itself gained the ability
>>to operate in that manner than third party tools could handle trying to do
>>the optimizations where it only includes the things it actually needs in the
>>stdlib and excludes things it doesn't. The key thing here is that since
>>you're doing a single file binary, you don't need to have a Python which is
>>suitable to execute random Python code, you only need one that is suitable to
>>execute this particular code so you can specialize what that includes.
>
> I'd love to see Python itself gain such a tool, but if it had the critical
> pieces to execute in this way, that would enable a common approach to
> supporting this in third party tools, on a variety of platforms.

Stripping out unused code is a hard problem in a language as dynamic
as Python. It would be great to see it happen, but I'm not

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Paul Moore]

On 28 May 2015 at 17:28, Chris Barker  wrote:
> On Thu, May 28, 2015 at 9:23 AM, Chris Barker  wrote:
>>
>> Barry Warsaw wrote:
>> >> I do think single-file executables are an important piece to Python's
>> >> long-term competitiveness.
>>
>> Really? It seems to me that desktop development is dying. What are the
>> critical use-cases for a single file executable?
>
>
> oops, sorry -- I see this was addressed in another thread. Though I guess I
> still don't see why "single file" is critical, over "single thing to
> install" -- like a OS-X app bundle that can just be dragged into the
> Applications folder.

On Windows, there's a strong interest in "no install" download-and-run
applications (.Net has "xcopy deployment", portable applications are a
big deal to some people).

"Just unzip the distribution somewhere and run the exe" is definitely
a selling point for that audience. And "download an exe and just run
it" is even more so. But the latter is definitely an incremental
improvement - the former is the big deal (IMO).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[David Cournapeau]

On Fri, May 29, 2015 at 1:28 AM, Chris Barker  wrote:

> On Thu, May 28, 2015 at 9:23 AM, Chris Barker 
> wrote:
>
>> Barry Warsaw wrote:
>> >> I do think single-file executables are an important piece to Python's 
>> >> long-term
>> competitiveness.
>>
>> Really? It seems to me that desktop development is dying. What are the
>> critical use-cases for a single file executable?
>>
>
> oops, sorry -- I see this was addressed in another thread. Though I guess
> I still don't see why "single file" is critical, over "single thing to
> install" -- like a OS-X app bundle that can just be dragged into the
> Applications folder.
>

It is much simpler to deploy in an automated, recoverable way (and also
much faster), because you can't have parts of the artefact "unsynchronized"
with another part of the program. Note also that moving a python
installation in your fs is actually quite unlikely to work in interesting
usecases on unix because of the relocatability issue.

Another advantage: it makes it impossible for users to tamper an
application's content and be surprised things don't work anymore (a very
common source of issues, familiar to anybody deploying complex python
applications in the "enterprise world").

I recently started using some services written in go, and the single file
approach is definitely a big +. It makes *using* applications written in it
so much easier than python, even though I am complete newbie in go and
relatively comfortable with python.

One should keep in mind that go has some inherent advantages over python in
those contexts even if python were to gain single file distribution
tomorrow. Most of go stdlib is written in go now I believe, and it is much
more portable across linux systems on a given CPU arch compared to python.
IOW, it is more robust against ABI variability.

David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Donald Stufft]

On May 28, 2015 at 12:01:22 PM, Barry Warsaw (ba...@python.org) wrote:
> On May 28, 2015, at 11:39 AM, Donald Stufft wrote:
>  
> >You don’t need a "fully functioning Python" for a single file binary, you
> >only need enough to actually run your application. For example, if you're
> >making an application that can download files over HTTP, you don't need to
> >include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite,
> >csv, email, mailcap, mailbox, imaplib, nntplib, etc.
>  
> There are actually two related but different use cases to "single file
> executables".
>  
> The first is nicely solved by tools like pex, where you don't need to include
> a fully functional Python at the head of the zip file because the environment
> you're deploying it into will have enough Python to make the zip work. This
> can certainly result in smaller zip files. This is the approach I took with
> Snappy Ubuntu Core support for Python 3, based on the current situation that
> the atomic upgrade client is written in Python 3. If that changes and Python
> 3 is removed from the image, then this approach won't work.
>  
> pex (and others) does a great job at this, so unless there are things better
> refactored into upstream Python, I don't think we need to do much here.

Pex would be improved by having native support for importing .so’s from within
a zipfile via zipimport. It would also be improved by having good, built in
support for extraneous resources in the stdlib too. It’s doing pretty well on
it’s own though besides those quality of life improvements.

>  
> The second use case is as you describe: put a complete functional Python
> environment at the head of the zip file so you don't need anything in the
> target deployment environment. "Complete" can easily mean the entire stdlib,
> and although that would usually be more bloated than you normally need, hey,
> it's just some extra unused bits so who cares? . I think this would be
> an excellent starting point which can be optimized to trim unnecessary bits
> later, maybe by third party tools.
>  
> >Of course deciding which pieces you include in the zip file you're appending
> >to the end of Python is up to whatever tool builds this executable which
> >doesn't need to be part of Python itself. If Python itself gained the ability
> >to operate in that manner than third party tools could handle trying to do
> >the optimizations where it only includes the things it actually needs in the
> >stdlib and excludes things it doesn't. The key thing here is that since
> >you're doing a single file binary, you don't need to have a Python which is
> >suitable to execute random Python code, you only need one that is suitable to
> >execute this particular code so you can specialize what that includes.
>  
> I'd love to see Python itself gain such a tool, but if it had the critical
> pieces to execute in this way, that would enable a common approach to
> supporting this in third party tools, on a variety of platforms.

Right, it would be great to get it built into Python itself, but I consider that
less important than getting the critical pieces into Python. If those pieces are
there then we can iterate outside of the standard library and try different
approaches to *building* such a file, and eventually take a look at the 
landscape
and bless one approach (or not, if we don’t want to).

>  
> I do think single-file executables are an important piece to Python's
> long-term competitiveness.
>  

I completely agree. I talk to a lot of people about packaging of things, and 
while
I think there are some serious problems with huge parts of Go’s packaging and
distribution story the static linking and compiling down to a “single” file is 
not
one of them. People *really* enjoy this and it simplifies a ton of things for 
people.
They don’t have to worry about making sure a whole set of files are deployed, 
they
don’t have to worry about what version of Python is installed (or if any 
version is
installed), they don’t have to worry about what other things have been 
installed, they
just copy an executable and then they are good to go.

I think we could potentially do *better* than Go in this regard, because we can 
make
it possible to do both “static” and “dynamic” dependencies, as well as provide 
the
ecosystem around that to do things like provide visibility into what versions of
libraries are “compiled” into that single file executable, etc.

---  
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Guido van Rossum]

Single-file binaries are indeed important. (Though in most cases they don't
have to be totally stand-alone -- they can depend on a system python and
its stdlib. At least in typical datacenter setups.) Have people looked at
PEX (a format developed by Twitter) or Pants (which seems to be an
open-source tool that can build PEX files)?

Łukasz has told me that at Facebook they have a similar system that they
are now using to deploy Python 3 binaries.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Chris Barker]

I'm confused:

Doesn't py2exe (optionally) create a single file executable?

And py2app on the Mac creates an application bundle, but that is
more-or-less the equivalent on OS-X (you may not even be able to have a
single file executable that can access the Window Manager, for instance)

Depending on what extra packages you need, py2exe's single file doesn't
always work, but last I tried, it worked for a fair bit (I think all of the
stdlib).

I don't know what PyInstaller or others create. And I have no idea if there
is a linux option -- but it seems like the standard of practice for an
application for linux is a bunch of files scattered over the system anyway
:-)

Yes, the resulting exe is pretty big, but it does try to include only those
modules and packages that are used, and that kind of optimization could be
improved in any case.

So is something different being asked for here?

Barry Warsaw wrote:
>> I do think single-file executables are an important piece to Python's 
>> long-term
competitiveness.

Really? It seems to me that desktop development is dying. What are the
critical use-cases for a single file executable?

And I'd note that getting a good way to use Python to develop for iOS,
Android, and Mobile Windows is FAR more critical!  -- maybe that's the same
problem ?

-Chris


On Thu, May 28, 2015 at 8:39 AM, Donald Stufft  wrote:

>
>
> On May 28, 2015 at 11:30:37 AM, Steve Dower (steve.do...@microsoft.com)
> wrote:
> > Donald Stufft wrote:
> > > Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB
> Python script + whatever
> > other pieces you need.
> >
> > For contrast, here are the things you need on Windows to be able to get
> to an interactive
> > prompt (I don't know how other platforms get this down to 4KB...):
> >
> > * python.exe (or some equivalent launcher) 39KB
> > * python35.dll 3,788KB
> > * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not
> redistributable
> > so doesn't count here)
> > * 26 files in Lib 343KB
> >
> > This gets you to ">>>", and basically everything after that is going to
> fail for some reason.
> > That's an unavoidable 4,257KB.
> >
> > The rest of the stdlib adds another ~16MB once you exclude the test
> suite, so a fully functioning
> > Python is not cheap. (Using compressed .pyc's in a zip file can make a
> big difference here
> > though, assuming you're willing to trade CPU for HDD.)
> >
> > Cheers,
> > Steve
> >
> >
>
> You don’t need a "fully functioning Python" for a single file binary, you
> only
> need enough to actually run your application. For example, if you're making
> an application that can download files over HTTP, you don't need to include
> parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv,
> email,
> mailcap, mailbox, imaplib, nntplib, etc.
>
> Of course deciding which pieces you include in the zip file you're
> appending
> to the end of Python is up to whatever tool builds this executable which
> doesn't need to be part of Python itself. If Python itself gained the
> ability
> to operate in that manner than third party tools could handle trying to do
> the
> optimizations where it only includes the things it actually needs in the
> stdlib
> and excludes things it doesn't. The key thing here is that since you're
> doing
> a single file binary, you don't need to have a Python which is suitable to
> execute random Python code, you only need one that is suitable to execute
> this
> particular code so you can specialize what that includes.
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Chris Angelico]

On Fri, May 29, 2015 at 2:28 AM, Chris Barker  wrote:
> oops, sorry -- I see this was addressed in another thread. Though I guess I
> still don't see why "single file" is critical, over "single thing to
> install" -- like a OS-X app bundle that can just be dragged into the
> Applications folder.

There's also "single thing to uninstall", which IMO is more important.
If I download a tiny program that's supposed to just do one tiny
thing, and it has to install itself into Program Files, Common Files,
Windows\System32, and Documents & Settings\my-user-name\Applications,
then I have to hope it has a proper uninstaller. If it's a single
executable that just does its stuff (or, failing that, a single zip
file that I extract to anywhere and run the program), I can expect
that deleting that file (or directory) will get rid of it all. Of
course, it's entirely possible that it's gone and left its droppings
all over the system, but that's a matter of trust - a legit program
won't lie about that.

Is this a Windows-specific issue, or is it also intended for Linux and
Mac OS, where there'll already be a system Python (so a
single-file-executable would be used to be independent of the system
Python)?

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Donald Stufft]

On May 28, 2015 at 12:24:42 PM, Chris Barker (chris.bar...@noaa.gov) wrote:
> I'm confused:
>  
> Doesn't py2exe (optionally) create a single file executable?
>  
> And py2app on the Mac creates an application bundle, but that is
> more-or-less the equivalent on OS-X (you may not even be able to have a
> single file executable that can access the Window Manager, for instance)
>  
> Depending on what extra packages you need, py2exe's single file doesn't
> always work, but last I tried, it worked for a fair bit (I think all of the
> stdlib).
>  
> I don't know what PyInstaller or others create. And I have no idea if there
> is a linux option -- but it seems like the standard of practice for an
> application for linux is a bunch of files scattered over the system anyway
> :-)
>  
> Yes, the resulting exe is pretty big, but it does try to include only those
> modules and packages that are used, and that kind of optimization could be
> improved in any case.
>  
> So is something different being asked for here?

All of those solutions “work” to varying degrees of work, almost all of them 
rely
on hacks in order to make things “work” because the ability to do it isn’t built
into Python itself. If the critical pieces to execute in this way was built into
Python itself, then those tools would work a whole lot better than they 
currently
do.

>  
> Barry Warsaw wrote:
> >> I do think single-file executables are an important piece to Python's 
> >> long-term
> competitiveness.
>  
> Really? It seems to me that desktop development is dying. What are the
> critical use-cases for a single file executable?

The desktop isn’t dying, Mobile is becoming a very important thing of course,
but that’s just because people are using devices *more* to account for the
use of Mobile, they aren’t really using their Desktop’s less.

See: 
http://blogs.wsj.com/cmo/2015/05/26/mobile-isnt-killing-the-desktop-internet/

>  
> And I'd note that getting a good way to use Python to develop for iOS,
> Android, and Mobile Windows is FAR more critical! -- maybe that's the same
> problem ?
>  

It’s not the same problem, but it’s also not very relevant. Volunteer time isn’t
fungible, you get what people are willing to work on regardless of whether it
will help Python as a whole. It’s also not an either/or proposition, we can both
improve our ability to develop under iOS/Android/etc and improve our ability to
handle desktop applications.

---  
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Paul Moore]

On 28 May 2015 at 17:13, Barry Warsaw  wrote:
> On May 28, 2015, at 10:37 AM, Donald Stufft wrote:
>
>>I think docker is a pretty crummy answer to Go’s static binaries. What I would
>>love is for Python to get:
>>
>>* The ability to import .so modules via zipzimport (ideally without a
>>temporary   directory, but that might require newer APIs from libc and such).
>
> +1 - Thomas Wouters mentioned at the language summit some work being done on
> glibc to add dlopen_from_memory() (sp?) which would allow for loading .so
> files directly from a zip.  Not sure what the status is of that, but it would
> be a great addition.

+1 but it needs to be cross-platform - py2exe has something similar
for Windows, which we should expect to use if the glibc solution is
Unix-only.

>>* The ability to create a “static” Python that links everything it needs into
>>the binary to do a zipimport of everything else (including the stdlib).
>
> +1

+0 - I suppose it would be nice, but how important is this really? If
all we do is end up with a single massive file instead of a directory,
then I don't see we've gained much.

Smallish C programs tend to hit the 10-100k executable size. Offhand I
don't know how big a typical go "single file executable" is. If I
could bundle up my Python script (something simple, let's say a "Hello
world" for argument's sake) and get a 10-20k single-file executable,
this would be worthwhile. If the resulting exe was 5MB (which is what
today's solutions that bundle the Python DLL and full zipped stdlib
tend to weigh in at) then I'm not interested.

>>*The ability to execute a zipfile that has been concat onto the end of the
>>Python binary.
>
> +1

+1. This would be immensely useful to tack on the front of a pyz
archive. Even just the existing (39K on Windows) python.exe which
relies on the Python installation would be great. Basically it could
replace the setuptools/distlib "script wrapper" executables with
something that doesn't need to run 2 processes just to run a script.

>>We’d probably want some more quality of life improvements around accessing
>>resources from within that zip file as well, but that can be done as a
>>library easier than the above three things can.
>
> E.g. you really should be using the pkg_resources APIs for loading resources
> from your packages, otherwise you're gonna have problems with zip
> executables.  We've talked before about adopting some of these APIs into
> Python's stdlib.  pkgutil is a start, and the higher level APIs from
> pkg_resources should probably go there.

+1. We need the APIs available in the stdlib, then 3rd party libraries
have a solution when users complain "your library isn't zipimport
safe". (Of course "well, we have to support Python 2.7/3.4" remains a
response for some time yet :-() Then we can start working on the
culture change where library authors start expecting their code to be
deployed in single-file "run-from-zip" applications.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Steve Dower]

Donald Stufft wrote:
> Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python 
> script + whatever other pieces you need.

For contrast, here are the things you need on Windows to be able to get to an 
interactive prompt (I don't know how other platforms get this down to 4KB...):

* python.exe (or some equivalent launcher) 39KB
* python35.dll 3,788KB
* vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not 
redistributable so doesn't count here)
* 26 files in Lib 343KB

This gets you to ">>>", and basically everything after that is going to fail 
for some reason. That's an unavoidable 4,257KB.

The rest of the stdlib adds another ~16MB once you exclude the test suite, so a 
fully functioning Python is not cheap. (Using compressed .pyc's in a zip file 
can make a big difference here though, assuming you're willing to trade CPU for 
HDD.)

Cheers,
Steve
 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Brian Curtin]

On Thu, May 28, 2015 at 11:23 AM, Chris Barker  wrote:
> I'm confused:
>
> Doesn't py2exe (optionally) create a single file executable?
>
> And py2app on the Mac creates an application bundle, but that is
> more-or-less the equivalent on OS-X (you may not even be able to have a
> single file executable that can access the Window Manager, for instance)
>
> Depending on what extra packages you need, py2exe's single file doesn't
> always work, but last I tried, it worked for a fair bit (I think all of the
> stdlib).
>
> I don't know what PyInstaller or others create. And I have no idea if there
> is a linux option -- but it seems like the standard of practice for an
> application for linux is a bunch of files scattered over the system anyway
> :-)
>
> Yes, the resulting exe is pretty big, but it does try to include only those
> modules and packages that are used, and that kind of optimization could be
> improved in any case.
>
> So is something different being asked for here?
>
> Barry Warsaw wrote:
>>> I do think single-file executables are an important piece to Python's
>>> long-term competitiveness.
>
> Really? It seems to me that desktop development is dying. What are the
> critical use-cases for a single file executable?

Donald mentioned one earlier: command line utilities. I want a single
CLI I can deploy to my customers that doesn't make them have to
install Python or even know it's Python at all. My users write code in
all types of languages on all OSes, but I should be able to produce
one thing that they can all use. Donald himself initiated the CLI in
particular I'm talking about, but Go is picking up steam here as we
have other utilities that quickly solved the "write one thing, every
user can run it immediately, no one knows/cares what it's written in"

When I worked on Ubuntu One, I was the Windows guy responsible for
making sure the end-user experience was the same there as it was on
Ubuntu. On Ubuntu we were a part of the base install and didn't have
to worry about much. On Windows we had none of that, not even the C
runtime, so we had some pre-installer work to do, and then a bunch of
py2exe hacking to make everything play nicely and transparently.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Gregory P. Smith]

On Thu, May 28, 2015 at 9:08 AM Guido van Rossum  wrote:

> Wow. Such thread. :-)
>
> This patch could save companies like Dropbox a lot of money. We run a ton
> of Python code in large datacenters, and while we are slow in moving to
> Python 3, we're good at updating to the latest 2.7.
>

Dropbox should be compiling its own interpreter with whatever patches it
deems appropriate. The people it'll save resources for are companies not
enlightened enough to do that: thousands of them, generally small or
non-tech focused :)

The patch is forward and backward compatible.I'm strongly in favor.
>

+1 I'm in favor as well.  I mostly wanted to make sure that people were
aware of profile-opt builds and that it was being compared.  Sounds like
both benefit, even used together.  Win win.

This is a 100% API compatible change.  It just rearranges the interpreter
loop on compilers enlightened enough to allow it.  I was always bummed that
it didn't make it into 2.7 itself.  But given the world+dog is going to
have 2.7 around and kicking for a long time, lets save the world some CPU
cycles (read: carbon) for little effort.  Very practical.  Good for the
world.

People who need to save orders of magnitude more cycles shouldn't use an
interpreter. ie: PyPy. Or consider the costs of moving to a compiled
language.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Paul Moore]

On 28 May 2015 at 17:47, Guido van Rossum  wrote:
> Single-file binaries are indeed important. (Though in most cases they don't
> have to be totally stand-alone -- they can depend on a system python and its
> stdlib. At least in typical datacenter setups.) Have people looked at PEX (a
> format developed by Twitter) or Pants (which seems to be an open-source tool
> that can build PEX files)?

There appear to be problems with pex on Windows. I've reported a
couple of bugs, which appear to have been fixed (although I don't know
the timeframe, I lost interest before a fix was implemented). But
there are new ones I just found on a quick test.

I'd like pex to work, it looks like a nice tool. But I don't want to
be their Windows support resource, and it seems like they may not
currently have anyone else :-)

Paul

(I'll probably go and lie down now and stop banging the cross-platform
drum for a while :-))
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Chris Angelico]

On Fri, May 29, 2015 at 3:04 AM, Brian Curtin  wrote:
> Donald mentioned one earlier: command line utilities. I want a single
> CLI I can deploy to my customers that doesn't make them have to
> install Python or even know it's Python at all. My users write code in
> all types of languages on all OSes, but I should be able to produce
> one thing that they can all use. Donald himself initiated the CLI in
> particular I'm talking about, but Go is picking up steam here as we
> have other utilities that quickly solved the "write one thing, every
> user can run it immediately, no one knows/cares what it's written in"

Unix-like systems have this courtesy of the shebang, so as long as
there's some sort of Python installed, people don't need to know or
care that /usr/local/bin/mailmail is implemented in Python. Maybe the
solution is to push for Windows to always include a Python
interpreter, which would allow a tiny stub to go and call on that?
Obviously a full shebang concept would be a huge change to Windows,
but if a 4KB executable can go and locate the rest of Python, or open
up a web browser saying "Please install OS update KB123456", that
would do it for most end users.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Steve Dower]

Donald Stufft wrote:
> On May 28, 2015 at 11:30:37 AM, Steve Dower (steve.do...@microsoft.com) wrote:
>> Donald Stufft wrote:
>> > Well Python 3.4.3 binary is 4kb for me, so you'd have that + your
>> > 1KB Python script + whatever
>> other pieces you need.
>>
>> For contrast, here are the things you need on Windows to be able to
>> get to an interactive prompt (I don't know how other platforms get this down
>> to 4KB...):
>>
>> * python.exe (or some equivalent launcher) 39KB
>> * python35.dll 3,788KB
>> * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not
>> redistributable so doesn't count here)
>> * 26 files in Lib 343KB
>>
>> This gets you to ">>>", and basically everything after that is going to fail
> for some reason.
>> That's an unavoidable 4,257KB.
>>
>> The rest of the stdlib adds another ~16MB once you exclude the test
>> suite, so a fully functioning Python is not cheap. (Using compressed
>> .pyc's in a zip file can make a big difference here though, assuming
>> you're willing to trade CPU for HDD.)
>>
>> Cheers,
>> Steve
>>
>>
> 
> You don’t need a "fully functioning Python" for a single file binary, you only
> need enough to actually run your application. For example, if you're making an
> application that can download files over HTTP, you don't need to include parts
> of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email,
> mailcap, mailbox, imaplib, nntplib, etc.
> 
> Of course deciding which pieces you include in the zip file you're appending 
> to
> the end of Python is up to whatever tool builds this executable which doesn't
> need to be part of Python itself. If Python itself gained the ability to 
> operate
> in that manner than third party tools could handle trying to do the
> optimizations where it only includes the things it actually needs in the 
> stdlib
> and excludes things it doesn't. The key thing here is that since you're doing 
> a
> single file binary, you don't need to have a Python which is suitable to 
> execute
> random Python code, you only need one that is suitable to execute this
> particular code so you can specialize what that includes.

Agreed, but the minimally functioning Python is barely under 5MB. That will be 
considered bloated and won't help us compete with Go, so we should find a 
better way to fix Python application distribution and stop getting so hung up 
on putting everything into a single executable file.

Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Barry Warsaw]

On May 28, 2015, at 09:23 AM, Chris Barker wrote:

>Barry Warsaw wrote:
>>I do think single-file executables are an important piece to Python's
>>long-term competitiveness.
>
>Really? It seems to me that desktop development is dying. What are the
>critical use-cases for a single file executable?
>
>And I'd note that getting a good way to use Python to develop for iOS,
>Android, and Mobile Windows is FAR more critical!  -- maybe that's the same
>problem ?

Well, in my world they are the same problem!  With mobile, IoT, etc. you can't
or shouldn't assume that Python will be available in the base environment.
There are ways to deploy to the various new world of devices that perhaps
don't require single-file executables, but those are more complicated to
manage, and they still have to ship the Python environment along with the
application code.

Cheers,
-Barry

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Donald Stufft]

On May 28, 2015 at 12:54:34 PM, Chris Angelico (ros...@gmail.com) wrote:
> On Fri, May 29, 2015 at 2:28 AM, Chris Barker wrote:
> > oops, sorry -- I see this was addressed in another thread. Though I guess I
> > still don't see why "single file" is critical, over "single thing to
> > install" -- like a OS-X app bundle that can just be dragged into the
> > Applications folder.
>  
> There's also "single thing to uninstall", which IMO is more important.
> If I download a tiny program that's supposed to just do one tiny
> thing, and it has to install itself into Program Files, Common Files,
> Windows\System32, and Documents & Settings\my-user-name\Applications,
> then I have to hope it has a proper uninstaller. If it's a single
> executable that just does its stuff (or, failing that, a single zip
> file that I extract to anywhere and run the program), I can expect
> that deleting that file (or directory) will get rid of it all. Of
> course, it's entirely possible that it's gone and left its droppings
> all over the system, but that's a matter of trust - a legit program
> won't lie about that.
>  
> Is this a Windows-specific issue, or is it also intended for Linux and
> Mac OS, where there'll already be a system Python (so a
> single-file-executable would be used to be independent of the system
> Python)?
>  

I think it’s an issue for all platforms, even when there is a system Python
that can be used.

Here’s why:

* Even on Linux systems Python isn’t always a guaranteed thing to be installed,
  for instance Debian works just fine without any Python installed.

* On OS X you have the system Python yes, but that is in an unknown state. It
  could have any number of changes made to it or things installed or what have
  you.

* Even if you have Python installed already, is it the right one? What if it’s
  an ancient RHEL box that has 2.6 or (heaven forbid) 2.4? What if it’s a not
  ancient box that has Python 2.7 but you want to deploy your app in Python 3?

* What if you have Python installed already, but it’s been patched by the place
  you got it from and now the behavior is different than what you expected?

etc etc.

---  
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Brett Cannon]

 On Thu, May 28, 2015, 12:14 Barry Warsaw  wrote:

Go seems to be popular where I work.  It is replacing Python in a number of
places, although Python (and especially Python 3) is still a very important
part of our language toolbox.

There are several reasons why Go is gaining popularity.  Single-file
executables is definitely a reason; it makes deployment very easy, even if
it
increases the maintenance burden (e.g. without shared libraries, you have
multiple copies of things so when a security fix is required for one of
those
things you have to recompile the world).

Start up times and memory footprint are also factors.  Probably not much to
be
done about the latter, but perhaps PEP 432 can lead to improvements in the
former.  (Hey Nick, I'm guessing you'll want to bump that one back to 3.6.)

Certainly better support for multi-cores comes up a lot.  It should be a
SMoE
to just get rid of the GIL once and for all .

One thing I've seen more than once is that new development happens in Python
until the problem is understood, then the code is ported to Go.  Python's
short path from idea to working code, along with its ability to quickly
morph
as requirements and understanding changes, its batteries included
philosophy,
and its "fits-your-brain" consistency are its biggest strengths!

On May 28, 2015, at 10:37 AM, Donald Stufft wrote:

>I think docker is a pretty crummy answer to Go’s static binaries. What I
would
>love is for Python to get:
>
>* The ability to import .so modules via zipzimport (ideally without a
>temporary   directory, but that might require newer APIs from libc and
such).

+1 - Thomas Wouters mentioned at the language summit some work being done on
glibc to add dlopen_from_memory() (sp?) which would allow for loading .so
files directly from a zip.  Not sure what the status is of that, but it
would
be a great addition.

>* The ability to create a “static” Python that links everything it needs
into
>the binary to do a zipimport of everything else (including the stdlib).

+1

>*The ability to execute a zipfile that has been concat onto the end of the

>Python binary.

+1

>I think that if we get all of that, we could easily create a single file
>executable with real, native support from Python by simply compiling Python
>in that static mode and then appending a zip file containing the standard
>library and any other distributions we need to the end of it.
>
>We’d probably want some more quality of life improvements around accessing
>resources from within that zip file as well, but that can be done as a
>library easier than the above three things can.

E.g. you really should be using the pkg_resources APIs for loading resources
from your packages, otherwise you're gonna have problems with zip
executables.  We've talked before about adopting some of these APIs into
Python's stdlib.  pkgutil is a start, and the higher level APIs from
pkg_resources should probably go there.

 Donald Stuff proposed importlib.resources a little while back to handle
the storage-agnostic api dor reading data and I have been thinking about it
for years. I plan to make it happen in Python 3.6.

-brett


Cheers,
-Barry

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/brett%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Barry Warsaw]

On May 28, 2015, at 12:44 PM, Donald Stufft wrote:

>Pex would be improved by having native support for importing .so’s from within
>a zipfile via zipimport. It would also be improved by having good, built in
>support for extraneous resources in the stdlib too.

Completely agree on both points.  Having an API for importing .so's from a zip
would be really useful.  Today that can be implemented as "copy to tempdir"
and tomorrow the implementation could optionally dlopen_from_memory() without
any client code changing.

>Right, it would be great to get it built into Python itself, but I consider
>that less important than getting the critical pieces into Python. If those
>pieces are there then we can iterate outside of the standard library and try
>different approaches to *building* such a file, and eventually take a look at
>the landscape and bless one approach (or not, if we don’t want to).

Sounds good to me!

Cheers,
-Barry


pgp1R37cL1HEu.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Ryan Gonzalez]

py2exe tends to invoke DLL hell if you have various versions of VS or Office or 
both installed. Because Windows.


On May 28, 2015 11:23:57 AM CDT, Chris Barker  wrote:
>I'm confused:
>
>Doesn't py2exe (optionally) create a single file executable?
>
>And py2app on the Mac creates an application bundle, but that is
>more-or-less the equivalent on OS-X (you may not even be able to have a
>single file executable that can access the Window Manager, for
>instance)
>
>Depending on what extra packages you need, py2exe's single file doesn't
>always work, but last I tried, it worked for a fair bit (I think all of
>the
>stdlib).
>
>I don't know what PyInstaller or others create. And I have no idea if
>there
>is a linux option -- but it seems like the standard of practice for an
>application for linux is a bunch of files scattered over the system
>anyway
>:-)
>
>Yes, the resulting exe is pretty big, but it does try to include only
>those
>modules and packages that are used, and that kind of optimization could
>be
>improved in any case.
>
>So is something different being asked for here?
>
>Barry Warsaw wrote:
>>> I do think single-file executables are an important piece to
>Python's long-term
>competitiveness.
>
>Really? It seems to me that desktop development is dying. What are the
>critical use-cases for a single file executable?
>
>And I'd note that getting a good way to use Python to develop for iOS,
>Android, and Mobile Windows is FAR more critical!  -- maybe that's the
>same
>problem ?
>
>-Chris
>
>
>On Thu, May 28, 2015 at 8:39 AM, Donald Stufft 
>wrote:
>
>>
>>
>> On May 28, 2015 at 11:30:37 AM, Steve Dower
>(steve.do...@microsoft.com)
>> wrote:
>> > Donald Stufft wrote:
>> > > Well Python 3.4.3 binary is 4kb for me, so you'd have that + your
>1KB
>> Python script + whatever
>> > other pieces you need.
>> >
>> > For contrast, here are the things you need on Windows to be able to
>get
>> to an interactive
>> > prompt (I don't know how other platforms get this down to 4KB...):
>> >
>> > * python.exe (or some equivalent launcher) 39KB
>> > * python35.dll 3,788KB
>> > * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is
>not
>> redistributable
>> > so doesn't count here)
>> > * 26 files in Lib 343KB
>> >
>> > This gets you to ">>>", and basically everything after that is
>going to
>> fail for some reason.
>> > That's an unavoidable 4,257KB.
>> >
>> > The rest of the stdlib adds another ~16MB once you exclude the test
>> suite, so a fully functioning
>> > Python is not cheap. (Using compressed .pyc's in a zip file can
>make a
>> big difference here
>> > though, assuming you're willing to trade CPU for HDD.)
>> >
>> > Cheers,
>> > Steve
>> >
>> >
>>
>> You don’t need a "fully functioning Python" for a single file binary,
>you
>> only
>> need enough to actually run your application. For example, if you're
>making
>> an application that can download files over HTTP, you don't need to
>include
>> parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite,
>csv,
>> email,
>> mailcap, mailbox, imaplib, nntplib, etc.
>>
>> Of course deciding which pieces you include in the zip file you're
>> appending
>> to the end of Python is up to whatever tool builds this executable
>which
>> doesn't need to be part of Python itself. If Python itself gained the
>> ability
>> to operate in that manner than third party tools could handle trying
>to do
>> the
>> optimizations where it only includes the things it actually needs in
>the
>> stdlib
>> and excludes things it doesn't. The key thing here is that since
>you're
>> doing
>> a single file binary, you don't need to have a Python which is
>suitable to
>> execute random Python code, you only need one that is suitable to
>execute
>> this
>> particular code so you can specialize what that includes.
>>
>> ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>>
>https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>>
>
>
>
>-- 
>
>Christopher Barker, Ph.D.
>Oceanographer
>
>Emergency Response Division
>NOAA/NOS/OR&R(206) 526-6959   voice
>7600 Sand Point Way NE   (206) 526-6329   fax
>Seattle, WA  98115   (206) 526-6317   main reception
>
>chris.bar...@noaa.gov
>
>
>
>
>___
>Python-Dev mailing list
>Python-Dev@python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailma

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Ryan Gonzalez]

I agree that size is an issue, but is it really that bad? Just compare it to 
the recent "web surge" where everyone is writing desktop apps in HTML5+CSS+JS 
and bundling a huge WebKit engine in their apps binary.

Python on Windows is seriously in a bad state. IMO, what needs to be 
prioritized is the ability to make exes that *actually work* with nicer GUI 
abilities. py2exe gives too many random DLL errors and PyInstaller is an ugly 
hack that just shoves 20 DLLs with your executable. Mixed with the fact that 
TkInter looks even uglier when built via py2exe and almost everything else 
(PyGI, PySide, etc.) requires yet another 20 DLLs (PySide threw in Qt DLLs that 
I didn't even use!), it's sad. Really sad.

This is most of the reason I write programs that I plan on distributing to 
various crowds in some statically compiled language (C++, Nim, Felix, not Go) 
with (when necessary) a statically-linked GUI library. Less DLL hell, less free 
files, etc.

Oh yeah, and add to that the problems with running both Python 2 and 3 on 
Windows while using some binaries that want 3 and others that want 2. It's 
painful.


On May 28, 2015 10:30:33 AM CDT, Steve Dower  wrote:
>Donald Stufft wrote:
>> Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB
>Python script + whatever other pieces you need.
>
>For contrast, here are the things you need on Windows to be able to get
>to an interactive prompt (I don't know how other platforms get this
>down to 4KB...):
>
>* python.exe (or some equivalent launcher) 39KB
>* python35.dll 3,788KB
>* vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not
>redistributable so doesn't count here)
>* 26 files in Lib 343KB
>
>This gets you to ">>>", and basically everything after that is going to
>fail for some reason. That's an unavoidable 4,257KB.
>
>The rest of the stdlib adds another ~16MB once you exclude the test
>suite, so a fully functioning Python is not cheap. (Using compressed
>.pyc's in a zip file can make a big difference here though, assuming
>you're willing to trade CPU for HDD.)
>
>Cheers,
>Steve
> 
>___
>Python-Dev mailing list
>Python-Dev@python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Chris Angelico]

On Fri, May 29, 2015 at 3:20 AM, Donald Stufft  wrote:
> On May 28, 2015 at 12:54:34 PM, Chris Angelico (ros...@gmail.com) wrote:
>> Is this a Windows-specific issue, or is it also intended for Linux and
>> Mac OS, where there'll already be a system Python (so a
>> single-file-executable would be used to be independent of the system
>> Python)?
>>
>
> I think it’s an issue for all platforms, even when there is a system Python
> that can be used.
>
> Here’s why:
>
> * Even on Linux systems Python isn’t always a guaranteed thing to be 
> installed,
>   for instance Debian works just fine without any Python installed.
>
> * On OS X you have the system Python yes, but that is in an unknown state. It
>   could have any number of changes made to it or things installed or what have
>   you.
>
> * Even if you have Python installed already, is it the right one? What if it’s
>   an ancient RHEL box that has 2.6 or (heaven forbid) 2.4? What if it’s a not
>   ancient box that has Python 2.7 but you want to deploy your app in Python 3?
>
> * What if you have Python installed already, but it’s been patched by the 
> place
>   you got it from and now the behavior is different than what you expected?

The independence argument. Yep, reasonable. The trouble is that if
everyone seeks to be independent of the system Python, it makes the
system Python redundant with every single Python app; and the bundled
Python will have to be platform-specific (not just OS, but
architecture, word size, possibly OS version in some cases, etc, etc).
And that independence also means you miss out on
security/compatibility updates from upstream, so you're having to
manage your own updates.

I'd still much rather see a small stub that goes and calls on a system
Python than something that has to duplicate all of Python into every
binary, but it's a choice for the application developer to make. If
there can at least be an easy way to strip off the self-executability
header and get back the base Python application, that would be a big
help - you could have a script that goes through all your executables,
looks for the signature that says "hi, I'm a Python 2.7.10
self-runner", and strips it off in favour of a 2.7.11 that fixes some
critical issue. Or you could strip that off and run the underlying
Python code on a different OS. Binary blobs are so often unworkable.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Paul Moore]

On 28 May 2015 at 18:04, Brian Curtin  wrote:
> Donald mentioned one earlier: command line utilities. I want a single
> CLI I can deploy to my customers that doesn't make them have to
> install Python or even know it's Python at all.

Yep, that's the killer for me as well.

I know it's unrealistic in some sense, but my benchmark is what does a
"Hello, world" program look like? In C, it's a 38K executable with no
external dependencies. What does it look like in Python? (I'm not too
worried if it's a bit bigger, but if it's a *lot* bigger that starts
to be noticeable - "Python generates bloated exes").

What I'd like to be able to do is to write Python ports of a range of
core Unix utilities (comm, cut, join, od, seq, tr, ...) and have them
be viable alternatives to my current C builds (a series of single-file
100-200k static exes).

On 28 May 2015 at 18:15, Chris Angelico  wrote:
> Unix-like systems have this courtesy of the shebang, so as long as
> there's some sort of Python installed, people don't need to know or
> care that /usr/local/bin/mailmail is implemented in Python. Maybe the
> solution is to push for Windows to always include a Python
> interpreter, which would allow a tiny stub to go and call on that?

Unfortunately (and believe me, I've been down this road many times) on
Windows *only* the exe format is a "first-class" executable.
Executable scripts and shebangs are very useful, but there are always
corner cases where they don't work *quite* like an exe. On Windows,
you have to be prepared to ship an exe if you want to compete with
languages that generate exes.

Having said that, the trick of appending a zipfile to an exe (or
similar) is already common practice in the Python world, and works
really well. Vinay Sanjip's pyzzer is a good example of this approach.

On 28 May 2015 at 16:45, Steve Dower  wrote:
> Agreed, but the minimally functioning Python is barely under 5MB. That will be
> considered bloated and won't help us compete with Go, so we should find a 
> better
> way to fix Python application distribution and stop getting so hung up on 
> putting
> everything into a single executable file.

There's a perception issue here. You can compile C# code into small
exes (that's how people think) and "all" you need is the .net runtime
installed. If we shipped a "pyc" compiler that "compiled" Python code
into small exes that "just needed the Python runtime installed" would
that feel the same to people? Would they be happy to view that as
comparable to go compiled executables? (I assume go *doesn't* rely on
a separate runtime, though).

Nevertheless, I would like to understand how Unix can manage to have a
Python 3.4.3 binary at 4kb. Does that *really* have no external
dependencies (other than the C library)? Are we really comparing like
with like here?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Usability of the limited API

[Steve Dower]

Zach has a patch to automate putting the right exports in python3.dll, which 
I'm strongly in favor of, but it was rejected because people may have added 
APIs that aren't meant to be stable.

Right now, you can #include a number of prototypes that aren't actually 
available because there are two places to update and so one (in this case, the 
DLL) doesn't get updated.

I think the current plan is to remove everything not currently in the DLL from 
the stable ABI and force people to add them back manually. This way we can 
enable the generator without committing to a large set of new APIs.

I don't have the issue number handy, but it should be near the top of the 
recently modified list.

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Moore
Sent: ‎5/‎28/‎2015 7:12
To: Python Dev
Subject: [Python-Dev] Usability of the limited API

With Python 3.5 shipping an embeddable copy of the interpreter on
Windows, I thought I'd try out a simple embedded interpreter as an
experiment. I wanted to use the limited API, as I'd rather it were
easy to upgrade the interpreter without recompiling the embedding app.

But the "Very high-level embedding" example in the docs doesn't
compile with the limited API.

#include 

int
main(int argc, char *argv[])
{
wchar_t *program = Py_DecodeLocale(argv[0], NULL);
if (program == NULL) {
fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
exit(1);
}
Py_SetProgramName(program);  /* optional but recommended */
Py_Initialize();
PyRun_SimpleString("from time import time,ctime\n"
   "print('Today is', ctime(time()))\n");
Py_Finalize();
PyMem_RawFree(program);
return 0;
}

The Py_DecodeLocale/Py_SetProgramName/PyMem_RawFree bit can probably
be replaced by a Py_SetProgramName call specifying a static value,
it's not exactly crucial. (Py_DecodeLocale appears to be defined as in
the limited API by the headers, but not exported from python3.dll, by
the way, which implies that something's out of sync).

But PyRun_SimpleString doesn't appear to be exposed in the limited
API, even though https://docs.python.org/3/c-api/veryhigh.html doesn't
mention this, and https://docs.python.org/3/c-api/stable.html says
that functions not part of the stable API will be marked as such.

I dumped out the exported symbols from python3.dll, which is the
simplest way I could think of finding out what is in the limited API
(it's hardly user friendly, but never mind). And frustratingly, none
of the very high level PyRun_XXX APIs are available.

At this point, I think I'll probably just give up and use the full
API, but it does make me question whether the limited API is actually
usable as it stands.

I was hoping to be able to suggest as an application bundling option
that people could write a trivial wrapper script in C to fire up a
Python script, and bundle that along with its dependencies and the
embeddable Python distribution. Looks like that's doable, but only
using the full API, which makes upgrading the bundled Python
interpreter a bit messier. Ah, well, no huge loss :-(

But after this experiment, I do wonder - is the limited API really a
viable option for embedders?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Wes Turner]

On Thu, May 28, 2015 at 11:38 AM, Paul Moore  wrote:

> On 28 May 2015 at 16:58, Barry Warsaw  wrote:
> > On May 28, 2015, at 11:39 AM, Donald Stufft wrote:
> >
> >>You don’t need a "fully functioning Python" for a single file binary, you
> >>only need enough to actually run your application. For example, if you're
> >>making an application that can download files over HTTP, you don't need
> to
> >>include parts of the stdlib like xmlrpc, pickle, shelve, marshall,
> sqlite,
> >>csv, email, mailcap, mailbox, imaplib, nntplib, etc.
> >
> > There are actually two related but different use cases to "single file
> > executables".
> >
> > The first is nicely solved by tools like pex, where you don't need to
> include
> > a fully functional Python at the head of the zip file because the
> environment
> > you're deploying it into will have enough Python to make the zip work.
> This
> > can certainly result in smaller zip files.  This is the approach I took
> with
> > Snappy Ubuntu Core support for Python 3, based on the current situation
> that
> > the atomic upgrade client is written in Python 3.  If that changes and
> Python
> > 3 is removed from the image, then this approach won't work.
> >
> > pex (and others) does a great job at this, so unless there are things
> better
> > refactored into upstream Python, I don't think we need to do much here.
>
> One problem with pex is that it doesn't appear to work on Windows (I
> just gave it a try, and got errors because it relies on symlinks).
>
> IMO, any solution to "distributing Python applications" that is
> intended to compete with the idea that "go produces nice single-file
> executables" needs to be cross-platform. At the moment, zipapp (and in
> general, the core support for running applications from a zip file)
> handles this for the case where you're allowed to assume an already
> installed Python interpreter. The proviso here, as Donald pointed out,
> is that it doesn't handle C extensions.
>
> The biggest problem with 3rd-party solutions is that they don't always
> support the full range of platforms that Python supports. That's fine
> for a 3rd party tool, but if we want to have a response to people
> asking how to bundle their application written in Python, we need a
> better answer than "if you're on Windows, use py2exe, or if you're on
> Unix use pex, or maybe..."
>
> Python has core support for the equivalent of Java's jar format in
> zipapp. It's not well promoted (and doesn't support C extensions) but
> it's a pretty viable option for a lot of situations.
>
> > The second use case is as you describe: put a complete functional Python
> > environment at the head of the zip file so you don't need anything in the
> > target deployment environment.  "Complete" can easily mean the entire
> stdlib,
> > and although that would usually be more bloated than you normally need,
> hey,
> > it's just some extra unused bits so who cares? .  I think this
> would be
> > an excellent starting point which can be optimized to trim unnecessary
> bits
> > later, maybe by third party tools.
>
> Tools like py2exe and cx_Freeze do this, and are pretty commonly used
> on Windows. An obvious example of use is Mercurial. If you're looking
> at this scenario, a good place to start would probably be
> understanding why cx_Freeze isn't more commonly used on Unix (AFAIK,
> it supports Unix, but I've only ever really heard of it being used on
> Windows).
>

Esky https://github.com/cloudmatrix/esky/

* supports "py2exe, py2app, cxfreeze and bbfreeze"
* builds a zip archive containing an .exe
* manages (failed) [auto-]updates

PEX https://pantsbuild.github.io/pex_design.html

* adds an executable header to a (topo-sorted?) ZIP file with a minimal path

* pipsi https://github.com/mitsuhiko/pipsi/blob/master/pipsi.py
  * installs packages with console_scripts into separate virtualenvs with
minimal sys.paths and ~/.local/bin)

At the end of the day I still need packaging or configmgmt or NIX
for checksums (a manifest wrapped around the executable wrapper).


>
> I suspect "single file executables" just aren't viewed as a desirable
> solution on Unix. Although Donald referred to a 4K binary, which
> probably means just a stub exe that depends on system-installed .so
> files, likely including Python (I'm just guessing here). It's easy to
> do something similar on Windows, but it's *not* what most Windows
> users think of when you say a "single file executable for a Python
> program" (because there's no system package manager doing dependencies
> for you).
>

NuGet, Chocolatey, -> OneGet

It's a topologically sorted adjacency list + build + install + uninstall
scripts.


>
> Again, platform-specific answers are one thing, and are relatively
> common, but having a good cross-platform answer at the language level
> (a section on docs.python.org "How to ship your Python program") is
> much harder.
>
> >>Of course deciding which pieces you include in the zip file you're
> appending
> >>to the e

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Terry Reedy]

On 5/28/2015 10:55 AM, Steve Dower wrote:


And it would look like a 20MB+ file just for a simple 1KB Python
script...

For Windows at least, I'd prefer to have some app-style installer
generation (e.g. http://pynsist.readthedocs.org/en/latest/) which,
combined with the embeddable Python distro (new for 3.5.0b1 in case
anyone missed it), can simply extract everything into an install
directory and run it from there. None of the items on the list above
are needed for or would help with this.


What I wish, of course, is that Windows just came with Python3, the way 
that DOS came with BASIC, so people could publish and trade Python 
programs the way we once did with BASIC programs.  Then a simple 1KB 
Python script would just take an extra 1KB on disk.  To me, the removal 
of a simple, builtin programming language for everyone was the biggest 
Windows mistake.


Failing that, maybe PSF & edu-sig could somehow encourage universities 
that requires students to have a computer to recommend or even require 
that Python be loaded so students could at least assume that others 
students have Python already loaded.  Python is the BASIC of the 21 century.


Somehow, trying to make it easier to have 50 duplicate copies of Python 
on a system seems the wrong direction to go.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[M.-A. Lemburg]

You might want to have a look at eGenix PyRun, which gives you
an almost complete Python runtime in 4-13MB (depending on what
startup performance needs you have):

http://www.egenix.com/products/python/PyRun/

On 28.05.2015 17:58, Barry Warsaw wrote:
> On May 28, 2015, at 11:39 AM, Donald Stufft wrote:
> 
>> You don’t need a "fully functioning Python" for a single file binary, you
>> only need enough to actually run your application. For example, if you're
>> making an application that can download files over HTTP, you don't need to
>> include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite,
>> csv, email, mailcap, mailbox, imaplib, nntplib, etc.
> 
> There are actually two related but different use cases to "single file
> executables".
> 
> The first is nicely solved by tools like pex, where you don't need to include
> a fully functional Python at the head of the zip file because the environment
> you're deploying it into will have enough Python to make the zip work.  This
> can certainly result in smaller zip files.  This is the approach I took with
> Snappy Ubuntu Core support for Python 3, based on the current situation that
> the atomic upgrade client is written in Python 3.  If that changes and Python
> 3 is removed from the image, then this approach won't work.
> 
> pex (and others) does a great job at this, so unless there are things better
> refactored into upstream Python, I don't think we need to do much here.
> 
> The second use case is as you describe: put a complete functional Python
> environment at the head of the zip file so you don't need anything in the
> target deployment environment.  "Complete" can easily mean the entire stdlib,
> and although that would usually be more bloated than you normally need, hey,
> it's just some extra unused bits so who cares? .  I think this would be
> an excellent starting point which can be optimized to trim unnecessary bits
> later, maybe by third party tools.

See above. This is what eGenix PyRun provides.

Our main motivation is to have a binary which works on all
Unix platforms, without having to rely on the way too many
system dependent Python distribution (with all their quirks
and whistles ;-)).

On Windows, we use py2exe at the moment, but a port of PyRun
to Windows would be possible as well. You'd still need the
separate Python DLL, though in order to stay compatible to
C extensions which link against this.

As for application packaging: we don't have a need to put
everything into a single ZIP file or even concatenate such
a ZIP file to PyRun (which is possible: just add sys.executable to
sys.path to import from the executable).

We have plans to create a tool to make such packaging possible,
though, since it's handy to have for building small executable
apps, e.g. to drive installations or larger applications.

>> Of course deciding which pieces you include in the zip file you're appending
>> to the end of Python is up to whatever tool builds this executable which
>> doesn't need to be part of Python itself. If Python itself gained the ability
>> to operate in that manner than third party tools could handle trying to do
>> the optimizations where it only includes the things it actually needs in the
>> stdlib and excludes things it doesn't. The key thing here is that since
>> you're doing a single file binary, you don't need to have a Python which is
>> suitable to execute random Python code, you only need one that is suitable to
>> execute this particular code so you can specialize what that includes.
> 
> I'd love to see Python itself gain such a tool, but if it had the critical
> pieces to execute in this way, that would enable a common approach to
> supporting this in third party tools, on a variety of platforms.
> 
> I do think single-file executables are an important piece to Python's
> long-term competitiveness.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 28 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Chris Angelico]

On Fri, May 29, 2015 at 3:52 AM, Paul Moore  wrote:
> On 28 May 2015 at 18:15, Chris Angelico  wrote:
>> Unix-like systems have this courtesy of the shebang, so as long as
>> there's some sort of Python installed, people don't need to know or
>> care that /usr/local/bin/mailmail is implemented in Python. Maybe the
>> solution is to push for Windows to always include a Python
>> interpreter, which would allow a tiny stub to go and call on that?
>
> Unfortunately (and believe me, I've been down this road many times) on
> Windows *only* the exe format is a "first-class" executable.
> Executable scripts and shebangs are very useful, but there are always
> corner cases where they don't work *quite* like an exe. On Windows,
> you have to be prepared to ship an exe if you want to compete with
> languages that generate exes.

I'm aware of that. When I said "a tiny stub", I was thinking in terms
of a small executable. The idea is that its sole purpose is to locate
Python someplace else, and chain to it; that has to be actual
executable code, complete with the 512-byte "MZ" header and
everything, to ensure compatibility. But it should be able to be
small, tight, and easy to verify correctness of, so there aren't (in
theory!) security exploits in the header itself.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables

[Carl Meyer]

On 05/28/2015 11:52 AM, Paul Moore wrote:
[snip]
> Nevertheless, I would like to understand how Unix can manage to have a
> Python 3.4.3 binary at 4kb. Does that *really* have no external
> dependencies (other than the C library)? Are we really comparing like
> with like here?

I don't know what Donald was looking at, but I'm not seeing anything
close to that 4k figure here. (Maybe he's on OS X, where framework
builds have a "stub" executable that just execs the real one?)

On my Ubuntu Trusty system, the system Python 3.4 executable is 3.9M,
and the one I compiled myself from source, without any special options,
is almost 12M. (Not really sure what accounts for that difference -
Ubuntu system Python uses shared libraries for more stuff?)

Carl



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Donald Stufft]

On May 28, 2015 at 2:11:02 PM, Terry Reedy (tjre...@udel.edu) wrote:
> On 5/28/2015 10:55 AM, Steve Dower wrote:
>  
> > And it would look like a 20MB+ file just for a simple 1KB Python
> > script...
> >
> > For Windows at least, I'd prefer to have some app-style installer
> > generation (e.g. http://pynsist.readthedocs.org/en/latest/) which,
> > combined with the embeddable Python distro (new for 3.5.0b1 in case
> > anyone missed it), can simply extract everything into an install
> > directory and run it from there. None of the items on the list above
> > are needed for or would help with this.
>  
> What I wish, of course, is that Windows just came with Python3, the way
> that DOS came with BASIC, so people could publish and trade Python
> programs the way we once did with BASIC programs. Then a simple 1KB
> Python script would just take an extra 1KB on disk. To me, the removal
> of a simple, builtin programming language for everyone was the biggest
> Windows mistake.
>  
> Failing that, maybe PSF & edu-sig could somehow encourage universities
> that requires students to have a computer to recommend or even require
> that Python be loaded so students could at least assume that others
> students have Python already loaded. Python is the BASIC of the 21 century.
>  
> Somehow, trying to make it easier to have 50 duplicate copies of Python
> on a system seems the wrong direction to go.
>  
> --
> Terry Jan Reedy
>  
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io  
>  

Honestly, I’m on an OS that *does* ship Python (OS X) and part of me hopes
that they stop shipping it. It’s very rare that someone ships Python as
part of their OS without modifying it in some way, and those modifications
almost always cause pain to some set of users (and since I work on pip, they
tend to come to us with the weirdo problems). Case in point: Python on OS X
adds some preinstalled software, but they put this pre-installed software before
site-packages in sys.path, so pip can’t upgrade those pre-installed software
packages at all. 

---  
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables

[Donald Stufft]

On May 28, 2015 at 2:33:25 PM, Carl Meyer (c...@oddbird.net) wrote:
> On 05/28/2015 11:52 AM, Paul Moore wrote:
> [snip]
> > Nevertheless, I would like to understand how Unix can manage to have a
> > Python 3.4.3 binary at 4kb. Does that *really* have no external
> > dependencies (other than the C library)? Are we really comparing like
> > with like here?
> 
> I don't know what Donald was looking at, but I'm not seeing anything
> close to that 4k figure here. (Maybe he's on OS X, where framework
> builds have a "stub" executable that just execs the real one?)
> 
> On my Ubuntu Trusty system, the system Python 3.4 executable is 3.9M,
> and the one I compiled myself from source, without any special options,
> is almost 12M. (Not really sure what accounts for that difference -
> Ubuntu system Python uses shared libraries for more stuff?)
> 
> Carl
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io 
> 

The problem is I'm an idiot and did du -h against ``which python``, which of
course resolved to the symlink of python that points to python3.4. The real
executable on my OSX box is 2.6M (built using pyenv).

--- 
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Terry Reedy]

On 5/28/2015 12:44 PM, Donald Stufft wrote:


I do think single-file executables are an important piece to Python's
long-term competitiveness.



I completely agree. I talk to a lot of people about packaging of things, and 
while
I think there are some serious problems with huge parts of Go’s packaging and
distribution story the static linking and compiling down to a “single” file is 
not
one of them.


How about the following compromise between assuming the presence of 
installed python and ignoring the possibility of installed python: we 
provide (as least for Windows) a C startup file that checks for the 
needed version of python and offers to download and start the installer 
if not present.  A user would see 'Python 3.5 is needed to run this 
file; shall I download and install it?' just once on a particular machine.


--
Terry Jan Reedy


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Chris Barker]

OK, I'm really confused here:

1) what the heck is so special about go all of a sudden? People have been
writing and deploying single file executables built with C and ++, and
whatever else? forever. (and indeed, it was a big sticking point for me
when I introduced python in my organization)

2) Why the sudden interest in this as core a Python issue? I've been using
Python for desktop apps, on primarily Windows and the Mac for years -- and
had to deal with py2exe, py2app, etc. forever. And it has been a very very
common question on the various mailing lists for ages: how do I deploy
this? how do I make it easy to install? The answer from the developers of
cPython itself has always been that that's a third party problem -- and go
look for py2exe and friends to solve it. And that it is a solved-enough
problem. The biggest "unsolved" issues are that you get a really  big
application.

Don't get me wrong -- I've wanted for years for it to be easier to deploy
python-based apps as a single thinking for users to easily install and
uninstall where they don't need to know it's python -- but what the heck is
different now?

3) There was mention of a platform-neutral way to do this. Isn't that
simply impossible? The platforms are different in all the ways that matter
for this problem: both technical differences, and conventions. Which isn't
to say you couldn't have one API to produce a single "thing" executable, so
it would look like one solution for multiple platforms to the user. But the
end product should be (would have to be) a different beast altogether.

And doesn't PyInstaller already provide that (may it can't do
single-file...)

Anyway -- if there really is a new interest in this problem such that
people will put some time into, here are some thoughts I've had for ages:

The big one is Separation of concerns: in order to build a single "thing"
executable, you need three things:
  a) An API to for the developer to specify what they want
  b) Figure out what needs to be included -- what extra modules, etc.
  c) A way to package it all up: App bundle on the Mac, single file
executable on Windows (statically linked? zip file, ???)

That third one -- (c) is inherently platform dependent -- and there "is
more than one way to do it" even on one platform. But it sure would be nice
if the API between a) b), and c)  could be unified so we could mix and
match different implementations.

And, of course, if cPython itself could be built in a way that makes
step(c) easier/less kludgy great!

-Chris


On Thu, May 28, 2015 at 9:54 AM, Donald Stufft  wrote:

>
>
> On May 28, 2015 at 12:24:42 PM, Chris Barker (chris.bar...@noaa.gov)
> wrote:
> > I'm confused:
> >
> > Doesn't py2exe (optionally) create a single file executable?
> >
> > And py2app on the Mac creates an application bundle, but that is
> > more-or-less the equivalent on OS-X (you may not even be able to have a
> > single file executable that can access the Window Manager, for instance)
> >
> > Depending on what extra packages you need, py2exe's single file doesn't
> > always work, but last I tried, it worked for a fair bit (I think all of
> the
> > stdlib).
> >
> > I don't know what PyInstaller or others create. And I have no idea if
> there
> > is a linux option -- but it seems like the standard of practice for an
> > application for linux is a bunch of files scattered over the system
> anyway
> > :-)
> >
> > Yes, the resulting exe is pretty big, but it does try to include only
> those
> > modules and packages that are used, and that kind of optimization could
> be
> > improved in any case.
> >
> > So is something different being asked for here?
>
> All of those solutions “work” to varying degrees of work, almost all of
> them rely
> on hacks in order to make things “work” because the ability to do it isn’t
> built
> into Python itself. If the critical pieces to execute in this way was
> built into
> Python itself, then those tools would work a whole lot better than they
> currently
> do.
>
> >
> > Barry Warsaw wrote:
> > >> I do think single-file executables are an important piece to Python's
> long-term
> > competitiveness.
> >
> > Really? It seems to me that desktop development is dying. What are the
> > critical use-cases for a single file executable?
>
> The desktop isn’t dying, Mobile is becoming a very important thing of
> course,
> but that’s just because people are using devices *more* to account for the
> use of Mobile, they aren’t really using their Desktop’s less.
>
> See:
> http://blogs.wsj.com/cmo/2015/05/26/mobile-isnt-killing-the-desktop-internet/
>
> >
> > And I'd note that getting a good way to use Python to develop for iOS,
> > Android, and Mobile Windows is FAR more critical! -- maybe that's the
> same
> > problem ?
> >
>
> It’s not the same problem, but it’s also not very relevant. Volunteer time
> isn’t
> fungible, you get what people are willing to work on regardless of whether
> it
> will help Python as a whole. It’s also not an either/or

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Sturla Molden]

Donald Stufft  wrote:

> Honestly, I’m on an OS that *does* ship Python (OS X) and part of me hopes
> that they stop shipping it. It’s very rare that someone ships Python as
> part of their OS without modifying it in some way, and those modifications
> almost always cause pain to some set of users (and since I work on pip, they
> tend to come to us with the weirdo problems). Case in point: Python on OS X
> adds some preinstalled software, but they put this pre-installed software 
> before
> site-packages in sys.path, so pip can’t upgrade those pre-installed software
> packages at all. 

Many Unix tools need Python, so Mac OS X (like Linux distros and FreeBSD)
will always need a system Python. Yes, it would be great if could be called
spython or something else than python. But the main problem is that it is
used by end-users as well, not just the operating system. 

Anyone who use Python on OSX should install their own Python. The system
Python should be left alone as it is. 

If the system Python needs updating, it is the responsibility of Apple to
distribute the upgrade. Nobody should attempt to use pip to update the
system Python. Who knows what side-effects it might have. Preferably pip
should have a check for it and bluntly refuse to do it.

Sturla

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Paul Moore]

On 28 May 2015 at 19:22, Chris Angelico  wrote:
>> Unfortunately (and believe me, I've been down this road many times) on
>> Windows *only* the exe format is a "first-class" executable.
>> Executable scripts and shebangs are very useful, but there are always
>> corner cases where they don't work *quite* like an exe. On Windows,
>> you have to be prepared to ship an exe if you want to compete with
>> languages that generate exes.
>
> I'm aware of that. When I said "a tiny stub", I was thinking in terms
> of a small executable. The idea is that its sole purpose is to locate
> Python someplace else, and chain to it; that has to be actual
> executable code, complete with the 512-byte "MZ" header and
> everything, to ensure compatibility. But it should be able to be
> small, tight, and easy to verify correctness of, so there aren't (in
> theory!) security exploits in the header itself.

OK, cool. I'm sort of working on that as a bit of a side project - a
tiny stub exe that you can prepend to a Python zipapp which runs it
via the standard embedding APIs. It's little more than an idea at the
moment, but I don't think it'll be too hard to implement...

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Parasa, Srinivas Vamsi]

Sorry for missing Julian's question. The GCC version used for the benchmarks is 
4.8.2
Will look into the discussion at 
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=39284 and will investigate it.

> Julian Taylor jtaylor.debian at googlemail.com 
> Thu May 28 13:30:59 CEST 2015
> won't this need python compiled with gcc 5.1 to have any effect? Which
> compiler version was used for the benchmark?
> the issue that negated most computed goto improvements
> (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=39284) was only closed
> very recently (r212172, 9f4ec746affbde1)

-Original Message-
From: Matthias Klose [mailto:d...@ubuntu.com] 
Sent: Thursday, May 28, 2015 5:01 AM
To: Parasa, Srinivas Vamsi; 'python-dev@python.org'
Subject: Re: [Python-Dev] Computed Goto dispatch for Python 2

On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote:
> Hi All,
> 
> This is Vamsi from Server Scripting Languages Optimization team at Intel 
> Corporation.
> 
> Would like to submit a request to enable the computed goto based dispatch in 
> Python 2.x (which happens to be enabled by default in Python 3 given its 
> performance benefits on a wide range of workloads). We talked about this 
> patch with Guido and he encouraged us to submit a request on Python-dev 
> (email conversation with Guido shown at the bottom of this email).
> 
> Attached is the computed goto patch (along with instructions to run) for 
> Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin  at 
> http://bugs.python.org/issue4753). We built and tested this patch for Python 
> 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP 
> CPU with 18 cores, hyper-threading off, turbo off).
> 
> Below is a summary of the performance we saw on the "grand unified python 
> benchmarks" suite (available at https://hg.python.org/benchmarks/). We made 3 
> rigorous runs of the following benchmarks. In each rigorous run, a benchmark 
> is run 100 times with and without the computed goto patch. Below we show the 
> average performance boost for the 3 rigorous runs.
> 
> Python 2.7.10 (original) vs Computed Goto performance Benchmark

-1

As Gregory pointed out, there are other options to build the interpreter, and 
we are missing data how these compare with your patch.

I assume, you tested with the Intel compiler, so it would be good to see 
results for other compilers as well (GCC, clang).  Please could you provide the 
data for LTO and profile guided optimized builds (maybe combined too)?  I'm 
happy to work with you on setting up these builds, but currently don't have the 
machine resources to do so myself.

If the benefits show up for these configurations too, then I'm +/-0 on this 
patch.

Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Brett Cannon]

On Thu, May 28, 2015 at 3:25 PM Chris Barker  wrote:

> OK, I'm really confused here:
>
> 1) what the heck is so special about go all of a sudden? People have been
> writing and deploying single file executables built with C and ++, and
> whatever else? forever. (and indeed, it was a big sticking point for me
> when I introduced python in my organization)
>

Because Go is much easier to use to write those CLI apps than C or C++. The
barrier now is low enough that the ease of development plus the ability to
do statically compiled binaries is enticing folks to drop Python and Ruby
for Go when making a CLI app.


>
> 2) Why the sudden interest in this as core a Python issue? I've been using
> Python for desktop apps, on primarily Windows and the Mac for years -- and
> had to deal with py2exe, py2app, etc. forever. And it has been a very very
> common question on the various mailing lists for ages: how do I deploy
> this? how do I make it easy to install? The answer from the developers of
> cPython itself has always been that that's a third party problem -- and go
> look for py2exe and friends to solve it. And that it is a solved-enough
> problem. The biggest "unsolved" issues are that you get a really  big
> application.
>

Anecdotal evidence suggests Go's user base has a decent amount of converts
from Python and it's currently the only language that seems to be siphoning
people out of our community. You do hear stories of people skipping Python
3 and going to Go, but considering how much more work that is than writing
Python 2/3 code I believe that typically happens when the organization
wanted to jump ship anyway and the Python 3 transition just gave them an
excuse to rewrite their stuff (plus we all know how enticing it can be to
play with a shiny new piece of tech if given the chance).


>
> Don't get me wrong -- I've wanted for years for it to be easier to deploy
> python-based apps as a single thinking for users to easily install and
> uninstall where they don't need to know it's python -- but what the heck is
> different now?
>

Active user loss where the biggest reason people are leaving that we can
actively fix now is easy app deployment (the other is performance; some
might argue concurrency but concurrent.futures and async/await water that
down somewhat for me).


>
> 3) There was mention of a platform-neutral way to do this. Isn't that
> simply impossible? The platforms are different in all the ways that matter
> for this problem: both technical differences, and conventions. Which isn't
> to say you couldn't have one API to produce a single "thing" executable, so
> it would look like one solution for multiple platforms to the user. But the
> end product should be (would have to be) a different beast altogether.
>

I think it's to have a single tool to do it for any platform, not to have
the technical nuts and bolts be the same necessarily. I think it's also to
figure out if there is anything the interpreter and/or stdlib can do to
facilitate this.

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (3.5): remove STORE_MAP, since it's unused

[Serhiy Storchaka]

On 28.05.15 22:40, benjamin.peterson wrote:

https://hg.python.org/cpython/rev/ac891c518d4e
changeset:   96342:ac891c518d4e
branch:  3.5
parent:  96339:6f05f83c7010
user:Benjamin Peterson 
date:Thu May 28 14:40:08 2015 -0500
summary:
   remove STORE_MAP, since it's unused


Wouldn't it break support of .pyc files compiled with older versions of 
Python 3?



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (3.5): remove STORE_MAP, since it's unused

[Benjamin Peterson]

On Thu, May 28, 2015, at 15:52, Serhiy Storchaka wrote:
> On 28.05.15 22:40, benjamin.peterson wrote:
> > https://hg.python.org/cpython/rev/ac891c518d4e
> > changeset:   96342:ac891c518d4e
> > branch:  3.5
> > parent:  96339:6f05f83c7010
> > user:Benjamin Peterson 
> > date:Thu May 28 14:40:08 2015 -0500
> > summary:
> >remove STORE_MAP, since it's unused
> 
> Wouldn't it break support of .pyc files compiled with older versions of 
> Python 3?

Those won't work anyway because the PYC magic version has changed.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Chris Barker]

On Thu, May 28, 2015 at 12:25 PM, Sturla Molden 
wrote:

> Many Unix tools need Python, so Mac OS X (like Linux distros and FreeBSD)
> will always need a system Python. Yes, it would be great if could be called
> spython or something else than python. But the main problem is that it is
> used by end-users as well, not just the operating system.
>

I think it's great for it to be used by end users as a system library /
utility. i.e. like you would a the system libc -- so if you can write a
little python script that only uses the stdlib -- you can simply deliver
that script.

But if you want to go an install a bunch of extra non-standard packages
(or, for heaven's sake, want a version with bug fixes!), they you really
are better off installing a new python you can control.

The system
> Python should be left alone as it is.
>

absolutely!

By the way, py2app will build an application bundle that depends on the
system python, indeed, that's all it will do if you run it with the system
python, as Apple has added some non-redistributable bits in there. But
things get kin dof confusing if you want to rely on non-system packages...

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Paul Moore]

On 28 May 2015 at 20:47, Brett Cannon  wrote:
> I think it's to have a single tool to do it for any platform, not to have
> the technical nuts and bolts be the same necessarily. I think it's also to
> figure out if there is anything the interpreter and/or stdlib can do to
> facilitate this.

Precisely. At the moment, the story seems to be "if you're on Windows,
use py2exe, if you're on OSX use py2app, or on Unix, ..., or..."

What would be a compelling story is "to build your app into a single
file executable, do "python -m build ". The machinery behind
the build can be as different as necessary - but being able to use the
same command on every platform is the goal.

zipapp is a start down that direction, but there's a *lot* more to be
done before we have a story good enough to address the growing trend
towards wanting a strong single-file deployment solution.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (3.5): remove STORE_MAP, since it's unused

[Ned Batchelder]

On 5/28/15 3:52 PM, Serhiy Storchaka wrote:

On 28.05.15 22:40, benjamin.peterson wrote:

https://hg.python.org/cpython/rev/ac891c518d4e
changeset:   96342:ac891c518d4e
branch:  3.5
parent:  96339:6f05f83c7010
user:Benjamin Peterson 
date:Thu May 28 14:40:08 2015 -0500
summary:
   remove STORE_MAP, since it's unused


Wouldn't it break support of .pyc files compiled with older versions 
of Python 3?
.pyc files are not compatible across versions of Python.  3.5 changed 
the meaning of the BUILD_MAP opcode, for example.


--Ned.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Sturla Molden]

On 28/05/15 21:37, Chris Barker wrote:


I think it's great for it to be used by end users as a system library /
utility. i.e. like you would a the system libc -- so if you can write a
little python script that only uses the stdlib -- you can simply deliver
that script.


No it is not, because someone will be 'clever' and try to upgrade it 
with pip or install packages into it.


Sturla

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Nick Coghlan]

On 29 May 2015 05:25, "Chris Barker"  wrote:
>
> OK, I'm really confused here:
>
> 1) what the heck is so special about go all of a sudden? People have been
writing and deploying single file executables built with C and ++, and
whatever else? forever. (and indeed, it was a big sticking point for me
when I introduced python in my organization)

For scientific Python folks, the equivalent conversations I have are about
Julia.

If you're not used to thinking of Python's competitive position as "best
orchestration language, solid competitor in any given niche", then the rise
of niche specific competitors like Go & Julia can feel terrifying, as the
relatively narrow user base changes the trade-offs you can make in the
language & ecosystem design to better optimise them for that purpose.

We don't need to debate the accuracy of that perception of risk, though. If
it motivates folks to invest time & energy into providing one-obvious-way
to do cross-platform single-file distribution, lower barriers to adoption
for PyPy, and work on a Rust-style memory ownership based model for
concurrent execution of subinterpreters across multiple cores, then the
community wins regardless :)

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Glenn Linderman]

On 5/28/2015 12:26 PM, Paul Moore wrote:

On 28 May 2015 at 19:22, Chris Angelico  wrote:

Unfortunately (and believe me, I've been down this road many times) on
Windows *only* the exe format is a "first-class" executable.
Executable scripts and shebangs are very useful, but there are always
corner cases where they don't work *quite* like an exe. On Windows,
you have to be prepared to ship an exe if you want to compete with
languages that generate exes.

I'm aware of that. When I said "a tiny stub", I was thinking in terms
of a small executable. The idea is that its sole purpose is to locate
Python someplace else, and chain to it; that has to be actual
executable code, complete with the 512-byte "MZ" header and
everything, to ensure compatibility. But it should be able to be
small, tight, and easy to verify correctness of, so there aren't (in
theory!) security exploits in the header itself.

OK, cool. I'm sort of working on that as a bit of a side project - a
tiny stub exe that you can prepend to a Python zipapp which runs it
via the standard embedding APIs. It's little more than an idea at the
moment, but I don't think it'll be too hard to implement...


Paul,

I've been using zipapps for some years now, and it would be really cool 
to have what I think you are talking about here:


1) Build zipapp as normal.  It likely depends on some minimal Python 
version.
2) Prepend stub .exe (Windows) or !# line (Unix) that declares a version 
of Python to actually use. This can be as broad as Python 2 or Python 3 
(not sure how to specify that either works, especially on Windows), or 
more detailed/restrictive by specifying  2.n or 3.n.  On Windows, it 
would find the newest qualifying installation and run it, and Unix, the 
symlinks are set up to do that already, is my understanding, so the 
proper !# line does that for you.


This would be something I could use and benefit from immediately upon it 
being available, so I laud your idea, and hope you have a successful 
implementation, and look forward to using it.  It would largely replace 
the need for the py.exe launcher for some classes of applications.


Of course, per other disccusions, this doesn't solve the problem for:

A) machine without Python installed
B) programs that need binary extensions

Other discussions have suggested:

3) The stub could offer to download and install Python

A corollary:

4) The stub could offer to download and install the needed binary 
extensions as well as Python. This would require the installation 
uniformity of something like pip, so perhaps would be restricted to 
extensions available via pip.  And it would be much enhanced by some 
technique where the zipapp would contain metadata readable by the stub, 
that would declare the list of binary extensions required. Or, of 
course, it could even declare non-binary extension that are not packaged 
with the zipapp, if the process is smooth, the modules available via 
pip, etc., as a tradeoff.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Matthias Klose]

On 05/28/2015 06:13 PM, Barry Warsaw wrote:
> Go seems to be popular where I work.  It is replacing Python in a number of
> places, although Python (and especially Python 3) is still a very important
> part of our language toolbox.
> 
> There are several reasons why Go is gaining popularity.  Single-file
> executables is definitely a reason; it makes deployment very easy, even if it
> increases the maintenance burden (e.g. without shared libraries, you have
> multiple copies of things so when a security fix is required for one of those
> things you have to recompile the world).

And the very same place where you are working is investing in getting shared
libraries working for Go.  Single binaries may be popular for distributing end
user applications, but definitely not for distributing a core OS or a SDK.
Sorry, you didn't yet arrive in distro land ...

Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Chris Barker]

On Thu, May 28, 2015 at 10:32 AM, Ryan Gonzalez  wrote:

> py2exe tends to invoke DLL hell if you have various versions of VS or
> Office or both installed. Because Windows.
>

uh, yes -- Windows applications invoke dll hell..nothign to be done
about that!

-Chris


>
> On May 28, 2015 11:23:57 AM CDT, Chris Barker 
> wrote:
>
>> I'm confused:
>>
>> Doesn't py2exe (optionally) create a single file executable?
>>
>> And py2app on the Mac creates an application bundle, but that is
>> more-or-less the equivalent on OS-X (you may not even be able to have a
>> single file executable that can access the Window Manager, for instance)
>>
>> Depending on what extra packages you need, py2exe's single file doesn't
>> always work, but last I tried, it worked for a fair bit (I think all of the
>> stdlib).
>>
>> I don't know what PyInstaller or others create. And I have no idea if
>> there is a linux option -- but it seems like the standard of practice for
>> an application for linux is a bunch of files scattered over the system
>> anyway :-)
>>
>> Yes, the resulting exe is pretty big, but it does try to include only
>> those modules and packages that are used, and that kind of optimization
>> could be improved in any case.
>>
>> So is something different being asked for here?
>>
>> Barry Warsaw wrote:
>> >> I do think single-file executables are an important piece to Python's 
>> >> long-term
>> competitiveness.
>>
>> Really? It seems to me that desktop development is dying. What are the
>> critical use-cases for a single file executable?
>>
>> And I'd note that getting a good way to use Python to develop for iOS,
>> Android, and Mobile Windows is FAR more critical!  -- maybe that's the same
>> problem ?
>>
>> -Chris
>>
>>
>> On Thu, May 28, 2015 at 8:39 AM, Donald Stufft  wrote:
>>
>>>
>>>
>>> On May 28, 2015 at 11:30:37 AM, Steve Dower (steve.do...@microsoft.com)
>>> wrote:
>>> > Donald Stufft wrote:
>>> > > Well Python 3.4.3 binary is 4kb for me, so you'd have that + your
>>> 1KB Python script + whatever
>>> > other pieces you need.
>>> >
>>> > For contrast, here are the things you need on Windows to be able to
>>> get to an interactive
>>> > prompt (I don't know how other platforms get this down to 4KB...):
>>> >
>>> > * python.exe (or some equivalent launcher) 39KB
>>> > * python35.dll 3,788KB
>>> > * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not
>>> redistributable
>>> > so doesn't count here)
>>> > * 26 files in Lib 343KB
>>> >
>>> > This gets you to ">>>", and basically everything after that is going
>>> to fail for some reason.
>>> > That's an unavoidable 4,257KB.
>>> >
>>> > The rest of the stdlib adds another ~16MB once you exclude the test
>>> suite, so a fully functioning
>>> > Python is not cheap. (Using compressed .pyc's in a zip file can make a
>>> big difference here
>>> > though, assuming you're willing to trade CPU for HDD.)
>>> >
>>> > Cheers,
>>> > Steve
>>> >
>>> >
>>>
>>> You don’t need a "fully functioning Python" for a single file binary,
>>> you only
>>> need enough to actually run your application. For example, if you're
>>> making
>>> an application that can download files over HTTP, you don't need to
>>> include
>>> parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv,
>>> email,
>>> mailcap, mailbox, imaplib, nntplib, etc.
>>>
>>> Of course deciding which pieces you include in the zip file you're
>>> appending
>>> to the end of Python is up to whatever tool builds this executable which
>>> doesn't need to be part of Python itself. If Python itself gained the
>>> ability
>>> to operate in that manner than third party tools could handle trying to
>>> do the
>>> optimizations where it only includes the things it actually needs in the
>>> stdlib
>>> and excludes things it doesn't. The key thing here is that since you're
>>> doing
>>> a single file binary, you don't need to have a Python which is suitable
>>> to
>>> execute random Python code, you only need one that is suitable to
>>> execute this
>>> particular code so you can specialize what that includes.
>>>
>>> ---
>>> Donald Stufft
>>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>>
>>>
>>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>>>
>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R(206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115   (206) 526-6317   main reception
>>
>> chris.bar...@noaa.gov
>>
>> --
>>
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>>
>>
> --
> Sent from my An

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Donald Stufft]

On May 28, 2015 at 5:50:32 PM, Matthias Klose (d...@ubuntu.com) wrote:
> On 05/28/2015 06:13 PM, Barry Warsaw wrote:
> > Go seems to be popular where I work. It is replacing Python in a number of
> > places, although Python (and especially Python 3) is still a very important
> > part of our language toolbox.
> >
> > There are several reasons why Go is gaining popularity. Single-file
> > executables is definitely a reason; it makes deployment very easy, even if 
> > it
> > increases the maintenance burden (e.g. without shared libraries, you have
> > multiple copies of things so when a security fix is required for one of 
> > those
> > things you have to recompile the world).
>  
> And the very same place where you are working is investing in getting shared
> libraries working for Go. Single binaries may be popular for distributing end
> user applications, but definitely not for distributing a core OS or a SDK.
> Sorry, you didn't yet arrive in distro land ...
>  
>

I don’t think anyone is claiming that single file should be the *only* way, just
that for a sizable set of people it is a very attractive way.

---  
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Paul Sokolovsky]

Hello,

On Thu, 28 May 2015 23:48:59 +0200
Matthias Klose  wrote:

[]

> And the very same place where you are working is investing in getting
> shared libraries working for Go.  Single binaries may be popular for
> distributing end user applications, but definitely not for
> distributing a core OS or a SDK. Sorry, you didn't yet arrive in
> distro land ...

Of course it did. Like, Ubuntu 14.04LTS ships Go 1.2. No, it starts
with the fact that when you don't have Go installed and type "go", it
suggests to install gccgo, which just segfaults on running. Then you
figure out that you need to install "golang", and that's 1.2, and a lot
of things simply don't work with that version, like "go get" reports
that a package not found, while it perfectly exists. So, let Go stay
what it is - a corporate toy lingo for press-releases. That's until
Google has thought that it generated enough buzz and it's time to shut
it down like their numerous other projects. (Isn't Go old already and
"everyone" uses Rust?)


-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Usability of the limited API

[Nick Coghlan]

On 29 May 2015 01:04, "Steve Dower"  wrote:
>
> Paul Moore wrote:
> > On 28 May 2015 at 15:28, Steve Dower  wrote:
> >> I don't have the issue number handy, but it should be near the top of
> >> the recently modified list.
> >
> > I recall seeing that issue. I'm fine with that - getting the two in
sync is
> > obviously worth doing (and clearly in hand). I'm personally not sure
whether
> > automating the exposure of symbols is the correct approach, as I'm not
sure
> > people typically even consider the stable API when adding functions. Is
the
> > default (what you get if somebody just blindly adds a symbol with no
thought for
> > the stable API) to expose it or not? If the default is that it's not
exposed,
> > then automation seems reasonable, otherwise I'm not so sure.
>
> Now I'm at my desk, the issue is http://bugs.python.org/issue23903
>
> I believe new symbols are considered stable by default, so perhaps we
actually want a test that will generate a C file that imports everything
"stable" and will break the buildbots if someone adds something new without
explicitly adding it to the list of stable functions?

The stable CPython ABI is actually tracked on
http://upstream-tracker.org/versions/python_stable_api.html

Ideally we'd be running those checks automatically as part of our own QA
with http://ispras.linuxbase.org/index.php/ABI_compliance_checker, similar
to Antoine's regular refleak checks.

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Ian Cordasco]

On Thu, May 28, 2015 at 5:08 PM, Paul Sokolovsky  wrote:
> Hello,
>
> On Thu, 28 May 2015 23:48:59 +0200
> Matthias Klose  wrote:
>
> []
>
>> And the very same place where you are working is investing in getting
>> shared libraries working for Go.  Single binaries may be popular for
>> distributing end user applications, but definitely not for
>> distributing a core OS or a SDK. Sorry, you didn't yet arrive in
>> distro land ...
>
> Of course it did. Like, Ubuntu 14.04LTS ships Go 1.2. No, it starts
> with the fact that when you don't have Go installed and type "go", it
> suggests to install gccgo, which just segfaults on running. Then you
> figure out that you need to install "golang", and that's 1.2, and a lot
> of things simply don't work with that version, like "go get" reports
> that a package not found, while it perfectly exists. So, let Go stay
> what it is - a corporate toy lingo for press-releases. That's until
> Google has thought that it generated enough buzz and it's time to shut
> it down like their numerous other projects. (Isn't Go old already and
> "everyone" uses Rust?)
>
>
> --
> Best regards,
>  Paul  mailto:pmis...@gmail.com
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.com

Note that as much as I love Rust, it still isn't the replacement for
Go. It doesn't have a stable ABI so if you distribute a binary and
that person has a different version of Rust 1.x installed, it won't be
guaranteed to work (and, at this point, probably won't work anyway).
Go is just more popular because it's been around longer and it (as far
as a single developer is concerned) gets rid of the dependency mess.
That's why developers like it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Nick Coghlan]

On 29 May 2015 00:52, "Paul Moore"  wrote:
>
> +1. The new embeddable Python distribution for Windows is a great step
> forward for this. It's not single-file, but it's easy to produce a
> single-directory self-contained application with it. I don't know if
> there's anything equivalent for Linux/OSX - maybe it's something we
> should look at for them as well (although the whole "static binaries"
> concept seems to be fairly frowned on in the Unix world, from what
> I've seen).

Correct - in the absence of the capacity to rebuild and redeploy the world
at the click of a button, widespread deployment of static binaries poses an
appallingly high security risk. It isn't an accident that Linux container
orchestration is co-evolving with Linux container formats.

Those efforts are mostly focused on network services & GUI applications,
though. For portable console applications, Go is still one of the nicest
options currently available, as the relatively limited ability to
interoperate with the system provided C/C++ libraries makes it much harder
to create dependencies between the app and the platform. It's similar to
Java in that respect, but without the dependency on a large language
runtime like the JVM.

In that vein, it might be interesting to see what could be done with
MicroPython in terms of providing a lightweight portable Python runtime
without CPython's extensive integration with the underlying OS.

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Barry Warsaw]

On May 28, 2015, at 11:48 PM, Matthias Klose wrote:

>And the very same place where you are working is investing in getting shared
>libraries working for Go.  Single binaries may be popular for distributing end
>user applications, but definitely not for distributing a core OS or a SDK.

Yep, I mentioned this in an earlier message (probably buried in the
centithread avalanche).  Both static and dynamic linking have valid use
cases.

Cheers,
-Barry

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] time-based releases (was Re: Preserving the definition order of class namespaces.)

[Antoine Pitrou]

On Thu, 28 May 2015 08:48:11 +1000
Nick Coghlan  wrote:
> 
> I just remembered one of the biggest causes of pain: Windows binaries for
> projects that aren't using the stable ABI. It used to regularly take 6+
> months for the Windows ecosystem to catch up after each 2.x release.

You're right, compatibility of C extension builds under Windows is
pretty much the killer.

> After all, the real difference between the alphas and the final releases
> isn't about anything *we* do, it's about the testing *other people* do that
> picks up gaps in our test coverage. A gated trunk makes it more feasible
> for other projects to do continuous integration against it.

Long ago (before I became a core developer) we had "community
buildbots" for that. They didn't receive any attention or maintenance
from third-party projects.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Paul Sokolovsky]

Hello,

On Fri, 29 May 2015 08:38:44 +1000
Nick Coghlan  wrote:

[]

> In that vein, it might be interesting to see what could be done with
> MicroPython in terms of providing a lightweight portable Python
> runtime without CPython's extensive integration with the underlying
> OS.

Thanks for mentioning that. That's definitely what I have on my mind,
actually I wanted to do long ago a Lua-esque hack of being able to cat
together an interpreter and a script, so resulting executable would
just run the script. What stopped me is that it would be Lua-esque-ly
useless, as how much useful one can do with a bare script without
dependencies?

And MicroPython definitely has some puzzle pieces for a generic
solution, but so far not a complete picture:

1. There're frozen modules support, but they're that - modules,
packages not supported for freezing so far.

2. Then frozen modules require recompiling, and that's not real-world
scalable.

3. Standard library is already "distributed" (vs CPython's monolithic),
but half of modules are dummy so far.


That said, making a demo of self-contained webapp server in 350-400K is
definitely on my TODO list (package support for frozen modules is the
only blocker for that).


> Cheers,
> Nick.



-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Nick Coghlan]

On 29 May 2015 2:16 am, "Barry Warsaw"  wrote:
>
> Go seems to be popular where I work.  It is replacing Python in a number
of
> places, although Python (and especially Python 3) is still a very
important
> part of our language toolbox.
>
> There are several reasons why Go is gaining popularity.  Single-file
> executables is definitely a reason; it makes deployment very easy, even
if it
> increases the maintenance burden (e.g. without shared libraries, you have
> multiple copies of things so when a security fix is required for one of
those
> things you have to recompile the world).
>
> Start up times and memory footprint are also factors.  Probably not much
to be
> done about the latter, but perhaps PEP 432 can lead to improvements in the
> former.  (Hey Nick, I'm guessing you'll want to bump that one back to
3.6.)

Yep. I got the feature branch mostly working again just after PyCon (since
several folks expressed interest in helping with it), and thanks to Eric
Snow, the biggest blocker to further progress (splitting the import system
initialisation into two distinct phases) has already been addressed for 3.5
(that's not merged into the feature branch in my sandbox repo yet, though).

PEP 432 itself isn't likely to change startup time for the full interpreter
runtime very much (as it's mostly about rearranging how we call the
existing setup steps, rather than changing the steps themselves), but
having more of the C API available earlier in the bootstrapping cycle will
hopefully lay a foundation for future improvements.

The intent is also to make embedding *much* easier, and have it be trivial
to skip initialising any subsystems that a given application doesn't need.

> Certainly better support for multi-cores comes up a lot.  It should be a
SMoE
> to just get rid of the GIL once and for all .

Eric's been looking into this as well, and we think there's a plausible
path forward based on changing the way subinterpreters work such that the
GIL can be changed to a read/write lock, and each subinterpreter gets its
own Local Interpreter Lock.

Expect to hear more on that front before too long :)

> One thing I've seen more than once is that new development happens in
Python
> until the problem is understood, then the code is ported to Go.  Python's
> short path from idea to working code, along with its ability to quickly
morph
> as requirements and understanding changes, its batteries included
philosophy,
> and its "fits-your-brain" consistency are its biggest strengths!

Right, Go is displacing C/C++ in that regard (moreso than Python itself),
and now that Rust has hit 1.0, I expect we'll see it becoming another
contender for this task. Rust's big advantage over Go in that regard is
being compatible with the C/C++ ecosystem, including Python's cffi.

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Donald Stufft]

On May 28, 2015 at 7:40:26 PM, Nick Coghlan (ncogh...@gmail.com) wrote:
> >
> > One thing I've seen more than once is that new development happens  
> in Python
> > until the problem is understood, then the code is ported to Go.  
> Python's
> > short path from idea to working code, along with its ability  
> to quickly morph
> > as requirements and understanding changes, its batteries  
> included philosophy,
> > and its "fits-your-brain" consistency are its biggest strengths!  
>  
>  
> Right, Go is displacing C/C++ in that regard (moreso than Python  
> itself), and now that Rust has hit 1.0, I expect we'll see it becoming  
> another contender for this task. Rust's big advantage over Go  
> in that regard is being compatible with the C/C++ ecosystem,  
> including Python's cffi.
>  

I’m not sure if I’m reading this right or not, but just to be clear, I’ve
seen a number of people express the sentiment that they are switching from
Python to Go and that the deployment story is one of the reasons. It’s not
just people switching from C/C++.

---  
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Can someone configure the buildbots to build the 3.5 branch?

[Larry Hastings]

The buildbots currently live in a state of denial about the 3.5 branch.  
Could someone whisper tenderly in their collective shell-like ears so 
that they start building 3.5, in addition to 3.4 and trunk?


Thank you,


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Victor Stinner]

2015-05-28 18:07 GMT+02:00 Guido van Rossum :
> This patch could save companies like Dropbox a lot of money. We run a ton of
> Python code in large datacenters, and while we are slow in moving to Python
> 3, we're good at updating to the latest 2.7.

I'm not sure that backporting optimizations would motivate companies
like Dropbox to drop Python 2. It's already hard to convince someone
to migrate to Python 3.

Why not continue to enhance Python 3 instead of wasting our time with
Python 2? We have limited resources in term of developers to maintain
Python.

(I'm not talking about fixing *bugs* in Python 2 which is fine with me.)

--

By the way, I just wrote sixer, a new tool to generate patches to port
OpenStack to Python 3 :-)
https://pypi.python.org/pypi/sixer

It's based on regex, so it's less reliable than 2to3, 2to6 or
modernize, but it's just enough for my specific use case. On
OpenStack, it's not possible to send one giant patch "hello, this is
python 3". Code is modified by small and incremental changes.

Come on in the Python 3 world and... always look on the bright side of
life ( https://www.youtube.com/watch?v=VOAtCOsNuVM )!

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Larry Hastings]

On 05/28/2015 05:58 PM, Victor Stinner wrote:

Why not continue to enhance Python 3 instead of wasting our time with
Python 2? We have limited resources in term of developers to maintain
Python.


Uh, guys, you might as well call off the debate.  Benjamin already 
checked it in.


   https://hg.python.org/cpython/rev/17d3bbde60d2


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Guido van Rossum]

On Thu, May 28, 2015 at 5:58 PM, Victor Stinner 
wrote:

> 2015-05-28 18:07 GMT+02:00 Guido van Rossum :
> > This patch could save companies like Dropbox a lot of money. We run a
> ton of
> > Python code in large datacenters, and while we are slow in moving to
> Python
> > 3, we're good at updating to the latest 2.7.
>
> I'm not sure that backporting optimizations would motivate companies
> like Dropbox to drop Python 2. It's already hard to convince someone
> to migrate to Python 3.
>
> Why not continue to enhance Python 3 instead of wasting our time with
> Python 2? We have limited resources in term of developers to maintain
> Python.
>

As a matter of fact (unknown to me earlier) Dropbox is already using a
private backport of this patch. So it won't affect Dropbox's decision
anyway. But it might make life easier for other, smaller companies that
don't have the same resources.

However this talk of "wasting our time with Python 2" needs to stop, and if
you think that making Python 2 less attractive will encourage people to
migrate to Python 3, think again. Companies like Intel are *contributing*
by offering this backport up publicly.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Ryan Gonzalez]

YESSS!!!

On Thu, May 28, 2015 at 8:09 PM, Larry Hastings  wrote:

>  On 05/28/2015 05:58 PM, Victor Stinner wrote:
>
> Why not continue to enhance Python 3 instead of wasting our time with
> Python 2? We have limited resources in term of developers to maintain
> Python.
>
>
> Uh, guys, you might as well call off the debate.  Benjamin already checked
> it in.
>
> https://hg.python.org/cpython/rev/17d3bbde60d2
>
>
> */arry*
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>
>


-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something’s wrong.
http://kirbyfan64.github.io/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Eric Snow]

On Thu, May 28, 2015 at 5:01 PM, Paul Sokolovsky  wrote:
> That said, making a demo of self-contained webapp server in 350-400K is
> definitely on my TODO list (package support for frozen modules is the
> only blocker for that).

It may be worth taking this over to import-...@python.org for more discussion.

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Gregory P. Smith]

On Thu, May 28, 2015 at 6:10 PM Larry Hastings  wrote:

> On 05/28/2015 05:58 PM, Victor Stinner wrote:
>
> Why not continue to enhance Python 3 instead of wasting our time with
> Python 2? We have limited resources in term of developers to maintain
> Python.
>
>
> Uh, guys, you might as well call off the debate.  Benjamin already checked
> it in.
>
> https://hg.python.org/cpython/rev/17d3bbde60d2
>
>
Thanks Benjamin! :)


>
>
> */arry*
>  ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Thomas Wouters]

On Thu, May 28, 2015 at 6:13 PM, Barry Warsaw  wrote:

> Go seems to be popular where I work.  It is replacing Python in a number of
> places, although Python (and especially Python 3) is still a very important
> part of our language toolbox.
>
> There are several reasons why Go is gaining popularity.  Single-file
> executables is definitely a reason; it makes deployment very easy, even if
> it
> increases the maintenance burden (e.g. without shared libraries, you have
> multiple copies of things so when a security fix is required for one of
> those
> things you have to recompile the world).
>
> Start up times and memory footprint are also factors.  Probably not much
> to be
> done about the latter, but perhaps PEP 432 can lead to improvements in the
> former.  (Hey Nick, I'm guessing you'll want to bump that one back to 3.6.)
>
> Certainly better support for multi-cores comes up a lot.  It should be a
> SMoE
> to just get rid of the GIL once and for all .
>
> One thing I've seen more than once is that new development happens in
> Python
> until the problem is understood, then the code is ported to Go.  Python's
> short path from idea to working code, along with its ability to quickly
> morph
> as requirements and understanding changes, its batteries included
> philosophy,
> and its "fits-your-brain" consistency are its biggest strengths!
>
> On May 28, 2015, at 10:37 AM, Donald Stufft wrote:
>
> >I think docker is a pretty crummy answer to Go’s static binaries. What I
> would
> >love is for Python to get:
> >
> >* The ability to import .so modules via zipzimport (ideally without a
> >temporary   directory, but that might require newer APIs from libc and
> such).
>
> +1 - Thomas Wouters mentioned at the language summit some work being done
> on
> glibc to add dlopen_from_memory() (sp?) which would allow for loading .so
> files directly from a zip.  Not sure what the status is of that, but it
> would
> be a great addition.
>

It's dlopen_with_offset:
https://sourceware.org/bugzilla/show_bug.cgi?id=11767. There's also a patch
that's sort-of dlopen_from_memory (dlopen_ehdr), but it requires a lot of
manual setup to map the right bits to the right places; dlopen_with_offset
is a lot simpler.

Building a Python application into a single file doesn't require
dlopen_with_offset, *iff* you build everything from source. It's not easy
to do this -- Python's setup.py and third-party's uses of distutils don't
allow this -- but it's mostly possible using the old Modules/Setup file.
(Or you could do what we routinely do at Google with third-party packages
and re-implement the build in your own build system ;P)



>
> >* The ability to create a “static” Python that links everything it needs
> into
> >the binary to do a zipimport of everything else (including the stdlib).
>
>
This is possible (with some jumping through hoops) using Modules/Setup and
some post-processing of the standard library. It would be a lot easier if
we got rid of distutils for building Python (or for everything) -- or made
it output a Modules/Setup-like file :) (For those who don't remember,
Modules/Setup was the file we used to build stdlib extension modules before
we had distutils, and it's parsed and incorporated into the regular
Makefile. It can build both static and dynamic extension modules.)


> +1
>
> >*The ability to execute a zipfile that has been concat onto the end of
> the
> >Python binary.
>

This is already possible, just not with the regular 'python' binary. All it
takes is fifty lines of C or so, a tiny application that embeds Python,
sets sys.path[0] to argv[0], and uses the runpy module to execute something
from the ZIP file. There are some issues with this approach (like what
sys.executable should be :) but they're mostly cosmetic


>
> +1
>
> >I think that if we get all of that, we could easily create a single file
> >executable with real, native support from Python by simply compiling
> Python
> >in that static mode and then appending a zip file containing the standard
> >library and any other distributions we need to the end of it.
> >
> >We’d probably want some more quality of life improvements around accessing
> >resources from within that zip file as well, but that can be done as a
> >library easier than the above three things can.
>
> E.g. you really should be using the pkg_resources APIs for loading
> resources
> from your packages, otherwise you're gonna have problems with zip
> executables.  We've talked before about adopting some of these APIs into
> Python's stdlib.  pkgutil is a start, and the higher level APIs from
> pkg_resources should probably go there.
>
> Cheers,
> -Barry
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/thomas%40python.org
>



-- 
Thomas Wouters 

Hi! I'm an email virus! Think twice before sending your email to help me
sp

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Chris Barker]

Getting lost as to what thread this belongs in...

But another tack to take toward a single executable is Cython's embedding
option:

https://github.com/cython/cython/wiki/EmbeddingCython

This is a quick and dirty way to create a C executable that will then run
the cythonized code, all linked to the python run time.

At this point, it still requires the python shared lib, and I think any
other compiled extension is shared, too. And if you run Cython on all the
python code and modules you use, you'll have a LOT of shared libs. But
perhaps one could re-do the linking step of all that and get a single
compiled exe.

and IIUC, the way Windows dll hell works, if you stuff the whole pile into
one dir -- you will get a single executable directory, if not a single file.

and about a 2X performance boost, as well, when you cythonize pure Python,
at least in my limited experience.

Just a thought.

-Chris








-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Obtaining stack-frames from co-routine objects

[Ben Leslie]

Hi all,

Apologies in advance; I'm not a regular, and this may have been
handled already (but I couldn't find it when searching).

I've been using the new async/await functionality (congrats again to
Yury on getting that through!), and I'd like to get a stack trace
between the place at which blocking occurs and the outer co-routine.

For example, consider this code:

"""
async def a():
await b()

async def b():
await switch()

@types.coroutine
def switch():
yield

coro_a = a()
coro_a.send(None)
"""

At this point I'd really like to be able to somehow get a stack trace
similar to:

test.py:2
test.py:4
test.py:9

Using the gi_frame attribute of coro_a, I can get the line number of
the outer frame (e.g.: line 2), but from there there is no way to
descend the stack to reach the actual yield point.

I thought that perhaps the switch() co-routine could yield the frame
object returned from inspect.currentframe(), however once that
function yields that frame object has f_back changed to None.

A hypothetical approach would be to work the way down form the
outer-frame, but that requires getting access to the co-routine object
that the outer-frame is currently await-ing. Some hypothetical code
could be:

"""
def show(coro):
print("{}:{}".format(coro.gi_frame.f_code.co_filename,
coro.gi_frame.f_lineno))
if dis.opname[coro.gi_code.co_code[coro.gi_frame.f_lasti + 1]] ==
'YIELD_FROM':
show(coro.gi_frame.f_stack[0])
"""

This relies on the fact that an await-ing co-routine will be executing
a YIELD_FROM instruction. The above code uses a completely
hypothetical 'f_stack' property of frame objects to pull the
co-routine object which a co-routine is currently await-ing from the
stack. I've implemented a proof-of-concept f_stack property in the
frameobject.c just to test out the above code, and it seems to work.

With all that, some questions:

1) Does anyone else see value in trying to get the stack-trace down to
the actual yield point?
2) Is there a different way of doing it that doesn't require changes
to Python internals?
3) Assuming no to #2 is there a better way of getting the information
compared to the pretty hacking byte-code/stack inspection?

Thanks,

Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python 3.5 schedule addendum adding a new Python 3.5.0 beta, this weekend

[Larry Hastings]

On behalf of the Python 3.5 release team:

Due to a particularly bad bug ( http://bugs.python.org/issue24285 ), 
we're going to issue a new beta of Python 3.5 this weekend.  This will 
not change the rest of the schedule; it'll just bump the remaining beta 
numbers up by 1.  Thus the schedule is now as follows:


   - 3.5.0 beta 1: May 24, 2015
   - 3.5.0 beta 2: May 31, 2015
   - 3.5.0 beta 3: July 5, 2015
   - 3.5.0 beta 4: July 26, 2015
   - 3.5.0 candidate 1: August 9, 2015
   - 3.5.0 candidate 2: August 23, 2015
   - 3.5.0 candidate 3: September 6, 2015
   - 3.5.0 final: September 13, 2015


May you live in interesting times,


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] time-based releases (was Re: Preserving the definition order of class namespaces.)

[Nick Coghlan]

On 29 May 2015 9:17 am, "Antoine Pitrou"  wrote:
>
> On Thu, 28 May 2015 08:48:11 +1000
> Nick Coghlan  wrote:

> > After all, the real difference between the alphas and the final releases
> > isn't about anything *we* do, it's about the testing *other people* do
that
> > picks up gaps in our test coverage. A gated trunk makes it more feasible
> > for other projects to do continuous integration against it.
>
> Long ago (before I became a core developer) we had "community
> buildbots" for that. They didn't receive any attention or maintenance
> from third-party projects.

Right, but it's hard to integrate against trunk when trunk itself may be
broken. If we had a way of publishing "known good" commit hashes that
passed the test suite on all the stable buildbots, that could potentially
provide a basis for integration testing without needing to switch to merge
gating first.

Do we know if BuildBot offers an interface for that?

Cheers,
Nick.

>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com