Re: Packaging pypy
Hi Matthias (2011.11.29_14:21:18_+0200) > maybe for binary packages, but there is no reason why a pypy extension > couldn't > be built from the same source packages. Could you summarize why it needs to > be > a separate stack? Sorry, discussion on IRC yesterday, made me realise that I was unclear. I think we should package it on it's own for now (a separate, empty, stack). Working out the details of how to build binary packages for it and cpython requires a little thought still, and we can modify the PyPy package once we've worked that out. Providing a PyPy + virtualenv will probably give most people who desperately want PyPy, now, something to play with (and it could make it into the upcoming Ubuntu release). (This needs virtualenv 1.6.1, #632982) As to the integration: We had a reasonable discussion on #debian-python. To summarise the major points: How to tell dh_python2 about it: We'd probably want something like X-PyPy-Version to state PyPy support. The logical follow up is, how about X-Alternate-Python-Stacks ? It's tempting to consider other alternative Python implementations. But I don't think any of them besides PyPy are anywhere near a point where we could consider packaging modules for them. I think we may want to express PyPy version restrictions for packages which could only build their C extensions in a PyPy >= X (PyPy is still expanding Cpyext, adding more of the Python C-API) For pure python modules, it's probably not useful, but is an appropriate way to state PyPy support How to handle dependencies between modules: === Does every binary package that supports cpython and pypy have to have a versioned dependency for pypy support for every dependency? One solution is Provides pypy-foo. Does that need to be pypy1.7-foo ? After all, C-extensions are going to have to be built for each pypy version, and we'll have to do transitions, just like cpython. (Although, less painful, presumably, as PyPy isn't undergoing language changes, like cpython - for 2.x) Or we just live with a fair amount of PyPy module brokenness while we are bootstrapping the modules. XB-PyPy-Versions? SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 465 6908 C: +27 72 419 8559 UCT: x3127 -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/2030165448.gf29...@bach.rivera.co.za
Re: Packaging pypy
Hi Barry (2011.11.29_18:00:07_+0200) > While chatting about this in irc (#debian-python on oftc), I mistakenly > thought that PyPy supported PEP 3147, but I think it's only PEP 3149. 3147 > shouldn't be that difficult to support - what is your thought on adding that > to PyPy? It would mean one less symlink farm. One symlink farm for the entire PyPy (!py3k PyPy) sounds good. It'd even be very unexpected for a pure-python module to have code differences to have code differences between PyPy and cPython. But it's quite plausible (although probably still rare-ish) for modules with extensions to have variation in pure-python code. SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 465 6908 C: +27 72 419 8559 UCT: x3127 -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/2030165857.gg29...@bach.rivera.co.za
Re: Packaging pypy
Hello, Am 30.11.2011 17:54, schrieb Stefano Rivera: We'd probably want something like X-PyPy-Version to state PyPy support. The logical follow up is, how about X-Alternate-Python-Stacks ? I was contemplating that long term, once Nuitka is really a useful accelerator, it should be allowed to compile modules, whole packages, or programs at install time. Will that fit into this? Yours, Kay -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4ed67d6d.7000...@gmx.de
Re: Packaging pypy
Hi Kay (2011.11.30_21:01:01_+0200) > >The logical follow up is, how about X-Alternate-Python-Stacks ? > > I was contemplating that long term, once Nuitka is really a useful > accelerator, it should be allowed to compile modules, whole > packages, or programs at install time. > > Will that fit into this? I know little about Nuitka, but I don't think so. Nuitka isn't something that python modules will want to support at build time, is it? As I understand it, it could compile *applications* to C using Python's C API. What could it usefully do for a pure-python module package? SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 465 6908 C: +27 72 419 8559 UCT: x3127 -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/2030195946.gh29...@bach.rivera.co.za
Re: Packaging pypy
Hello Stefano, Am 30.11.2011 20:59, schrieb Stefano Rivera: Hi Kay (2011.11.30_21:01:01_+0200) The logical follow up is, how about X-Alternate-Python-Stacks ? I was contemplating that long term, once Nuitka is really a useful accelerator, it should be allowed to compile modules, whole packages, or programs at install time. Will that fit into this? I know little about Nuitka, but I don't think so. Nuitka isn't something that python modules will want to support at build time, is it? As I understand it, it could compile *applications* to C using Python's C API. What could it usefully do for a pure-python module package? It could turn every "module.py" into a "module.so" with more or less dubious benefits. With for StringIO e.g. that's potentially a cool thing to do, for other things, it likely isn't at all. It too (once Nuitka reaches that milestone) would e.g. turn ctypes bindings to faster, in its case, statically compiled native call bindings. I admit, it's probably 2 years away, that this is a real issue, but you know, if the infrastructure added for PyPy would help me too, that would be great. For Nuitka I would like to mix and match PyPy and CPython module alternatives freely for compilation. The mode, where a Python program is condensed to a single executable, is of course independent, and unrelated. But yeah, if a module or whole package could be made faster for everything, and all it took was adding an indication to the Debian package, that would be great. Yours, Kay -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4ed6ac94.6050...@gmx.de
Re: Packaging pypy
On 30 November 2011 22:22, Kay Hayen wrote: > It could turn every "module.py" into a "module.so" with more or less > dubious benefits. Note that there are a few differences between compiled and pure Python modules. E.g. - Tracebacks to errors won't show code from compiled files. - Functions assigned to a class attribute get bound to instances (i.e. get automatically passed self) if they're defined in Python, but compiled functions do not. I had to fix something recently where a standard library function had moved from pure-Python to C, and broken code that bound it to a class. - I can arbitrarily add extra attributes to instances of Python classes, but not to instances of compiled classes. Also, I occasionally go and tweak installed Python files, which is obviously impractical if they have to be translated and compiled. I know you're not suggesting we turn this on tomorrow. But I think replacing pure-python modules with compiled modules should always be a deliberate choice, not something distributions do automatically. StringIO already has compiled equivalents, of course - cStringIO, and the compiled io package in Python 2.7 and 3.x. Thomas
Re: Packaging pypy
Hello Thomas, It could turn every "module.py" into a "module.so" with more or less dubious benefits. Note that there are a few differences between compiled and pure Python modules. E.g. - Tracebacks to errors won't show code from compiled files. That is only if the ".py" file doesn't exist, right? Why wouldn't it. To me, compilation is not about removing the source code. Not at all, it's only about acceleration. - Functions assigned to a class attribute get bound to instances (i.e. get automatically passed self) if they're defined in Python, but compiled functions do not. I had to fix something recently where a standard library function had moved from pure-Python to C, and broken code that bound it to a class. Nuitka's "compiled_function" is way more compatible that "PyCFunction" if that is your concern. It has "func_code", etc. and works file in descriptors. Monkey patching, use in properties, metaclasses, whatever, is supposed to work with it just fine, and tested in the CPython test suites I run. In principle the only differences are: a) The "func.func_code" has no bytecode object (not going to change) b) The "func.func_defaults" are not there (changing later this year) It's really supposed to be as good as before. Now already. I do not pass the "hg" test suite with "compiled hg" yet. But once it does (needs at least func_defaults to work, and I believe I do not enforce enough evaluation order apparently yet, these things are not well covered by the test suite), that kinds of proves a lot, because hg is really very dynamic code. BTW: I am not sure a compiled "hg" itself is beneficial, because it's likely performance bound by IO. - I can arbitrarily add extra attributes to instances of Python classes, but not to instances of compiled classes. Nuitka is also more compatible here, although I am not so sure, if compiled classes of any kind have that problem. Also, I occasionally go and tweak installed Python files, which is obviously impractical if they have to be translated and compiled. True enough. I would hate it too. Lets say, the meta path importer that would be hooked in site.py checks dates. Then you would only become slower (use the modified version), and/or would have it compiled again at import time (slower to start, once, but permissions would be an issue, you like edit as root, but you run as mortal user). I don't think it would have to be impractical. I wouldn't want to live in a world, where Python programs become hard to modify. I know you're not suggesting we turn this on tomorrow. But I think replacing pure-python modules with compiled modules should always be a deliberate choice, not something distributions do automatically. Fair enough. Even if one can approx. the amount of transparency of the measure, it e.g. should be considered if it's useful at all, by the maintainer of a module or program, and would be something to turn on via configuration of course. StringIO already has compiled equivalents, of course - cStringIO, and the compiled io package in Python 2.7 and 3.x. I just meant to give an example. I believe PyPy does port C implemented modules to Python, because they can then execute it faster. To a compiler, be it JIT or not, Python is potentially better input. BUT: Right now, my priority is to get the release out that has a Debian package, and to find a sponsor for it, so Nuitka is available to Debian users to experiment with it. Still I am already in a position to state that Nuitka created ".so" and ".pyc" files have no practical differences except speed of creation and sizes. So on a design level, these could be treated the same. This "being compatible" part is currently basically the only priority at all. I have this plan, where that level of compatibility is the overarching first goal to achieve, even before performance numbers are even near impressive or measured a lot. All, I would like now and here, is just to understand, how PyPy is fit into things, and if that's a possible path for deeper Nuitka integration later on too, that would be great. If it's not, no problem, at all. I am no big fan of over complicating one solution, because potential other uses. I wouldn't ask for it. I just saw that PyPy suggested "Alternate" and not "PyPy" and a hint that they of course would like to have a universal solution. Yours, Kay -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4ed6c131.3080...@gmx.de
Re: Packaging pypy
On 30 November 2011 23:50, Kay Hayen wrote: > That is only if the ".py" file doesn't exist, right? Why wouldn't it. To > me, compilation is not about removing the source code. Not at all, it's > only about acceleration. > > Nuitka's "compiled_function" is way more compatible that "PyCFunction" if > that is your concern. > Neat! I'll try to get round to having a look at Nuitka at some point (in terms of using it, not packaging it - sorry, I can't help there). Thomas
