Re: [Python-Dev] externals?
Hi, On Sun, Nov 18, 2012 at 8:22 AM, Georg Brandl wrote: > One way would be to use one hg repo per version, and (maybe, if needed) > a master repo that has them as subrepos. Or have all versions in the same repo as usual (with branches), but have hg subrepos point to different repos: ones extracted from the main repo by containing only the correct branch. But it might be a bit delicate to pull this off. (hg clone takes a "-r" option and copies only things needed for the given revision or branch, but apparently we can't pass this option automatically to the cloning of subrepos. (Maybe it points out that subrepos are a hack best done without altogether, which is what we did in pypy.)) A bientôt, Armin. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] externals?
Am 18.11.2012 10:00, schrieb Armin Rigo: > Hi, > > On Sun, Nov 18, 2012 at 8:22 AM, Georg Brandl wrote: >> One way would be to use one hg repo per version, and (maybe, if needed) >> a master repo that has them as subrepos. > > Or have all versions in the same repo as usual (with branches), but > have hg subrepos point to different repos: ones extracted from the > main repo by containing only the correct branch. But it might be a > bit delicate to pull this off. (hg clone takes a "-r" option and > copies only things needed for the given revision or branch, but > apparently we can't pass this option automatically to the cloning of > subrepos. (Maybe it points out that subrepos are a hack best done > without altogether, which is what we did in pypy.)) Yep. Anyway, if every external version goes into a branch, then we don't need subrepos anyway. That is a better idea than mine. Since you can use (e.g.) "hg clone -r tk-8.5" or download a tarball specific to a branch, nobody should need to get the whole externals history on clone. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way)
Yes! For many years I have been very frustrated by the install-centric nature of python. I am biased, of course, by the fact that I am developing an application where python is embedded, an application that needs to run out of the box. A developer may have many many versions (branches) of the application on his drive, and each needs to work separately. We have managed to isolate things, by patching python (and contributing that patch) to override the default library seach path (and ignore environment paths) when python is started up thogh the api. All well and good. But recently we have started in increasing amount to use external libraries and packages and we have been introduced to the dependency hell that is public python packages. In this install-centric world, developers reference huge external packages without a second thought, which cause large dependency trees. Using a simple tool may require whole HTTP frameworks to be downloaded. What is worse is when there are versioning conflicts between those dependencies. I don't have a well formed solution in mind, but I would see it desirable to have a way for someone to release his package with all its dependencies as a self-contained and isolated unit. E.g. if package foo.py relies on functionality from version 1.7 of bar.py, then that functionality could be bottled up for foo´s exclusive usage. Another package, baz.py, could then also make use of bar, but version 1.8. The two bar versions would be isolated. Perhaps this is just a pipedream. Even unpossible. But it doesn't harm to try to think about better ways to do things. K -Original Message- From: Christian Tismer [mailto:[email protected]] Sent: 15. nóvember 2012 23:10 To: Kristján Valur Jónsson Cc: [email protected] Subject: Generally boared by installation (Re: [Python-Dev] Setting project home path the best way) Hi guys, I am bored of installing things. Bored of things that happen to not work for some minor reasons. Reasons that are not immediately obvious. Things that don't work because some special case was not handled. Things that compile for half an hour and then complain that something is not as expected. May it be a compiler, a library, a command like pip or easy-install, a system like macports or homebrew, virtualenv, whatsoever. These things are all great if they work. When they do not work, the user is in some real trouble. And he reads hundreds Of blogs and sites and emails, which all answer a bit of slightly related questions, but all in all - This is not how Python should work !! I am really bored and exhausted and annoyed by those packages which Pretend to make my life eadier, but they don't really. Something is really missing. I want something that is easy to use in all cases, also if it fails. Son't get me wrong, I like things like pip or virtualenv or homebrew. I just think they have to be rewritten completely. They have the wrong assumption that things work! The opposite should be the truth: by default, things go wrong. Correctness is very fragile. I am thinking of a better concept that is harder to break. I thin to design a setup tool that is much more checking itself and does not trust in any assumption. Scenario: After hours and hours, I find how to modify setup.py to function almost correctly for PySide. This was ridiculously hard to do! Settings for certain directories, included and stuff are not checked when they could be, but after compiling a lot of things! After a lot of tries and headaches, I find out that virtualenv barfs on a simple link like ./.Python, the executable, when switching from stock Python to a different (homebrew) version!! This was obviously never tested well, so it frustrates me quite a lot. I could fill a huge list full of complaints like that if I had time. But I don't. Instead, I think installation scripts are generally still wrong by concept today and need to be written in a different way. I would like to start a task force and some sprints about improving this situation. My goal is some unbreakable system of building blocks that are self-contained with no dependencies, that have a defined interface to talk to, and that know themselves completely by introspection. They should not work because they happen to work around all known defects, but by design and control. Whoever is interested to work with me on this is hereby honestly welcomed! Cheers - chris Sent from my Ei4Steve On Nov 15, 2012, at 10:17, Kristján Valur Jónsson wrote: > When python is being run from a compile environment, it detects this by > looking for "Lib" folders in directories above the one containing the > executable. > (I always thought that this "special" execution mode, hardwired in, > was a bit odd, and suggested that this could be made a function of pep405) > Anyway, keeping your executable as part of the tree is the trick I use, and > to make things nice I put
Re: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way)
On Sun, Nov 18, 2012 at 8:54 PM, Kristján Valur Jónsson < [email protected]> wrote: > I don't have a well formed solution in mind, but I would see it desirable > to have a way for someone to release his package with all its dependencies > as a self-contained and isolated unit. E.g. if package foo.py relies on > functionality from version 1.7 of bar.py, then that functionality could be > bottled up for foo´s exclusive usage. > Another package, baz.py, could then also make use of bar, but version 1.8. > The two bar versions would be isolated. > > Perhaps this is just a pipedream. Even unpossible. But it doesn't harm > to try to think about better ways to do things. > > Easily bundling dependencies is a key principle behind the ability to execute directories and zipfiles that contain a top level __main__.py file that was added back in 2.6 (although the zipfile version doesn't play nicely with extension modules). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Register-based VM for CPython
Interesting work indeed. From profiling CPython it has long been clear to me that enormous gains can be made by making instruction dispatching faster. A huge amount of time is spent in the evaluation loop. I have also been making small inroads to offline bytecode optimization. Identifying common patterns and introducing special opcodes to deal with them. Obviously using register addressing makes such an approach more effective. (Working with code objects is fun and exciting, btw, and the reason for my patch http://bugs.python.org/issue16475) K From: Python-Dev [mailto:[email protected]] On Behalf Of Victor Stinner Sent: 17. nóvember 2012 01:13 To: Python Dev Subject: [Python-Dev] Register-based VM for CPython The WPython project is similar to my work (except that it does not use registers). It tries also to reduce the overhead of instruction dispatch by using more complex instructions. http://code.google.com/p/wpython/ Using registers instead of a stack allow to implement more optimizations (than WPython). For example, it's possible to load constants outside loops and merge "duplicate constant loads". I also implemented more aggressive and experimental optimizations (disabled by default) which may break applications: move loads of attributes and globals outside of loops, and replace binary operations with inplace operations. For example, "x=[]; for ...: x.append(...)" is optimized to something like "x=[]; x_append=x.append; for ...: x_append(...)", and "x = x + 1" is replaced with "x += 1". Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Register-based VM for CPython
On Sat, 17 Nov 2012 11:17:40 +0100 Armin Rigo wrote: > Hi Victor, > > On Sat, Nov 17, 2012 at 2:13 AM, Victor Stinner > wrote: > > The major drawback of the register approach (at least of my implementation) > > is that it changes the lifetime of objects. Newly created objects are only > > "destroyed" at the exit of the function, whereas the stack-based VM destroys > > "immediatly" objects (thanks to the reference counter). PyPy has similar > > issues with its different garbage collector. > > That is not strictly correct. PyPy, Jython and IronPython have > non-immediate destructors, but as far as I can tell they all avoid to > keep objects alive for an unbounded amount of time. This important > difference is visible if the function calls other code that takes a > long while to run: in your approach, the objects created by the > function itself will stay alive for the whole duration, while the > other interpreters will all release them soon after they are not > referenced any more --- not instantly like CPython but still soon. Agreed with Armin. Also, I would point out that the reference counting behaviour is an important feature of *C*Python (to the point that we have test cases checking against reference cycles), so we can't break it nilly-willy. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] externals?
Zitat von Armin Rigo : Hi, On Sun, Nov 18, 2012 at 8:22 AM, Georg Brandl wrote: One way would be to use one hg repo per version, and (maybe, if needed) a master repo that has them as subrepos. Or have all versions in the same repo as usual (with branches), but have hg subrepos point to different repos: ones extracted from the main repo by containing only the correct branch. But it might be a bit delicate to pull this off. (hg clone takes a "-r" option and copies only things needed for the given revision or branch, but apparently we can't pass this option automatically to the cloning of subrepos. (Maybe it points out that subrepos are a hack best done without altogether, which is what we did in pypy.)) I'd like to stress that we don't need any versioning here. wget and tar would be sufficient, except that it's Windows, so we have neither wget nor tar. However, including a PowerShell script may be an option; most developers will have PowerShell already on their system. AFAICT, PowerShell can do HTTP downloads and extract zip files. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Register-based VM for CPython
On 17.11.12 03:13, Victor Stinner wrote: The major drawback of the register approach (at least of my implementation) is that it changes the lifetime of objects. Newly created objects are only "destroyed" at the exit of the function, whereas the stack-based VM destroys "immediatly" objects (thanks to the reference counter). It should not be a problem. Just register instructions should clear input registers if they are not binded to named local variables. I.e. "a = b + c * d" should be compiled to: BINARY_MUL_REG R1, 'c', 'd' BINARY_ADD_REG 'a', 'b', R1 # R1 cleared ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Register-based VM for CPython
2012/11/18 Antoine Pitrou : > Also, I would point out that the reference counting behaviour is an > important feature of *C*Python (to the point that we have test cases > checking against reference cycles), so we can't break it nilly-willy. The tests about reference cycles are just tests that garbage is collected, a Python language feature. Those aren't technically CPython specific. -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Register-based VM for CPython
On Sun, 18 Nov 2012 09:37:57 -0500 Benjamin Peterson wrote: > 2012/11/18 Antoine Pitrou : > > Also, I would point out that the reference counting behaviour is an > > important feature of *C*Python (to the point that we have test cases > > checking against reference cycles), so we can't break it nilly-willy. > > The tests about reference cycles are just tests that garbage is > collected, a Python language feature. Those aren't technically CPython > specific. We do have tests which check reference cycles are not created. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Register-based VM for CPython
2012/11/18 Antoine Pitrou : > On Sun, 18 Nov 2012 09:37:57 -0500 > Benjamin Peterson wrote: > >> 2012/11/18 Antoine Pitrou : >> > Also, I would point out that the reference counting behaviour is an >> > important feature of *C*Python (to the point that we have test cases >> > checking against reference cycles), so we can't break it nilly-willy. >> >> The tests about reference cycles are just tests that garbage is >> collected, a Python language feature. Those aren't technically CPython >> specific. > > We do have tests which check reference cycles are not created. Oh, those. Those are implementation details. -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] externals?
On Sun, Nov 18, 2012 at 6:18 AM, wrote: > > Zitat von Armin Rigo : > > >> Hi, >> >> On Sun, Nov 18, 2012 at 8:22 AM, Georg Brandl wrote: >>> >>> One way would be to use one hg repo per version, and (maybe, if needed) >>> a master repo that has them as subrepos. >> >> >> Or have all versions in the same repo as usual (with branches), but >> have hg subrepos point to different repos: ones extracted from the >> main repo by containing only the correct branch. But it might be a >> bit delicate to pull this off. (hg clone takes a "-r" option and >> copies only things needed for the given revision or branch, but >> apparently we can't pass this option automatically to the cloning of >> subrepos. (Maybe it points out that subrepos are a hack best done >> without altogether, which is what we did in pypy.)) > > > I'd like to stress that we don't need any versioning here. wget and > tar would be sufficient, except that it's Windows, so we have neither > wget nor tar. However, including a PowerShell script may be an option; > most developers will have PowerShell already on their system. AFAICT, > PowerShell can do HTTP downloads and extract zip files. I would hope we can just write a simple Python script to do this, rather than require PowerShell. I'm 99.9% certain anyone building Python on Windows will already have a version of Python installed. Plus, they're going to need it anyway to build OpenSSL (see PCbuild/build_ssl.py and the references to it in VS projects). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Register-based VM for CPython
On Sun, 18 Nov 2012 11:27:32 -0500 Benjamin Peterson wrote: > 2012/11/18 Antoine Pitrou : > > On Sun, 18 Nov 2012 09:37:57 -0500 > > Benjamin Peterson wrote: > > > >> 2012/11/18 Antoine Pitrou : > >> > Also, I would point out that the reference counting behaviour is an > >> > important feature of *C*Python (to the point that we have test cases > >> > checking against reference cycles), so we can't break it nilly-willy. > >> > >> The tests about reference cycles are just tests that garbage is > >> collected, a Python language feature. Those aren't technically CPython > >> specific. > > > > We do have tests which check reference cycles are not created. > > Oh, those. Those are implementation details. At this point I'm not sure which statement you are arguing against. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] externals?
On 11/18/2012 12:05 PM, Brian Curtin wrote: On Sun, Nov 18, 2012 at 6:18 AM, wrote: Zitat von Armin Rigo : Or have all versions in the same repo as usual (with branches), but have hg subrepos point to different repos: ones extracted from the main repo by containing only the correct branch. But it might be a bit delicate to pull this off. (hg clone takes a "-r" option and copies only things needed for the given revision or branch, but apparently we can't pass this option automatically to the cloning of subrepos. (Maybe it points out that subrepos are a hack best done without altogether, which is what we did in pypy.)) I'd like to stress that we don't need any versioning here. wget and tar would be sufficient, except that it's Windows, so we have neither wget nor tar. However, including a PowerShell script may be an option; most developers will have PowerShell already on their system. AFAICT, PowerShell can do HTTP downloads and extract zip files. I would hope we can just write a simple Python script to do this, rather than require PowerShell. I'm 99.9% certain anyone building Python on Windows will already have a version of Python installed. Plus, they're going to need it anyway to build OpenSSL (see PCbuild/build_ssl.py and the references to it in VS projects). After reading the thread, I realize that I do not actually want externam dependency files moved to hg. I and others are not going to push changes back, so we do not need hg clones. What would be good would to be able to access the files and use them to build python without svn installed. I don't know the best way to do that, but if tarred or zipped releases were made for each version that should be downloaded, our urllib, tarfile/ziplib, and any other modules needed should be sufficient to transfer and unpack. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
