Re: [Python-Dev] PEP 370, open questions
On 17/01/2008, Christian Heimes <[EMAIL PROTECTED]> wrote: > * Should the Windows installer add %APPDATA%/Python/Scripts to PATH? The Windows installers currently do not add the main Python\Scripts directory to PATH, so they shouldn't add this one either. Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
On 07:55 am, [EMAIL PROTECTED] wrote: >A CC of the mail goes to the authors of setuptools, virtual python, >working env and virtual env. What's your opinion on the PEP? Do you >have >some input and advice for me? I wrote a similar tool called Combinator (http://divmod.org/trac/wiki/DivmodCombinator) which Divmod uses quite intensively. I think a significant portion of the Twisted community also uses it for development. Luckily you don't need to CC me specially as I'm here already ;-). One of the features it provides is a user site-packages directory much in the same style that the PEP proposes, so I'll comment on my experiences maintaining it. >The PEP 370 (http://www.python.org/dev/peps/pep-0370) "per user site >packages directory" has several open questions: > >* Are the directories for Windows, Mac and Unix fine? >* Mac: Should framework and non-framework builds of Python use > the same directories? One of the problems we've encountered over and over again with Combinator is that MacOS has a dual personality. Its personality is not just an issue of framework vs. non-framework build, but a fundamental question of "is this platform a UNIX or is it not a UNIX". An example: Let's say you have a GNU autotools project in C, which we'll call "Xxx", and a set of Python bindings, PyXxx. Combinator deals with this by using ~/.local, and providing scripts to set up PATH and LD_LIBRARY_PATH to point to ~/.local/bin and ~/.local/lib, respectively. I'm not suggesting that Python come with such a tool (although it might be handy, at least until this convention catches on with distributors), but it should at least make it clear how one would achieve the following straightforwardly: cd xxx-0.2.4 ./configure --prefix ~/.local make make install cd ../pyxxx-0.0.1 python setup.py install --prefix ~/.local Using Combinator, the user is now fully set up to "import xxx". But only if they're using Linux, because I made the same mistake (which we will probably be correcting at some point soon) of using Python's *existing* per-user site-packages directory of ~/Library/Python on the mac, and not adding ~/.local. On the Mac, our user now has a problem: given that ~/Library/Python/ doesn't follow the /usr or /usr/local style filesystem layout, autotools-based projects will not build their libraries in the right places using a single command-line option. (I guess maybe you could do something with --bindir and --sbindir and --exec-prefix or whatever, I don't know. I tend to get bored and wander off when I have to pass more than 3 options to a configure script.) A user familiar with the Mac alone might know what to do at this point (to be honest, I don't!), but I do know that people familiar with both platforms are confused by this apparently gratuitous inconsistency. They follow familiar procedure on a Linux box, it works great, then they do the same thing on a Mac (with the same shell, an apparently similar version of Python) and it doesn't. Keep in mind that "./configure --prefix" is familiar procedure from a lot more places than Combinator. For example, on shared hosting where I didn't have root, I've used this same trick without Combinator, building Python itself with --prefix ~/.local and editing .bashrc to modify the appropriate env vars. My suggestion would be to have *both* ~/.local *and* ~/Library/Python be honored on the Mac, because there really isn't much harm in doing so. Perhaps it would make sense for non-framework builds to not honor ~/Library/Python, but I am pretty certain, based on experience fielding requests from confused users, that framework builds should still honor ~/.local/lib/python.../site-packages. Windows has this problem less because everything has to be done so completely differently. In any event, the really important thing from my perspective is that the PEP should explain how this very common use-case for per-user installation of libraries can be accomplished on each of the "big three" platforms. This explanation should be put somewhere that users can easily find it when they are building libraries. I don't know what the "right way" to do this on Windows is though, so I can't offer much help there. Something to do with MinGW and intense suffering, I would guess. >* The patch also adds a usecustomize hook to site. Is it useful and > should it stay? Should this be "usercustomize"? I thought it was a typo but I see the same typo in the PEP. I have often wished for something like this for debugging, and it might have other uses, but there should be a caution against abuse :). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
On 17 Jan, 2008, at 8:55, Christian Heimes wrote: The PEP 370 (http://www.python.org/dev/peps/pep-0370) "per user site packages directory" has several open questions: * Are the directories for Windows, Mac and Unix fine? The Mac directories look fine to me. Is it worthwhile to note in the PEP that the directories for the Mac are already used in Python 2.5? * Mac: Should framework and non-framework builds of Python use the same directories? Yes, because that makes supporting users easier and keeps things nice and simple (no need to ask which kind of Python they installed to determine where their user directory is). Distutils should already link extensions in a way that allows you to have a python2.6 unix- build and framework-build that share the same site-packages directory. Ronald smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
Ronald Oussoren wrote: > The Mac directories look fine to me. > > Is it worthwhile to note in the PEP that the directories for the Mac are > already used in Python 2.5? Good point! >> * Mac: Should framework and non-framework builds of Python use >> the same directories? > > Yes, because that makes supporting users easier and keeps things nice > and simple (no need to ask which kind of Python they installed to > determine where their user directory is). Distutils should already link > extensions in a way that allows you to have a python2.6 unix-build and > framework-build that share the same site-packages directory. Have you read glyph's posting? He brought up some good points why Python on Mac should use ~/.local. Framework builds have an additional import directory in ~/Library/Python Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
> * Are the directories for Windows, Mac and Unix fine? Regarding Windows, I personally think its OK (but I also personally think the status-quo is OK too). As has already been mentioned in this thread, Windows actually provides an API you can call to determine various "well known" folders. I assume the PEP literally means "the environment variable APPDATA" instead of the result of SHGetSpecialFolderPath(CSIDL_APPDATA); it might be worth mentioning that (and ideally *why* the env-var is preferred). Similarly, a justification for why CSIDL_APPDATA/%APPDATA% is preferred over CSIDL_LOCAL_APPDATA - ie, why we are suggesting Python is a "roaming" application (eg, this implies that in certain Unix environments, it is likely that ~/.local/ will follow users as they log into different machines on the network - I've no idea how true that is) > * Should the Windows installer add %APPDATA%/Python/Scripts to PATH? I'd recommend not; IIUC, Python's main binary directory isn't added to the path, but even if it was, it will only be useful if .py is also on PATHEXT. Either way though, I'd suggest that particular decision is outside the scope of the PEP and a question for the windows installer assuming the PEP be accepted. Maybe some general hand-waving along the lines of "env vars on Windows are left to the installer, where such global issues are considered in a wider context" would do :) Cheers, Mark ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
Mark Hammond wrote: >> * Are the directories for Windows, Mac and Unix fine? > > Regarding Windows, I personally think its OK (but I also personally think > the status-quo is OK too). As has already been mentioned in this thread, > Windows actually provides an API you can call to determine various "well > known" folders. I assume the PEP literally means "the environment variable > APPDATA" instead of the result of SHGetSpecialFolderPath(CSIDL_APPDATA); it > might be worth mentioning that (and ideally *why* the env-var is preferred). > Similarly, a justification for why CSIDL_APPDATA/%APPDATA% is preferred over > CSIDL_LOCAL_APPDATA - ie, why we are suggesting Python is a "roaming" > application (eg, this implies that in certain Unix environments, it is > likely that ~/.local/ will follow users as they log into different machines > on the network - I've no idea how true that is) Isn't SHGetSpecialFolderPath() XP and newer only? Python 2.6 is going to support 2000 and newer. I've written a patch for the ntpath module which adds a getshellfolders() function. The function uses SHGetFolderPathW to build a dict of id to path mappings. The patch is available at http://bugs.python.org/issue1763. Maybe you have some time to review it? As far as I understand and remember the Windows API SHGetFolderPath() and friends are retrieving the information from the registry. The shell folder keys are (IIRC) REG_EXPAND_SZ keys which are expanded with the function ExpandEnvironmentStrings(). So at the end CSIDL_APPDATA uses the %APPDATA% env var -- unless I'm totally mistaken. Or do you suggest that I'm using the win32 API calls instead of the env var? Point taken! I'll justify why I view Python as a roaming app. All company and university Linux boxes I've used in the past had exported $HOME via NFS. So ~/.local is roamed. > I'd recommend not; IIUC, Python's main binary directory isn't added to the > path, but even if it was, it will only be useful if .py is also on PATHEXT. > Either way though, I'd suggest that particular decision is outside the scope > of the PEP and a question for the windows installer assuming the PEP be > accepted. Maybe some general hand-waving along the lines of "env vars on > Windows are left to the installer, where such global issues are considered > in a wider context" would do :) *hehe* I love your idea. It should be trivial to write a view lines of Python code which modify PATH in HKCU. I'm going to write a simple example and drop it into Tools/scripts/. Christian PS: Have you tested the new PCbuild directory for VS 2008? Martin said that you know how to create a new set of base addresses for pre-linking. dllbase_nt.txt is probably outdated. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
On Thu, 17 Jan 2008 08:55:51 +0100, Christian Heimes <[EMAIL PROTECTED]> wrote: > >* Should the site package directory also be ignored if process > gid != effective gid? If it should, I think the PEP should explain the attack this defends against in more detail. The current brief mention of "security issues" is a bit hand-wavey. For example, what is the relationship between security, this feature, and the PYTHONPATH environment variable? Isn't the attack of putting malicious code into a user site-packages directory the same as the attack of putting it into a directory in PYTHONPATH? Jean-Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
> Isn't SHGetSpecialFolderPath() XP and newer only? MSDN documents: Minimum operating systems Windows 2000, Windows NT 4.0 with Internet Explorer 4.0, Windows 98, Windows 95 with Internet Explorer 4.0 > The patch is available at > http://bugs.python.org/issue1763. Maybe you have some time to review > it? Sure - I'll do my best to do that in the next 24 hours - feel free to prod if I don't :) > As far as I understand and remember the Windows API SHGetFolderPath() > and friends are retrieving the information from the registry. The shell > folder keys are (IIRC) REG_EXPAND_SZ keys which are expanded with the > function ExpandEnvironmentStrings(). So at the end CSIDL_APPDATA uses > the %APPDATA% env var -- unless I'm totally mistaken. I'm not sure exactly how that is implemented, and I'm not necessarily arguing you must call that function (eg, it's not hard to argue the env-var may be preferred, as it offers an easy override) - it's just that the PEP shouldn't be silent about the issue. Something I meant to mention was that supporting multiple "per-user" directories could be considered - that way CSIDL_LOCAL_APPDATA could be added later. YAGNI is easy to justify here, but now is the time to consider such issues. > PS: Have you tested the new PCbuild directory for VS 2008? Martin said > that you know how to create a new set of base addresses for pre- > linking. > dllbase_nt.txt is probably outdated. I'm afaid I haven't. The base-addresses are fairly easily tested - just import each and every module listed while running under the debugger - if you see messages in the output window telling you about relocation, you have a problem :) pywin32 is screaming for a new release (it didn't manage a single one in 2007!) and that is likely to happen before I get to play with the new-fangled stuff Mark ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
Christian Heimes wrote: > I'll justify why I view Python as a roaming app. All > company and university Linux boxes I've used in the past had exported > $HOME via NFS. So ~/.local is roamed. I think there is a slight subtlety here: the exported NFS $HOME is more equivalent to the HOMEDRIVE/HOMEPATH which comes from the HOME directory in NT/AD. ie it is simply a share pointed to by a drive letter available wherever the user logs on. Roaming profiles actually *copy* the data from your network versions of USERPROFILE to the local machine [need to check they still do this; a while since I've administered this kind of setup]. The difference therefore is that installing large quantities of Python modules into a roaming profile path will involve their being copied to-and-fro on logon/logoff which, historically at least, was a known cause of slow startup/shutdown. I'll try to confirm if this is still the case. My own feeling was to use ~/.local on Windows as well (or whatever is agreed on for *nix) and let os.path.expanduser handle it. But... from r54364, USERPROFILE takes priority over HOMEDRIVE/HOMEPATH. Obviously Georg had some rationale there although a quick search of python-dev doesn't throw anything up. If we decide anything else here, though, we would seem to be somewhat in conflict with that interpretation of home/~ as USERPROFILE. TJG ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
On Thu, 17 Jan 2008 13:09:34 +0100, Christian Heimes <[EMAIL PROTECTED]> wrote: >Jean-Paul Calderone wrote: >> If it should, I think the PEP should explain the attack this defends >> against in more detail. The current brief mention of "security issues" >> is a bit hand-wavey. For example, what is the relationship between >> security, this feature, and the PYTHONPATH environment variable? Isn't >> the attack of putting malicious code into a user site-packages directory >> the same as the attack of putting it into a directory in PYTHONPATH? > >The PYTHONPATH env var has the same security implications. However a >user has multiple ways to avoid problems. For example the user can use >the -E flag or set up sudo to ignore the environment. I'm not sure how sudo gets involved. sudo doesn't set the euid, it sets the uid. This is about programs with the setuid bit set. (I assume this doesn't also apply to Python programs that explicitly make use of the seteuid() call, since this will probably only be checked at interpreter startup before any Python application code has run.) > >The uid and gid tests aren't really required. They just provide an extra >safety net if a user forgets to add the -s flag to a suid app. It's not much of a safety net if PYTHONPATH still allows injection of arbitrary code. It's just needless additional complexity for no benefit. On the other hand, if all of the other mechanisms for modifying how imports work is also made to behave this way, then maybe there's a point. Jean-Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python] Re: PEP 370, open questions
Christian Heimes wrote: > Tim Golden wrote: > >> Christian Heimes wrote: >> >>> I'll justify why I view Python as a roaming app. All >>> company and university Linux boxes I've used in the past had exported >>> $HOME via NFS. So ~/.local is roamed. >>> >> I think there is a slight subtlety here: the exported NFS >> $HOME is more equivalent to the HOMEDRIVE/HOMEPATH which >> comes from the HOME directory in NT/AD. ie it is simply >> a share pointed to by a drive letter available wherever >> the user logs on. Roaming profiles actually *copy* the >> data from your network versions of USERPROFILE to the >> local machine [need to check they still do this; a while >> since I've administered this kind of setup]. >> >> The difference therefore is that installing large quantities >> of Python modules into a roaming profile path will involve >> their being copied to-and-fro on logon/logoff which, historically >> at least, was a known cause of slow startup/shutdown. I'll >> try to confirm if this is still the case. >> > > I can't comment on the matter. I've not used roaming user profiles on > Windows for more than five years. Can someone with more experience shed > some like on the matter? > Roaming profiles still load and save the profile on logon/logoff. I think using the home directory (or subdirectory thereof) is preferable as that can *either* be on a shared drive *or* part of the roaming profile. Michael Foord ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
Jean-Paul Calderone wrote: > If it should, I think the PEP should explain the attack this defends > against in more detail. The current brief mention of "security issues" > is a bit hand-wavey. For example, what is the relationship between > security, this feature, and the PYTHONPATH environment variable? Isn't > the attack of putting malicious code into a user site-packages directory > the same as the attack of putting it into a directory in PYTHONPATH? The PYTHONPATH env var has the same security implications. However a user has multiple ways to avoid problems. For example the user can use the -E flag or set up sudo to ignore the environment. The uid and gid tests aren't really required. They just provide an extra safety net if a user forgets to add the -s flag to a suid app. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
On 17 Jan, 2008, at 9:40, [EMAIL PROTECTED] wrote: On 07:55 am, [EMAIL PROTECTED] wrote: The PEP 370 (http://www.python.org/dev/peps/pep-0370) "per user site packages directory" has several open questions: * Are the directories for Windows, Mac and Unix fine? * Mac: Should framework and non-framework builds of Python use the same directories? One of the problems we've encountered over and over again with Combinator is that MacOS has a dual personality. Its personality is not just an issue of framework vs. non-framework build, but a fundamental question of "is this platform a UNIX or is it not a UNIX". IMHO the question is wrong. MacOS X is a UNIX platform (and Apple has the paperwork to prove it), but at the same time it has a non- traditional filesystem layout. The "darwin" system at the foundation of OSX is a classic unix system, with the usual filesystem layout, but the bits that make OSX OSX use a completely diffent filesystem layout. The framework build of Python definitly targets the upper layer of the OSX stack, not just the Unix core. This sometimes causes confusion, but mostly from developers wandering over from a Linux system that complain that OSX isn't Linux. Note that even Darwin is not Linux, there are several differences that cause problems for naive porters. Two of those: Darwin uses different binary formats for shared libraries and plugins; the darwin linker hardcodes the full path to shared libraries into executables (without a runtime search path). And that's only the start of fun, universal (fat) binaries make live even more interesting. An example: Let's say you have a GNU autotools project in C, which we'll call "Xxx", and a set of Python bindings, PyXxx. Combinator deals with this by using ~/.local, and providing scripts to set up PATH and LD_LIBRARY_PATH to point to ~/.local/bin and ~/.local/lib, respectively. ~/Library/Combinator would be a better installation root on OSX, that location fits better into guidelines from Apple and also avoids completely hiding the Combinator data from the user. I'm not suggesting that Python come with such a tool (although it might be handy, at least until this convention catches on with distributors), but it should at least make it clear how one would achieve the following straightforwardly: This is probably off-topic for python-dev, but how is combinator different from zc.buildout and virtualenv? I'm particularly liking virtualenv at the moment as it allows me to have seperate trees for projects without having several full python installations. cd xxx-0.2.4 ./configure --prefix ~/.local make make install cd ../pyxxx-0.0.1 python setup.py install --prefix ~/.local Using Combinator, the user is now fully set up to "import xxx". But only if they're using Linux, because I made the same mistake (which we will probably be correcting at some point soon) of using Python's *existing* per-user site-packages directory of ~/Library/Python on the mac, and not adding ~/.local. Why? Just because users can't remember on which platform they are developing ;-)? That said, there's a libpython.a file in the framework build for basicly that reason: enough users complained about there not being a python library they could link to that it was easier to add a symlink then trying to educate everyone. On the Mac, our user now has a problem: given that ~/Library/Python/ doesn't follow the /usr or /usr/local style filesystem layout, autotools-based projects will not build their libraries in the right places using a single command-line option. (I guess maybe you could do something with --bindir and --sbindir and --exec-prefix or whatever, I don't know. I tend to get bored and wander off when I have to pass more than 3 options to a configure script.) You could teach Combinator about running configure scripts ;-). A user familiar with the Mac alone might know what to do at this point (to be honest, I don't!), but I do know that people familiar with both platforms are confused by this apparently gratuitous inconsistency. This is not a gratuitous inconsistency. The Mac developers (mostly Jack Jansen, but I was involved as well) decided that following the platform conventions is a good thing. They follow familiar procedure on a Linux box, it works great, then they do the same thing on a Mac (with the same shell, an apparently similar version of Python) and it doesn't. Keep in mind that "./configure --prefix" is familiar procedure from a lot more places than Combinator. For example, on shared hosting where I didn't have root, I've used this same trick without Combinator, building Python itself with --prefix ~/.local and editing .bashrc to modify the appropriate env vars. My suggestion would be to have *both* ~/.local *and* ~/Library/ Python be honored on the Mac, because there really isn't much harm in doing so. Perhaps it would m
Re: [Python-Dev] PEP 370, open questions
Tim Golden wrote: > Christian Heimes wrote: >> I'll justify why I view Python as a roaming app. All >> company and university Linux boxes I've used in the past had exported >> $HOME via NFS. So ~/.local is roamed. > > I think there is a slight subtlety here: the exported NFS > $HOME is more equivalent to the HOMEDRIVE/HOMEPATH which > comes from the HOME directory in NT/AD. ie it is simply > a share pointed to by a drive letter available wherever > the user logs on. Roaming profiles actually *copy* the > data from your network versions of USERPROFILE to the > local machine [need to check they still do this; a while > since I've administered this kind of setup]. > > The difference therefore is that installing large quantities > of Python modules into a roaming profile path will involve > their being copied to-and-fro on logon/logoff which, historically > at least, was a known cause of slow startup/shutdown. I'll > try to confirm if this is still the case. I can't comment on the matter. I've not used roaming user profiles on Windows for more than five years. Can someone with more experience shed some like on the matter? > My own feeling was to use ~/.local on Windows as well > (or whatever is agreed on for *nix) and let os.path.expanduser > handle it. But... from r54364, USERPROFILE takes priority > over HOMEDRIVE/HOMEPATH. Obviously Georg had some rationale > there although a quick search of python-dev doesn't throw > anything up. If we decide anything else here, though, we > would seem to be somewhat in conflict with that interpretation > of home/~ as USERPROFILE. The PEP already explains why I don't want to write to ~/ on Windows. I had considered it shortly but MSDN advices against it. Is %USERPROFILE% not equal to %HOMEDRIVE%%HOMEPATH%? Maybe Gregor did a mistake because he thought that both point to the same location. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
[EMAIL PROTECTED] wrote: > One of the problems we've encountered over and over again with > Combinator is that MacOS has a dual personality. Its personality is not > just an issue of framework vs. non-framework build, but a fundamental > question of "is this platform a UNIX or is it not a UNIX". MacOS isn't the only platform that has this problem. I use cygwin under Windows, and I wish Python (whether or not a cygwin build) would also use ~/.local. > On the Mac, our user now has a problem: given that ~/Library/Python/ > doesn't follow the /usr or /usr/local style filesystem layout, > autotools-based projects will not build their libraries in the right > places using a single command-line option. (I guess maybe you could do > something with --bindir and --sbindir and --exec-prefix or whatever, I > don't know. I tend to get bored and wander off when I have to pass more > than 3 options to a configure script.) I guess Windows isn't as bad as MacOS in this case, because there's no standard to follow and conflict with. Everything under whatever directory we choose won't likely match other packages, so we can do whatever we want. Still, I'd prefer something under $HOME. > Windows has this problem less because everything has to be done so > completely differently. True, except that cygwin tries hard to make it look like Unix. I'd rather Python under Windows really look like Unix, but I'm probably in a minority. And yes, I do share computers (physically and through Terminal Server), so per-user local libraries would be nice. Eric. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
Christian Heimes wrote: > Tim Golden wrote: >> The difference therefore is that installing large quantities >> of Python modules into a roaming profile path will involve >> their being copied to-and-fro on logon/logoff which, historically >> at least, was a known cause of slow startup/shutdown. I'll >> try to confirm if this is still the case. > > I can't comment on the matter. I've not used roaming user profiles on > Windows for more than five years. Can someone with more experience shed > some like on the matter? I'll try to confirm with our support guys here. >> My own feeling was to use ~/.local on Windows as well >> (or whatever is agreed on for *nix) and let os.path.expanduser >> handle it. But... from r54364, USERPROFILE takes priority >> over HOMEDRIVE/HOMEPATH. Obviously Georg had some rationale >> there although a quick search of python-dev doesn't throw >> anything up. If we decide anything else here, though, we >> would seem to be somewhat in conflict with that interpretation >> of home/~ as USERPROFILE. > > The PEP already explains why I don't want to write to ~/ on Windows. I > had considered it shortly but MSDN advices against it. Entirely understood, but then... > Is %USERPROFILE% not equal to %HOMEDRIVE%%HOMEPATH%? No. On my machine, for example: HOMEDRIVE=H: HOMEPATH=\ HOMESHARE=\\vogbs022\it\goldent USERPROFILE=C:\Documents and Settings\goldent However, using an account without a mapped home drive then yes, they are the same. TJG ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
On 12:02 pm, [EMAIL PROTECTED] wrote: On 17 Jan, 2008, at 9:40, [EMAIL PROTECTED] wrote: On 07:55 am, [EMAIL PROTECTED] wrote: The framework build of Python definitly targets the upper layer of the OSX stack, not just the Unix core. This sometimes causes confusion, but mostly from developers wandering over from a Linux system that complain that OSX isn't Linux. The framework build targets both layers, as I understand it - and it sounds like you're saying it too, since it's not "just" the UNIX core. Solaris isn't Linux either, by the way. These conventions hold across far more than one operating system :). Note that even Darwin is not Linux, there are several differences that cause problems for naive porters. Two of those: Darwin uses different binary formats for shared libraries and plugins; the darwin linker hardcodes the full path to shared libraries into executables (without a runtime search path). Distutils should take care of this distinction in Python's case, no? Xxx's autotools generate a shared library, PyXxx's setup.py generates a plugin (or "dylib", is that still what they're called these days?). An example: Let's say you have a GNU autotools project in C, which we'll call "Xxx", and a set of Python bindings, PyXxx. Combinator deals with this by using ~/.local, and providing scripts to set up PATH and LD_LIBRARY_PATH to point to ~/.local/bin and ~/.local/lib, respectively. ~/Library/Combinator would be a better installation root on OSX, that location fits better into guidelines from Apple and also avoids completely hiding the Combinator data from the user. I disagree, but Combinator's a complex beast and has a lot of other moving parts which need to live in specific places. Its main purpose is to manage your import path to easily switch between different development branches of multiple projects, and so most of its data is already in locations that the user has specified. A key thing about ~/.local in this case is that it isn't specific to Combinator. It's any per-user installed dependency libraries for development purposes, not necessarily on Combinator-managed projects, and not even necessarily Python projects. This is probably off-topic for python-dev, but how is combinator different from zc.buildout and virtualenv? We are definitely veering in that direction, but it probably bears a brief description just so folks here can see how it does and doesn't apply to the PEP. zc.buildout and virtualenv are primarily heterogeneous deployment tools, with development being just a different type of deployment. They're ways of installing Python packages into an isolated, configured environment. Combinator is primarily a development tool. Although I've used it as a deployment tool (as you can use zc.buildout as a development tool) that's not really its focus. Combinator has no "installation" step for most projects. ~/.local is a special case, reserved for common unchanging dependencies that require building; most code managed by Combinator lives in ~/Projects/YourProject/trunk or ~/Projects/YourProject/branches. (I *used* to be a Mac guy. Can you tell? :-)) The idea with zc.buildout is you are installing application A which requires library X version Q, and installing application B which requires library X version M; you want to keep those separated. The idea with combinator is that you are *developing* application A, and you want to make sure that it continues working with both version Q and M, so you can easily do chbranch X releases/Q # the most common combinator command trial a.test chbranch X releases/M trial a.test It also has specific integration with subversion for creating and merging branches. chbranch will try to check out releases/Q if there isn't already such a directory present, based on svn metadata. When you create a branch to start working on a feature, your environment is automatically reconfigured to import code from that branch. When you merge a branch to trunk, it is adjusted to load code from the merged trunk again. Hopefully some day soon it will also have integration with bazaar too. Why? Just because users can't remember on which platform they are developing ;-)? That said, there's a libpython.a file in the framework build for basicly that reason: enough users complained about there not being a python library they could link to that it was easier to add a symlink then trying to educate everyone. The system installation of Python on the mac has a nod in this direction as well. /usr/bin/python is also present, as is /usr/lib/pythonX.Yÿ0Cas symlinks between the two locations. You could teach Combinator about running configure scripts ;-). Better yet, perhaps somebody should teach configure about MacOS, and about per-user installation ;). But the real question here is not what Combinator should do, but what Python should do. This is not a gratuitous inconsistency. The
Re: [Python-Dev] PEP 370, open questions
On 12:19 pm, [EMAIL PROTECTED] wrote: >[EMAIL PROTECTED] wrote: >MacOS isn't the only platform that has this problem. I use cygwin >under >Windows, and I wish Python (whether or not a cygwin build) would also >use ~/.local. I would like to agree. MacOS has an advantage here though. Windows, even without cygwin, but doubly so once cygwin gets involved, has a schizophrenic idea of what "~" means. (I believe other messages in this thread have made reference to issues with expanduser.) You definitely are going to have a hard time getting non-cygwin and cygwin python to agree on the location of your home directory. I rather brutally modified my last Windows installation to get those two values to line up, and it required surgery for each user created, and caused a number of "interesting issues" with various tools writing fake cygwin UNIX paths and others thinking they were Windows drive-letter paths... >>Windows has this problem less because everything has to be done so >>completely differently. > >True, except that cygwin tries hard to make it look like Unix. I'd >rather Python under Windows really look like Unix, but I'm probably in >a >minority. And yes, I do share computers (physically and through >Terminal Server), so per-user local libraries would be nice. If you can figure out how to make this work without raising any of the bogeymen I just made reference to, I would be strongly in favor. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
On 12:26 pm, [EMAIL PROTECTED] wrote: >On Thu, 17 Jan 2008 13:09:34 +0100, Christian Heimes <[EMAIL PROTECTED]> >wrote: >>The uid and gid tests aren't really required. They just provide an >>extra >>safety net if a user forgets to add the -s flag to a suid app. >It's not much of a safety net if PYTHONPATH still allows injection of >arbitrary code. It's just needless additional complexity for no >benefit. By confusing users' expectations, it may actually be *worse* to add this "safety net" than to do nothing. It should be obvious right now that tightly controlling the environment is a requirement of any suid Python code. However, talking about different behavior in the case of differing euid and uid might confuse some developers and/or administrators into thinking that Python was doing all it needed to. There's also the confusion that the value of $HOME is actually the relevant thing for controlling "user-installed" imports, not the (E)UID. I think it would be good to have a look at the security implications of this and other environment-dependent execution, including $PYTHONPATH and $PYTHONSTARTUP, in a separate PEP. It might be good to change the way some of these things work, but in either case it would be good to have an unambiguous declaration of the *expected* security properties and potential attack vectors against the Python interpreter, for both developers and system administrators. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
On 17/01/2008, Tim Golden <[EMAIL PROTECTED]> wrote:
> > Is %USERPROFILE% not equal to %HOMEDRIVE%%HOMEPATH%?
>
> No. On my machine, for example:
>
> HOMEDRIVE=H:
> HOMEPATH=\
> HOMESHARE=\\vogbs022\it\goldent
>
> USERPROFILE=C:\Documents and Settings\goldent
>
> However, using an account without a mapped home
> drive then yes, they are the same.
Given this information (and the earlier comments about roaming
profiles) I would suggest that Python should use %HOMEDRIVE%HOMEPATH%.
This matches os.path.expanduser('~'). My previous comments were based
on my experience on a machine without a mapped home drive, so I'd
missed the distinction.
Yes, this does contradict the Microsoft guideline that you shouldn't
write to USERPROFILE, but Microsoft themselves set HOMEDRIVE and
HOMEPATH, so they vioated the guidelines, not us :-).
The point here is that we want "the user's home". This is clearly
%HOMEDRIVE%%HOMEPATH% on Windows, guidelines notwithstanding.
Paul.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
> The best we can hope for is to point to HOMEDRIVE/PATH as
> Paul suggested (*possibly* rewiring os.path.expanduser to
> try that first, depending on Win2K stuff)
os.path.expanduser uses HOMEDRIVE/HOMEPATH. It checks for an explicit
HOME (set manually by the user, this isn't a Windows standard name)
first. (At least in 2.5, see ntpath.py)
I think consistency with os.path.expanduser('~') is a significant benefit.
> and make sure
> the user is aware of what might happen if the HOMEDRIVE/PATH
> is really a pointer to a network-based profile.
Given that os.path.expanduser allows HOME to override
HOMEDRIVE/HOMEPATH, users in this position could set an explicit HOME.
The one downside of following expanduser is that Christian's code is
in C, where ntpath.expanduser is in Python, so there are 2 versions to
keep in sync. Maybe Christian could expose his C implementation, which
ntpath.expanduser could then reuse?
Paul.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
On 17 Jan, 2008, at 14:22, [EMAIL PROTECTED] wrote: On 12:02 pm, [EMAIL PROTECTED] wrote: On 17 Jan, 2008, at 9:40, [EMAIL PROTECTED] wrote: On 07:55 am, [EMAIL PROTECTED] wrote: The framework build of Python definitly targets the upper layer of the OSX stack, not just the Unix core. This sometimes causes confusion, but mostly from developers wandering over from a Linux system that complain that OSX isn't Linux. The framework build targets both layers, as I understand it - and it sounds like you're saying it too, since it's not "just" the UNIX core. Solaris isn't Linux either, by the way. These conventions hold across far more than one operating system :). I know, but somehow most people I run into that even know that there are more unix flavors than Linux also know that while all unices are simular they are not exactly the same. Note that even Darwin is not Linux, there are several differences that cause problems for naive porters. Two of those: Darwin uses different binary formats for shared libraries and plugins; the darwin linker hardcodes the full path to shared libraries into executables (without a runtime search path). Distutils should take care of this distinction in Python's case, no? Xxx's autotools generate a shared library, PyXxx's setup.py generates a plugin (or "dylib", is that still what they're called these days?). Distutils does take care of it, but that doesn't stop users from complaining that 'cc -shared' doesn't result in a workable python extension ;-). BTW. dylibs are shared libraries, there is no common suffix for bundles. Python uses '.so' for its bundle, adding to the confusion for folks used to systems using ELF libraries ;-) An example: Let's say you have a GNU autotools project in C, which we'll call "Xxx", and a set of Python bindings, PyXxx. Combinator deals with this by using ~/.local, and providing scripts to set up PATH and LD_LIBRARY_PATH to point to ~/.local/bin and ~/.local/lib, respectively. ~/Library/Combinator would be a better installation root on OSX, that location fits better into guidelines from Apple and also avoids completely hiding the Combinator data from the user. I disagree, but Combinator's a complex beast and has a lot of other moving parts which need to live in specific places. Its main purpose is to manage your import path to easily switch between different development branches of multiple projects, and so most of its data is already in locations that the user has specified. A key thing about ~/.local in this case is that it isn't specific to Combinator. It's any per-user installed dependency libraries for development purposes, not necessarily on Combinator-managed projects, and not even necessarily Python projects. Ah, so ~/.local is "just" a per-user variation on /usr/local? I didn't even know of that convention before this thread started, I tend to use ~/local (without dot) instead. Why? Just because users can't remember on which platform they are developing ;-)? That said, there's a libpython.a file in the framework build for basicly that reason: enough users complained about there not being a python library they could link to that it was easier to add a symlink then trying to educate everyone. The system installation of Python on the mac has a nod in this direction as well. /usr/bin/python is also present, as is /usr/lib/ pythonX.Yÿ0Cas symlinks between the two locations. You could teach Combinator about running configure scripts ;-). Better yet, perhaps somebody should teach configure about MacOS, and about per-user installation ;). I value my life to much to even think about developing this feature for auto*. Ronald smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
On 17 Jan, 2008, at 14:22, [EMAIL PROTECTED] wrote: As long as the default installation location is still ~/Library/ Python/ X.Y for framework builds I wouldn't mind too much. Is there a default "installation" location? When we started this part of the discussion, it was just which paths should be on sys.path. I'm pretty sure I've seen a suggestion for a distutils options for installing into your home location (something like "python setup.py install --user"). Ronald smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
Paul Moore wrote:
> On 17/01/2008, Tim Golden <[EMAIL PROTECTED]> wrote:
>>> Is %USERPROFILE% not equal to %HOMEDRIVE%%HOMEPATH%?
>> No. On my machine, for example:
>>
>> HOMEDRIVE=H:
>> HOMEPATH=\
>> HOMESHARE=\\vogbs022\it\goldent
>>
>> USERPROFILE=C:\Documents and Settings\goldent
>>
>> However, using an account without a mapped home
>> drive then yes, they are the same.
>
> Given this information (and the earlier comments about roaming
> profiles) I would suggest that Python should use %HOMEDRIVE%HOMEPATH%.
> This matches os.path.expanduser('~'). My previous comments were based
> on my experience on a machine without a mapped home drive, so I'd
> missed the distinction.
>
> Yes, this does contradict the Microsoft guideline that you shouldn't
> write to USERPROFILE, but Microsoft themselves set HOMEDRIVE and
> HOMEPATH, so they vioated the guidelines, not us :-).
>
> The point here is that we want "the user's home". This is clearly
> %HOMEDRIVE%%HOMEPATH% on Windows, guidelines notwithstanding.
A few datapoints here from WinXP SP2. I'll try Win2k when I
get home this evening if no-one's done it before then.
--
My support guys have temporarily switched me over to a roaming
profile so I could check what happens. As
expected, all the non-Local... folders are copied to and from
the network share specified as ProfilePath on my AD entry. (In
fact, our policy forbids profiles bigger than a few tens of Mb
so I had to hive off my existing profile and create a new one
to do this test).
Ignoring Terminal Services -- which have their own profiles --
and Cygwin -- which does its own thing -- we have four
situations: Users with and without mapped HOME drives; and with
and without network (roaming) profiles.
USERPROFILE points to c:\docu..\goldent regardless of whether
it's a network-based profile or not. The difference is what
happens when you log on/off.
If the home drive is mapped, HOMEDRIVE/PATH point to that mapping
If the home drive isn't mapped, HOMEDRIVE/PATH point to
USERPROFILE.
APPDATA is a non-local part of othe profile and as such is
copied to and from the network share if the profile is roaming.
--
There doesn't seem to be a clear solution.
If we point to APPDATA, HOMEDRIVE/PATH or USERPROFILE
we might be bulking up the content a user has to copy
to and fro. And local space policies might hinder things.
If we point to local settings\application data then the
packages won't roam with the user.
The best we can hope for is to point to HOMEDRIVE/PATH as
Paul suggested (*possibly* rewiring os.path.expanduser to
try that first, depending on Win2K stuff) and make sure
the user is aware of what might happen if the HOMEDRIVE/PATH
is really a pointer to a network-based profile.
TJG
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
Paul Moore wrote:
>> The best we can hope for is to point to HOMEDRIVE/PATH as
>> Paul suggested (*possibly* rewiring os.path.expanduser to
>> try that first, depending on Win2K stuff)
>
> os.path.expanduser uses HOMEDRIVE/HOMEPATH. It checks for an explicit
> HOME (set manually by the user, this isn't a Windows standard name)
> first. (At least in 2.5, see ntpath.py)
Ummm... see my earlier point a few posts back which refers
to r54364 which gave priority to USERPROFILE over HOMEDRIVE/HOMEPATH
> I think consistency with os.path.expanduser('~') is a significant benefit.
Agreed entirely.
> Given that os.path.expanduser allows HOME to override
> HOMEDRIVE/HOMEPATH, users in this position could set an explicit HOME.
You're right of course (and that's what I do now, in fact). It's
just that it doesn't feel quite right relying on a nonstandard
env var. It means if I log on to another machine and want to
take advantage of these theoretically roaming Python modules
I'd have to remember to set my HOME var to something.
> The one downside of following expanduser is that Christian's code is
> in C, where ntpath.expanduser is in Python, so there are 2 versions to
> keep in sync. Maybe Christian could expose his C implementation, which
> ntpath.expanduser could then reuse?
If it had its priorities switched around, Christian's patch could simply
call os.path.expanduser.
I'll hold off now. The situation's clear enough and I'm not really
offering anything here. There hard part's going to be making a
decision. I think my proposal amounts to:
1) Switch the priority of env var checks in os.path.expanduser
so that HOMEDRIVE/PATH comes before USERPROFILE.
2) Have the new code use os.path.expanduser under Windows
3) Document the fact that installing in a per-user way might
be writing considerable amounts to a network-based profile.
TJG
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
Ronald Oussoren wrote: > I'm pretty sure I've seen a suggestion for a distutils options for > installing into your home location (something like "python setup.py > install --user"). If you have read the PEP then I'm pretty sure that you have read about the python setup.py install --user thing. ;) Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
Tim Golden wrote:
> If it had its priorities switched around, Christian's patch could simply
> call os.path.expanduser.
No, I can't use expanduser("~") because there is no sensible way to get
from the path to APPDATA. In it's infinite wisdom MS has decided to
localize the name of its shell folders. On my German WinXP box the
APPDATA folder is called "Anwendungsdaten".
> I'll hold off now. The situation's clear enough and I'm not really
> offering anything here. There hard part's going to be making a
> decision. I think my proposal amounts to:
>
> 1) Switch the priority of env var checks in os.path.expanduser
> so that HOMEDRIVE/PATH comes before USERPROFILE.
> 2) Have the new code use os.path.expanduser under Windows
> 3) Document the fact that installing in a per-user way might
> be writing considerable amounts to a network-based profile.
3) is sensible but I'd prefer 4):
4) Default to the %APPDATA% env var but give the user a chance to
overwrite the policy with the env var PYTHONUSERBASE, or whatever is a
good name for it. Did I mention that I'm not good in naming things? *g*
Christian
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
Paul Moore wrote: > The one downside of following expanduser is that Christian's code is > in C, where ntpath.expanduser is in Python, so there are 2 versions to > keep in sync. Maybe Christian could expose his C implementation, which > ntpath.expanduser could then reuse? What code are you talking about? Except for the -s option PEP 370 is implemented in Python. Or are you talking about my os.path.getshellfolder() patch? Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] ntpath r54364 (was: PEP 370, open questions)
On 17/01/2008, Tim Golden <[EMAIL PROTECTED]> wrote: > Ummm... see my earlier point a few posts back which refers > to r54364 which gave priority to USERPROFILE over HOMEDRIVE/HOMEPATH Sorry. I'd not realised this was a post-2.5 revision (ie, not in released code yet...) Looking at the change, it appears that the point is to support ~user, which it does by getting the current user's home, and replacing the last part, on the assumption that it is the user name. This works for USERPROFILE (in my experience, I don't know if there are corner cases where it isn't true) but does *not* work if HOME is set, and only works otherwise by preferring USERPROFILE over HOMEDRIVE/HOMEPATH (and I believe that preference is misleading, as USERPROFILE is always set if HOMEDRIVE/PATH is, so there's no point checking for HD/HP if USERDIR fails). I'd recommend that this change be reverted. To correctly get another user's home directory would involve reading (via the registry, or maybe some utility function I couldn't find at a quick glance) the value of HOMEDRIVE/HOMEPATH which is set for the other user. (In fact, given that I guess these could be set in a login script, it's entirely possible that the information simply isn't available). I don't believe that ~user is a useful concept on Windows, and I'd rather that code using it fail than give what are quite probably unreliable answers. If someone wants to support ~user where user is the name of the current user, then that could be covered by an explicit check that the name matches, and if so use the code for '~', but personally I don't think that's worth it. Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ntpath r54364
Christian Heimes wrote: > Paul Moore wrote: >> I'd recommend that this change be reverted. To correctly get another >> user's home directory would involve reading (via the registry, or >> maybe some utility function I couldn't find at a quick glance) the >> value of HOMEDRIVE/HOMEPATH which is set for the other user. (In fact, >> given that I guess these could be set in a login script, it's entirely >> possible that the information simply isn't available). > > I concur! The changes should be reverted. The implementation is most > likely to fail in more elaborate cases, e.g. c:\users\a\alpha or > c:\users\groupa\name. The proper way for a ~user implementation is a > bunch of Win32 API calls. Frustratingly, I don't believe there's *any* canonical way to find ~user without actually going through the whole process of getting a token and impersonating them. If they've logged onto this machine already you can have a good go by following the code posted the other day on c.l.py [1] (although I'm now not sure about the l10n aspects which Christian mentioned). Any technique of going up one (from your own profile) and then down one is no better than assuming that all users are in /home on a *nix box. As to a logged-on user's *own* home path, I'd prefer HOMEDRIVE/PATH over USERPROFILE since the latter is at least explictly named HOMEsomething, although the latter can be the fallback. Both are present on Win2k & on WinXP. Can't check WinNT or Win9x but I don't know if we're looking to support those or not. TJG [1] http://groups.google.com/group/comp.lang.python/msg/71b20a67a9ca76dd ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ntpath r54364 (was: PEP 370, open questions)
Paul Moore wrote: > I'd recommend that this change be reverted. To correctly get another > user's home directory would involve reading (via the registry, or > maybe some utility function I couldn't find at a quick glance) the > value of HOMEDRIVE/HOMEPATH which is set for the other user. (In fact, > given that I guess these could be set in a login script, it's entirely > possible that the information simply isn't available). I concur! The changes should be reverted. The implementation is most likely to fail in more elaborate cases, e.g. c:\users\a\alpha or c:\users\groupa\name. The proper way for a ~user implementation is a bunch of Win32 API calls. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
[EMAIL PROTECTED] wrote: > My suggestion would be to have *both* ~/.local *and* ~/Library/Python be > honored on the Mac, because there really isn't much harm in doing so. > Perhaps it would make sense for non-framework builds to not honor > ~/Library/Python, but I am pretty certain, based on experience fielding > requests from confused users, that framework builds should still honor > ~/.local/lib/python.../site-packages. I'm taking your suggestion under consideration: ~/.local for standard and framework builds and ~/Library/Python for framework builds only. > In any event, the really important thing from my perspective is that the > PEP should explain how this very common use-case for per-user > installation of libraries can be accomplished on each of the "big three" > platforms. This explanation should be put somewhere that users can > easily find it when they are building libraries. > > I don't know what the "right way" to do this on Windows is though, so I > can't offer much help there. Something to do with MinGW and intense > suffering, I would guess. Good point, but is it the job of the PEP to explain such problems in detail? The PEP should consider the problem but the explanation belongs to a tutorial - IMHO. >> * The patch also adds a usecustomize hook to site. Is it useful and >> should it stay? > > Should this be "usercustomize"? I thought it was a typo but I see the > same typo in the PEP. I have often wished for something like this for > debugging, and it might have other uses, but there should be a caution > against abuse :). Yeah! I've a tendency to drag typos through my code because I heavily use copy 'n paste. A while ago I copied a typo from C code into the Python unit tests and later into the docs, too Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ntpath r54364
Tim Golden wrote: > Frustratingly, I don't believe there's *any* canonical > way to find ~user without actually going through the whole > process of getting a token and impersonating them. If > they've logged onto this machine already you can have > a good go by following the code posted the other day > on c.l.py [1] (although I'm now not sure about the l10n > aspects which Christian mentioned). Any technique of > going up one (from your own profile) and then down one > is no better than assuming that all users are in /home > on a *nix box. A while ago I've seen a variant of ExpandEnvironmentStrings that takes a user token, too. It's called ExpandEnvironmentStringsForUser(). http://msdn2.microsoft.com/en-us/library/aa373507(VS.85).aspx Before anybody gets exciting about the good news: Here is the bad news. A user token can't be obtained easily. In fact it requires a login + password or a process handler. :( > As to a logged-on user's *own* home path, I'd prefer > HOMEDRIVE/PATH over USERPROFILE since the latter is at > least explictly named HOMEsomething, although the latter > can be the fallback. Both are present on Win2k & on WinXP. > Can't check WinNT or Win9x but I don't know if we're looking > to support those or not. Python 2.6 targets Windows 2000 and newer. So no, we don't have to support NT and the 9x series any more. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
Ronald Oussoren wrote: > Ah, so ~/.local is "just" a per-user variation on /usr/local? I didn't > even know of that convention before this thread started, I tend to use > ~/local (without dot) instead. I assume ~/.local was first introduced by the freedesktop.org people. On my box it's only used for some desktop related apps like ~/.local/share/Trash or XFC4. http://www.freedesktop.org/wiki/Specifications/basedir-spec Contrary to ~/local, the dot local directory doesn't show up, unless the user unhides dot files. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
On Jan 17, 2008, at 4:05 PM, Christian Heimes wrote: > I assume ~/.local was first introduced by the freedesktop.org > people. On > my box it's only used for some desktop related apps like > ~/.local/share/Trash or XFC4. I've only seen ~/.local/ defined there, and only ~/.local/share/ in that case. Anything beyond that is grass-roots, and still pretty obscure. That said, I think extending usage of ~/.local/ is fine for things that aren't supposed to be edited or directly used by users. ~/.local/ bin/ seems suspect. -Fred -- Fred Drake ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] building _ctypes in trunk fails first time around
If I run "make clean" and then "make", builting _ctypes fails with this message: *** WARNING: renaming "_ctypes" since importing it failed: No module named _weakref Typing "make" a second time fixes this -- it seems a simple matter of _ctypes being built before _weakref. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370, open questions
Paul Moore gmail.com> writes: > > Yes, this does contradict the Microsoft guideline that you shouldn't > write to USERPROFILE, but Microsoft themselves set HOMEDRIVE and > HOMEPATH, so they vioated the guidelines, not us . > > The point here is that we want "the user's home". This is clearly > %HOMEDRIVE%%HOMEPATH% on Windows, guidelines notwithstanding. > To add another twist to this the HOMEDRIVE mapping is not guarantied at all on windows startup (http://www.google.com/search?q=windows+homedrive+mapping). This has caused a lot of pain for me in the past. Can this be changed into using %APPDATA% by default and changing to %HOMEDRIVE%%HOMEPATH% based on some MSI switch? Regards, Suraj ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] What to do for bytes in 2.6?
I believe the issue of whether and how to backport bytes (and bytearray?) from 3.0 to 2.6 has come up before, but I don't think we've come to any kind of conclusion. It's quite subtle. In a private email, Thomas Wouters and I discussed this: [Guido] > > Perhaps the biggest question in my mind is what to do with bytes in > > 2.6. Make it an alias for str? Or a new type that doesn't mix with > > either str or unicode? But then how do we get read() calls and the > > like on binary files or sockets to return bytes instances? Perhaps a > > trivial str subclass? Or return bytes when -3 is specified? I need to > > bring this up on the list. [Thomas] > I don't know yet. I'm not sure yet what to do with strings and unicode in > 2.6 in -3 mode (or perhaps through a future-import, with some hackery.) > Perhaps a *set* of str-subclasses makes sense: a will-be-unicode subclass, > and a will-be-bytes subclass. String literals and text-mode input would > produce will-be-unicode, binary input and byte literals will-be-bytes. > Switching between -3 and normal mode for that would be changing a few > pointers in the core, making all operations return normal strings in normal > mode. And the would-be types can produce whatever warning they wish -- they > need not worry about speed. But I'm not sure if that'll work. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What to do for bytes in 2.6?
[GvR] > I believe the issue of whether and how to backport bytes > (and bytearray?) from 3.0 to 2.6 has come up before, but > I don't think we've come to any kind of conclusion. My recommendation is to leave it out of 2.6. Not every 3.0 concept has to be backported. This particular one doesn't have a straight-forward integration. It duplicates some existing functionality and in general doesn't make life better for the 2.6 coder. The only benefit I can see is that it lets you write code that is a step closer to 3.0; however, it has not been our goal to write one piece of code that runs under both major releases. I think both 2.x and 3.0 are better served if 2.x cleanly stays with the str/uncode model and 3.0 sticks with the bytes/text model. Commingling the two muddies the waters and conflates the distinctions. I think it best to make that jump all at once. Multiple-ways-to-do-it should not be the motto for 2.6. Instead, let's provide the best transition tools possible and keep both 2.6 and 3.0 as clean as possible (no duplicative or half-intergrated functionality). Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What to do for bytes in 2.6?
On Jan 17, 2008 6:00 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote: > [GvR] > > I believe the issue of whether and how to backport bytes > > (and bytearray?) from 3.0 to 2.6 has come up before, but > > I don't think we've come to any kind of conclusion. > > My recommendation is to leave it out of 2.6. > > Not every 3.0 concept has to be backported. This particular one doesn't have > a straight-forward integration. It duplicates some existing functionality > and in general doesn't make life better for the 2.6 coder. The only benefit > I can see is that it lets you write code that is a step closer to 3.0; > however, it has not been our goal to write one piece of code that runs under > both major releases. Indeed. However, it *is* one of our stated goals that 2.6 will provide a variety of features that will make source-to-source conversion more likely to succeed. I'm currently expecting that the str/unicode/bytes issue is by far the biggest issue where automated conversion is unlikely to be effective. *If* we provide some kind of "backport" of bytes (even if it's just an alias for or trivial subclass of str), it should be part of a strategy that makes it easier to write code that runs under 2.6 and can be automatically translated to run under 3.0 with the same semantics. I realize that this is a lofty goal and that I haven't thought through how to get there -- maybe it won't be possible anyway, but I think it's worth a try. (I'll think about it more later.) > I think both 2.x and 3.0 are better served if 2.x cleanly stays with the > str/uncode model and 3.0 sticks with the bytes/text model. Commingling the > two muddies the waters and conflates the distinctions. I think it best to > make that jump all at once. > > Multiple-ways-to-do-it should not be the motto for 2.6. Instead, let's > provide the best transition tools possible and keep both 2.6 and 3.0 as clean > as possible (no duplicative or half-intergrated functionality). Well, 2.6 *is* one of the transition tools. We should probably do a case study on the back of an envelope on how to take existing code that deals with encoded text, decoded text, and binary data, and how to restructure it for automatic translation to correct 3.0 code. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What to do for bytes in 2.6?
> *If* we provide some kind of "backport" of > bytes (even if it's just an alias for or trivial > subclass of str), it should be part of a strategy > that makes it easier to write code that > runs under 2.6 and can be automatically translated > to run under 3.0 with the same semantics. If it's just an alias or trivial subclass, then we haven't added anything that can't be done trivially by the 2-to-3 tool. I'm thinking that this is a deeper change. It doesn't serve either 2.6 or 3.0 to conflate str/unicode model with the bytes/text model. Mixing the two in one place just creates a mess in that one place. I'm sure we're thinking that this is just an optional transition tool, but the reality is that once people write 2.6 tools that use the new model, then 2.6 users are forced to deal with that model. It stops being optional or something in the future, it becomes a mental jump that needs to be made now (while still retaining the previous model in mind for all the rest of the code). I don't think you need a case study to forsee that it will be unpleasant to work with a code base that commingles the two world views. One other thought. I'm guessing that apps that would care about the distinction are already using unicode and are already treating text as distinct from arrays of bytes. Instead, it's backwards thinking 20th-century neanderthal ascii-bound folks like myself who are going to have transition issues. It would be nice for us knuckle-draggers to not have to face the issue until 3.0. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Rationale for NamedTemporaryFile revisited [SEC=UNCLASSIFIED]
I found this posting, and those following it, as I too am baffled that
NamedTemporaryFile doesn't let you read the generated file - even within the
same script.
For unit testing it is very commonplace to generate a test file on the fly
and then use it as input the function being tested. NamedTemporaryFile would
be the perfect candidate for this except that the file seems to disappear
immediately after it has been closed. This is contrary to Dustin's comment
which states that the lifetime extends until the object is garbage collected.
The following script illustrates this isn't the case:
from tempfile import NamedTemporaryFile
fid = NamedTemporaryFile(mode='w',
suffix='.tmp',
dir='.')
fid.write('My temp file')
fid.close()
# Read it (fid hasn't been garbage collected yet)
print fid.name
# But the file itself has gone
fid2 = open(fid.name)
print fid2.readlines()
Can someone please help point me to the best way of creating temporary files
in Python that can be read in immediately after their creation.
Thanks
Ole Nielsen, Geoscience Australia
On Sun, Mar 18, 2007 at 11:49:55AM +1200, Greg Ewing wrote:
> I've just discovered the hard way that NamedTemporaryFile
> automatically deletes the file when you close it. That
> doesn't seem very useful to me, since surely the reason
> you're using NamedTemporaryFile instead of TemporaryFile
> is that you want to do something else with it afterwards?
> What's the rationale for this behaviour?
For both TemporaryFile and NamedTemporaryFile, the rationale is that
"temporary" extends until the object is garbage collected.
TemporaryFile is deleted immediately (to prevent other applications from
modifying your temporary file). NamedTemporaryFile is inteded for use
when you need access to the file by filename during the lifetime of the
file. In either case, when the object goes, the file should too.
Dustin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What to do for bytes in 2.6?
On Jan 17, 2008 7:11 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote: > > *If* we provide some kind of "backport" of > > bytes (even if it's just an alias for or trivial > > subclass of str), it should be part of a strategy > > that makes it easier to write code that > > runs under 2.6 and can be automatically translated > > to run under 3.0 with the same semantics. > > If it's just an alias or trivial subclass, then we > haven't added anything that can't be done trivially > by the 2-to-3 tool. I suggest you study how the 2to3 tool actually works before asserting this. Consider the following function. def stuff(blah): foo = "" while True: bar = blah.read(1024) if bar == "": break foo += bar return foo Is it reading text or binary data from stream blah? We can't tell. If it's meant to be reading text, 2to3 should leave it alone. But if it's meant to be reading binary data, 2to3 should change the string literals to bytes literals (b"" in this case). (If it's used for both, there's no hope.) As it stands, 2to3 hasn't a chance to decide what to do, so it will leave it alone -- but the "translated" code will be wrong if it was meant to be reading bytes. However, if the two empty string literals were changed to b"", we would know it was reading bytes. 2to3 could leave it alone, but at least the untranslated code would be correct for 2.6 and the translated code would be correct for 3.0. This may seem trivial (because we do all the work, and 2to3 just leaves stuff alone), but having b"" and bytes as aliases for "" and str in 2.6 would mean that we could write 2.6 code that correctly expresses the use of binary data -- and we could use u"" and unicode for code using text, and 2to3 would translate those to "" and str and the code would be correct 3.0 text processing code. Note that we really can't make 2to3 assume that all uses of str and "" are referring to binary data -- that would mistranslate the vast majority of code that does non-Unicode-aware text processing, which I estimate is the majority of small and mid-size programs. > I'm thinking that this is a deeper change. > It doesn't serve either 2.6 or 3.0 to conflate > str/unicode model with the bytes/text model. > Mixing the two in one place just creates a mess > in that one place. > > I'm sure we're thinking that this is just an optional > transition tool, but the reality is that once people > write 2.6 tools that use the new model, > then 2.6 users are forced to deal with that model. > It stops being optional or something in the future, > it becomes a mental jump that needs to be made now > (while still retaining the previous model in mind > for all the rest of the code). This may be true. But still, 2.6 *will* run 2.5 code without any effort, so we will be able to mix modules using the 2.5 style and modules using the 3.0 style (or at least some aspects of 3.0 style) in one interpreter. Neither 2.5 nor 3.0 will support this combination. That's why 2.6 is so important it's a stepping stone. > I don't think you need a case study to forsee that > it will be unpleasant to work with a code base > that commingles the two world views. Well, you shouldn't commingle the two world view in a single module or package. But that would just be bad style -- you shouldn't use competing style rules within a package either (like using words_with_underscores and camelCaseWords for method names). > One other thought. I'm guessing that apps that would > care about the distinction are already using unicode > and are already treating text as distinct from arrays > of bytes. Yes, but 99% of these still accept str instances in positions where they require text. The problem is that the str type and its literals are ambiguous -- their use is not enough to be able to guess whether text or data is meant. Just being able to (voluntarily! on a per-module basis!) use a different type name and literal style for data could help forward-looking programmers get started on making the distinction clear, thus getting ready for 3.0 without making the jump just yet (or maintaining a 2.6 and a 3.0 version of the same package easily, using 2to3 to automatically generate the 3.0 version from the 2.6 code base). > Instead, it's backwards thinking 20th-century > neanderthal ascii-bound folks like myself who are going > to have transition issues. It would be nice for us > knuckle-draggers to not have to face the issue until 3.0. Oh, you won't. Just don't use the -3 command-line flag and don't put "from __future__ import " at the top of your modules, and you won't have to change your ways at all. You can continue to distribute your packages in 2.5 syntax that will also work with 2.6, and your users will be happy (as long as they don't want to use your code on 3.0 -- but if you want to give them that, *that* is when you will finally be forced to face the issue. :-) Note that I believe that the -3 flag should not change semantics -- it should only add warnings. Sema
Re: [Python-Dev] Rationale for NamedTemporaryFile revisited [SEC=UNCLASSIFIED]
Don't close it until you're done with it. Isn't that obvious?
On Jan 17, 2008 8:30 PM, <[EMAIL PROTECTED]> wrote:
> I found this posting, and those following it, as I too am baffled that
> NamedTemporaryFile doesn't let you read the generated file - even within the
> same script.
> For unit testing it is very commonplace to generate a test file on the fly
> and then use it as input the function being tested. NamedTemporaryFile would
> be the perfect candidate for this except that the file seems to disappear
> immediately after it has been closed. This is contrary to Dustin's comment
> which states that the lifetime extends until the object is garbage collected.
> The following script illustrates this isn't the case:
>
> from tempfile import NamedTemporaryFile
> fid = NamedTemporaryFile(mode='w',
> suffix='.tmp',
> dir='.')
>
> fid.write('My temp file')
> fid.close()
>
> # Read it (fid hasn't been garbage collected yet)
> print fid.name
>
> # But the file itself has gone
> fid2 = open(fid.name)
> print fid2.readlines()
>
>
> Can someone please help point me to the best way of creating temporary files
> in Python that can be read in immediately after their creation.
>
> Thanks
> Ole Nielsen, Geoscience Australia
>
>
>
> On Sun, Mar 18, 2007 at 11:49:55AM +1200, Greg Ewing wrote:
> > I've just discovered the hard way that NamedTemporaryFile
> > automatically deletes the file when you close it. That
> > doesn't seem very useful to me, since surely the reason
> > you're using NamedTemporaryFile instead of TemporaryFile
> > is that you want to do something else with it afterwards?
> > What's the rationale for this behaviour?
>
> For both TemporaryFile and NamedTemporaryFile, the rationale is that
> "temporary" extends until the object is garbage collected.
> TemporaryFile is deleted immediately (to prevent other applications from
> modifying your temporary file). NamedTemporaryFile is inteded for use
> when you need access to the file by filename during the lifetime of the
> file. In either case, when the object goes, the file should too.
>
> Dustin
>
>
>
>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What to do for bytes in 2.6?
Raymond Hettinger writes: > One other thought. I'm guessing that apps that would > care about the distinction are already using unicode > and are already treating text as distinct from arrays > of bytes. Indeed. Mailman, for instance. Yet Mailman still has problems with (broken) wire protocol that sneaks past the gate, and causes some exception that is only handled by the top-level "no matter what goes wrong, we're not going to lose this post" handler (which literally shunts it into a queue that only human admins look it -- it's not Mailman's problem any more.) However, I am not sure it would help Mailman to catch such bugs to move from the str/unicode paradigm to the bytes/text paradigm. The problem Mailman faces is that there is no (single) Japanese foyer where the characters have to exchange their muddy "bytes" shoes for nice clean "unicode" slippers. Instead, there are a number of ways to get in, and the translation takes place (and sometimes not) at different stages. But this is not a Python issue; it has to do with Mailman's design. So I don't think this would be improved if we changed the paradigm forcibly. I don't see a benefit to apps like Mailman from changing over in 2.x. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rationale for NamedTemporaryFile revisited [SEC=UNCLASSIFIED]
Thank you very much for the quick reply. I believe we have to close the file in order be able to read it in - in this case to feed a unittest. I actually tried to read it in before closing it, but (as I suspected) the data wasn't available. We use unit testing for pretty much all functions and classes in our system. Some functions read files and do something with them. We therefore need a standardised and robust way to generate temporary files that exist on the filesystem long enough for the function being tested to read them. We are happy to write a wrapper ourselves that will do just that - but before doing so we had a look at the standard functionality. Hence the question about NamedTemporaryFile: What is the purpose of creating a name if it disappears after the file object is closed? I agree with Dustin that it would make sense for the file to live on the disk for as long as the File object is available to the script, closed or not. But I don't think that is the way NamedTemporaryFile currently works as illustrated in the test script in my first posting. Any help or comments, please? Cheers Ole Nielsen, Geoscience Australia -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Guido van Rossum Sent: Friday, 18 January 2008 3:45 To: Nielsen Ole Cc: [email protected] Subject: Re: [Python-Dev] Rationale for NamedTemporaryFile revisited [SEC=UNCLASSIFIED] Don't close it until you're done with it. Isn't that obvious? On Jan 17, 2008 8:30 PM, <[EMAIL PROTECTED]> wrote: > I found this posting, and those following it, as I too am baffled that > NamedTemporaryFile doesn't let you read the generated file - even within the > same script. > For unit testing it is very commonplace to generate a test file on the fly > and then use it as input the function being tested. NamedTemporaryFile would > be the perfect candidate for this except that the file seems to disappear > immediately after it has been closed. This is contrary to Dustin's comment > which states that the lifetime extends until the object is garbage collected. > The following script illustrates this isn't the case: > > from tempfile import NamedTemporaryFile > fid = NamedTemporaryFile(mode='w', > suffix='.tmp', > dir='.') > > fid.write('My temp file') > fid.close() > > # Read it (fid hasn't been garbage collected yet) > print fid.name > > # But the file itself has gone > fid2 = open(fid.name) > print fid2.readlines() > > > Can someone please help point me to the best way of creating temporary files > in Python that can be read in immediately after their creation. > > Thanks > Ole Nielsen, Geoscience Australia > > > > On Sun, Mar 18, 2007 at 11:49:55AM +1200, Greg Ewing wrote: > > I've just discovered the hard way that NamedTemporaryFile > > automatically deletes the file when you close it. That > > doesn't seem very useful to me, since surely the reason > > you're using NamedTemporaryFile instead of TemporaryFile > > is that you want to do something else with it afterwards? > > What's the rationale for this behaviour? > > For both TemporaryFile and NamedTemporaryFile, the rationale is that > "temporary" extends until the object is garbage collected. > TemporaryFile is deleted immediately (to prevent other applications from > modifying your temporary file). NamedTemporaryFile is inteded for use > when you need access to the file by filename during the lifetime of the > file. In either case, when the object goes, the file should too. > > Dustin > > > > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What to do for bytes in 2.6?
"Guido van Rossum" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] | Is it reading text or binary data from stream blah? We can't tell. If | it's meant to be reading text, 2to3 should leave it alone. But if it's | meant to be reading binary data, 2to3 should change the string | literals to bytes literals (b"" in this case). (If it's used for both, | there's no hope.) As it stands, 2to3 hasn't a chance to decide what to | do, so it will leave it alone -- but the "translated" code will be | wrong if it was meant to be reading bytes. It seems that the main purpose of adding bytes (as more or less a synonym for str when used as bytes) is to aid 2to3 translation. So I think I would favor it being part of a future import. | Note that I believe that the -3 flag should not change semantics -- it | should only add warnings. Semantic changes must either be backwards | compatible or be requested explicitly with a __forward__ import (which | 2to3 can remove). Were you planning to make bytes a __future__ (or __forward__?) import? I think making it so should satisfy Raymond's concerns. Even if whatever you eventually do is technically backwards compatible, he is suggesting that conceptually, it is not. I see some validity to that view. tjr ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What to do for bytes in 2.6?
> having b"" and bytes as aliases for "" and > str in 2.6 would mean that we could write 2.6 code that correctly > expresses the use of binary data -- and we could use u"" and unicode > for code using text, and 2to3 would translate those to "" and str and > the code would be correct 3.0 text processing code. I see. There's a healthy benefit for 2to3 translation that cannot be accomplished in any other way. This may potentially more than offset the negative impact to the 2.x world where it complexifies the language without any immediate payoff. FWIW, I'm sold on the rationale. Hopefully, this can be confined to just annotation and aliasing but not porting back any C API changes or code that depends on the bytes/text distinction. I worry that as soon as the annotation is made available, it will ripple throughout the code and pervade the language so that it cannot be ignored by Py2.6 coders. It's a bit of a Pandora's Box. > Just being able to (voluntarily! on a > per-module basis!) use a different type name and literal style for > data could help forward-looking programmers get started on making the > distinction clear, thus getting ready for 3.0 without making the jump > just yet (or maintaining a 2.6 and a 3.0 version of the same package > easily, using 2to3 to automatically generate the 3.0 version from the > 2.6 code base). Are we going to be "forward-looking" and change all of the standard library modules? Is this going to pop-up everywhere and become something you have to know about whether or not you're a volunteering forward-looking programmer? I hope not. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What to do for bytes in 2.6?
On 04:43 am, [EMAIL PROTECTED] wrote: >Just being able to (voluntarily! on a >per-module basis!) use a different type name and literal style for >data could help forward-looking programmers get started on making the >distinction clear, thus getting ready for 3.0 without making the jump >just yet (or maintaining a 2.6 and a 3.0 version of the same package >easily, using 2to3 to automatically generate the 3.0 version from the >2.6 code base). Yes! Yes! Yes! A thousand times yes! :-D This is *the* crucial feature which will make porting large libraries like Twisted to 3.0 even possible. Thank you, Guido. To the critics of this idea: libraries which process text, if they are meant to be correct, will need to deal explicitly with the issue of what data-types they believe to be text, what methods they will call on them, and how they deal with them. You cannot get away from this. It is not an issue reserved for the "pure" future of 3.0; if your code doesn't handle these types correctly now, it has bugs in it *now*. (In fact I am fixing some code with just such a bug in it right now :).) It is definitely possible to make your library code do the right thing for different data types, continue to support str literals in 2.6, and eventually require text / unicode input (after an appropriate deprecation period, of course). And it will be a lot easier if the translations imposed by 2to3 are as minimal as possible. >Note that I believe that the -3 flag should not change semantics -- it >should only add warnings. Semantic changes must either be backwards >compatible or be requested explicitly with a __forward__ import (which >2to3 can remove). This also sounds great. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rationale for NamedTemporaryFile revisited [SEC=UNCLASSIFIED]
[EMAIL PROTECTED] wrote:
> Thank you very much for the quick reply.
>
> I believe we have to close the file in order be able to read it in - in this
> case to feed a unittest. I actually tried to read it in before closing it,
> but (as I suspected) the data wasn't available.
>
Did you try seeking back to the beginning before attempting your read?
> We use unit testing for pretty much all functions and classes in our system.
> Some functions read files and do something with them. We therefore need a
> standardised and robust way to generate temporary files that exist on the
> filesystem long enough for the function being tested to read them.
>
Well you can hardly believe this requirement is so novel that its
absence hasn't been missed in Python, then.
> We are happy to write a wrapper ourselves that will do just that - but before
> doing so we had a look at the standard functionality. Hence the question
> about NamedTemporaryFile: What is the purpose of creating a name if it
> disappears after the file object is closed? I agree with Dustin that it would
> make sense for the file to live on the disk for as long as the File object is
> available to the script, closed or not. But I don't think that is the way
> NamedTemporaryFile currently works as illustrated in the test script in my
> first posting.
>
> Any help or comments, please?
>
The mistake you made was to change the mode from the default "w+b". The
following code works on both Windows and Linux:
from tempfile import NamedTemporaryFile
fid = NamedTemporaryFile(mode='w+b',
suffix='.tmp',
dir='.')
fid.write('My temp file')
fid.seek(0)
data = fid.read()
print data
fid.close()
By the way, this is the sort of topic normally reserved for
comp.lang.python, whose denizens would have straightened you out in no
time flat.
regards
Steve
> Cheers
> Ole Nielsen, Geoscience Australia
>
>
> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Guido
> van Rossum
> Sent: Friday, 18 January 2008 3:45
> To: Nielsen Ole
> Cc: [email protected]
> Subject: Re: [Python-Dev] Rationale for NamedTemporaryFile revisited
> [SEC=UNCLASSIFIED]
>
> Don't close it until you're done with it. Isn't that obvious?
>
> On Jan 17, 2008 8:30 PM, <[EMAIL PROTECTED]> wrote:
>> I found this posting, and those following it, as I too am baffled that
>> NamedTemporaryFile doesn't let you read the generated file - even within
> the
>> same script.
>> For unit testing it is very commonplace to generate a test file on the fly
>> and then use it as input the function being tested. NamedTemporaryFile
> would
>> be the perfect candidate for this except that the file seems to disappear
>> immediately after it has been closed. This is contrary to Dustin's comment
>> which states that the lifetime extends until the object is garbage
> collected.
>> The following script illustrates this isn't the case:
>>
>> from tempfile import NamedTemporaryFile
>> fid = NamedTemporaryFile(mode='w',
>> suffix='.tmp',
>> dir='.')
>>
>> fid.write('My temp file')
>> fid.close()
>>
>> # Read it (fid hasn't been garbage collected yet)
>> print fid.name
>>
>> # But the file itself has gone
>> fid2 = open(fid.name)
>> print fid2.readlines()
>>
>>
>> Can someone please help point me to the best way of creating temporary
> files
>> in Python that can be read in immediately after their creation.
>>
>> Thanks
>> Ole Nielsen, Geoscience Australia
>>
>>
>>
>> On Sun, Mar 18, 2007 at 11:49:55AM +1200, Greg Ewing wrote:
>>> I've just discovered the hard way that NamedTemporaryFile
>>> automatically deletes the file when you close it. That
>>> doesn't seem very useful to me, since surely the reason
>>> you're using NamedTemporaryFile instead of TemporaryFile
>>> is that you want to do something else with it afterwards?
>>> What's the rationale for this behaviour?
>> For both TemporaryFile and NamedTemporaryFile, the rationale is that
>> "temporary" extends until the object is garbage collected.
>> TemporaryFile is deleted immediately (to prevent other applications from
>> modifying your temporary file). NamedTemporaryFile is inteded for use
>> when you need access to the file by filename during the lifetime of the
>> file. In either case, when the object goes, the file should too.
>>
>> Dustin
>>
>>
>>
>>
>> ___
>> Python-Dev mailing list
>> [email protected]
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
--
Steve Holden+1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-d
Re: [Python-Dev] building _ctypes in trunk fails first time around
Guido van Rossum schrieb: > If I run "make clean" and then "make", builting _ctypes fails with this > message: > > *** WARNING: renaming "_ctypes" since importing it failed: No module > named _weakref > > Typing "make" a second time fixes this -- it seems a simple matter of > _ctypes being built before _weakref. > What a pity. Can the order be changed in which the extensions are built, or should the importing be tried after all the extensions are ready? Or should _weakref be made a builtin module? Thomas ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What to do for bytes in 2.6?
Guido van Rossum schrieb: > I believe the issue of whether and how to backport bytes (and > bytearray?) from 3.0 to 2.6 has come up before, but I don't think > we've come to any kind of conclusion. It's quite subtle. In a private > email, Thomas Wouters and I discussed this: > > [Guido] >> > Perhaps the biggest question in my mind is what to do with bytes in >> > 2.6. Make it an alias for str? Or a new type that doesn't mix with >> > either str or unicode? But then how do we get read() calls and the >> > like on binary files or sockets to return bytes instances? Perhaps a >> > trivial str subclass? Or return bytes when -3 is specified? I need to >> > bring this up on the list. > > [Thomas] >> I don't know yet. I'm not sure yet what to do with strings and unicode in >> 2.6 in -3 mode (or perhaps through a future-import, with some hackery.) >> Perhaps a *set* of str-subclasses makes sense: a will-be-unicode subclass, >> and a will-be-bytes subclass. String literals and text-mode input would >> produce will-be-unicode, binary input and byte literals will-be-bytes. >> Switching between -3 and normal mode for that would be changing a few >> pointers in the core, making all operations return normal strings in normal >> mode. And the would-be types can produce whatever warning they wish -- they >> need not worry about speed. But I'm not sure if that'll work. > Is the bytes type required for PEP3118 'Revising the buffer protocol'? I just started to work on implementing this PEP for ctypes, in the hope that these changes can be merged into trunk. Thomas ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
