Re: [Python-Dev] is sys.modules not meant to be replaced?
Eric Snow wrote: > p.s. I tried opening a tracker ticket on this, but it wouldn't go > through. I'll try again later. Adding the following line to /etc/hosts makes the bug tracker fast when python.org is down: 127.0.0.1 python.org This is because bugs.python.org works fine, but it links to CSS files and images that are on python.org. Forcing DNS lookups for python.org to return localhost makes the requests for these resources fail fast. A more long-term solution would be to duplicate all the resources to bugs.python.org... Petri ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Rebooting PEP 394 (aka Support the /usr/bin/python2 symlink upstream)
On Wed, 20 Jul 2011 01:53:09 -0500 Kerrick Staley wrote: > On Mon, Jul 18, 2011 at 3:03 AM, Ned Deily wrote: > > I think adding the requirement to mandate hard link vs soft link usage > > is an unnecessary and unwarranted attempt at optimization. For > > instance, IIRC, the OS X installers don't use any hard links: that may > > complicate the install, plus hard links on OS X HFS* file systems are a > > bit of a kludge and not necessarily more efficient than symlinks. It's > > not a big deal but perhaps the wording should be changed to make a > > suggestion about hard links vs syminks rather than mandate which should > > be used. > > Ah, OK. The wording's been changed so that symbolic links will be > installed on Mac OS X and hard links elsewhere (although maybe > symbolic links are also better on certain other platforms; I'm not > sure). > > I do think that specific instructions must be given (rather than just > a suggestion) because it's indicating what must be done to CPython. > The instructions *should* be as close as possible to what the > installer already does, but I'm not entirely sure what the installer > does by default, and the hard-link recommendation was based off a > cursory inspection of my own system, so further input from yourself > and the rest of python-dev would be appreciated. I think the recommendation should be symbolic links for all systems. Hard links are generally harder to discover, while it is trivial to find out that a given file is a symbolink link, and to which other file. The optimization is probably not useful in the real world (our executables are relatively small, and people worried about a couple of megabytes can always go for the shared library option). Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Rebooting PEP 394 (aka Support the /usr/bin/python2 symlink upstream)
On Sun, 24 Jul 2011 23:16:55 +0200 Antoine Pitrou wrote: > > I think the recommendation should be symbolic links for all systems. > Hard links are generally harder to discover, while it is trivial to > find out that a given file is a symbolink link, and to which other file. > The optimization is probably not useful in the real world (our > executables are relatively small, and people worried about a couple of > megabytes can always go for the shared library option). Oh, I don't even know why I thought hard links would be a size optimization anyway. I guess I thought "file copy" instead of "symbolic link". Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Comments of the PEP 3151
Arguments in favor of specific errors - Using specific errors avoids the need of "import errno". The import is sometimes done in the except block, which may raise a new error (and may be slow). I like specific exceptions because it avoids the need of re-raise unwanted exception. It makes the code more readable: we ignore explicitly specific errors instead of re-raised unwanted exceptions and implicitly ignore some errors. For example: try: os.mkdir(dir) except OSError as err: if err.errno != errno.EEXIST: raise becomes try: os.mkdir(dir) except FileExistsError: pass By the way, is it faster to not handle and than re-raise unwanted exceptions? (I don't like tests like "err.errno != errno.EEXIST", it's confusing to me. It uses errno identifier twice: once as an attribute, once as a module name.) Make specific errors builtins - I agree that it's better to make specific errors builtins. If we move exceptions to more specific modules, like io and socket, you have to load these modules to catch an exception and remember in which module the exception is. An advantage of builtin specific exceptions is to avoid an import (import errno). Making them builtin may also avoid a bootstrap issue (catch a specific error at startup). Choice of the specific errors - I don't understand how Antoine decided which errno should have an exception or not. For example, EINTR and EPIPE are handled in many modules of the standard library but don't have their exception, whereas EISDIR (IsADirectoryError) and ENOTDIR (NotADirectoryError) are very rare (EISDIR is never handled in the standard library) but have their exception. If we add EINTR, I don't know if it's better to add it to BlockingIOError or to create a new exception (InterruptError?). Another good candidate is EINVAL. Would it be possible to have an (exhaustive?) list of errno with their popularity? At least, their popularity in the Python standard library? Raise a specific error -- Does raising a specific error (e.g. raise FileNotFound()) set errno and message attributes (to something different than None)? If we provide an error message error: should it be localized? The description of FileDescriptorError tells about the "default error message". It we use a localized message, it's not possible to preallocate or cache instances. I don't see how we could choose a default errno value, because some exceptions handle multiple errno. For example, PermissionError.errno can be EACCES or EPERM. Do specific errors slows down raising exceptions? Do raising an IOError (or a "legacy" error) use a O(1) lookup to choose the exception class? PermissionError --- EACCES and EPERM have a different meaning. Even that they are very similar, we might provide two different exceptions. EPERM is only used once in the Python stdlib, so we might only provide AccesError. On Linux, EPERM usually indicates an operation requiring root priviledge. EACCES can be related to filesystem permissions (read-only, user is not allowed to write, etc.) or can be an mmap error. Deprecation --- Because IOError, OSError, select.error, etc. are well known exceptions, I'm not in favor of removing them from Python 3. It would make porting from Python 2 worse. If we don't remove them, they should not be deprecated. I'm in favor of adding a note in the documentation of all legacy exceptions to advice to use IOError or specific exceptions instead. I suppose that these notes will have to indicate a Python version. Test the implementation --- Did someone test Blender, Django or another major applications on the implementation of the PEP 3151? -1 on FileSystemError - I'm not sure that we need FileSystemError or ConnectionError. Careless code will use IOError, whereas careful code will use an explicit list like (ConnectionAbortedError, ConnectionRefusedError, ConnectionResetError). If we remove IsADirectoryError and NotADirectoryError, FileSystemError only contains FileExistsError and FileNotFoundError. I don't think that these errors can occur on a same function. For example, rmdir() only raises ENOTDIR. I only see one advantage of FileSystemError: it doesn't handle FileDescriptorError. Advice usage of FileSystemError would avoid to hide real bugs like FileDescriptorError. I don't really care of ConnectionError. Anyway, FileSystemError and ConnectionError can be added later if needed. WindowsError and winerror attribute --- I don't like the idea of having a winerror attribute platforms other than Windows. Why not using hasattr(err, "winerror") instead? WindowsError is already specific to Windows. I don't see any usage of the winerror attribute in the Python stdlib. shutil module uses: try: WindowsError exce
Re: [Python-Dev] Comments of the PEP 3151
Hello, > By the way, is it faster to not handle and than re-raise unwanted > exceptions? You mean "is it faster to not handle than re-raise unwanted exceptions?". It probably is, yes. > I don't understand how Antoine decided which errno should have an > exception or not. Mostly experience with the stdlib combined with intuition. The list is not cast in stone. > If we add EINTR, I don't know if it's better to add it to > BlockingIOError or to create a new exception (InterruptError?). EINTR is not related to non-blocking I/O, so it shouldn't map to BlockingIOError. > Would it be possible to have an (exhaustive?) list of errno with their > popularity? At least, their popularity in the Python standard library? How do you suggest to do that? > Does raising a specific error (e.g. raise FileNotFound()) set errno and > message attributes (to something different than None)? No, it doesn't. "errno" should IMO only be set if it's truely returned by a system routine. As for the message, like with every other exception it should be supplied by the developer. > Do specific errors slows down raising exceptions? Do raising an IOError > (or a "legacy" error) use a O(1) lookup to choose the exception class? It uses a dict. You could run some benchmarks if you want, but I doubt it makes much sense to worry about that. > EACCES and EPERM have a different meaning. Even that they are very > similar, we might provide two different exceptions. EPERM is only used > once in the Python stdlib, so we might only provide AccesError. > > On Linux, EPERM usually indicates an operation requiring root > priviledge. EACCES can be related to filesystem permissions (read-only, > user is not allowed to write, etc.) or can be an mmap error. I'd have to research that a bit more. Perhaps EACCES can be mapped to AccessError and EPERM to PermissionError, but I think the proximity of the two concepts will make things confusing, for little benefit. > Because IOError, OSError, select.error, etc. are well known exceptions, > I'm not in favor of removing them from Python 3. It would make porting > from Python 2 worse. If we don't remove them, they should not be > deprecated. Ok. Does anyone else have an opinion on deprecations? (not deprecating them means less work for me, anyway) > Test the implementation > --- > > Did someone test Blender, Django or another major applications on the > implementation of the PEP 3151? Does Django have a working Python 3 port yet? > WindowsError and winerror attribute > --- > > I don't like the idea of having a winerror attribute platforms other > than Windows. Well, there isn't, as you can see in the tests (test_windows_error in Lib/test/test_pep3151.py). > shutil module uses: > > (...) > > try: > copystat(src, dst) > except OSError as why: > if WindowsError is not None and isinstance(why, WindowsError): > # Copying file access times may fail on Windows > pass That's the kind of code the PEP hopes to discourage :) > Lock.acquire() doesn't raise an exception on timeout: just remove "(for > example in Lock.acquire())". Oops, will fix. > Should FileNotFound handle ENODEV? (see test_ossaudiodev) That's not really the same thing. For example, mmap says: [ENODEV] The fildes argument refers to a file whose type is not supported by mmap(). (http://www.opengroup.org/sud/sud1/xsh/mmap.htm) Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Comments of the PEP 3151
Antoine Pitrou writes: > > Did someone test Blender, Django or another major applications on the > > implementation of the PEP 3151? > > Does Django have a working Python 3 port yet? Define "working". Martin ported Django to Python 3 years ago as proof of concept, but never claimed it was ready for production use or as the main line of development. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] is sys.modules not meant to be replaced?
On Sun, Jul 24, 2011 at 12:54 AM, Nick Coghlan wrote: > On Sun, Jul 24, 2011 at 3:05 PM, Eric Snow > wrote: >> Are there other objects in the interpreter state that are exposed in >> sys that would have the same problem? > > Rebinding (rather than mutating) any global state in any module is > always dubious due to the potential for direct references to the > previous value. (This is a large part of the reason why imp.reload() > often doesn't work in practice) > > sys.modules, sys.path, sys.argv, sys.stdout, et al are all subject to > that caveat. For mutable state, the onus is typically on the developer > performing the substitution to decide whether or not those > consequences are acceptable (usually, they're not, hence you get > idioms like "sys.path[:] = saved_copy" to revert sys.path to a saved > value). For immutable state (such as sys.stdout), where modification > in place isn't possible, the obligation often goes the other way, with > developers deliberately avoiding cached references in order to > correctly react to runtime changes. I agree with what you are saying wholeheartedly, but still think there is something fishy with the way that sys.modules works. I'll take it from here to a tracker issue though. :) -eric > sys.modules is by far the worst case though, since dropping modules > out of that cache is only safe for modules that correctly support > imp.reload(). This is not the case for any module that mutates other > global state (or is otherwise referenced from other modules), nor does > it work properly for extension modules. > > Cheers, > Nick. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Comments of the PEP 3151
On Mon, Jul 25, 2011 at 9:56 AM, Victor Stinner wrote: > By the way, is it faster to not handle and than re-raise unwanted > exceptions? Yes, but probably not that much faster given the overhead of instantiating the exception and unwinding the stack in the first place. > Choice of the specific errors > - > > I don't understand how Antoine decided which errno should have an > exception or not. > > For example, EINTR and EPIPE are handled in many modules of the standard > library but don't have their exception, whereas EISDIR > (IsADirectoryError) and ENOTDIR (NotADirectoryError) are very rare > (EISDIR is never handled in the standard library) but have their > exception. We do tend to call isdir() fairly often, though. Part of the appeal of isdir() is the ugliness of using EAFP when you have to do explicit errno checks. > If we add EINTR, I don't know if it's better to add it to > BlockingIOError or to create a new exception (InterruptError?). InterruptedError seems like a reasonable candidate for addition to me - catch and retry in that case is something developers are likely to want to do. Perhaps EPIPE should map to FileDescriptorError along with EBADF, with different messages based on the exact error code? Potentially renamed to DescriptorStateError? I'm not sure of the reason for singling out EBADF for special treatment in this case - there are several other errno values that indicate a descriptor is no longer in a usable state (ECONSHUTDOWN would be another one, in the same vein as EPIPE). To be honest though, what's the use case for *catching* FileDescriptorError without catching IOError in general? Perhaps this one should be dropped entirely, to be handled by broadly catching IOError? > Another good candidate is EINVAL. As with EBADF, I'm having trouble visualising a clear use case for handling things like EINVAL and EOPNOTSUP differently from other kinds of IO errors. > Would it be possible to have an (exhaustive?) list of errno with their > popularity? At least, their popularity in the Python standard library? Probably, but the stdlib is more a *generator* of low level exceptions, so our code likely isn't representative enough to get a decent set of numbers. > If we provide an error message error: should it be localized? The > description of FileDescriptorError tells about the "default error > message". It we use a localized message, it's not possible to > preallocate or cache instances. We don't localize anything else, so there's no reason to start here. > PermissionError > --- > > EACCES and EPERM have a different meaning. Even that they are very > similar, we might provide two different exceptions. EPERM is only used > once in the Python stdlib, so we might only provide AccesError. > > On Linux, EPERM usually indicates an operation requiring root > priviledge. EACCES can be related to filesystem permissions (read-only, > user is not allowed to write, etc.) or can be an mmap error. Code that cares can still fall back to exc.errno == EPERM. I don't think we'd be doing anyone any favours by exposing subtle distinctions like this at the Python level. > Deprecation > --- > > Because IOError, OSError, select.error, etc. are well known exceptions, > I'm not in favor of removing them from Python 3. It would make porting > from Python 2 worse. If we don't remove them, they should not be > deprecated. > > I'm in favor of adding a note in the documentation of all legacy > exceptions to advice to use IOError or specific exceptions instead. I > suppose that these notes will have to indicate a Python version. +1 for grandfathering in the old exception names, but documenting the recommended alternatives as of 3.3. > -1 on FileSystemError > - > > I'm not sure that we need FileSystemError or ConnectionError. Careless > code will use IOError, whereas careful code will use an explicit list > like (ConnectionAbortedError, ConnectionRefusedError, > ConnectionResetError). > > If we remove IsADirectoryError and NotADirectoryError, FileSystemError > only contains FileExistsError and FileNotFoundError. I don't think that > these errors can occur on a same function. For example, rmdir() only > raises ENOTDIR. > > I only see one advantage of FileSystemError: it doesn't handle > FileDescriptorError. Advice usage of FileSystemError would avoid to hide > real bugs like FileDescriptorError. And that's precisely why FileSystemError is worthwhile - to separate out when the FS is objecting, rather than there being something wrong with the internal application state or when a previously valid descriptor has become unusable for some reason. It may also be reasonable to return a new DeviceNotAvailableError for ENODEV and EBUSY (as a new FileSystemError subclass). > I don't really care of ConnectionError. Anyway, FileSystemError and > ConnectionError can be added later if needed. But the use case for grouping them is quite obvious - there's
Re: [Python-Dev] is sys.modules not meant to be replaced?
On Mon, Jul 25, 2011 at 2:50 PM, Eric Snow wrote: > I agree with what you are saying wholeheartedly, but still think there > is something fishy with the way that sys.modules works. I'll take it > from here to a tracker issue though. :) Yup - the import system has a whole mess of interrelated global state that really needs to be on a class that can use descriptors to correctly invalidate caches when attributes are mutated or rebound. The ImportEngine GSoC project is the first step along the long, winding path towards doing something about that :) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] is sys.modules not meant to be replaced?
> I agree with what you are saying wholeheartedly, but still think there > is something fishy with the way that sys.modules works. I'll take it > from here to a tracker issue though. :) Well, there is a short answer to you question: sys.modules is *not* meant to be replaced. Doing so will result in undefined behavior. So there is nothing fishy about it. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
