[issue1764407] The -m switch does not use the builtin __main__ module
Nick Coghlan added the comment: Fixed committed to SVN as r57461 -- resolution: -> fixed status: open -> closed _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1764407> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1056] test_cmd_line starts python without -E
Nick Coghlan added the comment: Fixed for 2.6 in rev 58103 (Is the head still being merged to the py3k branch? Or does this need to be forward-ported manually?) -- nosy: +ncoghlan resolution: -> fixed status: open -> closed __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1056> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1739468] Add a -z interpreter flag to execute a zip file
Nick Coghlan added the comment: I like PJE's approach, and the patch works for me. About the only thing I'd change is to switch the expression in PyImport_GetImporter to a simple chain of if-statements in order to: - silence the warning from GCC about an unused value - make it more obvious to a reader what the function is doing An optimising compiler is going to produce similar code either way, and it took me a moment to realise that the && operations are being used purely for their short-circuiting effect, even though there is no real advantage to using an expression instead of a statement at that point in the code. Adding a simple test of the functionality to test_cmd_line would also be good. _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1739468> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1705170] contextmanager eats StopIteration
Changes by Nick Coghlan: -- assignee: -> ncoghlan _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1705170> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1705170] contextmanager eats StopIteration
Nick Coghlan added the comment: Fixed for 2.6 in rev 58766. I'm not sure if it will be possible to get this into 2.5.2. Leaving open against 2.5 until it is checked in on the maintenance branch. -- components: +Library (Lib) -None resolution: -> accepted versions: +Python 2.5 _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1705170> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1705170] contextmanager eats StopIteration
Nick Coghlan added the comment: Close, but not quite. The problem is that the 'value' argument may be None if instantiation of the exception hasn't been forced before __exit__ gets called. >>> class TestWith(object): ... def __enter__(self): ... pass ... def __exit__(self, exc_type, exc_value, exc_tb): ... print exc_type, exc_value, exc_tb ... >>> from __future__ import with_statement >>> with TestWith(): iter([]).next() ... None Traceback (most recent call last): File "", line 1, in StopIteration That 'None' in the middle there is the problem - contextmanager.__exit__ needs to be detecting that and instantiating the passed in exception if it isn't already instantiated. Something like the following at the start of the else clause should do the trick: if value is None: value = type() -- nosy: +ncoghlan _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1705170> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1291] test_resource fails on recent linux systems (
Nick Coghlan added the comment: I just hit this as well when rerunning the 2.5 tests before checking something else in. The test itself appears to be fine, but the call to f.close() outside the try/except block attempting to flush the file to disk and raising an IOError. Didn't something like this get fixed recently? Did the new IO module in py3k have a similar problem? (assigning to Neal to make a call on the importance for 2.5.2) -- assignee: -> nnorwitz components: +Interpreter Core -Extension Modules nosy: +ncoghlan, nnorwitz priority: -> high resolution: -> accepted __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1291> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1705170] contextmanager eats StopIteration
Nick Coghlan added the comment: Done in rev 58901 -- resolution: accepted -> fixed status: open -> closed _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1705170> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1291] test_resource fails on recent linux systems (
Nick Coghlan added the comment: I just compared the 2.5 test_resource with the trunk test_resource - the latter has been modified to remove the file size limitation before it attempts to close the file, eliminating the test failure without changing the underlying behaviour of f.close. __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1291> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1739468] Add a -z interpreter flag to execute a zip file
Nick Coghlan added the comment:
Attached an updated version of PJE's patch with the suggested cleanups
and a new unit test file (test_cmd_line_script.py). Finding the
roundtuits to finish the latter is actually what has taken me so long.
The basic tests and the directory tests are currently working, but for
some reason the zipfile tests are attempting to load __main__ using
pkgutil.ImpLoader instead of the zipimport module.
I'm posting the patch anyway to see if anyone else can spot where it's
going wrong before I find some more time to try and figure it out for
myself.
Added file: http://bugs.python.org/file8767/runmain_with_tests.diff
_
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue1739468>
_Index: Python/import.c
===
--- Python/import.c (revision 59036)
+++ Python/import.c (working copy)
@@ -104,7 +104,6 @@
};
#endif
-static PyTypeObject NullImporterType; /* Forward reference */
/* Initialize things */
@@ -167,7 +166,7 @@
/* adding sys.path_hooks and sys.path_importer_cache, setting up
zipimport */
- if (PyType_Ready(&NullImporterType) < 0)
+ if (PyType_Ready(&PyNullImporter_Type) < 0)
goto error;
if (Py_VerboseFlag)
@@ -1088,7 +1087,7 @@
}
if (importer == NULL) {
importer = PyObject_CallFunctionObjArgs(
- (PyObject *)&NullImporterType, p, NULL
+ (PyObject *)&PyNullImporter_Type, p, NULL
);
if (importer == NULL) {
if (PyErr_ExceptionMatches(PyExc_ImportError)) {
@@ -1106,6 +1105,20 @@
return importer;
}
+PyAPI_FUNC(PyObject *)
+PyImport_GetImporter(PyObject *path) {
+PyObject *importer=NULL, *path_importer_cache=NULL, *path_hooks=NULL;
+
+ if ((path_importer_cache = PySys_GetObject("path_importer_cache"))) {
+ if ((path_hooks = PySys_GetObject("path_hooks"))) {
+ importer = get_path_importer(path_importer_cache,
+path_hooks, path);
+ }
+ }
+ Py_XINCREF(importer); /* get_path_importer returns a borrowed reference
*/
+ return importer;
+}
+
/* Search the path (default sys.path) for a module. Return the
corresponding filedescr struct, and (via return arguments) the
pathname and an open file. Return NULL if the module is not found. */
@@ -3049,7 +3062,7 @@
};
-static PyTypeObject NullImporterType = {
+PyTypeObject PyNullImporter_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
"imp.NullImporter",/*tp_name*/
sizeof(NullImporter), /*tp_basicsize*/
@@ -3096,7 +3109,7 @@
{
PyObject *m, *d;
- if (PyType_Ready(&NullImporterType) < 0)
+ if (PyType_Ready(&PyNullImporter_Type) < 0)
goto failure;
m = Py_InitModule4("imp", imp_methods, doc_imp,
@@ -3118,8 +3131,8 @@
if (setint(d, "PY_CODERESOURCE", PY_CODERESOURCE) < 0) goto failure;
if (setint(d, "IMP_HOOK", IMP_HOOK) < 0) goto failure;
- Py_INCREF(&NullImporterType);
- PyModule_AddObject(m, "NullImporter", (PyObject *)&NullImporterType);
+ Py_INCREF(&PyNullImporter_Type);
+ PyModule_AddObject(m, "NullImporter", (PyObject *)&PyNullImporter_Type);
failure:
;
}
Index: Include/import.h
===
--- Include/import.h(revision 59036)
+++ Include/import.h(working copy)
@@ -24,6 +24,7 @@
#define PyImport_ImportModuleEx(n, g, l, f) \
PyImport_ImportModuleLevel(n, g, l, f, -1)
+PyAPI_FUNC(PyObject *) PyImport_GetImporter(PyObject *path);
PyAPI_FUNC(PyObject *) PyImport_Import(PyObject *name);
PyAPI_FUNC(PyObject *) PyImport_ReloadModule(PyObject *m);
PyAPI_FUNC(void) PyImport_Cleanup(void);
@@ -42,6 +43,7 @@
void (*initfunc)(void);
};
+PyAPI_DATA(PyTypeObject) PyNullImporter_Type;
PyAPI_DATA(struct _inittab *) PyImport_Inittab;
PyAPI_FUNC(int) PyImport_AppendInittab(char *name, void (*initfunc)(void));
Index: Lib/test/test_cmd_line.py
===
--- Lib/test/test_cmd_line.py (revision 59036)
+++ Lib/test/test_cmd_line.py (working copy)
@@ -1,3 +1,6 @@
+# Tests invocation of the interpreter with various command line arguments
+# All tests are executed with environment variables ignored
+# See test_cmd_line_script.py for testing of script execution
import test.test_support, unittest
import sys
Index: Lib/test/test_cmd_line_script.py
===
--- Lib/test/test_
[issue1739468] Add a -z interpreter flag to execute a zip file
Nick Coghlan added the comment:
I worked out what was wrong with my unit tests (I was incorrectly
including the path information when adding the test script to the zipfile)
I've updated the patch here, and will be committing the change once the
test suite finishes running.
--
versions: +Python 2.6
Added file: http://bugs.python.org/file8770/runmain_with_tests.diff
_
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue1739468>
_Index: Python/import.c
===
--- Python/import.c (revision 59036)
+++ Python/import.c (working copy)
@@ -104,7 +104,6 @@
};
#endif
-static PyTypeObject NullImporterType; /* Forward reference */
/* Initialize things */
@@ -167,7 +166,7 @@
/* adding sys.path_hooks and sys.path_importer_cache, setting up
zipimport */
- if (PyType_Ready(&NullImporterType) < 0)
+ if (PyType_Ready(&PyNullImporter_Type) < 0)
goto error;
if (Py_VerboseFlag)
@@ -1088,7 +1087,7 @@
}
if (importer == NULL) {
importer = PyObject_CallFunctionObjArgs(
- (PyObject *)&NullImporterType, p, NULL
+ (PyObject *)&PyNullImporter_Type, p, NULL
);
if (importer == NULL) {
if (PyErr_ExceptionMatches(PyExc_ImportError)) {
@@ -1106,6 +1105,20 @@
return importer;
}
+PyAPI_FUNC(PyObject *)
+PyImport_GetImporter(PyObject *path) {
+PyObject *importer=NULL, *path_importer_cache=NULL, *path_hooks=NULL;
+
+ if ((path_importer_cache = PySys_GetObject("path_importer_cache"))) {
+ if ((path_hooks = PySys_GetObject("path_hooks"))) {
+ importer = get_path_importer(path_importer_cache,
+path_hooks, path);
+ }
+ }
+ Py_XINCREF(importer); /* get_path_importer returns a borrowed reference
*/
+ return importer;
+}
+
/* Search the path (default sys.path) for a module. Return the
corresponding filedescr struct, and (via return arguments) the
pathname and an open file. Return NULL if the module is not found. */
@@ -3049,7 +3062,7 @@
};
-static PyTypeObject NullImporterType = {
+PyTypeObject PyNullImporter_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
"imp.NullImporter",/*tp_name*/
sizeof(NullImporter), /*tp_basicsize*/
@@ -3096,7 +3109,7 @@
{
PyObject *m, *d;
- if (PyType_Ready(&NullImporterType) < 0)
+ if (PyType_Ready(&PyNullImporter_Type) < 0)
goto failure;
m = Py_InitModule4("imp", imp_methods, doc_imp,
@@ -3118,8 +3131,8 @@
if (setint(d, "PY_CODERESOURCE", PY_CODERESOURCE) < 0) goto failure;
if (setint(d, "IMP_HOOK", IMP_HOOK) < 0) goto failure;
- Py_INCREF(&NullImporterType);
- PyModule_AddObject(m, "NullImporter", (PyObject *)&NullImporterType);
+ Py_INCREF(&PyNullImporter_Type);
+ PyModule_AddObject(m, "NullImporter", (PyObject *)&PyNullImporter_Type);
failure:
;
}
Index: Include/import.h
===
--- Include/import.h(revision 59036)
+++ Include/import.h(working copy)
@@ -24,6 +24,7 @@
#define PyImport_ImportModuleEx(n, g, l, f) \
PyImport_ImportModuleLevel(n, g, l, f, -1)
+PyAPI_FUNC(PyObject *) PyImport_GetImporter(PyObject *path);
PyAPI_FUNC(PyObject *) PyImport_Import(PyObject *name);
PyAPI_FUNC(PyObject *) PyImport_ReloadModule(PyObject *m);
PyAPI_FUNC(void) PyImport_Cleanup(void);
@@ -42,6 +43,7 @@
void (*initfunc)(void);
};
+PyAPI_DATA(PyTypeObject) PyNullImporter_Type;
PyAPI_DATA(struct _inittab *) PyImport_Inittab;
PyAPI_FUNC(int) PyImport_AppendInittab(char *name, void (*initfunc)(void));
Index: Lib/test/test_cmd_line.py
===
--- Lib/test/test_cmd_line.py (revision 59036)
+++ Lib/test/test_cmd_line.py (working copy)
@@ -1,3 +1,6 @@
+# Tests invocation of the interpreter with various command line arguments
+# All tests are executed with environment variables ignored
+# See test_cmd_line_script.py for testing of script execution
import test.test_support, unittest
import sys
Index: Lib/test/test_cmd_line_script.py
===
--- Lib/test/test_cmd_line_script.py(revision 0)
+++ Lib/test/test_cmd_line_script.py(revision 0)
@@ -0,0 +1,145 @@
+# Tests command line execution of scripts
+from __future__ import with_statement
+
+import unittest
+import os
+import os.path
+import sys
+import test
+import tempfile
[issue1739468] Add a -z interpreter flag to execute a zip file
Nick Coghlan added the comment: Committed as rev 59039 (now to see how the buildbots react for other platforms...) -- resolution: -> accepted status: open -> closed _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1739468> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1739468] Add a -z interpreter flag to execute a zip file
Nick Coghlan added the comment: Reverted status to open until I figure out why the tests are failing on the Mac OSX buildbot. -- resolution: accepted -> status: closed -> open _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1739468> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1487] PEP 366 implementation
New submission from Nick Coghlan:
Patch to implement PEP 366.
Note that it doesn't implement precisely the semantics described in the
version of the PEP posted in July, as some of those ideas didn't prove
feasible due to the fact that imp.new_module can't tell the difference
between normal modules and packages.
An updated version of the PEP will be posted shortly to correct those
problems.
--
components: Interpreter Core, Library (Lib)
files: pep_366_v1.diff
keywords: patch
messages: 57761
nosy: ncoghlan
priority: normal
severity: normal
status: open
title: PEP 366 implementation
type: rfe
versions: Python 2.6, Python 3.0
Added file: http://bugs.python.org/file8794/pep_366_v1.diff
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue1487>
__Index: Python/import.c
===
--- Python/import.c (revision 59120)
+++ Python/import.c (working copy)
@@ -2106,7 +2106,8 @@
{
static PyObject *namestr = NULL;
static PyObject *pathstr = NULL;
- PyObject *modname, *modpath, *modules, *parent;
+ static PyObject *pkgstr = NULL;
+ PyObject *pkgname, *modname, *modpath, *modules, *parent;
if (globals == NULL || !PyDict_Check(globals) || !level)
return Py_None;
@@ -2121,44 +2122,103 @@
if (pathstr == NULL)
return NULL;
}
+ if (pkgstr == NULL) {
+ pkgstr = PyString_InternFromString("__package__");
+ if (pkgstr == NULL)
+ return NULL;
+ }
*buf = '\0';
*p_buflen = 0;
- modname = PyDict_GetItem(globals, namestr);
- if (modname == NULL || !PyString_Check(modname))
- return Py_None;
+ pkgname = PyDict_GetItem(globals, pkgstr);
- modpath = PyDict_GetItem(globals, pathstr);
- if (modpath != NULL) {
- Py_ssize_t len = PyString_GET_SIZE(modname);
- if (len > MAXPATHLEN) {
+ if ((pkgname != NULL) && (pkgname != Py_None)) {
+ /* __package__ is set, so use it */
+ Py_ssize_t len;
+ if (!PyString_Check(pkgname)) {
PyErr_SetString(PyExc_ValueError,
- "Module name too long");
+ "__package__ set to non-string");
return NULL;
}
- strcpy(buf, PyString_AS_STRING(modname));
- }
- else {
- char *start = PyString_AS_STRING(modname);
- char *lastdot = strrchr(start, '.');
- size_t len;
- if (lastdot == NULL && level > 0) {
+ len = PyString_GET_SIZE(pkgname);
+ if (len == 0) {
+ if (level > 0) {
+ PyErr_SetString(PyExc_ValueError,
+ "Attempted relative import in
non-package");
+ return NULL;
+ }
+ return Py_None;
+ }
+ if (len > MAXPATHLEN) {
PyErr_SetString(PyExc_ValueError,
- "Attempted relative import in non-package");
+ "Package name too long");
return NULL;
}
- if (lastdot == NULL)
+ strcpy(buf, PyString_AS_STRING(pkgname));
+ } else {
+ /* __package__ not set, so figure it out and set it */
+ modname = PyDict_GetItem(globals, namestr);
+ if (modname == NULL || !PyString_Check(modname))
return Py_None;
- len = lastdot - start;
- if (len >= MAXPATHLEN) {
- PyErr_SetString(PyExc_ValueError,
- "Module name too long");
- return NULL;
+
+ modpath = PyDict_GetItem(globals, pathstr);
+ if (modpath != NULL) {
+ /* __path__ is set, so modname is already the package
name */
+ Py_ssize_t len = PyString_GET_SIZE(modname);
+ int error;
+ if (len > MAXPATHLEN) {
+ PyErr_SetString(PyExc_ValueError,
+ "Module name too long");
+ return NULL;
+ }
+ strcpy(buf, PyString_AS_STRING(modname));
+ error = PyDict_SetItem(globals, pkgstr, modname);
+
[issue11715] Building Python on multiarch Debian and Ubuntu
Nick Coghlan added the comment: It wouldn't surprise me at all if the laptop's links were a little off - I started with a Kubuntu image off VMWare's site quite some time ago, then dist-upgraded it through a couple of releases as they came out. -- ___ Python tracker <http://bugs.python.org/issue11715> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Nick Coghlan added the comment: The feature patch for sendmsg/recvmsg support came with a swathe of new tests, and the failures are in those new tests rather than anything breaking in the old ones. As Charles-François noted though, it doesn't look like the feature implementation itself is doing anything wrong, just that there are limits to what Mac OS X allows us to do with it (hence why I closed this feature request and opened issue #12958 to cover the task of updating the test suite to accurately reflect that situation). -- ___ Python tracker <http://bugs.python.org/issue6560> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12958] test_socket failures on Mac OS X
Nick Coghlan added the comment: OK, I've now looked into *why* the socket tests are throwing errors in tearDown, and it has to do with the way the threaded client/server tests in test_socket are set up. Specifically, ThreadableTest uses tearDown to reraise any exception raised in the client thread, and these are therefore outside the scope of the "expectedFailure" suppression in unittest. Now that I've tracked this down, it would be fairly straightforward to fix this specifically within test_socket.ThreadableTest by appropriately adjusting the definition of ThreadableTest.clientRun to discard exceptions encountered in tests flagged as expected failures. However, I'm wondering if that's the right thing to do. Perhaps it would make more sense to change unittest itself so that "expectedFailure" also suppresses tearDown errors. It doesn't seem all that unusual for a known failing test to also cause problems for the tearDown code. Added Michael to the nosy list to ask for his advice/opinion. In the meantime, I'll work on a patch that adjusts ThreadableTest directly. -- assignee: -> ncoghlan nosy: +michael.foord ___ Python tracker <http://bugs.python.org/issue12958> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1294232] Error in metaclass search order
Nick Coghlan added the comment: Looking at Daniel's updated patch is still on my to-do list, but I won't object if anyone else wants to take this forward (it will be at least a few weeks before I get to it). -- ___ Python tracker <http://bugs.python.org/issue1294232> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12958] test_socket failures on Mac OS X
Nick Coghlan added the comment: OK, I'll just deal with the problem directly in test_socket then. It looks like my latest attempt (suppressing unittest._ExpectedFailure in test_socket.ThreadableTest.clientRun) did the trick, so I'll push the updated tests some time this evening: http://www.python.org/dev/buildbot/all/builders/AMD64%20Snow%20Leopard%202%20custom/builds/44/steps/test/logs/stdio -- ___ Python tracker <http://bugs.python.org/issue12958> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10548] Error in setUp not reported as expectedFailure (unittest)
Nick Coghlan added the comment: As another data point, this question came up again in the context of issue #12958. The new test_socket.ThreadableTest uses tearDown() to pick up and reraise any exception that occurred in the client thread. This meant that my initial attempts at flagging some expected failures (due to Mac OS X limitations) didn't work - the client half of the failure was thrown in tearDown() and reported as an error. While I've determined how to avoid that problem in the test_socket case, the general question of whether or not we consider it legitimate to put common assertions in setUp() and tearDown(), or expect that test code explicitly cope with tearDown() failures that occur due to expected test failures still needs to be addressed. To my mind, bugs in my test infrastructure are going to get flushed out by tests that I'm neither skipping nor marking as expected failures. If I have a test that is known to fail in a way that invalidates the standard tearDown procedure for the test infrastructure, having to special case that situation in the tearDown code seems to go against the spirit of offering the "expectedFailure" decorator in the first place. I don't think the same reasoning holds for setUp though - there's no way for a failing test to reach back and force setUp to fail, so any errors raised there are always going to be infrastructure errors. -- nosy: +ncoghlan ___ Python tracker <http://bugs.python.org/issue10548> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13004] pprint: add option to truncate sequences
Changes by Nick Coghlan : -- dependencies: +general pprint rewrite ___ Python tracker <http://bugs.python.org/issue13004> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11682] PEP 380 reference implementation for 3.3
Changes by Nick Coghlan : Removed file: http://bugs.python.org/file22616/pep380-missing-docs.diff ___ Python tracker <http://bugs.python.org/issue11682> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis
Nick Coghlan added the comment: Attached patch should now be complete, including the documentation for the new keyword-only 'file' parameter on various dis module functions. -- Added file: http://bugs.python.org/file23197/issue11816_get_opinfo_branch_20110920.diff ___ Python tracker <http://bugs.python.org/issue11816> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11682] PEP 380 reference implementation for 3.3
Changes by Nick Coghlan : Added file: http://bugs.python.org/file22616/pep380-missing-docs.diff ___ Python tracker <http://bugs.python.org/issue11682> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11682] PEP 380 reference implementation for 3.3
Nick Coghlan added the comment: I have updated the bitbucket repo with changes to address most of Benjamin's review comments. A few points of note: - I agree in principle with the idea of splitting Yield and YieldFrom into distinct AST nodes, but I'd prefer to focus on getting the current implementation into the main repo first - in cleaning up the handling of missing send/close/throw attributes on subiterators I discovered that the PyObject_CallMethod* APIs were discarding exception information and coercing everything to a terse AttributeError. The branch now changes them to allow the error reported by the underlying call to PyObject_GetAttr to pass through unmodified. - the generator close() method treats an AttributeError as expected when looking for a close() method on the subiterator, but uses WriteUnraisable to deal with anything else. - I share Benjamin's suspicion that some of the INCREF/DECREF pairs in genobject.c aren't actually necessary, but I don't think it's worth messing with them at this stage. I haven't looked at Zbyszek's proposed doc changes at this point - the pull request has a lot of irrelevant line wrapping changes in it, so it makes it hard to review. -- ___ Python tracker <http://bugs.python.org/issue11682> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis
Nick Coghlan added the comment: 'Op' is just an abbreviation of 'operation'. So 'operation code' becomes 'opcode' and 'operation information' becomes 'opinfo'. The fact that it comes for the 'dis' module gives the context that the *kind* of operation we're talking about is a Python byte code instruction. When people are hacking on bytecode in the future, they'll likely end up using get_opinfo() a fair bit, so swapping the succinct 'opinfo' for the verbose 'bytecode_instruction' strikes me as a poor trade-off. -- ___ Python tracker <http://bugs.python.org/issue11816> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13062] Introspection generator and function closure state
New submission from Nick Coghlan : Based on the python-ideas thread about closures, I realised there are two features the inspect module could offer to greatly simplify some aspects of testing closure and generator behaviour: inspect.getclosure(func) Returns a dictionary mapping closure references from the supplied function to their current values. inspect.getgeneratorlocals(generator) Returns the same result as would be reported by calling locals() in the generator's frame of execution The former would just involve syncing up the names on the code object with the cell references on the function object, while the latter would be equivalent to doing generator.gi_frame.f_locals with some nice error checking for when the generator's frame is already gone (or the supplied object isn't a generator iterator). -- messages: 144606 nosy: ncoghlan priority: normal severity: normal status: open title: Introspection generator and function closure state type: feature request ___ Python tracker <http://bugs.python.org/issue13062> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13062] Introspection generator and function closure state
Nick Coghlan added the comment: Yep, that looks right to me. The eval loop then references those cells from the frame object during execution. -- ___ Python tracker <http://bugs.python.org/issue13062> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13062] Introspection generator and function closure state
Nick Coghlan added the comment:
Huh, I didn't actually realise getclosure() could be written as a one liner
until seeing Meador's version above:
{var : cell.cell_contents for var, cell in zip(func.__code__.co_freevars,
func.__closure__)}
--
___
Python tracker
<http://bugs.python.org/issue13062>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13075] PEP-0001 contains dead links
Nick Coghlan added the comment: These pages are all still on python.org - the links just need to be updated to point to the devguide equivalents (under http://docs.python.org/devguide) -- nosy: +ncoghlan ___ Python tracker <http://bugs.python.org/issue13075> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13053] Add Capsule migration documentation to "cporting"
Nick Coghlan added the comment: Mostly looks good - couple of minor comments in Reitveld. As far as the patch flow goes, the 2.x and 3.x branches are actually handled independently (they're too divergent for merging to make sense). So 2.7 and 3.2 will be independent commits, then the changes will be merged into default from the 3.2 branch. -- ___ Python tracker <http://bugs.python.org/issue13053> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13062] Introspection generator and function closure state
Nick Coghlan added the comment:
Because a generator can legitimately have no locals:
>>> def gen():
... yield 1
...
>>> g = gen()
>>> g.gi_frame.f_locals
{}
Errors should be reported as exceptions - AttributeError or TypeError if
there's no gi_frame and then ValueError or RuntimeError if gi_frame is None.
--
___
Python tracker
<http://bugs.python.org/issue13062>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13062] Introspection generator and function closure state
Nick Coghlan added the comment: The function case is simpler - AttributeError or TypeError if there's no __closure__ attribute, empty mapping if there's no closure. I've also changed my mind on the "no frame" generator case - since that mapping will evolve over time as the generator executes anyway, the empty mapping accurately reflects the "no locals currently defined" that applies when the generator either hasn't been started yet or has finished. People can use getgeneratorstate() to find that information if they need to know. -- ___ Python tracker <http://bugs.python.org/issue13062> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7689] Pickling of classes with a metaclass and copy_reg
Nick Coghlan added the comment: Specifically, 2.7.3. A date for that has not yet been set, but somewhere in the December/January time frame is likely. -- ___ Python tracker <http://bugs.python.org/issue7689> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13101] Module Doc viewer closes when browser window closes on Windows 8
Nick Coghlan added the comment: If that's the app I think it is (pydoc -g), we're probably going to kill it off in 3.3 in favour of the -b option. -- nosy: +ncoghlan ___ Python tracker <http://bugs.python.org/issue13101> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13101] Module Doc viewer closes when browser window closes on Windows 8
Nick Coghlan added the comment: Slight correction, pydoc.gui() is already gone in current hg tip. However, this error may be indicative of an underlying problem with webbrowser.open(url) throwing an exception. -- ___ Python tracker <http://bugs.python.org/issue13101> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13105] Please elaborate on how 2.x and 3.x are different heads
Nick Coghlan added the comment: This was from memory, so don't take it as gospel as far as the current security-fix-only branches go, but here's what I sent to Larry: - We maintain two independent heads in hg: 2.7 and default 3.2 is open for general bugfixes 2.5 (IIRC), 2.6 and 3.1 are open for security fixes Security fixes (if applicable to both heads) go: 2.5 -> 2.6 -> 2.7 3.1 -> 3.2 -> default General bug fixes (if applicable to both heads) go: 2.7 3.2 -> default New features are added to default only The relative ordering of 2.x and 3.x changes doesn't really matter - the important thing is not to merge them in *either* direction. I think you can theoretically do cherry-picking with Hg, but most people seem to just do independent commits to the two streams. - -- ___ Python tracker <http://bugs.python.org/issue13105> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12602] Missing using docs cross-references
Nick Coghlan added the comment: The 'using docs' are, oddly enough, the part of the docs called 'using' :) In particular, the part about the command line components (including the '
[issue8087] Unupdated source file in traceback
Nick Coghlan added the comment: (just reviewing the idea here, not the current patch) The problem of "stale code" (i.e. what was executed doesn't match what is displayed in the traceback) is a tricky and subtle one. There are a few different cases: 1. Source displayed does not match source on disk - these cases do happen, but they're almost always due to straight up bugs in the linecache or traceback modules. 2. Source has been changed, but module has not been reloaded - this is the case for "edited source file but forgot to reload module". I've certainly forgotten to do this myself, and I'm far from the only one. This is the case Diego's RFE targets, and I think it has some merit. 3. Source has been changed, module has been reloaded, but object reference is from previous version of module - the patch doesn't detect this. There are various ways we *could* detect it, but they all involve some fairly significant changes to the way compilation and module import work. Aspect 3 is a much deeper (and bigger) problem relating to native introspection support in function and class objects. But that doesn't make Diego's idea to improve Aspect 2 invalid - there have certainly been times when playing at the interactive prompt that such a warning would have reminded me that I needed to reload the module I was working on. -- nosy: +ncoghlan ___ Python tracker <http://bugs.python.org/issue8087> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13062] Introspection generator and function closure state
Nick Coghlan added the comment: No, the naming problem had occurred to me as well. Given the 'vars' builtin, perhaps 'getclosurevars' would do as the name? -- ___ Python tracker <http://bugs.python.org/issue13062> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13062] Introspection generator and function closure state
Nick Coghlan added the comment:
In reviewing Meador's patch (which otherwise looks pretty good), I had a
thought about the functionality and signature of getclosurevars().
Currently, it equates "closure" to "nonlocal scope", which isn't really true -
the function's closure is really the current binding of *all* of its free
variables, and that includes globals and builtins in addition to the lexically
scoped variables from outer scopes.
So what do people think about this signature:
ClosureVars = namedtuple("ClosureVars", "nonlocals globals builtins unbound")
def getclosurevars(func):
"""Returns a named tuple of dictionaries of the current nonlocal, global
and builtin references as seen by the body of the function. A final set of
unbound names is also provided."""
# figure out nonlocal_vars (current impl)
# figure out global_vars (try looking up names in f_globals)
# figure out builtin_vars (try looking up names in builtins)
# any leftover names go in unbound_vars
return ClosureVars(nonlocal_vars, global_vars, builtin_vars, unbound_vars)
Also, something that just occurred to me is that getclosurevars() should work
for already instantiated generator iterators as well as generator functions, so
the current typecheck may need to be made a bit more flexible.
--
___
Python tracker
<http://bugs.python.org/issue13062>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12602] Missing cross-references in Doc/using
Nick Coghlan added the comment: Don't feel bad about not recognising the context - this stuff wasn't documented at all for a long time, and it wasn't until Georg pointed me to the usage docs that I realised adding it there would be the right place. I should have remembered that and been less cryptic when creating this issue. It may suggest a meta-issue though - perhaps 'Documenting Python' should grow a devguide-style description of the Docs layout in source control, with a pointer from the devguide's directory layout description [1]? [1] http://docs.python.org/devguide/setup.html#directory-structure -- ___ Python tracker <http://bugs.python.org/issue12602> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13187] relative imports don't work when circular
Nick Coghlan added the comment: This is the same problem that all "from x import y" circular imports have - it isn't specific to relative imports. Closing as a duplicate of issue 992389 -- resolution: -> duplicate status: open -> closed superseder: -> attribute error after non-from import ___ Python tracker <http://bugs.python.org/issue13187> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue992389] attribute error due to circular import
Nick Coghlan added the comment: Changed the issue title to state clearly that the core issue is with circular imports that attempt to reference module contents at import time, regardless of the syntactic form used. All of the following module level code can fail due to this problem: from . import a from package import submodule from module import a import module module.a -- title: attribute error after non-from import -> attribute error due to circular import ___ Python tracker <http://bugs.python.org/issue992389> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue415492] Compiler generates relative filenames
Nick Coghlan added the comment: It's fairly easy to check this is still a problem: $ ./python Python 3.3.0a0 (default:a06ef7ab7321, Sep 22 2011, 13:41:29) [GCC 4.6.0 20110603 (Red Hat 4.6.0-10)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import setup >>> setup.__file__ 'setup.py' That's a relative path rather than an absolute one. If you edit sys.path to use '.' instead of '', it becomes clear that the import machinery is just doing a blind join of the sys.path entry with the relative location of the file: >>> del sys.modules['setup'] >>> sys.path[0] = '.' >>> import setup >>> setup.__file__ './setup.py' -- nosy: +brett.cannon, ncoghlan resolution: fixed -> status: closed -> open ___ Python tracker <http://bugs.python.org/issue415492> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8060] PEP 3101 string formatting missing engineering presentation type for floating point
Changes by Nick Coghlan : -- nosy: +ncoghlan ___ Python tracker <http://bugs.python.org/issue8060> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7475] codecs missing: base64 bz2 hex zlib hex_codec ...
Nick Coghlan added the comment: They were removed because adding new methods to builtin types violated the language moratorium. Now that the language moratorium is over, the transform/untransform convenience APIs should be added again for 3.3. It's an approved change, the original timing was just wrong. -- assignee: lemburg -> nosy: +ncoghlan ___ Python tracker <http://bugs.python.org/issue7475> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7475] codecs missing: base64 bz2 hex zlib hex_codec ...
Nick Coghlan added the comment: Sorry, I meant to state my rationale for the unassignment - I'm assuming this issue is covered by MAL's recent decision to step away from Unicode and codec maintenance issues. If that's incorrect, MAL can reclaim the issue, otherwise unassigning leaves it open for whoever wants to move it forward. -- ___ Python tracker <http://bugs.python.org/issue7475> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7475] codecs missing: base64 bz2 hex zlib hex_codec ...
Nick Coghlan added the comment:
Some further comments after getting back up to speed with the actual status of
this problem (i.e. that we had issues with the error checking and reporting in
the original 3.2 commit).
1. I agree with the position that the codecs module itself is intended to be a
type neutral codec registry. It encodes and decodes things, but shouldn't
actually care about the types involved. If that is currently not the case in
3.x, it needs to be fixed.
This type neutrality was blurred in 2.x by the fact that it only implemented
str->str translations, and even further obscured by the coupling to the
.encode() and .decode() convenience APIs. The fact that the type neutrality of
the registry itself is currently broken in 3.x is a *regression*, not an
improvement. (The convenience APIs, on the other hand, are definitely *not*
type neutral, and aren't intended to be)
2. To assist in producing nice error messages, and to allow restrictions to be
enforced on type-specific convenience APIs, the CodecInfo objects should grow
additional state as MAL suggests. To avoid redundancy (and inaccurate
overspecification), my suggested colour for that particular bikeshed is:
Character encoding codec:
.decoded_format = 'text'
.encoded_format = 'binary'
Binary transform codec:
.decoded_format = 'binary'
.encoded_format = 'binary'
Text transform codec:
.decoded_format = 'text'
.encoded_format = 'text'
I suggest using the fuzzy format labels mainly due to the existence of the
buffer API - most codec operations that consume binary data will accept
anything that implements the buffer API, so referring specifically to 'bytes'
in error messages would be inaccurate.
The convenience APIs can then emit errors like:
'a'.encode('rot_13') ==>
CodecLookupError: text <-> binary codec expected ('rot_13' is text <-> text)
'a'.decode('rot_13') ==>
CodecLookupError: text <-> binary codec expected ('rot_13' is text <-> text)
'a'.transform('bz2') ==>
CodecLookupError: text <-> text codec expected ('bz2' is binary <-> binary)
'a'.transform('ascii') ==>
CodecLookupError: text <-> text codec expected ('ascii' is text <-> binary)
b'a'.transform('ascii') ==>
CodecLookupError: binary <-> binary codec expected ('ascii' is text <->
binary)
For backwards compatibility with 3.2, codecs that do not specify their formats
should be treated as character encoding codecs (i.e. decoded format is 'text',
encoded format is 'binary')
--
___
Python tracker
<http://bugs.python.org/issue7475>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7475] codecs missing: base64 bz2 hex zlib hex_codec ...
Nick Coghlan added the comment:
Oops, typo in my second error example. The command should be:
b'a'.decode('rot_13')
(Since str objects don't offer a decode() method any more)
--
___
Python tracker
<http://bugs.python.org/issue7475>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7475] codecs missing: base64 bz2 hex zlib hex_codec ...
Nick Coghlan added the comment:
On Thu, Oct 20, 2011 at 8:34 AM, STINNER Victor wrote:
>> str.transform('bz2') ==> CodecLookupError
>
> A lookup error is surprising here. It may be a TypeError instead. The bz2 can
> be used with .transform, but not on str. So:
No, it's the same concept as the other cases - we found a codec with
the requested name, but it's not the kind of codec we wanted in the
current context (i.e. str.transform). It may be that the problem is
the user has a str when they expected to have a bytearray or a bytes
object, but there's no way for the codec lookup process to know that.
> - Lookup error if the codec cannot be used with encode/decode or
> transform/untransform
> - Type error if the value type is invalid
There's no way for str.transform to tell the difference between "I
asked for the wrong codec" and "I expected to have a bytes object
here, not a str object". That's why I think we need to think in terms
of format checks rather than type checks.
> (CodecLookupError doesn't exist, you propose to define a new exception who
> inherits from LookupError?)
Yeah, and I'd get that to handle the process of creating the nice
error messages. I think it may even make sense to build the filtering
options into codecs.lookup() itself:
def lookup(encoding, decoded_format=None, encoded_format=None):
info = _lookup(encoding) # The existing codec lookup algorithm
if ((decoded_format is not None and decoded_format !=
info.decoded_format) or
(encoded_format is not None and encoded_format !=
info.encoded_format)):
raise CodecLookupError(info, decoded_format, encoded_format)
Then the various encode, decode and transform methods can just pass
the appropriate arguments to 'codecs.lookup' without all having to
reimplement the format checking logic.
--
___
Python tracker
<http://bugs.python.org/issue7475>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13227] Option to make the lru_cache type specific
Nick Coghlan added the comment: Features looks good, but the current patch doesn't handle keyword arguments correctly (more details in the Reitveld review). -- nosy: +ncoghlan ___ Python tracker <http://bugs.python.org/issue13227> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7475] codecs missing: base64 bz2 hex zlib hex_codec ...
Nick Coghlan added the comment:
I'm fine with people needing to drop down to the lower level lookup() API if
they want the filtering functionality in Python code. For most purposes,
constraining the expected codec input and output formats really isn't a major
issue - we just need it in the core in order to emit sane error messages when
people misuse the convenience APIs based on things that used to work in 2.x
(like 'a'.encode('base64')).
At the C level, I'd adjust _PyCodec_Lookup to accept the two extra arguments
and add _PyCodec_EncodeText, _PyCodec_DecodeBinary, _PyCodec_TransformText and
_PyCodec_TransformBinary to support the convenience APIs (rather than needing
the individual objects to know about the details of the codec tagging
mechanism).
Making new codecs available isn't a backwards compatibility problem - anyone
relying on a particular key being absent from an extensible registry is clearly
doing the wrong thing.
Regarding the particular formats, I'd suggest that hex, base64, quopri, uu, bz2
and zlib all be flagged as binary transforms, but rot13 be implemented as a
text transform (Florent's patch has rot13 as another binary transform, but it
makes more sense in the text domain - this should just be a matter of adjusting
some of the data types in the implementation from bytes to str)
--
___
Python tracker
<http://bugs.python.org/issue7475>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13229] Add shutil.filter_walk
New submission from Nick Coghlan : I needed a depth-limited, filtered search of a directory tree recently and came up with the following wrapper around os.walk that brings in a few niceties like glob-style filtering, depth limiting and symlink traversal that is safe from infinite loops. It also emits a named tuple rather than the bare tuple emitted by os.walk: http://code.activestate.com/recipes/577913-selective-directory-walking/ I think this would make a nice addition to 3.3 as shutil.filter_walk, but it need tests, documentation and reformatting as a patch before it can go anywhere. -- components: Library (Lib) messages: 145999 nosy: ncoghlan priority: normal severity: normal stage: test needed status: open title: Add shutil.filter_walk type: feature request versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue13229> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13227] Option to make the lru_cache type specific
Nick Coghlan added the comment: Looks good to me (although it took me longer than it should have to figure out why you didn't need to store the keyword argument names a second time) -- ___ Python tracker <http://bugs.python.org/issue13227> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13237] subprocess docs should use emphasise convenience functions
New submission from Nick Coghlan : Many typical subprocess use cases can now be handled simply via the convenience functions: subprocess.call() subprocess.check_call() subprocess.check_output() However, readers of the documentation could be forgiven for not realising that, since the docs dive right in with Popen() and will scare most readers away in search of more user friendly APIs (or even other languages). The module documentation should be reordered to introduce the helper function first, then Popen afterwards. The "subprocess replacements" [1] section would ideally help address this, but it hasn't been updated to use the convenience function, instead using confusing direct calls to Popen. [1] http://docs.python.org/library/subprocess.html#subprocess-replacements -- assignee: docs@python components: Documentation messages: 146059 nosy: docs@python, ncoghlan priority: normal severity: normal stage: needs patch status: open title: subprocess docs should use emphasise convenience functions versions: Python 2.7, Python 3.2, Python 3.3 ___ Python tracker <http://bugs.python.org/issue13237> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
New submission from Nick Coghlan : I've been doing a few systems administration tasks with Python recently, and shell command invocation directly via the subprocess module is annoyingly clunky (even with the new convenience APIs). Since subprocess needs to avoid the shell by default for security reasons, I suggest we add the following simple helper functions to shutil: def call(cmd, *args, **kwds): if args or kwds: cmd = cmd.format(*args, **kwds) return subprocess.call(cmd, shell=True) def check_call(cmd, *args, **kwds): if args or kwds: cmd = cmd.format(*args, **kwds) return subprocess.check_call(cmd, shell=True) def check_output(cmd, *args, **kwds): if args or kwds: cmd = cmd.format(*args, **kwds) return subprocess.check_output(cmd, shell=True) Initially posted at: http://code.activestate.com/recipes/577891-simple-invocation-of-shell-commands/ Of course, even if others agree in principle, documentation and tests are still needed before this change can go anywhere. -- components: Library (Lib) messages: 146061 nosy: ncoghlan priority: normal severity: normal status: open title: Add shell command helpers to shutil module versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue13238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13237] subprocess docs should emphasise convenience functions
Changes by Nick Coghlan : -- title: subprocess docs should use emphasise convenience functions -> subprocess docs should emphasise convenience functions ___ Python tracker <http://bugs.python.org/issue13237> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment: And that's exactly the problem - a web developer's or security auditor's "shell injection" is a system administrator's "this language sucks". These wrappers are the kind of thing you want for shell invocations when using Python as a replacement for a shell script or rewriting something that was originally written in Perl, but they're a terrible idea if anything you're interpolating came from an untrusted data source. Currently, requiring "shell=True" in the arguments to the subprocess calls is considered a sufficient deterrent against people doing the wrong thing. I'm suggesting that requiring "import shutil" instead of "import subprocess" may be a similarly acceptable compromise that better serves the system administrators that choose to use Python for system automation tasks. -- ___ Python tracker <http://bugs.python.org/issue13238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment: Perhaps a better idea would be to use different names, so it's clearer at the point of invocation that the shell is being invoked (and hence shell injection attacks are a potential concern). For example: shell_call check_shell_call check_shell_output That would make large applications easier to audit (just search for 'shell_') while still making life easier for sysadmins. -- ___ Python tracker <http://bugs.python.org/issue13238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment: Of the 3 available options (mod style, string.Template and str.format), yes, str.format is the best choice. If people want the shell meaning of the braces, they can escape them by doubling them up in the command string. -- ___ Python tracker <http://bugs.python.org/issue13238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment: Initially, because I was suggesting the names shadow the subprocess convenience functions so they *had* to live in a different namespace. However, even after changing the names to explicitly include "shell", I'd like to keep them away from the general subprocess functionality - these wrappers are more convenient for shell operations than the subprocess ones, but it's that very convenience that makes them potentially dangerous in larger applications that may be interpolating data that untrusted users can manipulate. Since the intended audience is system administrators working on shell-like operations, the shell utility module seems like an appropriate place for them. Both the "import shutil" and the "shell" in the names would then serve as red flags on a code review or security audit. -- ___ Python tracker <http://bugs.python.org/issue13238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13237] subprocess docs should emphasise convenience functions
Nick Coghlan added the comment: You couldn't just move them - you'd need to change the wording of how they cross-link to each other, since the explanations of the convenience function currently assume you understand how Popen works. I'd like us to get to the point where you only need to understand the whole Popen swiss army knife if you actually want that level of flexibility. -- ___ Python tracker <http://bugs.python.org/issue13237> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment:
It's a flow thing. This idea was kicked off by the process of translating a
large Perl script to Python and paying attention to what the translation made
*worse*.
One of the big things it made worse was the translation of "qx" (quoted
executable) strings. In Perl, those are almost as neat and tidy as if you were
writing directly in the shell:
qx|ls -l $dirname|
The thought process isn't "build this command and then execute it", it's "run
this shell command".
Yes, you have to be careful that "dirname" is legal in the shell, but that
usually isn't a big problem in practice, because dirname came from a previous
listdir call, or you otherwise know that it's valid to interpolate it into the
command (and if it isn't, then the bug lies in the way 'dirname' was populated,
not in this operation).
Now, Python's never going to have implicit string interpolation, and that's
fine - we have explicit interpolation instead. A direct translation of the
above to idiomatic Python 2.7 code looks like the following:
subprocess.check_output("ls -l {}".format(dirname), shell=True)
Now, if I'm doing system administration tasks (like the script I was
translating), then I'm going to be doing that kind of thing a *lot*. I'm also
likely to be a sysadmin rather than a programmer, so rather than going "Oh, I
can write a helper function for this", my natural reaction is going to be "I'm
going to use a language that doesn't get in my way so much".
This proposal is aimed directly at making Python a better language for systems
administration by making shell invocation almost as easy as it is in Perl:
shutil.check_shell_output("ls -l {}", dirname)
Heck, if someone really wanted to, they could even do:
qx = shutil.check_shell_output
qx("ls -l {}", dirname)
However, this is also why I *don't* want these methods in subprocess - they put
the burden on the user to think about their data as if they were writing shell
scripts, because that data is going to get passed straight to the shell for
execution without any additional escaping. That's a reasonable idea for a shell
utility in shutil, it's not reasonable for a general purpose subprocess
manipulation utility in subprocess.
--
___
Python tracker
<http://bugs.python.org/issue13238>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment:
That's a fair point, but I think it actually *improves* the argument for better
helper functions, since we can have them automatically invoke shlex.quote() on
all of the arguments:
def _shell_format(cmd, args, kwds):
args = map(shlex.quote, args)
kwds = {k:shlex.quote(v) for k, v in kwds}
return cmd.format(*args, **kwds)
def _invoke_shell(func, cmd, args, kwds):
return func(_shell_format(cmd, args, kwds), shell=True)
def format_shell_call(cmd, *args, kwds):
return _shell_format(cmd, args, kwds)
def shell_call(cmd, *args, **kwds):
return _invoke_shell(subprocess.call, cmds, args, kwds)
def check_shell_call(cmd, *args, **kwds):
return _invoke_shell(subprocess.check_call, cmds, args, kwds)
def check_shell_output(cmd, *args, **kwds):
return _invoke_shell(subprocess.check_output, cmds, args, kwds)
--
___
Python tracker
<http://bugs.python.org/issue13238>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13105] Please elaborate on how 2.x and 3.x are different heads
Nick Coghlan added the comment: The devguide actually did align with what I said in my email, but this wasn't clear if you only read the "Forward Porting" section (you had to read the "Porting Between Major Versions" section further down the page as well). I added a new paragraph to the forward porting section so it describes the whole policy, leaving the following two sections to cover the mechanics of actually handling that with Mercurial. -- resolution: -> fixed status: open -> closed ___ Python tracker <http://bugs.python.org/issue13105> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1294232] Error in metaclass search order
Nick Coghlan added the comment: Fix has been applied to 3.x and hence will be in 3.3 and the next 3.2 release. I have adjusted the issue metadata to reflect the fact 2,7 still exhibits the problem, but the patch requires significant work to account for the 3.x vs 2.x changes in class creation before it can be backported. -- components: -Documentation stage: patch review -> needs patch versions: -Python 3.1, Python 3.2, Python 3.3 ___ Python tracker <http://bugs.python.org/issue1294232> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13252] new decumulate() function in itertools module
Nick Coghlan added the comment: OK, looking at the code I realised what you're trying to get at is the idea of reporting the differences between values in a series, such that: x = list(accumulate(seq)) assert x == list(accumulate(differences(x))) I don't think the use cases are there to justify the addition (not every iterator we can think of should be a building block in itertools), but if such a thing were to be added, "differences" would convey the intention significantly better than the invented term "decumulate". -- nosy: +ncoghlan ___ Python tracker <http://bugs.python.org/issue13252> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment:
Unfortunately, I don't think including implicit shlex.quote() calls is going to
have the effect I was originally looking for:
>>> subprocess.call("du -hs ../py*", shell=True)
593M../py3k
577M../py3k_pristine
479M../python27
300M../python31
381M../python32
288K../python_swallowed_whole
0
>>> subprocess.call("du -hs {}".format(shlex.quote("../py*")), shell=True)
du: cannot access `../py*': No such file or directory
1
However, tinkering with this did provide some other "feels like using the
shell" examples:
>>> subprocess.call("du -hs ~/devel", shell=True)
4.5G/home/ncoghlan/devel
0
>>> subprocess.call(["du","-hs","~/devel"])
du: cannot access `~/devel': No such file or directory
1
(I'm using the existing subprocess calls rather than the proposed API, since
I'm playing with this on the current hg tip without making any changes)
--
___
Python tracker
<http://bugs.python.org/issue13238>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13237] subprocess docs should emphasise convenience functions
Nick Coghlan added the comment: I'm reasonably happy with the changes I just checked in, but rather than doing multiple forward ports, my plan is to let them settle for a while, update them based on any feedback I get, then incorporate the final version into the 3.x series. -- ___ Python tracker <http://bugs.python.org/issue13237> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13237] subprocess docs should emphasise convenience functions
Changes by Nick Coghlan : -- assignee: docs@python -> ncoghlan ___ Python tracker <http://bugs.python.org/issue13237> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment:
Considering this further, I've realised that the idea of implicit quoting for
this style of helper function is misguided on another level - the parameters to
be interpolated may not even be strings yet, so attempting to quote them would
fail:
>>> subprocess.call("exit {}".format(1), shell=True)
1
>>> shlex.quote(1)
Traceback (most recent call last):
File "", line 1, in
File "/home/ncoghlan/devel/py3k/Lib/shlex.py", line 285, in quote
if _find_unsafe(s) is None:
TypeError: expected string or buffer
I'll note that none of these problems will be unique to the new convenience API
- they're all a general reflection of the impedance mismatch between typical
shell interfaces and filenames that can contain spaces.
--
___
Python tracker
<http://bugs.python.org/issue13238>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10197] subprocess.getoutput fails on win32
Nick Coghlan added the comment: Without knowing this issue existed, I recently started working on adding some convenience APIs for shell invocation to shutil: http://bugs.python.org/issue13238 I think the getstatus and getstatusoutput APIs were copied from the commands module in 3.0 without sufficient thought being given to whether or not they fit with the design principles of the subprocess module. IMO, both should be deprecated: - they're not cross-platform - they invoke the shell implicitly, which subprocess promises never to do -- nosy: +ncoghlan ___ Python tracker <http://bugs.python.org/issue10197> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment: I discovered a couple of APIs that were moved from the commands module to the subprocess module in 3.0: http://docs.python.org/dev/library/subprocess#subprocess.getstatusoutput However, they have issues, especially on Windows: http://bugs.python.org/issue10197 So part of this patch would include deprecating those two interfaces in favour of the new shutil ones - the existing APIs break the subprocess promise of never invoking the shell implicitly. -- ___ Python tracker <http://bugs.python.org/issue13238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment: After a bit of thought, I realised I could use the string.Formatter API to implement a custom formatter for the shell command helpers that auto-escapes whitespace while leaving the other shell metacharacters alone (so you can still interpolate paths containing wildcards, etc). People that want to bypass the autoescaping of whitespace can do the interpolation prior to the shell call, those that also want to escape shell metacharacters can use shlex.quote explicitly. Diff can be seen here: https://bitbucket.org/ncoghlan/cpython_sandbox/changeset/f19accc9a91b -- hgrepos: +85 ___ Python tracker <http://bugs.python.org/issue13238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13266] Support for unwrapping __wrapped__ functions in 'inspect' module
New submission from Nick Coghlan : I just got bitten by the singularly unhelpful results of doing inspect.getsource(generator_context_manager). Now that @functools.wraps adds the __wrapped__ attribute, perhaps inspect.getsource(f) should follow the wrapper chain by default? This would affect other inspect APIs as well, such as getgeneratorstate() and the new getclosurevars() and getgeneratorlocals() that will be available in 3.3 While the source code of the wrapper could still be accessed by doing inspect.getsource(f.__code__), this isn't a reliable alternative in general (e.g. it doesn't work when introspecting closure state, since that needs access to the function object to look things up correctly). Accordingly, there would need to be a way to disable the "follow wrapper chains behaviour". Alternatively (and more simply), we could just add an "inspect.unwrap(f)" function that followed a chain of __wrapped__ attributes and returned the last object in the chain (using an "already seen" set to avoid hit the recursion limit if someone sets up a wrapper loop). Applying this would then be a manual process when the initial output of an inspect function is clearly coming from a wrapper rather than the underlying function definition. -- components: Library (Lib) messages: 146420 nosy: ncoghlan priority: normal severity: normal status: open title: Support for unwrapping __wrapped__ functions in 'inspect' module versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue13266> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13266] Add inspect.unwrap(f) to easily unravel "__wrapped__" chains
Nick Coghlan added the comment: After a little thought, I think the explicit "unwrap" function is the only viable approach. Doing the unwrapping implicitly just has too many nasty corner cases to track down to ensure we aren't losing existing functionality. I'd also suggest that we add a __wrapped__ alias for the 'func' attribute of functools.partial objects. -- title: Support for unwrapping __wrapped__ functions in 'inspect' module -> Add inspect.unwrap(f) to easily unravel "__wrapped__" chains ___ Python tracker <http://bugs.python.org/issue13266> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13267] Add an option to disable importing orphaned bytecode files
Nick Coghlan added the comment: This feature is unnecessary now that PEP 3147 has been adopted. The way it works in 3.2+ is that orphaned bytecode files inside __pycache__ are always ignored, while bytecode files that live directly in the source directories are always imported. This handles both use cases nicely: - the implicitly created cache files are ignored if their source file has been deleted - bytecode only distribution is still supported by explicitly installing bytecode files instead of source files -- resolution: -> out of date stage: -> committed/rejected status: open -> closed ___ Python tracker <http://bugs.python.org/issue13267> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13237] subprocess docs should emphasise convenience functions
Nick Coghlan added the comment: Absent any further feedback, I think I'm done with the changes to the 2.7 subprocess docs. I'll let them sit for a few days, then do the forward port to 3.2 and default. There are a couple of additional changes I'll add to the 3.x versions: - mention redirecting to DEVNULL as a way of suppressing output when using call() and check_call() - mention the UTF-8 assumption involved in using the universal newlines mode for pipes in 3.x (i.e. the stuff I mistakenly checked in to 2.7 and then reverted) -- ___ Python tracker <http://bugs.python.org/issue13237> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13237] subprocess docs should emphasise convenience functions
Nick Coghlan added the comment: That's deliberate, as I'm only showing a selected subset of the full signature at that point and using the subprocess API's with positional arguments would lead to almost incomprehensible code. I'm not in any great hurry to forward port though, so I'm happy to wait until you get a chance to review the changes. -- ___ Python tracker <http://bugs.python.org/issue13237> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13237] subprocess docs should emphasise convenience functions
Nick Coghlan added the comment: We can only protect people from themselves so much - "shell=True" is invaluable when you actually want to invoke the shell, and the shell has much better tools for process invocation and pipeline processing than Python does (since shells are, in effect, domain specific languages dedicated to those tasks). If someone is blindly copying and pasting code from the internet, then shell injection attacks are likely to be the *least* of the security problems in anything they're building. The point of the examples is to demonstrate the return code handling and using the shell is the easiest way to do that. I'll add a note to the docstrings to be aware of the security issues with the parameter, though. As far as the keyword arguments go, no, I can't just drop the bare '*' from the abbreviated signature, because that would be making claims about the signature that are just plain *wrong* (there are other positional arguments that appear before 'stdin'). I'll add a note explaining that point somewhere in the 2.7 version, though. -- ___ Python tracker <http://bugs.python.org/issue13237> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13237] subprocess docs should emphasise convenience functions
Nick Coghlan added the comment: As the last checkin message says, I've made the documentation for the helper functions more self-contained. Each now has its own short "shell=True" warning with a pointer to the full explanation in the shared parameter description. There was also a copy-and-paste error in the Popen docs where the cross-reference for "shell=True" was a note rather than a warning, so I fixed that. For the helper functions, people are unlikely to pass in "stdout=PIPE" or "stderr=PIPE" in the first place, since there's no point (you can't get at the Popen object, so you can't at the streams in order to read from them). Accordingly, I downgraded the relevant boxes to be notes rather than warnings (the boxes down near Popen.wait() and the data attribute descriptions remain warnings). The latest version also uses the "keyword only" syntax for all three helper functions, but describes them with the following text in each case: = The arguments shown above are merely the most common ones, described below in :ref:`frequently-used-arguments` (hence the slightly odd notation in the abbreviated signature). = For 3.x, I'll reword that to: = The arguments shown above are merely the most common ones, described below in :ref:`frequently-used-arguments` (hence the use of keyword only argument notation in the abbreviated signature). = -- ___ Python tracker <http://bugs.python.org/issue13237> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment: I realised I could use the convert_field() option in the custom formatter to choose between several interpolation quoting options: default - str + shutil.quote_ascii_whitespace !q - str + shlex.quote !u - unquoted (i.e. no conversion, str.format default behaviour) !s - str (as usual) !r - repr (as usual) The most recent commit also exposes public APIs for the formatting aspects (shutil.quote_ascii_whitespace, shutil.shell_format, shutil.shell_format_map) -- keywords: +patch Added file: http://bugs.python.org/file23542/issue13238_shell_helpers.diff ___ Python tracker <http://bugs.python.org/issue13238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment:
Some examples:
>>> import shutil
>>> shutil.shell_call("du -hs {}", "../py*")
594M../py3k
579M../py3k_pristine
480M../python27
301M../python31
382M../python32
288K../python_swallowed_whole
0
>>> shutil.shell_call("du -hs {!q}", "../py*")
du: cannot access `../py*': No such file or directory
1
>>> shutil.shell_call("ls {}", "no file")
ls: cannot access no file: No such file or directory
2
>>> shutil.shell_call("ls {!u}", "no file")
ls: cannot access no: No such file or directory
ls: cannot access file: No such file or directory
2
--
___
Python tracker
<http://bugs.python.org/issue13238>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment: The first version I wrote *did* automatically invoke shlex.quote on all interpolated values, but that breaks wildcard handling. You can see that in the examples I posted above. With the default whitespace escaping (which allows spaces in filenames), wildcard matching still works (thus the list of directories matching the "../py*" pattern), but with full quoting it breaks (thus the "nothing named '../py*'" result). So I figured the simplest default was "you can have spaces in your filenames, but otherwise you can do what you want". Now that I have the 'unquoted' conversion specifier, I'm open to tweaking that scheme to allow the same chars that shlex.quote does *plus* specifically the wildcard matching chars (i.e. '*', '?'). -- ___ Python tracker <http://bugs.python.org/issue13238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13238] Add shell command helpers to shutil module
Nick Coghlan added the comment: Yeah, I was thinking about this a bit more and realised that I'd rejected the "quote everything by default" approach before I had the idea of providing a custom conversion specifier to disable the implicit string conversion and quoting. So perhaps a better alternative would be: default - str + shlex.quote !u - unquoted (i.e. normal str.format default behaviour) When you have a concise way to explicitly bypass it, making the default behaviour as safe as possible seems like a good way to go. -- ___ Python tracker <http://bugs.python.org/issue13238> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13298] Result type depends on order of operands for bytes and bytearray
New submission from Nick Coghlan :
In a recent python-ideas discussion of the differences between concatenation
and augmented assignment on lists, I pointed out the general guiding principle
behind Python's binary operation semantics was that the type of a binary
operation should not depend on the order of the operands. That is "X op Y" and
"Y op X" should either consistently create results of the same type ("1 + 1.1",
"1.1 + 1") or else throw an exception ("[] + ()", "() + []").
This principle is why list concatenation normally only works with other lists,
but will accept arbitrary iterables for augmented assignment. collections.deque
exhibits similar behaviour (i.e. strict on the binary operation, permissive on
augmented assignment).
However, bytes and bytearray don't follow this principle - they accept anything
that implements the buffer interface even in the binary operation, leading to
the following asymmetries:
>>> b'' + bytearray()
b''
>>> b'' + memoryview(b'')
b''
>>> bytearray() + b''
bytearray(b'')
>>> bytearray() + memoryview(b'')
bytearray(b'')
>>> memoryview(b'') + b''
Traceback (most recent call last):
File "", line 1, in
TypeError: unsupported operand type(s) for +: 'memoryview' and 'bytes'
>>> memoryview(b'') + bytearray(b'')
Traceback (most recent call last):
File "", line 1, in
TypeError: unsupported operand type(s) for +: 'memoryview' and 'bytearray'
Now, the latter two cases are due to a known problem where returning
NotImplemented from sq_concat or sq_repeat doesn't work properly (so none of
the relevant method implementations in the stdlib even try), but the bytes and
bytearray interaction is exactly the kind of type asymmetry the operand order
independence guideline is intended to prevent.
My question is - do we care enough to try to change this? If we do, then it's
necessary to decide on more appropriate semantics:
1. The "list" solution, permitting only the same type in binary operations
(high risk of breaking quite a lot of code)
2. Don't allow arbitrary buffers, but do allow bytes/bytearray interoperability
2a. always return bytes from mixed operations
2b. always return bytearray from mixed operations
2c. return the type of the first operand (ala set.__or__)
Or just accept that this really is more of a guideline than a rule and adjust
the documentation accordingly.
--
components: Interpreter Core
messages: 146669
nosy: ncoghlan
priority: normal
severity: normal
status: open
title: Result type depends on order of operands for bytes and bytearray
type: behavior
versions: Python 3.3
___
Python tracker
<http://bugs.python.org/issue13298>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13299] namedtuple row factory for sqlite3
New submission from Nick Coghlan : Currently, sqlite3 allows rows to be easily returned as ordinary tuples (default) or sqlite3.Row objects (which allow dict-style access). collections.namedtuple provides a much nicer interface than sqlite3.Row for accessing ordered data which uses valid Python identifiers for field names, and can also tolerate field names which are *not* valid identifiers. It would be convenient if sqlite3 provided a row factory along the lines of the one posted here: http://peter-hoffmann.com/2010/python-sqlite-namedtuple-factory.html (except with smarter caching on the named tuples) -- messages: 146670 nosy: ncoghlan priority: normal severity: normal stage: needs patch status: open title: namedtuple row factory for sqlite3 type: feature request versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue13299> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis
Nick Coghlan added the comment: Bitbucket repo and attached patch updated relative to current tip. -- Added file: http://bugs.python.org/file23569/issue11816_get_opinfo_branch_20111031.diff ___ Python tracker <http://bugs.python.org/issue11816> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis
Changes by Nick Coghlan : Removed file: http://bugs.python.org/file23197/issue11816_get_opinfo_branch_20110920.diff ___ Python tracker <http://bugs.python.org/issue11816> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis
Changes by Nick Coghlan : Removed file: http://bugs.python.org/file23095/issue11816_get_opinfo_branch_20110904.diff ___ Python tracker <http://bugs.python.org/issue11816> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis
Changes by Nick Coghlan : Removed file: http://bugs.python.org/file23019/issue11816_get_opinfo_branch_20110824.diff ___ Python tracker <http://bugs.python.org/issue11816> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis
Changes by Nick Coghlan : Removed file: http://bugs.python.org/file21599/dis.patch ___ Python tracker <http://bugs.python.org/issue11816> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11682] PEP 380 reference implementation for 3.3
Nick Coghlan added the comment: Bitbucket repo and attached patch updated relative to latest get_opinfo branch (which in turn was updated to apply cleanly against current CPython tip). (I still need to incorporate the doc updates and look into adding keyword argument support) -- Added file: http://bugs.python.org/file23570/issue11682_pep380_branch_2031.diff ___ Python tracker <http://bugs.python.org/issue11682> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11682] PEP 380 reference implementation for 3.3
Changes by Nick Coghlan : Removed file: http://bugs.python.org/file22616/pep380-missing-docs.diff ___ Python tracker <http://bugs.python.org/issue11682> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11682] PEP 380 reference implementation for 3.3
Changes by Nick Coghlan : Removed file: http://bugs.python.org/file23096/issue11682_pep380_branch_20110904.diff ___ Python tracker <http://bugs.python.org/issue11682> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13306] Add diagnostic tools to importlib?
New submission from Nick Coghlan : In discussing the module aliasing PEP on python-ideas, it occurred to me that we could potentially add some useful "sanity check" utilities to an "importlib.diagnostics" subpackage. For example: - scan sys.path looking for entries that are inside packages - scan sys.modules looking for entries with duplicate __file__ attributes - scan for shadowed modules (i.e. like find module, but don't stop when we get a hit, and report *all* modules found) Thoughts? -- messages: 146746 nosy: brett.cannon, ncoghlan priority: normal severity: normal stage: needs patch status: open title: Add diagnostic tools to importlib? type: feature request ___ Python tracker <http://bugs.python.org/issue13306> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13298] Result type depends on order of operands for bytes and bytearray
Nick Coghlan added the comment: We can just use this one - it was more in the nature of a question "is there anything we want to change about the status quo?" than a request for any specific change. I'm actually OK with buffer API based interoperability, but if we're going to offer that, we should be consistent: 1. bytes and bytearray should interoperate with anything supporting the buffer interface (which they already mostly do) 2. When they encounter each other, LHS wins (as with set() and frozenset()) 3. We should fix the operand coercion bug for C level concatenation slot implementations (already covered by issue #11477) 4. Update the documentation as needed Since we're tinkering with builtin behaviour, 1 & 2 should probably be brought up on python-dev once someone checks if there is anything other than .join() that needs updating. -- ___ Python tracker <http://bugs.python.org/issue13298> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13229] Add shutil.filter_walk
Nick Coghlan added the comment:
I should probably update that posted recipe to my latest version (which adds
"excluded_files" and "excluded_dirs" parameters).
However, since I've been dealing with remote filesystems where os.listdir() and
os.stat() calls from the local machine aren't possible lately, I also think we
may need to reconsider how this is structured and look at the idea of building
a more effective pipeline model that permits more efficient modes of
interaction.
Let's take 'os.walk' as the base primitive - the basis of the pipeline will
always be an iterator that produces 3-tuples of a base name, a list of
subdirectories and a list of files. The filtering pipeline elements will
require that the underlying walk include "topdown=True" and pay attention to
changes in the subdirectory list.
Then consider the following possible pipeline elements:
def filter_dirs(walk_iter, *include_filters, exclude_filters=()):
def should_include(dirname):
return any(fnmatch(dirname, include) for include in include_filters)
def should_exclude(dirname):
return any(fnmatch(dirname, include) for exclude in exclude_filters)
for dirpath, subdirs, files in walk_iter:
subdirs[:] = [subdir for subdir in subdirs
if should_include(subdir) and not
should_exclude(subdir)]
yield dirpath, subdirs, files
def filter_files(walk_iter, *include_filters, exclude_filters=()):
def should_include(dirname):
return any(fnmatch(dirname, include) for include in include_filters)
def should_exclude(dirname):
return any(fnmatch(dirname, include) for exclude in exclude_filters)
for dirpath, subdirs, files in walk_iter:
files[:] = [fname for fname in files
if should_include(fname) and not should_exclude(fname)]
yield dirpath, subdirs, files
def limit_depth(walk_iter, depth):
if depth < 0:
msg = "Depth limit greater than 0 ({!r} provided)"
raise ValueError(msg.format(depth))
sep = os.sep
for top, subdirs, files in walk_iter:
yield top, subdirs, files
initial_depth = top.count(sep)
if depth == 0:
subdirs[:] = []
break
for dirpath, subdirs, files in walk_iter:
yield dirpath, subdirs, files
current_depth = dirpath.count(sep) - initial_depth
if current_depth >= depth:
subdirs[:] = []
def detect_symlink_loops(walk_iter, onloop=None):
if onloop is None:
def onloop(path):
msg = "Symlink {!r} refers to a parent directory, skipping\n"
sys.stderr.write(msg.format(path))
sys.stderr.flush()
for top, subdirs, files in walk_iter:
yield top, subdirs, files
real_top = os.path.abspath(os.path.realpath(top))
break
for dirpath, subdirs, files in walk_iter:
if os.path.islink(dirpath):
# We just descended into a directory via a symbolic link
# Check if we're referring to a directory that is
# a parent of our nominal directory
relative = os.path.relpath(dirpath, top)
nominal_path = os.path.join(real_top, relative)
real_path = os.path.abspath(os.path.realpath(dirpath))
path_fragments = zip(nominal_path.split(sep), real_path.split(sep))
for nominal, real in path_fragments:
if nominal != real:
break
else:
if not onloop(dirpath):
subdirs[:] = []
continue
yield dirpath, subdirs, files
And pipeline terminators:
def walk_dirs(walk_iter):
for dirpath, subdirs, files in walk_iter:
yield dirpath
def walk_files(walk_iter):
for dirpath, subdirs, files in walk_iter:
for fname in files:
yield os.path.join(dirpath, fname)
def walk_all(walk_iter):
for dirpath, subdirs, files in walk_iter:
yield dirpath
for fname in files:
yield os.path.join(dirpath, fname)
The pipeline terminators could then be combined with ordinary iterable
consumers like comprehensions:
base_walk = detect_symlink_loops(os.walk(os.path.abspath(base_dir,
followlinks=True)))
depth_limited_walk = limit_depth(base_walk, 2)
filtered_walk = filter_dirs(filter_files(depth_limited_walk, "*.py"),
"*.pyp")
tree_info = {path, os.stat(path) for path in walk_all(filtered_walk)}
--
___
Python tracker
<http://bugs.python.org/issue13229>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13229] Add shutil.filter_walk
Nick Coghlan added the comment: This needs more thought - pypi package coming soon :) -- ___ Python tracker <http://bugs.python.org/issue13229> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
