[issue8844] Condition.wait() doesn't raise KeyboardInterrupt
Reid Kleckner added the comment: Added a patch that adds support for recomputing the timeout, plus a test for it. Can this still make it into 3.2, or is it too disruptive at this point in the release process? -- Added file: http://bugs.python.org/file18536/lock-interrupt-v4.diff ___ Python tracker <http://bugs.python.org/issue8844> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment: No, sorry, I just haven't gotten around to reproducing it on Linux. And I've even needed this functionality in the mean time, and we worked around it with the standard alarm trick! =/ -- ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment: I updated and committed the patch to the cpython hg repo in revision [c4a0fa6e687c]. -- ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment: On Mon, Mar 14, 2011 at 12:31 PM, Sridhar Ratnakumar wrote: > > Sridhar Ratnakumar added the comment: > > On 2011-03-14, at 9:18 AM, Reid Kleckner wrote: > >> I updated and committed the patch to the cpython hg repo in revision >> [c4a0fa6e687c]. > > Does this go to the main branch (py3.3) only? It is not clear from just > looking at http://hg.python.org/cpython/rev/c4a0fa6e687c/ Yes, it's a new feature, so I don't think it's appropriate to backport. Actually, I just noticed I forgot the update the doc patches. They should all say added in 3.3, not 3.2. Reid -- ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Changes by Reid Kleckner : -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11504] test_subprocess failure
Reid Kleckner added the comment: I can't reproduce this. I've tested on: 64-bit Linux (Debian lenny) OS X 10.6 Windows Vista 32-bit It seems reasonable to me that the interpreter should be able to initialize and write to stdout in less than half a second, but it seems to be failing consistently on that builder. I'd really rather not make the timeout longer, since it will increase testing time for everyone. Is there something about that builder that makes initialization take longer? -- ___ Python tracker <http://bugs.python.org/issue11504> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11504] test_subprocess failure
Reid Kleckner added the comment: I increased the timeout in [fd2b3eac6756] and the buildbot is passing now: http://python.org/dev/buildbot/all/builders/x86%20debian%20parallel%203.x -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue11504> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11504] test_subprocess failure
Reid Kleckner added the comment: :( Thanks for spotting these. Is there an easier way for me to be notified if these particular tests fail? Some of these are not in the "stable" builder set. Sorry to leave the builders broken for so long. I just upped the timeouts to 3 seconds. I guess the issue is that the builders are slow and also heavily loaded, so processes just don't get to start up as quick as we'd like them too. It might be worth adding some plumbing to have the child process signal the parent when it's initialized, but that seems like it's going to add a whole bunch more complexity to the test. Will close in a few days if there are no more broken buildbots. -- ___ Python tracker <http://bugs.python.org/issue11504> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11613] test_subprocess fails under Windows
Reid Kleckner added the comment: It is necessary, WaitForSingleObject takes its argument in milliseconds. It will make the exception message wrong, though, which I can fix. Reid On Sun, Mar 20, 2011 at 1:46 PM, Santoso Wijaya wrote: > > Santoso Wijaya added the comment: > > The timeout value given to wait() is multiplied by 1000 before being passed > to TimeoutExpired constructor. The multiplication is unnecessary since we > take the input unit as time unit second. > > -- > keywords: +patch > nosy: +santa4nt > Added file: http://bugs.python.org/file21310/timeoutsec.patch > > ___ > Python tracker > <http://bugs.python.org/issue11613> > ___ > -- ___ Python tracker <http://bugs.python.org/issue11613> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11613] test_subprocess fails under Windows
Reid Kleckner added the comment: The bot is green again as of ab2363f89058. Thanks for the heads up. -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue11613> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment: Thanks for fixing the negative timeout issue. I assumed incorrectly that a negative timeout would cause it to check and return immediately if it would otherwise block. As for the docs, the 3.2/3.3 issue was fixed in [[72e49cb7fcf5]]. I just added a Misc/NEWS entry for 3.3's What's New in [[9140f2363623]]. -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11757] test_subprocess.test_communicate_timeout_large_ouput failure on select(): negative timeout?
Reid Kleckner added the comment: I think the best behavior would be to go ahead and check one last time before raising the exception, so _remaining_time should turn a negative value into 0 (assuming that a timeout value of zero does the right thing for our use case). If people don't feel that is best, refactoring _remaining_time to incorporate the check in _check_timeout would also be good. -- ___ Python tracker <http://bugs.python.org/issue11757> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1054041] Python doesn't exit with proper resultcode on SIGINT
Reid Kleckner added the comment: Looks good to me. Do you need the TODO(gps)'s in there after implementing the behavior described? -- ___ Python tracker <http://bugs.python.org/issue1054041> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment: Pablo, so if I understand the issue you've run into correctly, you are using shell redirection to redirect stdout to a file, and then attempting to read from it using stdout=subprocess.PIPE. It seems to me like this behavior is expected, because the shell will close it's current stdout file descriptor and open a new one pointing at "file". When python tries to read from its end of the pipe, it complains that the fd has been closed. I can avoid the problem here either by not reading stdout or by not redirecting to a file. -- ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment:
> - why do you say Thread.join() uses a busy loop? is it because it uses
> Condition.wait()? If so, this will be solved in py3k by issue7316 (which you
> are welcome to review). Otherwise, I think there should be an upper bound on
> the sleeping granularity (say 2 seconds).
Yes, that's what I was referring to. I'm glad to hear the situation will
improve in the future!
> - the "if 'timeout' in kwargs" dance is a bit complicated. Why not simply
> "kwargs.pop('timeout', None)"?
Good call, done.
> - if it times out, communicate() should raise a specific exception. Bonus
> points
> if the exception holds the partial output as attributes (that's what we do
> for
> non-blocking IO in py3k), but if it's too difficult we can leave that out. I
> don't think returning None would be very good.
I agree. Does subprocess.TimeoutExpired sound good?
It won't be possible with the current implementation to put the partial output
in the exception, because read blocks. For example, in the Windows threaded
implementation, there's a background thread that just calls self.stdout.read(),
which blocks until its finished.
> - for consistency, other methods should probably raise the same exception. I
> think we can leave out the more complex scenarios such as "timing out but
> still processing the beginning of the output".
What do you mean still processing? I agree, they should all throw the same
exception. I think call and check_call should clean up after themselves by
killing the child processes they create, while communicate and wait should leave
that to the user.
I'm imagining something like this for communicate:
try:
(stdout, stderr) = p.communicate(timeout=123)
except subprocess.TimeoutExpired:
p.kill()
(stdout, stderr) = p.communicate() # Should not block long
And nothing special for check_call(cmd=[...], timeout=123).
--
Added file: http://bugs.python.org/file16093/subprocess-timeout.patch
___
Python tracker
<http://bugs.python.org/issue5673>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6642] returning after forking a child thread doesn't call Py_Finalize
New submission from Reid Kleckner : I attached a test case to reproduce. Here is what it does: - The main thread in the parent process starts a new thread and waits for it. - The child thread forks. - The child process creates a daemon thread, and returns. - The parent process (in the thread that forked) calls os.waitpid(childpid). What should happen is that the forked child process should terminate because it shouldn't wait for the daemon thread, and os.waitpid(childpid) should return after that, and then the main thread should return from thread.join(). What happens is that because it was a child thread that forked, the C stack starts inside of the pthread wrapper (or equivalent) instead of main. So when child process returns, it doesn't know that it is now the main thread, and it doesn't execute Py_Finalize. Furthermore, pthreads doesn't call sys_exit_group because it thinks that it is a lone thread returning, and it doesn't want to terminate the entire process group. When you look at it with 'ps f', this is what it looks like: 24325 pts/3Ss 0:01 bash 4453 pts/3Sl 0:00 \_ ./python thread_fork_hang.py 4459 pts/3Zl 0:07 | \_ [python] 4467 pts/3R+ 0:00 \_ ps f Here's the stack traces from the parent process: (gdb) bt #0 0x77bd0991 in sem_wait () from /lib/libpthread.so.0 #1 0x00587abd in PyThread_acquire_lock (lock=0x12bb680, waitflag=1) at ../../unladen2/Python/thread_pthread.h:349 #2 0x005b1660 in lock_PyThread_acquire_lock (self=0x77f37150, args=) at ../../unladen2/Modules/threadmodule.c:46 #3 0x0055b89d in _PyEval_CallFunction (stack_pointer=0x128ff20, na=, nk=0) at ../../unladen2/Python/eval.cc:4046 #4 0x0055644c in PyEval_EvalFrame (f=0x128fd60) at ../../unladen2/Python/eval.cc:2518 #5 0x0055b225 in PyEval_EvalCodeEx (co=0x77ef9670, globals=0x1, locals=0x2, args=0x123bbd8, argcount=1, kws=0x123bbe0, kwcount=0, defs=0x77e540e8, defcount=1, closure=0x0) at ../../unladen2/Python/eval.cc:3093 #6 0x0055b7b0 in _PyEval_CallFunction (stack_pointer=0x123bbe0, na=1, nk=0) at ../../unladen2/Python/eval.cc:4188 #7 0x0055644c in PyEval_EvalFrame (f=0x123ba40) at ../../unladen2/Python/eval.cc:2518 #8 0x0055b225 in PyEval_EvalCodeEx (co=0x77efea30, globals=0x1, locals=0x2, args=0x12038c8, argcount=1, kws=0x12038d0, kwcount=0, defs=0x77e54368, defcount=1, closure=0x0) at ../../unladen2/Python/eval.cc:3093 #9 0x0055b7b0 in _PyEval_CallFunction (stack_pointer=0x12038d0, na=1, nk=0) at ../../unladen2/Python/eval.cc:4188 #10 0x0055644c in PyEval_EvalFrame (f=0x1203750) at ../../unladen2/Python/eval.cc:2518 #11 0x0055b225 in PyEval_EvalCodeEx (co=0x77f55d50, globals=0x0, locals=0x0, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../../unladen2/Python/eval.cc:3093 #12 0x0055bc02 in PyEval_EvalCode (co=0x12bb680, globals=0x80, locals=0x0) at ../../unladen2/Python/eval.cc:552 #13 0x0057deb1 in PyRun_FileExFlags (fp=0x1121260, filename=0x7fffe6be "thread_fork_hang.py", start=, globals=0x10fa010, locals=0x10fa010, closeit=1, flags=0x7fffe290) at ../../unladen2/Python/pythonrun.c:1359 #14 0x0057e167 in PyRun_SimpleFileExFlags (fp=0x1121260, filename=0x7fffe6be "thread_fork_hang.py", closeit=1, flags=0x7fffe290) at ../../unladen2/Python/pythonrun.c:955 #15 0x004d8954 in Py_Main (argc=-134459232, argv=) at ../../unladen2/Modules/main.c:695 #16 0x76cdf1c4 in __libc_start_main () from /lib/libc.so.6 #17 0x004d7ae9 in _start () (gdb) thread 2 [Switching to thread 2 (Thread 0x40800950 (LWP 4458))]#0 0x77bd234f in waitpid () from /lib/libpthread.so.0 (gdb) bt #0 0x77bd234f in waitpid () from /lib/libpthread.so.0 #1 0x005b6adf in posix_waitpid (self=, args=) at ../../unladen2/Modules/posixmodule.c:5797 #2 0x0055b89d in _PyEval_CallFunction (stack_pointer=0x129cff8, na=, nk=0) at ../../unladen2/Python/eval.cc:4046 #3 0x0055644c in PyEval_EvalFrame (f=0x129ce60) at ../../unladen2/Python/eval.cc:2518 #4 0x0055b225 in PyEval_EvalCodeEx (co=0x77f558f0, globals=0x0, locals=0x0, args=0x77f98068, argcount=0, kws=0x1239f70, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../../unladen2/Python/eval.cc:3093 #5 0x005d98fc in function_call (func=0x77eefc80, arg=0x77f98050, kw=0x1286d20) at ../../unladen2/Objects/funcobject.c:524 #6 0x004dc68d in PyObject_Call (func=0x77eefc80, arg=0x77f98050, kw=0x1286d20) at ../../unladen2/Objects/abstract.c:2487 #7 0x005549d0 in _PyEval_CallFunctionVarKw (stack_pointer=0x129ce08, num_posargs=, num_kwargs=0, flags=) at ../../unladen2/Python/eval.cc:45 #8 0x00559de2 in PyEval_EvalFrame (f=0x129cc80) at ../../unladen2/Python/
[issue6643] joining a child that forks can deadlock in the forked child process
New submission from Reid Kleckner : This bug is similar to the importlock deadlock, and it's really part of a larger problem that you should release all locks before you fork. However, we can fix this in the threading module directly by freeing and resetting the locks on the main thread after a fork. I've attached a test case that inserts calls to sleep at the right places to make the following occur: - Main thread spawns a worker thread. - Main thread joins worker thread. - To join, the main thread acquires the lock on the condition variable (worker.__block.acquire()). == switch to worker == - Worker thread forks. == switch to child process == - Worker thread, which is now the only thread in the process, returns. - __bootstrap_inner calls self.__stop() to notify any other threads waiting for it that it returned. - __stop() tries to acquire self.__block, which has been left in an acquired state, so the child process hangs here. == switch to worker in parent process == - Worker thread calls os.waitpid(), which hangs, since the child never returns. So there's the deadlock. I think I should be able to fix it just by resetting the condition variable lock and any other locks hanging off the only thread left standing after the fork. -- components: Library (Lib) files: forkjoindeadlock.py messages: 91265 nosy: rnk severity: normal status: open title: joining a child that forks can deadlock in the forked child process versions: Python 2.6 Added file: http://bugs.python.org/file14647/forkjoindeadlock.py ___ Python tracker <http://bugs.python.org/issue6643> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6642] returning after forking a child thread doesn't call Py_Finalize
Changes by Reid Kleckner : -- versions: +Python 2.6 -Python 3.2 ___ Python tracker <http://bugs.python.org/issue6642> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] joining a child that forks can deadlock in the forked child process
Reid Kleckner added the comment: Here's a patch for 3.2 which adds the fix and a test case. I also verified that the problem exists in 3.1, 2.7, and 2.6 and backported the patch to those versions, but someone should review this one before I upload those. -- keywords: +patch versions: +Python 2.7, Python 3.1, Python 3.2 Added file: http://bugs.python.org/file14648/forkdeadlock.diff ___ Python tracker <http://bugs.python.org/issue6643> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6642] returning after forking a child thread doesn't call Py_Finalize
Reid Kleckner added the comment: Here's a patch against 2.6 for one way to fix it. I imagine it has problems, but I wanted to throw it out there as a straw man. This patch builds on the patch for http://bugs.python.org/issue6643 since some of the test cases will occasionally deadlock without it. -- keywords: +patch Added file: http://bugs.python.org/file14653/finalize-patch.diff ___ Python tracker <http://bugs.python.org/issue6642> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6800] os.exec* raises "OSError: [Errno 45] Operation not supported" in a multithreaded application
New submission from Reid Kleckner :
The test case is attached. On Mac OS X (and presumably FreeBSD, which
has the same behavior) when you try to exec from a process that has any
other threads in it, you get an OSError, "Operation not supported".
Here's the output on my MacBook:
Traceback (most recent call last):
File "daemon_exec.py", line 16, in
main()
File "daemon_exec.py", line 13, in main
os.execl('echo', 'hello world')
File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/os.py",
line 312, in execl
execv(file, args)
OSError: [Errno 45] Operation not supported
And on my Linux box:
hello world
Here's a similar bug that OCaml had to deal with:
http://caml.inria.fr/mantis/view.php?id=4666
I think it's reasonable for Python to declare this to be a limitation of
the OS, but considering that the other thread could be a daemon thread
that the user doesn't really care about, I think it would be reasonable
for Python to kill the other threads in the process before execing.
That's what happens on Linux, anyway.
I ran into this problem while trying to add a persistent background
compilation thread to unladen swallow, and wondered if having any other
threads would trigger the same exception.
It's tempting to just write this off, but I figured it should be
documented or left open as a low priority defect.
--
assignee: ronaldoussoren
components: Macintosh
files: daemon_exec.py
messages: 92054
nosy: rnk, ronaldoussoren
severity: normal
status: open
title: os.exec* raises "OSError: [Errno 45] Operation not supported" in a
multithreaded application
type: behavior
versions: Python 2.6
Added file: http://bugs.python.org/file14798/daemon_exec.py
___
Python tracker
<http://bugs.python.org/issue6800>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6800] os.exec* raises "OSError: [Errno 45] Operation not supported" in a multithreaded application
Reid Kleckner added the comment: Supposedly this bug also affects FreeBSD, but I can't verify it. I'd say the problem isn't going away, at least not for that platform, but I don't feel like it's worth bending over backwards to deal with it either. As far as it concerns unladen swallow, we'll bring down our background thread by another means. Unfortunately, there's no way to join pythreads, so I have to add a hack that just retries the execv call if errno == EOPNOTSUPP with an eventual timeout. -- ___ Python tracker <http://bugs.python.org/issue6800> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1068268] subprocess is not EINTR-safe
Changes by Reid Kleckner : -- nosy: +rnk ___ Python tracker <http://bugs.python.org/issue1068268> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6033] LOOKUP_METHOD and CALL_METHOD optimization
Changes by Reid Kleckner : -- nosy: +rnk ___ Python tracker <http://bugs.python.org/issue6033> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6033] LOOKUP_METHOD and CALL_METHOD optimization
Reid Kleckner added the comment: One thing I was wondering about the current patch is what about objects that have attributes that shadow methods? For example: class C(object): def foo(self): return 1 c = c() print c.foo() c.foo = lambda: 2 print c.foo() Shouldn't the above print 1 and 2? With the current patch, it seems that you might still print 1. There's also the possible performance drawback where you're loading builtin C methods, so the optimization fails, but you end up calling _PyType_Lookup twice. :( I'm doing the same optimization for unladen swallow, and these were some of the things I ran into. I think I'm going to write a PyObject_GetMethod that tries to get a method without binding it, but if it can't for any reason, it behaves just like PyObject_GetAttr and sets a status code. -- ___ Python tracker <http://bugs.python.org/issue6033> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
New submission from Reid Kleckner : I was looking for a way to run a subprocess with a timeout. While there are a variety of solutions on Google, I feel like this functionality should live in the standard library module. Apparently Guido thought this would be good in 2005 but no one did it: http://mail.python.org/pipermail/python-dev/2005-December/058784.html I'd be willing to implement it, but I'm not a core dev and I'd need someone to review it. I'll start working on a patch now, and if people think this is a good idea I'll submit it for review. My plan was to add a 'timeout' optional keyword argument to wait() and propagate that backwards through communicate(), call(), and check_call(). Does anyone object to this approach? -- components: Library (Lib) messages: 85256 nosy: rnk severity: normal status: open title: Add timeout option to subprocess.Popen type: feature request versions: Python 2.7, Python 3.1 ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment: Ugh. I made the assumption that there must be some natural and easy way to wait for a child process with a timeout in C, and it turns out it's actually a hard problem, which is why this isn't already implemented. So my initial hack for solving this problem in my own project was to run the subprocess, spawn a thread to wait on it, and then use the thread's wait method, which does accept a timeout. On further inspection, it turns out that Thread.wait() actually uses a busy loop to implement the timeout, which is what I was trying to avoid. If it's okay to have a busy loop there, is it okay to have one in Popen.wait()? Obviously, you don't need to busy loop if there is no timeout. -- ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment: I'd like some feedback on this patch. Is the API acceptable? Would it be better to throw an exception in wait() instead of returning None? What should communicate() return if it times out? I can't decide if it should try to return partial output, return None, or raise an exception. If it doesn't return partial output, that output is not recoverable. Maybe it should go into the exception object. On the other hand, the way that communicate() is implemented with threads on Windows makes it hard to "interrupt" the file descriptor read and return partial output. For that matter, I'm not even sure how to stop those extra threads, as you can see in the patch. If anyone can think of a way to avoid using threads entirely, that would be even better. What should call() and check_call() return when they timeout? If they return None, which is the current returncode attribute, there is no way of interacting with the process. They could throw an exception with a reference to the Popen object. -- keywords: +patch Added file: http://bugs.python.org/file13592/subprocess-timeout.patch ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8844] Condition.wait() doesn't raise KeyboardInterrupt
Reid Kleckner added the comment: I'd like to fix it, but I don't know if I'll be able to in time. It was something that bugged me while running the threading tests while working on Unladen. I'm imagining (for POSIX platforms) adding some kind of check for signals when the system call returns EINTR. If the signal handler raises an exception, like an interrupt should raise a KeyboardInterrupt, we can just give a different return code and propagate the exception. It also seems like this behavior can be extended gradually to different platforms, since I don't have the resources to change and test every threading implementation. -- ___ Python tracker <http://bugs.python.org/issue8844> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8844] Condition.wait() doesn't raise KeyboardInterrupt
Reid Kleckner added the comment: Here's a patch that makes Python-level lock acquisitions interruptible for py3k. There are many users of the C-level lock API, most of whom are not set up to deal with lock acquisition failure. I decided to make a new API function and leave the others alone. If possible, I think this should go out with 3.2. In that case, I was wondering if I should merge PyThread_acquire_lock_timed with my new PyThread_acquire_lock_timed_intr, since PyThread_acquire_lock_timed wasn't available in 3.1. Although it did go out in 2.7, we don't promise C API compatibility with the 2.x series, so I don't think it matters. I've tested this patch on Mac OS X and Linux. The whole test suite passes on both, along with the test that I added to test_threadsignals.py. I added a noop compatibility wrapper to thread_nt.h, but I haven't tested it or built it. When I get around to testing/fixing the subprocess patch on Windows, I'll make sure this works and the test is skipped. -- keywords: +patch Added file: http://bugs.python.org/file17929/lock-interrupt.diff ___ Python tracker <http://bugs.python.org/issue8844> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8844] Condition.wait() doesn't raise KeyboardInterrupt
Reid Kleckner added the comment: Here is a new version of a patch that updates recursive locks to have the same behavior. The pure Python RLock implementaiton should be interruptible by virtue of the base lock acquire primitive being interruptible. I've also updated the relevant documentation I could find. I've surely missed some, though. I also got rid of the _intr version of lock acquisition and simply added a new parameter to the _timed variant. The only two callers of it are the ones I updated in the _thread module. -- Added file: http://bugs.python.org/file17931/lock-interrupt.diff ___ Python tracker <http://bugs.python.org/issue8844> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] joining a child that forks can deadlock in the forked child process
Reid Kleckner added the comment: Here's an updated patch for py3k (3.2). The test still fails without the fix, and passes with the fix. Thinking more about this, I'll try summarizing the bug more coherently: When the main thread joins the child threads, it acquires some locks. If a fork in a child thread occurs while those locks are held, they remain locked in the child process. My solution is to do here what we do elsewhere in CPython: abandon radioactive locks and allocate fresh ones. -- Added file: http://bugs.python.org/file17932/thread-fork-join.diff ___ Python tracker <http://bugs.python.org/issue6643> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8844] Condition.wait() doesn't raise KeyboardInterrupt
Reid Kleckner added the comment: Oops, copy/paste oversight. =/ I wrote a test to verify that it handles signals, and then retries the lock acquire. -- Added file: http://bugs.python.org/file17935/lock-interrupt.diff ___ Python tracker <http://bugs.python.org/issue8844> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] joining a child that forks can deadlock in the forked child process
Reid Kleckner added the comment: I realized that in a later fix for unladen-swallow, we also cleared the condition variable waiters list, since it has radioactive synchronization primitives in it as well. Here's an updated patch that simplifies the fix by just using __init__() to completely reinitialize the condition variables and adds a test. This corresponds to unladen-swallow revisions r799 and r834. -- Added file: http://bugs.python.org/file17936/thread-fork-join.diff ___ Python tracker <http://bugs.python.org/issue6643> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8844] Condition.wait() doesn't raise KeyboardInterrupt
Reid Kleckner added the comment: Also, thanks for the quick reviews! -- ___ Python tracker <http://bugs.python.org/issue8844> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7576] Avoid warnings in PyModuleDef_HEAD_INIT
Reid Kleckner added the comment: This patch looks good to me, after digging through the relevant module code. I was confused though for a bit as to why PyModuleDef is a PyObject with a NULL type. It turns out that import.c wants to keep them in a dictionary, so it needs to be able to cast to PyObject* and to access the refcount. It never needs the type, though, so it's safe to leave it NULL. I think it might be worth adding comments explaining that in this patch or another. -- nosy: +rnk ___ Python tracker <http://bugs.python.org/issue7576> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Changes by Reid Kleckner : -- title: joining a child that forks can deadlock in the forked child process -> Throw away more radioactive locks that could be held across a fork in threading.py ___ Python tracker <http://bugs.python.org/issue6643> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Reid Kleckner added the comment: I completely agree, but the cat is out of the bag on this one. I don't see how we could get rid of fork until Py4K, and even then I'm sure there will be people who don't want to see it go, and I'd rather not spend my time arguing this point. The only application of fork that doesn't use exec that I've heard of is pre-forked Python servers. But those don't seem like they would be very useful, since with refcounting the copy-on-write behavior doesn't get you very many wins. The problem that this bandaid solves for me is that test_threading.py already tests thread+fork behaviors, and can fail non-deterministically. This problem was exacerbated while I was working on making the compilation thread. I don't think we can un-support fork and threads in the near future either, because subprocess.py uses fork, and libraries can use fork behind the user's back. -- ___ Python tracker <http://bugs.python.org/issue6643> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8844] Condition.wait() doesn't raise KeyboardInterrupt
Reid Kleckner added the comment: Alternatively, do you think it would be better to ignore interrupts when a timeout is passed? If a timeout is passed, the lock acquire will eventually fail in a deadlock situation, and the signal will be handled in the eval loop. However, if the timeout is sufficiently long, this is still a problem. I'd prefer to do that or use gettimeofday from _time than leave this as is. -- ___ Python tracker <http://bugs.python.org/issue8844> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9079] Make gettimeofday available in time module
Reid Kleckner added the comment: The patch looks good to me FWIW. I would be interested in using this perhaps in issue8844, which involves lock timeouts. It may be true that the POSIX API uses nanoseconds, but pythreads only exposes microsecond precision. In order to use it from the thread module, it need to get moved into Python/. My best guess at the best place to put it would be sysmodule.c, since that already wraps other system calls. Or, you know, we could live a little and make a new file for it. :) -- nosy: +rnk ___ Python tracker <http://bugs.python.org/issue9079> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6033] LOOKUP_METHOD and CALL_METHOD optimization
Reid Kleckner added the comment: I have an patch for unladen-swallow out for review here: http://codereview.appspot.com/160063/show It resolves the correctness issues I mentioned previously by emitting guards if necessary. If the type is predictable and uses slots, then we don't need to check the instance dict. It gives a 5% speedup on the unpickle benchmark. Presumably the other benchmarks do not do as many method calls. -- ___ Python tracker <http://bugs.python.org/issue6033> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment: I went through the trouble of building and testing Python on Windows Vista, and with some small modifications I got the tests I added to pass. Here's an updated patch. I'm still not really sure how those threads work on Windows, so I'd rather leave that TODO in until someone with Windows expertise checks it. -- Added file: http://bugs.python.org/file17994/subprocess-timeout-v4.patch ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6033] LOOKUP_METHOD and CALL_METHOD optimization
Reid Kleckner added the comment: Sorry, I was just posting it so Benjamin could see what this bought us. I'm not pushing to get this in CPython. The results are for JITed code. I forget what the interpreted results are. I think they are good for the microbenchmarks, but not as good for the macro. -- ___ Python tracker <http://bugs.python.org/issue6033> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment:
I forgot that I had to tweak the test as well as subprocess.py. I did a
.replace('\r', ''), but universal newlines is better.
Looking at the open questions I had about the Windows threads, I think it'll be
OK if the user follows the pattern of:
proc = subprocess.Popen(...)
try:
stdout, stderr = proc.communicate(timeout=...)
except subprocess.TimeoutExpired:
proc.kill()
stdout, stderr = proc.communicate()
If the child process is deadlocked and the user doesn't kill it, then the file
descriptors will be leaked and the daemon threads will also live on forever. I
*think* that's the worst that could happen. Or they could of course wakeup
during interpreter shutdown and cause tracebacks, but that's highly unlikely,
and already possible currently.
Anyway, I would say we can't avoid leaking the fds in that situation, because
we can't know if the user will eventually ask us for the data or not. If they
want to avoid the leak, they can clean up after themselves.
What's the next step for getting this in? Thanks to those who've taken time to
look at this so far.
--
Added file: http://bugs.python.org/file18028/subprocess-timeout-v5.patch
___
Python tracker
<http://bugs.python.org/issue5673>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9079] Make gettimeofday available in time module
Reid Kleckner added the comment: Right, it's one of the peculiarities of archive files (I think). When none of an object file's symbols are used from the main program, the object file is dropped on the floor, ie not included. This has bizarre consequences in C++ with static initializers, which get dropped. On Windows, the PyAPI_FUNC macros should prevent the linker from stripping the datetime stuff. Jeff Yasskin says you should create a noop function in your object file and call it from PyMain for force linkage of the object file you're building. -- ___ Python tracker <http://bugs.python.org/issue9079> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9079] Make gettimeofday available in time module
Reid Kleckner added the comment: I'd really rather not try to rely module loading from a threading primitive. :) I think if you follow Antoine's suggestion of adding _PyTime_Init (which does nothing in the body other than a comment) it should work fine. -- ___ Python tracker <http://bugs.python.org/issue9079> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment: I don't imagine this is going into 2.7.>0 at this point, so I ported the patch to py3k. I also added support to check_output for the timeout parameter and added docs for all of the methods/functions that now take a timeout in the module. The communicate docs include the pattern of: try: outs, errs = p.communicate(timeout=15) except subprocess.TimeoutExpired: p.kill() outs, errs = p.communicate() And check_output uses it. -- Added file: http://bugs.python.org/file18042/subprocess-timeout-py3k-v6.patch ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9079] Make gettimeofday available in time module
Reid Kleckner added the comment: I think you forgot to svn add pytime.c before making the diff. -- ___ Python tracker <http://bugs.python.org/issue9079> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Changes by Reid Kleckner : -- assignee: -> rnk keywords: +needs review -patch ___ Python tracker <http://bugs.python.org/issue6643> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6642] returning after forking a child thread doesn't call Py_Finalize
Changes by Reid Kleckner : -- assignee: -> rnk dependencies: +Throw away more radioactive locks that could be held across a fork in threading.py ___ Python tracker <http://bugs.python.org/issue6642> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8844] Condition.wait() doesn't raise KeyboardInterrupt
Reid Kleckner added the comment: Waiting until the portability hacks for gettimeofday make it into core Python. -- dependencies: +Make gettimeofday available in time module ___ Python tracker <http://bugs.python.org/issue8844> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2927] expose html.parser.unescape
Reid Kleckner added the comment: It's using the old Python 2 unicode string literal syntax. It also doesn't keep to 80 cols. I'd also rather continue using a lazily initialized dict instead of catching a KeyError for '. I also feel that with the changes to Unicode in py3k, the cp1252 stuff won't work as desired and should be cut. === Is anyone still interested in html.unescape or html.escape anyway? Every web framework seems to have their own support routines already. Otherwise I'd recommend close -> wontfix. -- nosy: +rnk ___ Python tracker <http://bugs.python.org/issue2927> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
Changes by Reid Kleckner : -- nosy: +rnk versions: +Python 3.2 -Python 3.1 ___ Python tracker <http://bugs.python.org/issue5872> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment: Uh oh, that was one of the fixes I made when I tested it on Windows. I may have failed to pick up those changes when I ported to py3k. I'll check it out tonight. -- ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment: When I ported the patch I tested on trunk + Windows to py3k, I messed that stuff up. I also had to fix a bunch of str vs. bytes issues this time around. On Windows, it uses TextIOWrapper to do the encoding, and on POSIX it uses os.write, so I have to do the encoding myself. :p This patch has been tested on Windows Vista and Mac OS X 10.5. -- Added file: http://bugs.python.org/file18101/subprocess-timeout-py3k-v7.patch ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9079] Make gettimeofday available in time module
Reid Kleckner added the comment: pytime.h looks like it got pasted into the file twice. Other than that, it looks good to me and the tests pass on OS X here. -- ___ Python tracker <http://bugs.python.org/issue9079> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9079] Make gettimeofday available in time module
Reid Kleckner added the comment: I think you used 'struct timeval *' in the function definition instead of '_PyTimeVal *'. -- ___ Python tracker <http://bugs.python.org/issue9079> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5673] Add timeout option to subprocess.Popen
Reid Kleckner added the comment: On Thu, Jul 22, 2010 at 9:05 AM, Alexander Belopolsky wrote: > > Alexander Belopolsky added the comment: > > The documentation should mention somewhere that timeout can be a float. For > example, as in time.sleep docstring: > > """ > sleep(seconds) > > Delay execution for a given number of seconds. The argument may be > a floating point number for subsecond precision. > """ > > I would also like to see some discussion of supported precision. Is is the > same as for time.sleep()? Does float precision ever affect timeout > precision? (On systems with nanosleep it may, but probably this has no > practical concequences.) I added info to wait and communicate, but left the docs for call, check_call, check_output all saying that their arguments are the same as Popen(...) and wait(...). > This can be done as a future enhancement, but I would like to see > datetime.timedelta as an acceptable type for timeout. This can be done by > adding duck-typed code in the error branch which would attempt to call > timeout.total_seconds() to extract a float. I'd prefer to leave it as a future enhancement. > Looking further, it appears that timeout can be anything that can be added to > a float to produce float. Is this an accident of implementation or a design > decision? Note that a result Fraction can be used as timeout but Decimal > cannot. Implementation detail. I don't think it should matter. > Zero and negative timeouts are accepted by subprocess.call(), but the result > is not documented. It looks like this still starts the process, but kills it > immediately. An alternative would be to not start the process at all or > disallow negative or maybe even zero timeouts altogether. I don't mind the > current choice, but it should be documented at least in > Popen.wait(timeout=None) section. > > + def wait(self, timeout=None, endtime=None): > """Wait for child process to terminate. Returns returncode > attribute.""" > > Docstring should describe timeout and endtime arguments. In fact I don't see > endtime documented anywhere. It is not an obvious choice > that endtime is ignored when timeout is given. An alternative would be to > terminate at min(now + timeout, endtime). I didn't intend for the endtime parameter to be documented, it is just a convenience for implementing communicate, which gets woken up at various times so it is easier to remember the final deadline rather than recompute the timeout frequently. > + delay = 0.0005 # 500 us -> initial delay of 1 ms > > I think this should be an argument to wait() and the use of busy loop should > be documented. > > + delay = min(delay * 2, remaining, .05) > > Why .05? It would probably be an overkill to make this another argument, but > maybe make it an attribute of Popen, say self._max_busy_loop_delay or a > shorter descriptive name of your choice. If you start it with '_', you don't > need to document it, but users may be able to mess with it if they suspect > that 0.05 is not the right choice. *Points to whoever impelemented it for Thread.wait(timeout=...)*. If it was good enough for that (until we got real lock acquisitions with timeouts), then I think it's good enough for this. > + endtime = time.time() + timeout > > Did you consider using datetime module instead of time module here? (I know, > you still need time.sleep() later, but you won't need to worry about variable > precision of time.time().) How does the datetime module help here? It seems like time.time uses roughly the same time sources that datetime.datetime.now does. One other thing I'm worried about here is that time.time can be non-increasing if the system clock is adjusted. :( Maybe someone should file a feature request for a monotonic clock. Reid -- ___ Python tracker <http://bugs.python.org/issue5673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
