[issue13812] multiprocessing package doesn't flush stderr on child exception
New submission from Jon Brandvein :
When a child process exits due to an exception, a traceback is written, but
stderr is not flushed. Thus I see a header like "Process 1:\n", but no
traceback.
I don't have a development environment or any experience with Mecurial, so I'm
afraid there's no patch, but it's a one-liner.
In /Lib/multiprocess/process.py :: Process._bootstrap
except:
exitcode = 1
import traceback
sys.stderr.write('Process %s:\n' % self.name)
sys.stderr.flush()
traceback.print_exc()
Append a "sys.stderr.flush()" to the suite.
It surprised me that flushing was even necessary. I would've thought that the
standard streams would all be closed just before the process terminated,
regardless of exit status. But I observe that unless I explicitly flush stdout
and stderr before terminating, the output is lost entirely, even if the exit is
not abnormal. This isn't the desired behavior, is it?
--
components: Library (Lib)
messages: 151508
nosy: brandj
priority: normal
severity: normal
status: open
title: multiprocessing package doesn't flush stderr on child exception
type: behavior
versions: Python 3.2, Python 3.3, Python 3.4
___
Python tracker
<http://bugs.python.org/issue13812>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13812] multiprocessing package doesn't flush stderr on child exception
Jon Brandvein added the comment: (Er, that should be /Lib/multiprocessing/process.py :: Process._bootstrap of course.) -- ___ Python tracker <http://bugs.python.org/issue13812> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13812] multiprocessing package doesn't flush stderr on child exception
Jon Brandvein added the comment: On Windows, the problem appears under Python 3.2.2 and 3.1.3, but not under 2.7.1. On Linux, I have not reproduced the problem on versions 2.6.3, 2.7.2, 3.1.1, or 3.2.2. So to summarize: - It seems there should be a stderr flush call on the line I indicated, for symmetry with the surrounding code. - Even without this line, it should still be flushed automatically upon child process exit, but this doesn't happen under Windows and Python 3.x. -- ___ Python tracker <http://bugs.python.org/issue13812> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13812] multiprocessing package doesn't flush stderr on child exception
Jon Brandvein added the comment: I've been looking over this package some more, and in particular, /Lib/multiprocessing/forking.py. There's plenty I don't understand, and I do have questions, if you would be willing to indulge me. I see that both the unix and windows codepaths define an "exit" alias for terminating the child process without performing any cleanup. On unix, this is os._exit (though it calls it directly in Popen.__init__() instead of using the alias). On Windows, it is the win32 ExitProcess() function. A quick check confirms that replacing this windows alias with "exit = sys.exit" causes flushing to occur. So my main question is: What's wrong with using sys.exit()? I would speculate it's either because there's shared state between child and parent, or to avoid propagating SystemExit through user code in the case freeze_support() was used. If forking.py is to terminate directly via the OS, I think it's forking.py's responsibility to flush stdout and stderr in main() on the Windows side, the way it does in Popen.__init__() on the unix side. I also want to point out that the sys.exit() in freeze_support() is unreachable due to the exit() in main(). So it no longer surprises me that the output is not being flushed in Python 3 under windows. What surprises me is that it *is* flushed in Python 2. I remember hearing something about differences between 2 and 3 in how they handle buffering, so I'm investigating this for my own curiosity. Incidentally, Python 2 still flushes when I remove newlines from my output text, so line buffering doesn't seem to be impacting my observations. -- ___ Python tracker <http://bugs.python.org/issue13812> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13812] multiprocessing package doesn't flush stderr on child exception
Jon Brandvein added the comment: Some more information: When I write to a new file created by open(), all versions flush correctly. However, if I reassign sys.stdout to that file, Python 3.x does not (again, under Windows). I wonder what it is that causes these other files to flush. (Note: I am testing by calling time.sleep() and inspecting the output file during and after the pause.) -- ___ Python tracker <http://bugs.python.org/issue13812> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13812] multiprocessing package doesn't flush stderr on child exception
Jon Brandvein added the comment: It turns out the file output was flushing due to garbage collection. When I created and held a global reference to it, it ceased to flush. Clearly, reassigning sys.stdout also held a reference to it. So it wasn't any kind of special sys.stdout-specific logic. I tried using os.fsync to determine whether data was being saved in an OS-level buffer. But since the OS may be free to ignore fsync, it's kind of worthless for this purpose. I also found that under Python 2.x, even a low-level exit like os._exit or multiprocessing.win32.ExitProcess, called from within a user-level function in the child, caused flushing. My best guess is that 1. Python 2.x is not buffering data at the Python level. I can't see how it could be and still flush it out when calling _exit(). 2. Python 3.x is buffering at the Python level, and the Python File object needs to be destroyed or explicitly flushed before the hard low-level exit() in forking.py. The solutions I can think of for Python 3.x are: 1. Replace "exit = win32.ExitProcess" with "exit = sys.exit". All outstanding file objects will be destroyed and flushed naturally as the interpreter is torn down. 2. Add an explicit stdout/stderr flush where appropriate in forking.py and process.py, to ensure tracebacks get written and to match the unix behavior. Leave it to the user to worry about flushing their own streams. 3. Continue to use win32.ExitProcess, but add some kind of mechanism for walking through all existing Python File objects and flushing/destroying them. This was a fleeting thought; it really amounts to reimplementing the behavior of destructing the Python interpreter. I'd really like to hear if there are good reasons for why (1) isn't how it's done currently. I'd also like to hear an explanation of Python 2.x's buffering. -- ___ Python tracker <http://bugs.python.org/issue13812> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13812] multiprocessing package doesn't flush stderr on child exception
Jon Brandvein added the comment: Regarding the patch: I'd also like to see sys.stdout.flush() and sys.stderr.flush() between "exitcode = self._boostrap()" and "exit(exitcode)" in /Lib/multiprocessing/forking.py :: main(). (The extra stderr flush would be for symmetry with Popen.__init__() for unix.) The remainder of this post is what I wrote before seeing your patch. Thank you for the explanation concerning libc and buffering. As for os._exit, I can see the need under unix, where the child process has a stack inherited from the parent. Also, allowing the child to cleanup shared resources would be disastrous for the parent. (I was going to propose the idea of making a sys.exit()-like function that did not rely on the exception mechanism to unwind the stack. But it wouldn't be usable here for this reason.) Under Windows, the child process's multiprocessing.forking.main() is launched either by directly calling main() on the command line, or from within freeze_support() in the user program. In the latter case, the user can be advised not to catch SystemExit around freeze_support(), just as they are already advised to make calling freeze_support() the first statement within "if __name__ == '__main__':". So I don't see any harm in switching to sys.exit() there. I think that in general, the differences between multiprocessing's behavior under unix and windows are a weakness, as is the lack of a precise specification for this behavior. At the same time, multiprocessing is a lot more convenient to use than the subprocess module when the child process is a Python program. In particular, I use multiprocessing for its support of passing pickled objects between parent and child. With subprocess, I don't think it's even possible to share a file descriptor between parent and child under Windows. I'm wondering whether it would be desirable to make an option for the unix side of multiprocessing to behave more like the windows one, and invoke the Python interpreter from the beginning with no trace of the parent's stack. That is, it'd follow a more traditional fork()-then-exec*() pattern. Then you'd be able to have the interpreter destruct naturally under both platforms, close resources, etc. Are there use cases that require just forking under unix? Is the performance significantly better without an exec*()? Perhaps it would be better If I took this discussion to python-ideas. -- ___ Python tracker <http://bugs.python.org/issue13812> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13841] multiprocessing should use sys.exit() where possible
New submission from Jon Brandvein : Currently the multiprocessing library calls a hard exit function (os._exit under unix, ExitProcess under Windows) to terminate the child process. Under unix, this is necessary because the child is forked without exec-ing. Calling sys.exit() would make it possible for the child to run code on the part of the stack inherited from the parent, via the exception handling mechanism. It might also allow the child to accidentally destroy shared state through an object destructor, even when that object was not explicitly relied on by the child. Under Windows, I do not see any benefit besides symmetry. Processes are not forked, and the only way control can pass to user code after executing the target function, is if the process is frozen and the user puts the call to freeze_support() inside a try block. This special case can be taken care of by advising the user not to do that. (We already tell the user where freeze_support() should be located.) Changing the multiprocessing exit routine from ExitProcess to sys.exit on Windows would ensure that all objects holding resources get properly destroyed. In particular, it would ensure that all file streams (including standard output and standard error) are flushed. This is especially important under Python 3, since the new IO system uses its own buffering which cannot be flushed by ExitProcess -- from the user's point of view, a potential regression from Python 2.x. Related issues: - #13812 would not have been a problem under windows. - If #8713 gets adopted, unix can use sys.exit() as well. -- components: Library (Lib) messages: 151835 nosy: brandj, jnoller, neologix, pitrou priority: normal severity: normal status: open title: multiprocessing should use sys.exit() where possible type: enhancement versions: Python 3.2, Python 3.3 ___ Python tracker <http://bugs.python.org/issue13841> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13812] multiprocessing package doesn't flush stderr on child exception
Jon Brandvein added the comment: Patch looks fine. I like the use of "finally" for the flush. -- ___ Python tracker <http://bugs.python.org/issue13812> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13841] multiprocessing should use sys.exit() where possible
Jon Brandvein added the comment: Good point. I don't think those particular behaviors are documented, so I'm not sure whether we need to worry about breaking them. -- ___ Python tracker <http://bugs.python.org/issue13841> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13853] SystemExit/sys.exit() doesn't print boolean argument
New submission from Jon Brandvein : Raising SystemExit manually, or calling sys.exit, with an argument of "True" or "False" results in no output to the screen. According to Doc/library/exceptions.rst and Doc/library/sys.rst, any object that is not an integer or None should be printed to stderr. Also, I'm not sure whether this is a bug, but "raise SystemExit(None)" differs from "sys.exit(None)", in that the former produces an exception with an args tuple of "(None,)", and the latter produces one with an empty args tuple. -- components: Interpreter Core messages: 151920 nosy: brandj priority: normal severity: normal status: open title: SystemExit/sys.exit() doesn't print boolean argument type: behavior versions: Python 2.7, Python 3.1, Python 3.2 ___ Python tracker <http://bugs.python.org/issue13853> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13854] multiprocessing: SystemExit from child with non-int, non-str arg causes TypeError
New submission from Jon Brandvein : In a child process, raising SystemExit or calling sys.exit with a non-integer, non-string argument value causes a TypeError at Lib/multiprocessing/process.py :: _bootstrap. This is from concatenating the argument with '\n' and writing it to stderr. Suggested fix: replace sys.stderr.write(e.args[0] + '\n') with sys.stderr.write(str(e.args[0]) + '\n') This problem also occurs when the value is None, but only for raising SystemExit (not calling sys.exit()). -- components: Library (Lib) messages: 151921 nosy: brandj, jnoller priority: normal severity: normal status: open title: multiprocessing: SystemExit from child with non-int, non-str arg causes TypeError type: behavior versions: Python 2.7, Python 3.1, Python 3.2 ___ Python tracker <http://bugs.python.org/issue13854> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
Jon Brandvein added the comment: > If such a function is added, I'd like the option to not indent empty lines: > trailing spaces are often not a good idea. >From dedent's documentation, it wasn't immediately clear to me that it ignores >blank lines when determining common whitespace. (In fact the comment in the >example suggests otherwise.) Perhaps a note could be added to the >documentation when this change is made? -- nosy: +brandj ___ Python tracker <http://bugs.python.org/issue13857> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13854] multiprocessing: SystemExit from child with non-int, non-str arg causes TypeError
Jon Brandvein added the comment: Also, as Brett pointed out to me in #13853, bool is a subclass of int, so they should follow the same code path. I suggest replacing elif type(e.args[0]) is int: exitcode = e.args[0] with something like elif isinstance(e.args[0], int): exitcode = e.args[0] which assumes that a subtype of int is convertible to int. -- ___ Python tracker <http://bugs.python.org/issue13854> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10482] subprocess and deadlock avoidance
Changes by Jon Brandvein : -- nosy: +brandj ___ Python tracker <http://bugs.python.org/issue10482> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator
Changes by Jon Brandvein : -- nosy: +brandj ___ Python tracker <http://bugs.python.org/issue4806> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21366] Document that return in finally overwrites prev value
New submission from Jon Brandvein: def foo(): try: return 1 finally; return 2 print(foo()) # 2 I've seen this peculiar case discussed on a few blogs lately, but was unable to find confirmation that this behavior is defined. In the try/finally section of Doc/reference/compound_stmts.rst, immediately after the sentence beginning > When a return, break, or continue statement is executed I propose adding something to the effect of: > A return statement in a finally clause overrides the value of any return > statement executed in the try suite. This wording also handles the case of nested try/finally blocks. -- assignee: docs@python components: Documentation messages: 217277 nosy: brandjon, docs@python priority: normal severity: normal status: open title: Document that return in finally overwrites prev value type: behavior ___ Python tracker <http://bugs.python.org/issue21366> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24948] Multiprocessing not timely flushing stack trace to stderr
Changes by Jon Brandvein : -- nosy: +brandjon ___ Python tracker <http://bugs.python.org/issue24948> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
