[issue41437] SIGINT blocked by socket operations like recv on Windows
New submission from Zhiming Wang :
I noticed that on Windows, socket operations like recv appear to always block
SIGINT until it's done, so if a recv hangs, Ctrl+C cannot interrupt the
program. (I'm a *nix developer investigating a behavioral problem of my program
on Windows, so please excuse my limited knowledge of Windows.)
Consider the following example where I spawn a TCP server that stalls
connections by 5 seconds in a separate thread, and use a client to connect to
it on the main thread. I then try to interrupt the client with Ctrl+C.
import socket
import socketserver
import time
import threading
interrupted = threading.Event()
class HoneypotServer(socketserver.TCPServer):
# Stall each connection for 5 seconds.
def get_request(self):
start = time.time()
while time.time() - start < 5 and not interrupted.is_set():
time.sleep(0.1)
return self.socket.accept()
class EchoHandler(socketserver.BaseRequestHandler):
def handle(self):
data = self.request.recv(1024)
self.request.sendall(data)
class HoneypotServerThread(threading.Thread):
def __init__(self):
super().__init__()
self.server = HoneypotServer(("127.0.0.1", 0), EchoHandler)
def run(self):
self.server.serve_forever(poll_interval=0.1)
def main():
start = time.time()
server_thread = HoneypotServerThread()
server_thread.start()
sock = socket.create_connection(server_thread.server.server_address)
try:
sock.sendall(b"hello")
sock.recv(1024)
except KeyboardInterrupt:
print(f"processed SIGINT {time.time() - start:.3f}s into the
program")
interrupted.set()
finally:
sock.close()
server_thread.server.shutdown()
server_thread.join()
if __name__ == "__main__":
main()
On *nix systems the KeyboardInterrupt is processed immediately. On Windows, the
KeyboardInterrupt is always processed more than 5 seconds into the program,
when the recv is finished.
I suppose this is a fundamental limitation of Windows? Is there any workaround
(other than going asyncio)?
Btw, I learned about SIGBREAK, which when unhandled seems to kill the process
immediately, but that means no chance of cleanup. I tried to handle SIGBREAK
but whenever a signal handler is installed, the behavior reverts to that of
SIGINT -- the handler is called only after 5 seconds have passed.
(I'm attaching a socket_sigint_sigbreak.py which is a slightly expanded version
of my sample program above, showing my attempt at handler SIGBREAK. Both
python .\socket_sigint_sigbreak.py --sigbreak-handler interrupt
and
python .\socket_sigint_sigbreak.py --sigbreak-handler exit
stall for 5 seconds.)
--
components: Windows
files: socket_sigint_sigbreak.py
messages: 374580
nosy: paul.moore, steve.dower, tim.golden, zach.ware, zmwangx
priority: normal
severity: normal
status: open
title: SIGINT blocked by socket operations like recv on Windows
type: behavior
versions: Python 3.8
Added file: https://bugs.python.org/file49348/socket_sigint_sigbreak.py
___
Python tracker
<https://bugs.python.org/issue41437>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42005] profile/cProfile CLI should catch BrokenPipeError
New submission from Zhiming Wang : Since profile/cProfile CLI interface prints a sorted list of stats, using head to limit output to the most relevant entries should be a fairly natural thing to do. Unfortunately, BrokenPipeError isn't caught, causing quite a bit of pollution to the output: $ python3 -m cProfile -m http.server -h | head usage: http.server [-h] [--cgi] [--bind ADDRESS] [--directory DIRECTORY] [port] positional arguments: port Specify alternate port [default: 8000] optional arguments: -h, --helpshow this help message and exit --cgi Run as CGI Server --bind ADDRESS, -b ADDRESS Traceback (most recent call last): File "/Users/zmwang/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/Users/zmwang/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/zmwang/.pyenv/versions/3.9.0/lib/python3.9/cProfile.py", line 180, in main() File "/Users/zmwang/.pyenv/versions/3.9.0/lib/python3.9/cProfile.py", line 173, in main runctx(code, globs, None, options.outfile, options.sort) File "/Users/zmwang/.pyenv/versions/3.9.0/lib/python3.9/cProfile.py", line 19, in runctx return _pyprofile._Utils(Profile).runctx(statement, globals, locals, File "/Users/zmwang/.pyenv/versions/3.9.0/lib/python3.9/profile.py", line 66, in runctx self._show(prof, filename, sort) File "/Users/zmwang/.pyenv/versions/3.9.0/lib/python3.9/profile.py", line 72, in _show prof.print_stats(sort) File "/Users/zmwang/.pyenv/versions/3.9.0/lib/python3.9/cProfile.py", line 42, in print_stats pstats.Stats(self).strip_dirs().sort_stats(sort).print_stats() File "/Users/zmwang/.pyenv/versions/3.9.0/lib/python3.9/pstats.py", line 431, in print_stats self.print_line(func) File "/Users/zmwang/.pyenv/versions/3.9.0/lib/python3.9/pstats.py", line 513, in print_line print(f8(tt/nc), end=' ', file=self.stream) BrokenPipeError: [Errno 32] Broken pipe Exception ignored in: <_io.TextIOWrapper name='' mode='w' encoding='utf-8'> BrokenPipeError: [Errno 32] Broken pipe I think the exception should be caught in order to suppress this noise. -- components: Library (Lib) messages: 378421 nosy: zmwangx priority: normal severity: normal status: open title: profile/cProfile CLI should catch BrokenPipeError type: behavior versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue42005> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42005] profile/cProfile CLI should catch BrokenPipeError
Change by Zhiming Wang : -- keywords: +patch pull_requests: +21617 stage: -> patch review pull_request: https://github.com/python/cpython/pull/22643 ___ Python tracker <https://bugs.python.org/issue42005> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39828] json.tool should catch BrokenPipeError
Change by Zhiming Wang : -- nosy: +zmwangx nosy_count: 5.0 -> 6.0 pull_requests: +21618 pull_request: https://github.com/python/cpython/pull/22643 ___ Python tracker <https://bugs.python.org/issue39828> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31281] fileinput inplace does not work with pathlib.Path
New submission from Zhiming Wang:
Consider
import fileinput
import pathlib
with fileinput.input(files=(pathlib.Path('in.txt'),), inplace=True) as fp:
for line in fp:
print(line, end='')
which results in
Traceback (most recent call last):
File "./pathlib-fileinput.py", line 6, in
for line in fp:
File "/Users/zmwang/.pyenv/versions/3.6.1/lib/python3.6/fileinput.py",
line 250, in __next__
line = self._readline()
File "/Users/zmwang/.pyenv/versions/3.6.1/lib/python3.6/fileinput.py",
line 331, in _readline
self._filename + (self._backup or ".bak"))
TypeError: unsupported operand type(s) for +: 'PosixPath' and 'str'
A trivial fix is converting the specified filename to str when assigning to
self._filename:
-self._filename = self._files[0]
+self._filename = str(self._files[0])
--
components: Library (Lib)
messages: 300860
nosy: zmwangx
priority: normal
severity: normal
status: open
title: fileinput inplace does not work with pathlib.Path
type: behavior
versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7
___
Python tracker
<http://bugs.python.org/issue31281>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31281] fileinput inplace does not work with pathlib.Path
Changes by Zhiming Wang : -- pull_requests: +3245 ___ Python tracker <http://bugs.python.org/issue31281> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35308] webbrowser regression: BROWSER env var not respected
New submission from Zhiming Wang : This is a regression in Python 3.7: $ BROWSER=w3m python3.6 -c 'import sys; import webbrowser; print(sys.version); print(webbrowser.get()); print(webbrowser._tryorder)' 3.6.7 (default, Nov 24 2018, 09:47:01) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] ['w3m', 'MacOSX', 'chrome', 'firefox', 'safari', 'w3m'] $ BROWSER=w3m python3.7 -c 'import sys; import webbrowser; print(sys.version); print(webbrowser.get()); print(webbrowser._tryorder)' 3.7.1 (default, Nov 24 2018, 09:35:18) [Clang 10.0.0 (clang-1000.11.45.5)] ['MacOSX', 'chrome', 'firefox', 'safari', 'w3m', 'w3m'] Note how Python 3.7.1 webbrowser decides to choose MacOSX as the default although BROWSER is explicitly set to w3m. (The above code does not run on Python 3.7.0 due to bpo-31014). This is because of misinterpretation of the `preferred` kwarg when bpo-31014 was fixed. See https://github.com/python/cpython/commit/25b804a9c21c735ce322877f105ebab2539ccfc1#diff-546766353a40839aa72374ecca5b0925R566. Browsers in BROWSER should be registered as `preferred=True` instead. -- components: Library (Lib) messages: 330377 nosy: zmwangx priority: normal severity: normal status: open title: webbrowser regression: BROWSER env var not respected type: behavior versions: Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35308> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35308] webbrowser regression: BROWSER env var not respected
Change by Zhiming Wang : -- nosy: +serhiy.storchaka ___ Python tracker <https://bugs.python.org/issue35308> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35308] webbrowser regression: BROWSER env var not respected
Change by Zhiming Wang : -- keywords: +patch pull_requests: +9946 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35308> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33618] Support TLS 1.3
Change by Zhiming Wang : -- nosy: +zmwangx ___ Python tracker <https://bugs.python.org/issue33618> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32616] Significant performance problems with Python 2.7 built with clang 3.x or 4.x
New submission from Zhiming Wang :
Python 2.7 could be significantly slower (5x in some cases) when compiled with
clang 3.x or 4.x, compared to clang 5.x. This is quite a problem on macOS,
since the latest clang from Apple (which comes with Xcode 9.2) is based on LLVM
4.x. This issue was first noticed by Bart Skowron and reported to the Homebrew
project.[1]
I ran some preliminary benchmarks (here[2] are the exact setup scripts) with
just a simple loop:
import time
def f(n):
while n > 0:
n -= 1
start = time.time()
f(5000)
stop = time.time()
print('%.6f' % (stop - start))
and here are my results:
- macOS 10.13.2 on a MacBook Pro:
2.082144/usr/bin/python2.7
7.964049/usr/local/bin/python2.7
8.750652dist/python27-apple-clang-900/bin/python2.7
8.476405dist/python27-clang-3.9/bin/python2.7
8.625660dist/python27-clang-4.0/bin/python2.7
1.760096dist/python27-clang-5.0/bin/python2.7
3.254814/usr/local/bin/python3.6
2.864716dist/python-master-apple-clang-900/bin/python3
3.071757dist/python-master-clang-3.9/bin/python3
2.925192dist/python-master-clang-4.0/bin/python3
2.908782dist/python-master-clang-5.0/bin/python3
- Ubuntu 17.10 in VirtualBox:
1.475095/usr/bin/python2.7
8.576817dist/python27-clang-3.9/bin/python2.7
8.165588dist/python27-clang-4.0/bin/python2.7
1.779193dist/python27-clang-5.0/bin/python2.7
1.728321dist/python27-gcc-5/bin/python2.7
1.570040dist/python27-gcc-6/bin/python2.7
1.604617dist/python27-gcc-7/bin/python2.7
2.323037/usr/bin/python3.6
2.964338dist/python-master-clang-3.9/bin/python3
3.054277dist/python-master-clang-4.0/bin/python3
2.734908dist/python-master-clang-5.0/bin/python3
2.490278dist/python-master-gcc-5/bin/python3
2.494691dist/python-master-gcc-6/bin/python3
2.642277dist/python-master-gcc-7/bin/python3
I haven't got time to run more rigorous benchmark suites (e.g., the
performance[3] package). I did try the floating point benchmark from
performance, and again saw a 2x difference in performance.
[1] https://github.com/Homebrew/homebrew-core/issues/22743
[2] https://gist.github.com/zmwangx/f8151ba8907ba8159a07fdd1528fc2b5
[3] https://pypi.python.org/pypi/performance
--
messages: 310395
nosy: zmwangx
priority: normal
severity: normal
status: open
title: Significant performance problems with Python 2.7 built with clang 3.x or
4.x
versions: Python 2.7
___
Python tracker
<https://bugs.python.org/issue32616>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32616] Significant performance problems with Python 2.7 built with clang 3.x or 4.x
Zhiming Wang added the comment: My benchmarks above do contain py37 (master) stats. -- ___ Python tracker <https://bugs.python.org/issue32616> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32616] Significant performance problems with Python 2.7 built with clang 3.x or 4.x
Zhiming Wang added the comment: Turns out python 2.7.10 doesn't suffer from the performance issue even when compiled with stock clang 4.x, and upon further investigation, I tracked down the commit that introduced the regression: commit 2c992a0788536087bfd78da8f2c62b30a461d7e2 Author: Benjamin Peterson Date: Thu May 28 12:45:31 2015 -0500 backport computed gotos (#4753) So Naoki was right that computed gotos is (solely) to blame here. -- ___ Python tracker <https://bugs.python.org/issue32616> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32616] Significant performance problems with Python 2.7 built with clang 3.x or 4.x
Zhiming Wang added the comment:
Yeah, Apple LLVM versions are a major headache. I resorted to feature
detection, using C++ coroutines support as the clang 5 distinguisher[1]:
$ cat /tmp/test/stub.cc
#include
int main() {
return 0;
}
$
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
-v
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin17.4.0
Thread model: posix
InstalledDir:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
$
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
-o stub stub.cc -fcoroutines-ts -stdlib=libc++
stub.cc:1:10: fatal error: 'experimental/coroutine' file not found
#include
^~~~
1 error generated.
$
/Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
-v
Apple LLVM version 9.1.0 (clang-902.0.31)
Target: x86_64-apple-darwin17.4.0
Thread model: posix
InstalledDir:
/Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
$
/Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
-o stub stub.cc -fcoroutines-ts -stdlib=libc++
Here Xcode.app is Xcode 9.2 and Xcode-beta.app is Xcode 9.3 beta 2.
The conclusion here seems to be that Apple LLVM 9.0.0 is based on LLVM 4, while
Apple LLVM 9.1.0 is based on LLVM 5.
[1]
http://releases.llvm.org/5.0.0/tools/clang/docs/ReleaseNotes.html#major-new-features
--
status: pending -> open
___
Python tracker
<https://bugs.python.org/issue32616>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31638] zipapp module should support compression
New submission from Zhiming Wang : Currently (up to 3.7.0a1) archives generated by the zipapp module are uncompressed, which are unnecessarily large for large projects. It's easy to add deflate compression support, and Python loads compressed archives just fine. I already have a patch available. I'll submit a PR soon. -- components: Library (Lib) messages: 303323 nosy: zmwangx priority: normal severity: normal status: open title: zipapp module should support compression type: enhancement versions: Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue31638> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31638] zipapp module should support compression
Change by Zhiming Wang : -- keywords: +patch pull_requests: +3803 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue31638> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35035] Documentation for email.utils is named email.util.rst
New submission from Zhiming Wang : Documentation for PSL module email.utils is named email.util.rst. See <https://docs.python.org/3/library/email.util.html>. This seems to violate the principle of least surprise. (I have a command line tool to open documentation for any PSL module, and I found this name mismatch when I used that with email.utils.) It should be named email.utils.rst instead, unless there's a specific reason not to. -- assignee: docs@python components: Documentation messages: 328190 nosy: docs@python, zmwangx priority: normal severity: normal status: open title: Documentation for email.utils is named email.util.rst versions: Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35035> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35035] Documentation for email.utils is named email.util.rst
Change by Zhiming Wang : -- keywords: +patch pull_requests: +9362 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35035> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
