[issue39853] Segmentation fault with urllib.request.urlopen and threads

Gilles Duboscq added the comment: Thanks Victor we'll look into moving away from this pattern. -- ___ Python tracker ___ ___ Python

[issue39853] Segmentation fault with urllib.request.urlopen and threads

STINNER Victor added the comment: > For us it happens when using multiprocessing: the main process forks 2 > processes and both use urlopen at roughly the same time. We are seeing this > on Python 3.7.2 on macOS 10.14.3. >From what I understood, calling fork() (and then continue to execute r

[issue39853] Segmentation fault with urllib.request.urlopen and threads

STINNER Victor added the comment: Oh wait, I removed that lock: commit 0de437de6210c2b32b09d6c47a805b23d023bd59 Author: Victor Stinner Date: Thu May 28 17:23:39 2020 +0200 bpo-25920: Remove socket.getaddrinfo() lock on macOS (GH-20177) On macOS, socket.getaddrinfo() no longer

[issue39853] Segmentation fault with urllib.request.urlopen and threads

Gilles Duboscq added the comment: I get '0' so it was not built with HAVE_GETHOSTBYNAME_R. -- ___ Python tracker ___ ___ Python-bug

[issue39853] Segmentation fault with urllib.request.urlopen and threads

STINNER Victor added the comment: "Segmentation fault with (...) threads (...) getaddrinfo" Aha, another victim on a getaddrinfo() implementation which is not thread safe. See this code in Modules/socketmodule.c: /* Lock to allow python interpreter to continue, but only allow one thread t

[issue39853] Segmentation fault with urllib.request.urlopen and threads

Gilles Duboscq added the comment: I'm not sure it's the same but we have seen stack traces looking like the one there: https://github.com/astropy/astropy/issues/9699 Current thread 0x7fffa857e3c0 (most recent call first): File "/sw/lib/python3.7/socket.py", line 748 in getaddrinfo Fil

[issue39853] Segmentation fault with urllib.request.urlopen and threads

STINNER Victor added the comment: If someone manages to reproduce the bug, please provide *at least* the Python traceback. You can use faulthandler to get it. It would also be very useful to get the C call stack using gdb (where command in gdb). In the meanwhile, I close the issue since only

[issue39853] Segmentation fault with urllib.request.urlopen and threads

Manjusaka added the comment: Hello Victor I have tried on both MacOS and Ubuntu 18.04 from 3.8.2 to the newest code in master, and can't reproduce this problem macOS-10.15.4-x86_64-i386-64bit Python 3.9.0a4+ (heads/master:9a81ab107a, Mar 24 2020, 02:06:30) [Clang 11.0.3 (clang-1103.0.32.26)]

[issue39853] Segmentation fault with urllib.request.urlopen and threads

STINNER Victor added the comment: Try to get the gdb traceback (C call stack). You may also try: python3 -X faulthandler urllib_segfault.py. But I expect a crash in Python finalization, where there is no Python frame and so empty traceback. -- __

[issue39853] Segmentation fault with urllib.request.urlopen and threads

STINNER Victor added the comment: Python 3.7.3 is outdated. Can you please try on newer Python version? I failed to reproduce the crash on Fedora 31. I tested: vstinner@apu$ python3.7 -VV Python 3.7.6 (default, Jan 30 2020, 09:44:41) [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] vstinner@apu$ pyt

[issue39853] Segmentation fault with urllib.request.urlopen and threads

Manjusaka added the comment: use the same GCC 8.3.0 to recompile the Python 3.7.3 still no fault maybe we need the core dump to figure it out. -- ___ Python tracker ___ _

[issue39853] Segmentation fault with urllib.request.urlopen and threads

Manjusaka added the comment: I have tried it on Python 3.7.3 Ubuntu 18.04 Linux-4.15.0-1060-aws-x86_64-with-debian-buster-sid Python 3.7.3 (default, Mar 23 2020, 18:15:26) [GCC 7.5.0] there is no segmentation fault would you mind sharing the core dump to help us find more detail about the

[issue39853] Segmentation fault with urllib.request.urlopen and threads

Change by STINNER Victor : -- nosy: +vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.py

[issue39853] Segmentation fault with urllib.request.urlopen and threads

New submission from Anne Archibald : This was discovered in the astropy test suite, where ThreadPoolExecutor is used to concurrently launch a lot of urllib.request.urlopen. This occurs when the URLs are local files; I'm not sure about other URL schemes. The problem appears to occur in python