[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Changes by Charles-François Natali : -- resolution: -> fixed status: open -> closed ___ Python tracker ___ ___ Python-bugs-list maili

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Roundup Robot added the comment: New changeset a9f10c3eff69 by Charles-François Natali in branch 'default': Issue #13303: Fix bytecode file default permission. http://hg.python.org/cpython/rev/a9f10c3eff69 -- ___ Python tracker

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Roundup Robot added the comment: New changeset 1238cdd288d2 by Charles-François Natali in branch 'default': Back out changeset b6336ba796d4 until fix for #13303. http://hg.python.org/cpython/rev/1238cdd288d2 -- ___ Python tracker

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Charles-François Natali added the comment: Here's a patch using the '.tmp' suffix. I also updated test_import. Note that the current test_import.test_execute_bit_not_copied is a no-op: """ fname = TESTFN + os.extsep + "py" create_empty_file(fname)

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Charles-François Natali added the comment: > So Python starts by removing the .tmp file, but it fails if another > process is already writing into the .tmp file. In this case, we do > nothing, which is not a problem: the other process will create the > file. unlink() does not fail, even if t

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

STINNER Victor added the comment: > 3) Fall back to the original ".tmp" suffix (with the risk > of stale tmp file). I missed something, what is the "stale tmp file" issue? Python/import.c uses: (void) unlink(filename); fd = open(filename, O_EXCL|O_CREAT|O_WRONLY|O_TRUNC ...); So Pytho

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Antoine Pitrou added the comment: > Thoughts? I would say go for the simpler (that's probably option 3). -- ___ Python tracker ___ _

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Vinay Sajip added the comment: What's the downside of option 2) ? -- nosy: +vinay.sajip ___ Python tracker ___ ___ Python-bugs-list m

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Changes by Vinay Sajip : -- resolution: fixed -> ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.p

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Charles-François Natali added the comment: mkstemp() creates the file with mode 0600, which can be surprising. I'm note sure about the best way to handle this: 1) change the file mode after creating it with fchmod(), using the source file mode. But we must also take into account the umask, so w

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Changes by Charles-François Natali : -- resolution: -> fixed stage: -> committed/rejected status: open -> closed ___ Python tracker ___

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Roundup Robot added the comment: New changeset 740baff4f169 by Charles-François Natali in branch 'default': Issue #13303: Fix a race condition in the bytecode file creation. http://hg.python.org/cpython/rev/740baff4f169 -- nosy: +python-dev ___ Pytho

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Antoine Pitrou added the comment: > Here's a patch for Python/import.c using mkstemp(3). AFAICT, mkstemp > should always be available (on Unix of course). This looks fine to me. The code simplification tastes good :) -- ___ Python tracker

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Charles-François Natali added the comment: Here's a patch for Python/import.c using mkstemp(3). AFAICT, mkstemp should always be available (on Unix of course). -- Added file: http://bugs.python.org/file23572/import_atomic_race-2.diff ___ Python trac

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Charles-François Natali added the comment: > (it does sound a bit overkill to me :-)) Well, it depends on what we want: - if having a stale bytecode file indefinitely is acceptable, then the '.tmp' suffix approach is OK - if not, then we should use pseudo-random files > If you go that way, yo

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Antoine Pitrou added the comment: > Here's a patch using pseudo-random filenames. > I used id(path) to generate a random name: it's faster than getpid(), > and doesn't pollute strace's output (I'm one of strace's biggest > fan :-). If you go that way, you should also modify Python/import.c to u

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Charles-François Natali added the comment: Here's a patch using pseudo-random filenames. I used id(path) to generate a random name: it's faster than getpid(), and doesn't pollute strace's output (I'm one of strace's biggest fan :-). -- Added file: http://bugs.python.org/file23571/impor

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Charles-François Natali added the comment: Then we won't write the bytecode any more. But it will be consistent. The solution would be to create a random temporary file (with e.g. mkstemp()): in case of crash, we'll let dangling temporary files, but we'll be able to update the bytecode. I'm no

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Antoine Pitrou added the comment: This looks good to me. I'm slightly worried about what happens when there's a stale tmp file (for example if the Python process crashed before renaming it). -- ___ Python tracker

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Charles-François Natali added the comment: Patch attached. -- keywords: +patch Added file: http://bugs.python.org/file23568/import_atomic_race.diff ___ Python tracker ___ __

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

Charles-François Natali added the comment: There's a race in _write_atomic(): """ # On POSIX-like platforms, renaming is atomic path_tmp = path + '.tmp' try: fd = _os.open(path_tmp, _os.O_EXCL | _os.O_CREAT | _os.O_WRONLY) with _io.FileIO(fd, 'wb')

[issue13303] Sporadic importlib failures: FileNotFoundError on os.rename()

New submission from STINNER Victor : Traceback (most recent call last): File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1186, in runtest_inner indirect_test() File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_ttk_textonl