[issue4944] os.fsync() doesn't work as expect in Windows

Emil Styrke added the comment: eryksun, thank you for the explanation, the penny finally dropped for me. It's the *other* child processes that keep the file open, not the parent. Actually this bug started hitting me after I *stopped* doing exactly what you suggest (fdopen on the temp file),

[issue4944] os.fsync() doesn't work as expect in Windows

eryksun added the comment: Emil, Your example child process opens the file with only read sharing, which fails with a sharing violation if some other process inherits the file handle with write access. The `with` block only prevents this in a single-threaded environment. When you spawn 10 chi

[issue4944] os.fsync() doesn't work as expect in Windows

Emil Styrke added the comment: I have experienced this issue with Python 2.7.8 and 2.7.9. It is almost the same issue as the OP experiences as far as I can tell: spawn some Python threads that each create a file, flush, fsync, close, then start a subprocess which uses the file through the Win

[issue4944] os.fsync() doesn't work as expect in Windows

Terry J. Reedy added the comment: To be a tracker bug, as opposed to feature request, behavior must disagree with the manual or doc string, not personal expectation. In any case, OP declined to test on 2.6 and suggested closing, so I am. This still remains on tracker to be searched. Anyone w

[issue4944] os.fsync() doesn't work as expect in Windows

Changes by Brian Curtin : -- nosy: +brian.curtin priority: -> normal stage: -> test needed ___ Python tracker ___ ___ Python-bugs-lis

[issue4944] os.fsync() doesn't work as expect in Windows

Javen Wang added the comment: I don't have Python2.6 installed. I just have ever read a bench article comparing the performance between different version of Python, including 2.5, 2.6 and 3.0. That article shows, for the same script, Python2.6 has longer running time than Python2.5. My applicati

[issue4944] os.fsync() doesn't work as expect in Windows

Amaury Forgeot d'Arc added the comment: python2.6 is not so different from python2.5. Which performance concerns do you have? ___ Python tracker ___ __

[issue4944] os.fsync() doesn't work as expect in Windows

Javen Wang added the comment: I'm using Python2.5 in which close_fds is not available in Windows. And I cannot use Python2.6 because I need to redirect the stdout and stderr and there's performance concern. I have questions on the root cause: a. why doesn't fopen() has sharing issue? b. wh

[issue4944] os.fsync() doesn't work as expect in Windows

Gabriel Genellina added the comment: Thanks for adapting the smaller example. I think I figured out what's the problem. The error reported by checkfile.c is 0x20 = ERROR_SHARING_VIOLATION "The process cannot access the file because it is being used by another process." I modified the subproc

[issue4944] os.fsync() doesn't work as expect in Windows

Javen Wang added the comment: The test code and script which can reproduce the problem has been uploaded. Added file: http://bugs.python.org/file12808/test_file_flush.py ___ Python tracker _

[issue4944] os.fsync() doesn't work as expect in Windows

Javen Wang added the comment: I did trace the order of file creation and process launch. It shows the file is indeed created before the process launch. I did another investigation. I added a new process, "cmd.exe /c copy Makefile NewMakefile", to copy the the Makefile created, just before launc

[issue4944] os.fsync() doesn't work as expect in Windows

Gabriel Genellina added the comment: > I'm quite certain that the makefile is generated before > the "make" > program is launched in separated process. Follow is the > original code > where the makefile is created (line 14) and a task is put > in queue (line > 19). It's executed in the main thre

[issue4944] os.fsync() doesn't work as expect in Windows

Javen Wang added the comment: I'm quite certain that the makefile is generated before the "make" program is launched in separated process. Follow is the original code where the makefile is created (line 14) and a task is put in queue (line 19). It's executed in the main thread (no parallel execu

[issue4944] os.fsync() doesn't work as expect in Windows

Gabriel Genellina added the comment: If you start the new thread *after* the file is closed, no race condition should exist. I'm sorry but I'm unable to follow the code you use to create threads and write the makefile; but you should verify that your original assertion "Once a makefile is gen

[issue4944] os.fsync() doesn't work as expect in Windows

Javen Wang added the comment: I narrowed down the root cause in the GIL of Python. I read the source code of implementing os.fsync and found it's using Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS to enclose the calling of _commit() in order to be thread safe. I tried to add them in my C exte

[issue4944] os.fsync() doesn't work as expect in Windows

Javen Wang added the comment: I agree multithread programming is kind of difficult and tricky. But I don't think there's a race condition in _MultiThreadBuildPlatform method, because I do have only one producer. And the consumer consumes the product only when it's done (i.e. the file is created

[issue4944] os.fsync() doesn't work as expect in Windows

Gabriel Genellina added the comment: I patiently waited for all those 150MB to download, modified Misc.py, run the specified commands and got this error: build.py... : error 7000: Failed to start command C:\Program Files\Microsoft Visual Studio 8\Vc\bin\nmake.exe / nologo -s t build [

[issue4944] os.fsync() doesn't work as expect in Windows

Javen Wang added the comment: Thank you very much for the trying. You might miss the step 4 in my previous message. The step is: C:\test\edk2> edksetup.bat C:\test\edk2> set PYTHONPATH=C:\test\tools\Source\Python C:\test\edk2> python.exe C:\test\tools\Source\Python\build\build.py -n 2 -p

[issue4944] os.fsync() doesn't work as expect in Windows

Amaury Forgeot d'Arc added the comment: I really tried, but I can't run your script. first PYTHONPATH must be set to some value (I tried with tools\Source\Python); Also the tool is complaining about missing WORKSPACE and EDK_TOOLS_PATH env vars, and a missing file "Conf\target.txt". This seems v

[issue4944] os.fsync() doesn't work as expect in Windows

Javen Wang added the comment: There would be more chances to see the problem by doing this: C:\test\edk2> python.exe C:\test\tools\Source\Python\build\build.py -n 2 -p IntelFrameworkModulePkg\IntelFrameworkModulePkg.dsc -a IA32 -s ___ Python tracker

[issue4944] os.fsync() doesn't work as expect in Windows

Javen Wang added the comment: The created files are on local drive. I saw the problem on the laptop (XP-SP2), desktop(XP-SP3) and server (Win2003). But there's no such problem on the Linux and Mac boxes. I tried to use fopen/fwrite in my extension according to your suggestion. The problem wasn'

[issue4944] os.fsync() doesn't work as expect in Windows

Javen Wang added the comment: gagenellina, My application is a little bit different from your test code. It won't wait for the exit of new process and there're file writing operations during makefile running. I changed the test code to be real multi-process and tried many file sizes. But I cann

[issue4944] os.fsync() doesn't work as expect in Windows

Amaury Forgeot d'Arc added the comment: python2.5 uses the functions of the fopen() family: fwrite(), fclose(). Does the problem reproduce if you use these functions in your extension module? Are your files located on the local hard drive, or on a network storage? -- nosy: +amaury.fo

[issue4944] os.fsync() doesn't work as expect in Windows

Changes by Gabriel Genellina : Added file: http://bugs.python.org/file12738/checkfile.c ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue4944] os.fsync() doesn't work as expect in Windows

Gabriel Genellina added the comment: I tried to reproduce the issue, interpreting your description, but failed. It worked fine in my setup. Perhaps you can add more elements until it fails. -- nosy: +gagenellina Added file: http://bugs.python.org/file12737/test_file_flush.py

[issue4944] os.fsync() doesn't work as expect in Windows

Changes by Ulrich Eckhardt : -- nosy: +eckhardt ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue4944] os.fsync() doesn't work as expect in Windows

New submission from Javen Wang : I encountered a very strange issue in file flush operation in Windows. Here's the scenario of my application: 1. The main thread of my application will create makefiles sequentially. 2. Once a makefile is generated, launch a separate process calling nmake