As part of profiling a Python process on Windows using Process Monitor.exe, I noticed that closing file handles using CloseFile() takes 1+ms. Contrast this with other I/O related system calls like WriteFile() that tend to take ~1us. (I /think/ it only takes a longer time if the file has been written to.) This is on Windows 10 running natively (no VM) when writing to an SSD. Files are opened with _fopen() in "a+" mode if it matters. I can also repro in "a" mode.
When writing thousands of files in rapid succession, this 1+ms pause (assuming synchronous I/O) piles up. Assuming a 1ms pause, writing 100,000 files spends 100s in CloseFile()! The process profile also shows the bulk of the time in CloseFile(), so this is a real hot spot. Both Mercurial and the Firefox build system need to write thousands of files in rapid order. Both are using Python. Both should experience significant performance improvements if we find a more efficient way to do bulk file writing (which currently is over 2x slower than Linux and OS X). Short of going full overlapped I/O (which will be a PITA with Python), the best I can think of is to perform CloseFile() on a background thread or pool of background threads. I was curious if anyone has dug into optimizing the writing of thousands of files on Windows and can share any insights so we can make the Firefox build system and Mercurial significantly faster on Windows. gps _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
