On Mon, Sep 28, 2015 at 6:45 PM, Ehsan Akhgari <ehsan.akhg...@gmail.com> wrote:
> On 2015-09-28 5:41 PM, Gregory Szorc wrote: > >> When writing thousands of files in rapid succession, this 1+ms pause >> (assuming synchronous I/O) piles up. Assuming a 1ms pause, writing 100,000 >> files spends 100s in CloseFile()! The process profile also shows the bulk >> of the time in CloseFile(), so this is a real hot spot. >> > > There is no CloseFile() on Windows. Did you mean CloseHandle()? > While this is probably something I should know, I confess to blindly copying results from Sysinternals' procmon utility, which reports file closes as the "CloseFile()" "operation." I reckon it is being intelligent and converting CloseHandle() to something more useful for reporting purposes. In my defense, procmon does report "operations" that I know are actual Windows functions. Kinda weird it is inconsistent. Who knows. The reason I'm asking is that CloseHandle() can close various types of > kernel objects, and if that is showing up in profiles, it's worth to verify > that the handle passed to it is actually coming from CreateFile(Ex). > Procmon is reporting lots of CreateFile() calls. And I'm 100% certain the underlying C code is calling CreateFile(). > Closing handles on a background thread doesn't help with performance if > you're invoking sub-processes that need to close a handle and wait for the > operation to finish. It would help if you provided more details on the > constraints you're dealing with, e.g., where do these handles come from? > Are they being created by one long running process or by several short > lived ones? etc. Another idea to experiment with is leaking the handles > and letting the kernel close them for you when your process is terminated. > I _think_ (but I'm not sure) that won't count towards the handle of the > process to become signaled so if you're spawning a process that needs to > close the file and wait for that to finish, that may be faster. > I'm dealing with a single threaded single long-running process that performs synchronous I/O, 1 open file at a time. CreateFile, CloseHandle, CreateFile, CloseHandle, ... I'm pretty sure leaking handles is out of the question, as we need to write to thousands or even tens of thousands of files and this will exhaust open files limits. > > On the topic of performance on Windows (but not directly related to your > question), beware of the ~60ms CreateProcess overhead < > https://llvm.org/bugs/show_bug.cgi?id=20253#c4>. Depending on the > context, one could kill both of these perf issues with one stone by doing > as many of the file manipulation in one process as you can, and closing the > handles on a background thread. > I'm well aware that new processes on Windows are much more expensive than on other architectures. You could say that parts of the Firefox build system go out of their way to avoid new processes because of this property :) _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
