On 8/21/21, 9:29 PM, "Bossart, Nathan" <bossa...@amazon.com> wrote: > I was curious about this, so I wrote a patch (attached) to store > multiple files per directory scan and tested it against the latest > patch in this thread (v9) [0]. Specifically, I set archive_command to > 'false', created ~20K WAL segments, then restarted the server with > archive_command set to 'true'. Both the v9 patch and the attached > patch completed archiving all segments in just under a minute. (I > tested the attached patch with NUM_FILES_PER_DIRECTORY_SCAN set to 64, > 128, and 256 and didn't observe any significant difference.) The > existing logic took over 4 minutes to complete. > > I'm hoping to do this test again with many more (100K+) status files, > as I believe that the v9 patch will be faster at that scale, but I'm > not sure how much faster it will be.
I ran this again on a bigger machine with 200K WAL files pending archive. The v9 patch took ~5.5 minutes, the patch I sent took ~8 minutes, and the existing logic took just under 3 hours. Nathan
