On Mon, Aug 23, 2021 at 11:50 AM Bossart, Nathan <bossa...@amazon.com> wrote: > To handle a "cheating" archive command, I'd probably need to add a > stat() for every time pgarch_readyXLog() returned something from > arch_files. I suspect something similar might be needed in Dipesh's > patch to handle backup history files and partial WAL files.
I think he's effectively got that already, although it's probably inside of pgarch_readyXLog(). The idea there is that instead of having a cache of files to be returned (as in your case) he just checks whether the next file in sequence happens to be present and if so returns that file name. To see whether it's present, he uses stat(). > In any case, I think Dipesh's patch is the way to go. It obviously > will perform better in the extreme cases discussed in this thread. I > think it's important to make sure the patch doesn't potentially leave > files behind to be picked up by a directory scan that might not > happen, but there are likely ways to handle that. In the worst case, > perhaps we need to force a directory scan every N files to make sure > nothing gets left behind. But maybe we can do better. It seems to me that we can handle that by just having the startup process notify the archiver every time some file is ready for archiving that's not the next one in the sequence. We have to make sure we cover all the relevant code paths, but that seems like it should be doable, and we have to decide on the synchronization details, but that also seems pretty manageable, even if we haven't totally got it sorted yet. The thing is, as soon as you go back to forcing a directory scan every N files, you've made it formally O(N^2) again, which might not matter in practice if the constant factor is low enough, but I don't think it will be. Either you force the scans every, say, 1000 files, in which case it's going to make the whole mechanism a lot less effective in terms of getting out from under problem cases -- or you force scans every, say, 1000000 files, in which case it's not really going to cause any missed files to get archived soon enough to make anyone happy. I doubt there is really a happy medium in there. I suppose the two approaches could be combined, too - remember the first N files you think you'll encounter and then after that try successive filenames until one is missing. That would be more resilient against O(N^2) behavior in the face of frequent gaps. But it might also be more engineering than is strictly required. -- Robert Haas EDB: http://www.enterprisedb.com
