On Sun, Feb 19, 2023 at 08:06:24PM +0530, Robert Haas wrote: > I mean, my idea was to basically just have one big callback: > ArchiverModuleMainLoopCB(). Which wouldn't return, or perhaps, would > only return when archiving was totally caught up and there was nothing > more to do right now. And then that callback could call functions like > AreThereAnyMoreFilesIShouldBeArchivingAndIfYesWhatIsTheNextOne(). So > it would call that function and it would find out about a file and > start an HTTP session or whatever and then call that function again > and start another HTTP session for the second file and so on until it > had as much concurrency as it wanted. And then when it hit the > concurrency limit, it would wait until at least one HTTP request > finished. At that point it would call > HeyEverybodyISuccessfullyArchivedAWalFile(), after which it could > again ask for the next file and start a request for that one and so on > and so forth.
This archiving implementation is not completely impossible with the current API infrastructure, either? If you consider the archiving as a two-step process where segments are first copied into a cheap, reliable area. Then these could be pushed in block in a more remote area like a S3 bucket? Of course this depends on other things like the cluster structure, but redundancy can be added with standby archiving, as well. I am not sure exactly how many requirements we want to push into a callback, to be honest, and surely more requirements pushed to the callback increases the odds of implementation mistakes, like a full loop. There already many ways to get it wrong with archiving, like missing a flush of the archived segment before the callback returns to ensure its durability.. -- Michael
signature.asc
Description: PGP signature
