In the lengthy thread on block-level incremental backup,[1] both Vignesh C[2] and Stephen Frost[3] have suggested storing a manifest as part of each backup, somethig that could be useful not only for incremental backups but also for full backups. I initially didn't think this was necessary,[4] but some of my colleagues figured out that my design was broken, because my proposal was to detect new blocks just using LSN, and that ignores the fact that CREATE DATABASE and ALTER TABLE .. SET TABLESPACE do physical copies without bumping page LSNs, which I knew but somehow forgot about. Fortunately, some of my colleagues realized my mistake in testing.[5] Because of this problem, for an LSN-based approach to work, we'll need to send not only an LSN, but also a list of files (and file sizes) that exist in the previous full backup; so, some kind of backup manifest now seems like a good idea to me.[6] That whole approach might still be dead on arrival if it's possible to add new blocks with old LSNs to existing files,[7] but there seems to be room to hope that there are no such cases.[8]
So, let's suppose we invent a backup manifest. What should it contain? I imagine that it would consist of a list of files, and the lengths of those files, and a checksum for each file. I think you should have a choice of what kind of checksums to use, because algorithms that used to seem like good choices (e.g. MD5) no longer do; this trend can probably be expected to continue. Even if we initially support only one kind of checksum -- presumably SHA-something since we have code for that already for SCRAM -- I think that it would also be a good idea to allow for future changes. And maybe it's best to just allow a choice of SHA-224, SHA-256, SHA-384, and SHA-512 right out of the gate, so that we can avoid bikeshedding over which one is secure enough. I guess we'll still have to argue about the default. I also think that it should be possible to build a manifest with no checksums, so that one need not pay the overhead of computing checksums if one does not wish. Of course, such a manifest is of much less utility for checking backup integrity, but you can still check that you've got the right files, which is noticeably better than nothing. The manifest should probably also contain a checksum of its own contents so that the integrity of the manifest itself can be verified. And maybe a few other bits of metadata, but I'm not sure exactly what. Ideas? Once we invent the concept of a backup manifest, what do we need to do with them? I think we'd want three things initially: (1) When taking a backup, have the option (perhaps enabled by default) to include a backup manifest. (2) Given an existing backup that has not got a manifest, construct one. (3) Cross-check a manifest against a backup and complain about extra files, missing files, size differences, or checksum mismatches. One thing I'm not quite sure about is where to store the backup manifest. If you take a base backup in tar format, you get base.tar, pg_wal.tar (unless -Xnone), and an additional tar file per tablespace. Does the backup manifest go into base.tar? Get written into a separate file outside of any tar archive? Something else? And what about a plain-format backup? I suppose then we should just write the manifest into the top level of the main data directory, but perhaps someone has another idea. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company [1] https://www.postgresql.org/message-id/flat/CA%2BTgmoYxQLL%3DmVyN90HZgH0X_EUrw%2BaZ0xsXJk7XV3-3LygTvA%40mail.gmail.com [2] https://www.postgresql.org/message-id/CALDaNm310fUZ72nM2n%3DcD0eSHKRAoJPuCyvvR0dhTEZ9Oytyzg%40mail.gmail.com [3] https://www.postgresql.org/message-id/20190916143817.GA6962%40tamriel.snowman.net [4] https://www.postgresql.org/message-id/CA%2BTgmoaj-zw4Mou4YBcJSkHmQM%2BJA-dAVJnRP8zSASP1S4ZVgw%40mail.gmail.com [5] https://www.postgresql.org/message-id/CAM2%2B6%3DXfJX%3DKXvpTgDvgd1rQjya_Am27j4UvJtL3nA%2BJMCTGVQ%40mail.gmail.com [6] https://www.postgresql.org/message-id/CA%2BTgmoYg9i8TZhyjf8MqCyU8unUVuW%2B03FeBF1LGDu_-eOONag%40mail.gmail.com [7] https://www.postgresql.org/message-id/CA%2BTgmoYT9xODgEB6y6j93hFHqobVcdiRCRCp0dHh%2BfFzZALn%3Dw%40mail.gmail.com and nearby messages [8] https://www.postgresql.org/message-id/20190916173933.GE6962%40tamriel.snowman.net
