On Fri, Aug 1, 2014 at 1:43 PM, desmodemone <desmodem...@gmail.com> wrote: > > > > 2014-08-01 18:20 GMT+02:00 Claudio Freire <klaussfre...@gmail.com>: > >> On Fri, Aug 1, 2014 at 12:35 AM, Amit Kapila <amit.kapil...@gmail.com> >> wrote: >> >> c) the map is not crash safe by design, because it needs only for >> >> incremental backup to track what blocks needs to be backuped, not for >> >> consistency or recovery of the whole cluster, so it's not an heavy cost >> >> for >> >> the whole cluster to maintain it. we could think an option (but it's >> >> heavy) >> >> to write it at every flush on file to have crash-safe map, but I not >> >> think >> >> it's so usefull . I think it's acceptable, and probably it's better to >> >> force >> >> that, to say: "if your db will crash, you need a fullbackup ", >> > >> > I am not sure if your this assumption is right/acceptable, how can >> > we say that in such a case users will be okay to have a fullbackup? >> > In general, taking fullbackup is very heavy operation and we should >> > try to avoid such a situation. >> >> >> Besides, the one taking the backup (ie: script) may not be aware of >> the need to take a full one. >> >> It's a bad design to allow broken backups at all, IMNSHO. > > > Hi Claudio, > thanks for your observation > First: the case it's after a crash of a database, and it's not something > happens every day or every week. It's something that happens in rare > conditions, or almost my experience is so. If it happens very often probably > there are other problems.
Not so much. In this case, the software design isn't software-crash safe, it's not that it's not hardware-crash safe. What I mean, is that an in-memory bitmap will also be out of sync if you kill -9 (or if one of the backends is killed by the OOM), or if it runs out of disk space too. Normally, a simple restart fixes it because pg will do crash recovery just fine, but now the bitmap is out of sync, and further backups are broken. It's not a situation I want to face unless there's a huge reason to go for such design. If you make it so that the commit includes flipping the bitmap, it can be done cleverly enough to avoid too much overhead (though it will have some), and you now have it so that any to-be-touched block is now part of the backup. You just apply all the bitmap changes in batch after a checkpoint, before syncing to disk, and before erasing the WAL segments. Simple, relatively efficient, and far more robust than an in-memory thing. Still, it *can* double checkpoint I/O on the worst case, and it's not an unfathomable case either. > Second: to avoid the problem to know if the db needed to have a full backup > to rebuild the map we could think to write in the map header the backup > reference (with an id and LSN reference for example ) so if the > someone/something try to do an incremental backup after a crash, the map > header will not have noone full backup listed [because it will be empty] , > and automaticcaly switch to a full one. I think after a crash it's a good > practice to do a full backup, to see if there are some problems on files or > on filesystems, but if I am wrong I am happy to know :) . After a crash I do not do a backup, I do a verification of the data (VACUUM and some data consistency checks usually), lest you have a useless backup. The backup goes after that. But, I'm not DBA guru. > Remember that I propose a map in ram to reduce the impact on performances, > but we could create an option to leave the choose to the user, if you want a > crash safe map, at every flush will be updated also a map file , if not, the > map will be in ram. I think the performance impact of a WAL-linked map isn't so big as to prefer the possibility of broken backups. I wouldn't even allow it. It's not free, making it crash safe, but it's not that expensive either. If you want to support incremental backups, you really really need to make sure those backups are correct and usable, and IMV anything short of full crash safety will be too fragile for that purpose. I don't want to be in a position of needing the backup and finding out it's inconsistent after the fact, and I don't want to encourage people to set themselves up for that by adding that "faster but unsafe backups" flag. I'd do it either safe, or not at all. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
