Re: [HACKERS] Differential backup

2010-04-28 Thread Greg Stark
On Tue, Apr 27, 2010 at 8:15 PM, Craig Ringer wrote: > > Another avenue possibly worth investigating may be using the in-heap > mvcc information to do SQL-level differential backups of individual > tables or of the whole database. think: > You can't use the mvcc information to do incremental back

Re: [HACKERS] Differential backup

2010-04-28 Thread Kevin Grittner
Hannu Krosing wrote: > On Tue, 2010-04-27 at 17:28 +0200, Csaba Nagy wrote: >> One use case we would have is to dump only the changes from the >> last backup of a single table. This table takes 30% of the DB >> disk space, it is in the order of ~400GB, and it's only inserted, >> never updated, t

Re: [HACKERS] Differential backup

2010-04-28 Thread Hannu Krosing
On Tue, 2010-04-27 at 17:28 +0200, Csaba Nagy wrote: > Hi all, > > On Tue, 2010-04-27 at 11:07 -0400, Merlin Moncure wrote: > > The block level case seems pretty much covered by the hot standby feature. > > One use case we would have is to dump only the changes from the last > backup of a single

Re: [HACKERS] Differential backup

2010-04-28 Thread Hannu Krosing
On Tue, 2010-04-27 at 10:32 -0400, Michael Tharp wrote: > On 04/27/2010 09:59 AM, Kevin Grittner wrote: > > Under what circumstances would PostgreSQL > > modify a file without changing the "last modified" timestamp or the > > file size? > > Do all OSes have sub-second precision mtimes? Because oth

Re: [HACKERS] Differential backup

2010-04-28 Thread Hannu Krosing
On Tue, 2010-04-27 at 12:14 -0400, Merlin Moncure wrote: > On Tue, Apr 27, 2010 at 11:13 AM, Kevin Grittner > wrote: > > Merlin Moncure wrote: > > > >> The proposal only seems a win to me if a fair percentage of the > >> larger files don't change, which strikes me as a relatively low > >> level c

Re: [HACKERS] Differential backup

2010-04-28 Thread Kevin Grittner
Hannu Krosing wrote: > I see the main value when doing pg_dump based backups Ah, now that makes more sense. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Differential backup

2010-04-28 Thread Hannu Krosing
On Tue, 2010-04-27 at 08:59 -0500, Kevin Grittner wrote: > Simon Riggs wrote: > > > Thinking about allowing a backup to tell which files have changed > > in the database since last backup. This would allow an external > > utility to copy away only changed files. > > > > Now there's a few ways o

Re: [HACKERS] Differential backup

2010-04-28 Thread Hannu Krosing
On Tue, 2010-04-27 at 14:30 +0100, Simon Riggs wrote: > Thinking about allowing a backup to tell which files have changed in the > database since last backup. This would allow an external utility to copy > away only changed files. > > Now there's a few ways of doing this and many will say this is

Re: [HACKERS] Differential backup

2010-04-27 Thread Craig Ringer
Simon Riggs wrote: > Is this route worthwhile? Or in some way unacceptable? >From an admin perspective, I think block-level differentials would be a lot more useful, especially if index storage could be safely excluded. IMO Pg really could use an "index bad or missing, rebuild on postmaster star

Re: [HACKERS] Differential backup

2010-04-27 Thread Robert Haas
On Tue, Apr 27, 2010 at 10:08 AM, Simon Riggs wrote: > On Tue, 2010-04-27 at 08:59 -0500, Kevin Grittner wrote: >> > An explicit mechanism where Postgres could authoritatively say >> > which files have changed would make many feel safer, especially >> > when other databases also do this. >> >> Why

Re: [HACKERS] Differential backup

2010-04-27 Thread Kevin Grittner
Merlin Moncure wrote: > Your archiving requirements are high. They are set by a Steering Committee composed of the Directory of State Courts and various District Court Administrators, Judges, Clerks of Court, and Registers in Probate who rely on this data and *really* want to be safe. I just

Re: [HACKERS] Differential backup

2010-04-27 Thread Merlin Moncure
On Tue, Apr 27, 2010 at 11:13 AM, Kevin Grittner wrote: > Merlin Moncure wrote: > >> The proposal only seems a win to me if a fair percentage of the >> larger files don't change, which strikes me as a relatively low >> level case to optimize for. > > That's certainly a situation we face, with a r

Re: [HACKERS] Differential backup

2010-04-27 Thread Csaba Nagy
Hi all, On Tue, 2010-04-27 at 11:07 -0400, Merlin Moncure wrote: > The block level case seems pretty much covered by the hot standby feature. One use case we would have is to dump only the changes from the last backup of a single table. This table takes 30% of the DB disk space, it is in the orde

Re: [HACKERS] Differential backup

2010-04-27 Thread Kevin Grittner
Merlin Moncure wrote: > The proposal only seems a win to me if a fair percentage of the > larger files don't change, which strikes me as a relatively low > level case to optimize for. That's certainly a situation we face, with a relatively slow WAN in the middle. http://archives.postgresql.o

Re: [HACKERS] Differential backup

2010-04-27 Thread Merlin Moncure
On Tue, Apr 27, 2010 at 10:32 AM, Michael Tharp wrote: > On 04/27/2010 09:59 AM, Kevin Grittner wrote: >> >> Under what circumstances would PostgreSQL >> modify a file without changing the "last modified" timestamp or the >> file size? > > Do all OSes have sub-second precision mtimes? Because othe

Re: [HACKERS] Differential backup

2010-04-27 Thread Michael Tharp
On 04/27/2010 09:59 AM, Kevin Grittner wrote: Under what circumstances would PostgreSQL modify a file without changing the "last modified" timestamp or the file size? Do all OSes have sub-second precision mtimes? Because otherwise I could see a scenario such at this: * File is modified * Bac

Re: [HACKERS] Differential backup

2010-04-27 Thread Florian Pflug
On Apr 27, 2010, at 16:08 , Simon Riggs wrote: > On Tue, 2010-04-27 at 08:59 -0500, Kevin Grittner wrote: >> Why? I must be missing something, because my feeling is that if you >> can't trust your OS to cover something like this, how can you trust >> any application *running* under that OS to do i

Re: [HACKERS] Differential backup

2010-04-27 Thread Florian Pflug
On Apr 27, 2010, at 15:50 , Alvaro Herrera wrote: > Simon Riggs wrote: >> Thinking about allowing a backup to tell which files have changed in the >> database since last backup. This would allow an external utility to copy >> away only changed files. >> >> Now there's a few ways of doing this and

Re: [HACKERS] Differential backup

2010-04-27 Thread Simon Riggs
On Tue, 2010-04-27 at 08:59 -0500, Kevin Grittner wrote: > > An explicit mechanism where Postgres could authoritatively say > > which files have changed would make many feel safer, especially > > when other databases also do this. > > Why? I must be missing something, because my feeling is that

Re: [HACKERS] Differential backup

2010-04-27 Thread Kevin Grittner
Simon Riggs wrote: > Thinking about allowing a backup to tell which files have changed > in the database since last backup. This would allow an external > utility to copy away only changed files. > > Now there's a few ways of doing this and many will say this is > already possible using file ac

Re: [HACKERS] Differential backup

2010-04-27 Thread Simon Riggs
On Tue, 2010-04-27 at 09:50 -0400, Alvaro Herrera wrote: > Simon Riggs wrote: > > > > Thinking about allowing a backup to tell which files have changed in the > > database since last backup. This would allow an external utility to copy > > away only changed files. > > > > Now there's a few ways o

Re: [HACKERS] Differential backup

2010-04-27 Thread Alvaro Herrera
Simon Riggs wrote: > > Thinking about allowing a backup to tell which files have changed in the > database since last backup. This would allow an external utility to copy > away only changed files. > > Now there's a few ways of doing this and many will say this is already > possible using file ac

[HACKERS] Differential backup

2010-04-27 Thread Simon Riggs
Thinking about allowing a backup to tell which files have changed in the database since last backup. This would allow an external utility to copy away only changed files. Now there's a few ways of doing this and many will say this is already possible using file access times. An explicit mechanis