Greetings,
* Robert Haas (robertmh...@gmail.com) wrote:
> On Tue, Sep 17, 2019 at 12:09 PM Stephen Frost wrote:
> > We need to be sure that we can detect if the WAL level has ever been set
> > to minimal between a full and an incremental and, if so, either refuse
> > to run the incremental, or pr
On Tue, Sep 17, 2019 at 12:09 PM Stephen Frost wrote:
> We need to be sure that we can detect if the WAL level has ever been set
> to minimal between a full and an incremental and, if so, either refuse
> to run the incremental, or promote it to a full, or make it a
> checksum-based incremental ins
Greetings,
* Robert Haas (robertmh...@gmail.com) wrote:
> On Mon, Sep 16, 2019 at 3:38 PM Stephen Frost wrote:
> > As discussed nearby, not everything that needs to be included in the
> > backup is actually going to be in the WAL though, right? How would that
> > ever be able to handle the case
On Mon, Sep 16, 2019 at 3:38 PM Stephen Frost wrote:
> As discussed nearby, not everything that needs to be included in the
> backup is actually going to be in the WAL though, right? How would that
> ever be able to handle the case where someone starts the server under
> wal_level = logical, take
On Mon, Sep 16, 2019 at 7:00 PM Robert Haas wrote:
>
> On Mon, Sep 16, 2019 at 4:31 AM Amit Kapila wrote:
> > This seems to be a blocking problem for the LSN based design.
>
> Well, only the simplest version of it, I think.
>
> > Can we think of using creation time for file? Basically, if the fi
On Mon, Sep 16, 2019 at 11:09 PM Stephen Frost wrote:
>
> Greetings,
>
> * Robert Haas (robertmh...@gmail.com) wrote:
> > On Mon, Sep 16, 2019 at 9:30 AM Robert Haas wrote:
> > > > Isn't some operations where at the end we directly call heap_sync
> > > > without writing WAL will have a similar pr
Greetings,
* Robert Haas (robertmh...@gmail.com) wrote:
> On Mon, Sep 16, 2019 at 1:10 PM Stephen Frost wrote:
> > I disagree with this on a couple of levels. The first is pretty simple-
> > we don't have all of the information. The user may have some reason to
> > believe that timestamp-based
On Mon, Sep 16, 2019 at 1:10 PM Stephen Frost wrote:
> I disagree with this on a couple of levels. The first is pretty simple-
> we don't have all of the information. The user may have some reason to
> believe that timestamp-based is a bad idea, for example, and therefore
> having an option to p
Greetings,
* Robert Haas (robertmh...@gmail.com) wrote:
> On Mon, Sep 16, 2019 at 9:30 AM Robert Haas wrote:
> > > Isn't some operations where at the end we directly call heap_sync
> > > without writing WAL will have a similar problem as well?
> >
> > Maybe. Can you give an example?
>
> Looking
Greetings,
* Robert Haas (robertmh...@gmail.com) wrote:
> On Mon, Sep 16, 2019 at 10:38 AM Stephen Frost wrote:
> > In a number of cases, trying to make sure that on a failover or copy of
> > the backup the next 'incremental' is really an 'incremental' is
> > dangerous. A better strategy to addr
On Mon, Sep 16, 2019 at 10:38 AM Stephen Frost wrote:
> In a number of cases, trying to make sure that on a failover or copy of
> the backup the next 'incremental' is really an 'incremental' is
> dangerous. A better strategy to address this, and the other issues
> realized on this thread recently
On Mon, Sep 16, 2019 at 9:30 AM Robert Haas wrote:
> > Isn't some operations where at the end we directly call heap_sync
> > without writing WAL will have a similar problem as well?
>
> Maybe. Can you give an example?
Looking through the code, I found two cases where we do this. One is
a bulk i
Greetings,
* Robert Haas (robertmh...@gmail.com) wrote:
> On Mon, Sep 16, 2019 at 4:31 AM Amit Kapila wrote:
> > Can we think of using creation time for file? Basically, if the file
> > creation time is later than backup-labels "START TIME:", then include
> > that file entirely. I think one big
On Mon, Sep 16, 2019 at 4:31 AM Amit Kapila wrote:
> This seems to be a blocking problem for the LSN based design.
Well, only the simplest version of it, I think.
> Can we think of using creation time for file? Basically, if the file
> creation time is later than backup-labels "START TIME:", th
On Mon, Sep 16, 2019 at 7:22 AM Robert Haas wrote:
>
> On Thu, Sep 12, 2019 at 9:13 AM Jeevan Chalke
> wrote:
> > I had a look over this issue and observed that when the new database is
> > created, the catalog files are copied as-is into the new directory
> > corresponding to a newly created dat
On Thu, Sep 12, 2019 at 9:13 AM Jeevan Chalke
wrote:
> I had a look over this issue and observed that when the new database is
> created, the catalog files are copied as-is into the new directory
> corresponding to a newly created database. And as they are just copied,
> the LSN on those pages are
On Fri, Sep 13, 2019 at 1:08 PM vignesh C wrote:
> Instead of reading the whole file here, we can read the file page by page.
> There is a possibility of data inconsistency if data is not read page by
> page, data will be consistent if read page by page as full page write will be
> enabled at t
On Mon, Sep 9, 2019 at 4:51 PM Jeevan Chalke
wrote:
>
>
>
> On Tue, Aug 27, 2019 at 4:46 PM vignesh C wrote:
>>
>> Few comments:
>> Comment:
>> + buf = (char *) malloc(statbuf->st_size);
>> + if (buf == NULL)
>> + ereport(ERROR,
>> + (errcode(ERRCODE_OUT_OF_MEMORY),
>> + errmsg("out of memory")))
Hi,
One of my colleague at EDB, Rajkumar Raghuwanshi, while testing this
feature reported an issue. He reported that if a full base-backup is
taken, and then created a database, and then took an incremental backup,
combining full backup with incremental backup is then failing.
I had a look over t
On Fri, Aug 30, 2019 at 6:52 PM Jeevan Ladhe
wrote:
> Here are some comments:
> Or maybe we can just say:
> "cannot verify checksum in file \"%s\"" if checksum requested, disable the
> checksum and leave it to the following message:
>
> + ereport(WARNING,
> + (errmsg("
On Tue, Sep 3, 2019 at 12:11 PM Dilip Kumar wrote:
> On Fri, Aug 16, 2019 at 3:54 PM Jeevan Chalke
> wrote:
> >
> 0003:
> +/*
> + * When to send the whole file, % blocks modified (90%)
> + */
> +#define WHOLE_FILE_THRESHOLD 0.9
>
> How this threshold is selected. Is it by some test?
>
Currentl
On Wed, Sep 4, 2019 at 5:21 PM Dilip Kumar wrote:
>
> I have not yet completed the review for 0004, but I have few more
> comments. Tomorrow I will try to complete the review and some testing
> as well.
>
> 1. It seems that the output full backup generated with
> pg_combinebackup also contains
On Tue, Aug 27, 2019 at 11:59 PM Robert Haas wrote:
> On Fri, Aug 16, 2019 at 6:23 AM Jeevan Chalke
> wrote:
> > [ patches ]
>
> Reviewing 0002 and 0003:
>
> - Commit message for 0003 claims magic number and checksum are 0, but
> that (fortunately) doesn't seem to be the case.
>
Oops, updated c
On Tue, Aug 27, 2019 at 4:46 PM vignesh C wrote:
> Few comments:
> Comment:
> + buf = (char *) malloc(statbuf->st_size);
> + if (buf == NULL)
> + ereport(ERROR,
> + (errcode(ERRCODE_OUT_OF_MEMORY),
> + errmsg("out of memory")));
> +
> + if ((cnt = fread(buf, 1, statbuf->st_size, fp)) > 0)
> + {
>
Hi,
Attached new set of patches adding support for the tablespace handling.
This patchset also fixes the issues reported by Vignesh, Robert, Jeevan
Ladhe,
and Dilip Kumar.
Please have a look and let me know if I missed any comments to account.
Thanks
--
Jeevan Chalke
Technical Architect, Prod
On Wed, Sep 4, 2019 at 10:08 PM Michael Paquier wrote:
> > For generating a
> > file, you can always emit the newest and "best" tar format, but for
> > reading a file, you probably want to be prepared for older or cruftier
> > variants. Maybe not -- I'm not super-familiar with the tar on-disk
> >
On Tue, Sep 03, 2019 at 08:59:53AM -0400, Robert Haas wrote:
> I think pg_basebackup is using homebrew code to generate tar files,
> but I'm reluctant to do that for reading tar files.
Yes. This code has not actually changed since its introduction.
Please note that we also have code which reads d
On Tue, Sep 3, 2019 at 12:46 PM Ibrar Ahmed wrote:
> I did that and have experience working on the TAR format. I was curious
> about what
> "best/newest" you are talking.
Well, why not go look it up?
On my MacBook, tar is documented to understand three different tar
formats: gnutar, ustar, and
On Tue, Sep 3, 2019 at 12:11 PM Dilip Kumar wrote:
>
> On Fri, Aug 16, 2019 at 3:54 PM Jeevan Chalke
> wrote:
> >
> 0003:
> +/*
> + * When to send the whole file, % blocks modified (90%)
> + */
> +#define WHOLE_FILE_THRESHOLD 0.9
>
> How this threshold is selected. Is it by some test?
>
>
> - ma
On Tue, Sep 3, 2019 at 7:39 PM Robert Haas wrote:
> On Tue, Sep 3, 2019 at 10:05 AM Ibrar Ahmed wrote:
> > +1 using the library to tar. But I think reason not using tar library is
> TAR is
> > one of the most simple file format. What is the best/newest format of
> TAR?
>
> So, I don't really wan
On Tue, Sep 3, 2019 at 8:00 PM Tom Lane wrote:
> Ibrar Ahmed writes:
> > +1 using the library to tar.
>
> Uh, *what* library?
>
I was just replying the Robert that he said
"But I think there must be a reason why tar libraries exist,
and I don't want to write a new one."
I said I am ok to use
Ibrar Ahmed writes:
> +1 using the library to tar.
Uh, *what* library?
pg_dump's pg_backup_tar.c is about 1300 lines, a very large fraction
of which is boilerplate for interfacing to pg_backup_archiver's APIs.
The stuff that actually knows specifically about tar looks to be maybe
a couple hundre
On Tue, Sep 3, 2019 at 10:05 AM Ibrar Ahmed wrote:
> +1 using the library to tar. But I think reason not using tar library is TAR
> is
> one of the most simple file format. What is the best/newest format of TAR?
So, I don't really want to go down this path at all, as I already
said. You can cer
On Tue, Sep 3, 2019 at 6:00 PM Robert Haas wrote:
> On Sat, Aug 31, 2019 at 3:41 PM Ibrar Ahmed wrote:
> > Are we using any tar library in pg_basebackup.c? We already have the
> capability
> > in pg_basebackup to do that.
>
> I think pg_basebackup is using homebrew code to generate tar files,
>
On Sat, Aug 31, 2019 at 3:41 PM Ibrar Ahmed wrote:
> Are we using any tar library in pg_basebackup.c? We already have the
> capability
> in pg_basebackup to do that.
I think pg_basebackup is using homebrew code to generate tar files,
but I'm reluctant to do that for reading tar files. For gener
On Fri, Aug 16, 2019 at 3:54 PM Jeevan Chalke
wrote:
>
0003:
+/*
+ * When to send the whole file, % blocks modified (90%)
+ */
+#define WHOLE_FILE_THRESHOLD 0.9
How this threshold is selected. Is it by some test?
- magic number, currently 0 (4 bytes)
I think in the patch we are using (#define
Hi Robert,
On Sat, Aug 31, 2019 at 8:29 AM Robert Haas wrote:
> On Thu, Aug 29, 2019 at 10:41 AM Jeevan Ladhe
> wrote:
> > Due to the inherent nature of pg_basebackup, the incremental backup also
> > allows taking backup in tar and compressed format. But, pg_combinebackup
> > does not understan
On Sat, Aug 31, 2019 at 7:59 AM Robert Haas wrote:
> On Thu, Aug 29, 2019 at 10:41 AM Jeevan Ladhe
> wrote:
> > Due to the inherent nature of pg_basebackup, the incremental backup also
> > allows taking backup in tar and compressed format. But, pg_combinebackup
> > does not understand how to res
On Thu, Aug 29, 2019 at 10:41 AM Jeevan Ladhe
wrote:
> Due to the inherent nature of pg_basebackup, the incremental backup also
> allows taking backup in tar and compressed format. But, pg_combinebackup
> does not understand how to restore this. I think we should either make
> pg_combinebackup sup
Here are some comments:
+/* The reference XLOG position for the incremental backup. */
+static XLogRecPtr refptr;
As Robert already pointed we may want to pass this as parameter around
instead
of a global variable. Also, can be renamed to something like:
incr_backup_refptr.
I see in your earlie
Hi,
I am doing some testing on pg_basebackup and pg_combinebackup patches. I
have also tried to create tap test for pg_combinebackup by taking
reference from pg_basebackup tap cases.
Attaching first draft test patch.
I have done some testing with compression options, both -z and -Z level is
worki
Due to the inherent nature of pg_basebackup, the incremental backup also
allows taking backup in tar and compressed format. But, pg_combinebackup
does not understand how to restore this. I think we should either make
pg_combinebackup support restoration of tar incremental backup or restrict
taking
On Fri, Aug 16, 2019 at 6:23 AM Jeevan Chalke
wrote:
> [ patches ]
Reviewing 0002 and 0003:
- Commit message for 0003 claims magic number and checksum are 0, but
that (fortunately) doesn't seem to be the case.
- looks_like_rel_name actually checks whether it looks like a
*non-temporary* relatio
On Fri, Aug 16, 2019 at 8:07 PM Ibrar Ahmed wrote:
>
> What do you mean by bigger file, a file greater than 1GB? In which case you
> get file > 1GB?
>
>
>
Few comments:
Comment:
+ buf = (char *) malloc(statbuf->st_size);
+ if (buf == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ err
On Fri, Aug 16, 2019 at 4:12 PM Ibrar Ahmed wrote:
>
>
>
>
> On Fri, Aug 16, 2019 at 3:24 PM Jeevan Chalke <
> jeevan.cha...@enterprisedb.com> wrote:
>
>>
>>
>> On Fri, Aug 2, 2019 at 6:43 PM vignesh C wrote:
>>
>>> Some comments:
>>> 1) There will be some link files created for tablespace, we m
On Fri, Aug 16, 2019 at 3:24 PM Jeevan Chalke <
jeevan.cha...@enterprisedb.com> wrote:
>
>
> On Fri, Aug 2, 2019 at 6:43 PM vignesh C wrote:
>
>> Some comments:
>> 1) There will be some link files created for tablespace, we might
>> require some special handling for it
>>
>
> Yep. I have that in
On Fri, Aug 9, 2019 at 6:07 AM Jeevan Ladhe
wrote:
> Hi Jeevan,
>
> I have reviewed the backup part at code level and still looking into the
> restore(combine) and functional part of it. But, here are my comments so
> far:
>
Thank you Jeevan Ladhe for reviewing the changes.
>
> The patches nee
On Fri, Aug 2, 2019 at 6:43 PM vignesh C wrote:
> Some comments:
> 1) There will be some link files created for tablespace, we might
> require some special handling for it
>
Yep. I have that in my ToDo.
Will start working on that soon.
> 2)
> Retry functionality is hanlded only for copying of
On Mon, Aug 12, 2019 at 4:57 PM Jeevan Chalke <
jeevan.cha...@enterprisedb.com> wrote:
>
>
> On Fri, Aug 9, 2019 at 6:36 PM Robert Haas wrote:
>
>> On Wed, Aug 7, 2019 at 5:46 AM Jeevan Chalke
>> wrote:
>> > So, do you mean we should just do fread() and fwrite() for the whole
>> file?
>> >
>> >
On Mon, Aug 12, 2019 at 7:57 AM Jeevan Chalke
wrote:
> Agree that we can certainly use open(), read(), write(), and close() here, but
> given that pg_basebackup.c and basbackup.c are using file operations, I think
> using fopen(), fread(), fwrite(), and fclose() will be better here, at-least
> for
On Mon, Aug 12, 2019 at 5:29 PM Jeevan Chalke <
jeevan.cha...@enterprisedb.com> wrote:
>
>
> On Fri, Aug 9, 2019 at 11:56 PM Jeevan Ladhe <
> jeevan.la...@enterprisedb.com> wrote:
>
>> Hi Robert,
>>
>> On Fri, Aug 9, 2019 at 6:40 PM Robert Haas wrote:
>>
>>> On Thu, Aug 8, 2019 at 8:37 PM Jeevan
On Fri, Aug 9, 2019 at 11:56 PM Jeevan Ladhe
wrote:
> Hi Robert,
>
> On Fri, Aug 9, 2019 at 6:40 PM Robert Haas wrote:
>
>> On Thu, Aug 8, 2019 at 8:37 PM Jeevan Ladhe
>> wrote:
>> > + if (!XLogRecPtrIsInvalid(previous_lsn))
>> > + appendStringInfo(labelfile, "PREVIOUS WAL LOCAT
On Fri, Aug 9, 2019 at 6:36 PM Robert Haas wrote:
> On Wed, Aug 7, 2019 at 5:46 AM Jeevan Chalke
> wrote:
> > So, do you mean we should just do fread() and fwrite() for the whole
> file?
> >
> > I thought it is better if it was done by the OS itself instead of
> reading 1GB
> > into the memory a
Hi Robert,
On Fri, Aug 9, 2019 at 6:40 PM Robert Haas wrote:
> On Thu, Aug 8, 2019 at 8:37 PM Jeevan Ladhe
> wrote:
> > + if (!XLogRecPtrIsInvalid(previous_lsn))
> > + appendStringInfo(labelfile, "PREVIOUS WAL LOCATION: %X/%X\n",
> > +(uint32) (previo
On Thu, Aug 8, 2019 at 8:37 PM Jeevan Ladhe
wrote:
> + if (!XLogRecPtrIsInvalid(previous_lsn))
> + appendStringInfo(labelfile, "PREVIOUS WAL LOCATION: %X/%X\n",
> +(uint32) (previous_lsn >> 32), (uint32)
> previous_lsn);
>
> May be we should rename to s
On Wed, Aug 7, 2019 at 5:46 AM Jeevan Chalke
wrote:
> So, do you mean we should just do fread() and fwrite() for the whole file?
>
> I thought it is better if it was done by the OS itself instead of reading 1GB
> into the memory and writing the same to the file.
Well, 'cp' is just a C program. I
Hi Jeevan,
I have reviewed the backup part at code level and still looking into the
restore(combine) and functional part of it. But, here are my comments so
far:
The patches need rebase.
+ if (!XLogRecPtrIsInvalid(previous_lsn))
+
On Wed, Aug 7, 2019 at 2:47 PM Jeevan Chalke
wrote:
>
>
> On Mon, Aug 5, 2019 at 7:13 PM Robert Haas wrote:
>
>> On Fri, Aug 2, 2019 at 9:13 AM vignesh C wrote:
>> > + rc = system(copycmd);
>>
>> I don't think this patch should be calling system() in the first place.
>>
>
> So, do you mean we s
On Mon, Aug 5, 2019 at 7:13 PM Robert Haas wrote:
> On Fri, Aug 2, 2019 at 9:13 AM vignesh C wrote:
> > + rc = system(copycmd);
>
> I don't think this patch should be calling system() in the first place.
>
So, do you mean we should just do fread() and fwrite() for the whole file?
I thought it
On Tue, Aug 6, 2019 at 11:31 PM Ibrar Ahmed wrote:
>
> I have not looked at the patch in detail, but just some nits from my side.
>
> On Fri, Aug 2, 2019 at 6:13 PM vignesh C wrote:
>
>> On Thu, Aug 1, 2019 at 5:06 PM Jeevan Chalke
>> wrote:
>> >
>> > On Tue, Jul 30, 2019 at 9:39 AM Jeevan Chal
I have not looked at the patch in detail, but just some nits from my side.
On Fri, Aug 2, 2019 at 6:13 PM vignesh C wrote:
> On Thu, Aug 1, 2019 at 5:06 PM Jeevan Chalke
> wrote:
> >
> > On Tue, Jul 30, 2019 at 9:39 AM Jeevan Chalke <
> jeevan.cha...@enterprisedb.com> wrote:
> >>
> >> I am almo
Greetings,
* Robert Haas (robertmh...@gmail.com) wrote:
> On Fri, Aug 2, 2019 at 9:13 AM vignesh C wrote:
> > + rc = system(copycmd);
>
> I don't think this patch should be calling system() in the first place.
+1.
Thanks,
Stephen
signature.asc
Description: PGP signature
On Fri, Aug 2, 2019 at 9:13 AM vignesh C wrote:
> + rc = system(copycmd);
I don't think this patch should be calling system() in the first place.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Thu, Aug 1, 2019 at 5:06 PM Jeevan Chalke
wrote:
>
> On Tue, Jul 30, 2019 at 9:39 AM Jeevan Chalke
> wrote:
>>
>> I am almost done writing the patch for pg_combinebackup and will post soon.
>
>
> Attached patch which implements the pg_combinebackup utility used to combine
> full basebackup wi
On Tue, Jul 30, 2019 at 9:39 AM Jeevan Chalke <
jeevan.cha...@enterprisedb.com> wrote:
>
>
>
> I am almost done writing the patch for pg_combinebackup and will post soon.
>
Attached patch which implements the pg_combinebackup utility used to combine
full basebackup with one or more incremental ba
On Wed, Jul 31, 2019 at 1:59 PM vignesh C wrote:
> I feel Robert's suggestion is good.
> We can probably keep one meta file for each backup with some basic information
> of all the files being backed up, this metadata file will be useful in the
> below case:
> Table dropped before incremental back
On Tue, Jul 30, 2019 at 1:58 AM Robert Haas wrote:
>
> On Wed, Jul 10, 2019 at 2:17 PM Anastasia Lubennikova
> wrote:
> > In attachments, you can find a prototype of incremental pg_basebackup,
> > which consists of 2 features:
> >
> > 1) To perform incremental backup one should call pg_basebackup
On Tue, Jul 30, 2019 at 1:28 AM Robert Haas wrote:
> On Wed, Jul 10, 2019 at 2:17 PM Anastasia Lubennikova
> wrote:
> > In attachments, you can find a prototype of incremental pg_basebackup,
> > which consists of 2 features:
> >
> > 1) To perform incremental backup one should call pg_basebackup
On Tue, Jul 30, 2019 at 1:58 AM Robert Haas wrote:
>
> I haven't had a chance to look at Jeevan's patch at all, or yours in
> any detail, as yet, so these are just some very preliminary comments.
> It will be good, however, if we can agree on who is going to do what
> part of this as we try to dr
Hi Jeevan
I reviewed first two patches -
0001-Add-support-for-command-line-option-to-pass-LSN.patch and
0002-Add-TAP-test-to-test-LSN-option.patch
from the set of incremental backup patches, and the changes look good to me.
I had some concerns around the way we are working around with the
On Wed, Jul 10, 2019 at 2:17 PM Anastasia Lubennikova
wrote:
> In attachments, you can find a prototype of incremental pg_basebackup,
> which consists of 2 features:
>
> 1) To perform incremental backup one should call pg_basebackup with a
> new argument:
>
> pg_basebackup -D 'basedir' --prev-back
On Fri, Jul 26, 2019 at 11:21 AM Jeevan Ladhe
wrote:
> Hi Vignesh,
>
> Please find my comments inline below:
>
> 1) If relation file has changed due to truncate or vacuum.
>> During incremental backup the new files will be copied.
>> There are chances that both the old file and new file
Hi Vignesh,
Please find my comments inline below:
1) If relation file has changed due to truncate or vacuum.
> During incremental backup the new files will be copied.
> There are chances that both the old file and new file
> will be present. I'm not sure if cleaning up of the
> o
Thanks Jeevan.
1) If relation file has changed due to truncate or vacuum.
During incremental backup the new files will be copied.
There are chances that both the old file and new file
will be present. I'm not sure if cleaning up of the
old file is handled.
2) Just a small thought
Hi Vignesh,
This backup technology is extending the pg_basebackup itself, which means
we can
still take online backups. This is internally done using pg_start_backup and
pg_stop_backup. pg_start_backup performs a checkpoint, and this checkpoint
is
used in the recovery process while starting the cl
Hi Jeevan,
The idea is very nice.
When Insert/update/delete and truncate/drop happens at various
combinations, How the incremental backup handles the copying of the
blocks?
On Wed, Jul 17, 2019 at 8:12 PM Jeevan Chalke
wrote:
>
>
>
> On Wed, Jul 17, 2019 at 7:38 PM Ibrar Ahmed wrote:
>>
>>
>>
On Wed, Jul 17, 2019 at 7:38 PM Ibrar Ahmed wrote:
>
>
> On Wed, Jul 17, 2019 at 6:43 PM Jeevan Chalke <
> jeevan.cha...@enterprisedb.com> wrote:
>
>> On Wed, Jul 17, 2019 at 2:15 PM Ibrar Ahmed
>> wrote:
>>
>>>
>>> At what stage you will apply the WAL generated in between the START/STOP
>>> bac
On Wed, Jul 17, 2019 at 6:43 PM Jeevan Chalke <
jeevan.cha...@enterprisedb.com> wrote:
> On Wed, Jul 17, 2019 at 2:15 PM Ibrar Ahmed wrote:
>
>>
>> At what stage you will apply the WAL generated in between the START/STOP
>> backup.
>>
>
> In this design, we are not touching any WAL related code.
On Wed, Jul 17, 2019 at 2:15 PM Ibrar Ahmed wrote:
>
> At what stage you will apply the WAL generated in between the START/STOP
> backup.
>
In this design, we are not touching any WAL related code. The WAL files will
get copied with each backup either full or incremental. And thus, the last
incr
On Wed, Jul 17, 2019 at 10:22 AM Jeevan Chalke <
jeevan.cha...@enterprisedb.com> wrote:
>
>
> On Thu, Jul 11, 2019 at 5:00 PM Jeevan Chalke <
> jeevan.cha...@enterprisedb.com> wrote:
>
>> Hi Anastasia,
>>
>> On Wed, Jul 10, 2019 at 11:47 PM Anastasia Lubennikova <
>> a.lubennik...@postgrespro.ru>
On Thu, Jul 11, 2019 at 5:00 PM Jeevan Chalke <
jeevan.cha...@enterprisedb.com> wrote:
> Hi Anastasia,
>
> On Wed, Jul 10, 2019 at 11:47 PM Anastasia Lubennikova <
> a.lubennik...@postgrespro.ru> wrote:
>
>> 23.04.2019 14:08, Anastasia Lubennikova wrote:
>> > I'm volunteering to write a draft patc
Hi Anastasia,
On Wed, Jul 10, 2019 at 11:47 PM Anastasia Lubennikova <
a.lubennik...@postgrespro.ru> wrote:
> 23.04.2019 14:08, Anastasia Lubennikova wrote:
> > I'm volunteering to write a draft patch or, more likely, set of
> > patches, which
> > will allow us to discuss the subject in more deta
23.04.2019 14:08, Anastasia Lubennikova wrote:
I'm volunteering to write a draft patch or, more likely, set of
patches, which
will allow us to discuss the subject in more detail.
And to do that I wish we agree on the API and data format (at least
broadly).
Looking forward to hearing your though
On Wed, Apr 24, 2019 at 12:57 PM Stephen Frost wrote:
> So, I had a thought about that when I was composing the last email and
> while I'm still unsure about it, maybe it'd be useful to mention it
> here- do we really need a list of every *file*, or could we reduce that
> down to a list of relatio
ed
database.
These are exactly the kind of issues that make me really wonder if this
is the right natural progression for pg_basebackup or any backup tool to
go in. Maybe there's some additional things we can do to make it harder
for someone to end up with a corrupted database when they restore
On Wed, Apr 24, 2019 at 9:28 AM Stephen Frost wrote:
> Looking at it from what I'm sitting, I brought up two ways that we
> could extend the protocol to "request from the server only those blocks
> where LSN >= threshold_value" with one being the modification to
> BASE_BACKUP and the other being a
Greetings,
* Robert Haas (robertmh...@gmail.com) wrote:
> On Mon, Apr 22, 2019 at 2:26 PM Stephen Frost wrote:
> > There was basically zero discussion about what things would look like at
> > a protocol level (I went back and skimmed over the thread before sending
> > my last email to specificall
ir dependencies for me, and has all
the restore options necessary built in as well.
Block level incremental backup would be amazing for my use case. I have
small updates / deletes that happen to data all over some of my largest
tables. With pgBackRest, since the diff/incremental backups are at the fi
22.04.2019 2:02, Robert Haas wrote:
I think we're getting closer to a meeting of the minds here, but I
don't think it's intrinsically necessary to rewrite the whole method
of operation of pg_basebackup to implement incremental backup in a
sensible way. One could instead just do a straightforward
On Mon, Apr 22, 2019 at 2:26 PM Stephen Frost wrote:
> There was basically zero discussion about what things would look like at
> a protocol level (I went back and skimmed over the thread before sending
> my last email to specifically see if I was going to get this response
> back..). I get the i
Greetings,
* Andres Freund (and...@anarazel.de) wrote:
> On 2019-04-22 14:26:40 -0400, Stephen Frost wrote:
> > I'm disappointed that the concerns about the trouble that end users are
> > likely to have with this didn't garner more discussion.
>
> My impression is that endusers are having a lot m
Hi,
On 2019-04-22 14:26:40 -0400, Stephen Frost wrote:
> I'm disappointed that the concerns about the trouble that end users are
> likely to have with this didn't garner more discussion.
My impression is that endusers are having a lot more trouble due to
important backup/restore features not bein
t exposes
> that information directly through some new replication command.
> There's a difference between something that's going in the wrong
> direction and something that's going in the right direction but not as
> far or as fast as you'd like. And I'm 99% sure tha
Greetings,
* Andres Freund (and...@anarazel.de) wrote:
> On 2019-04-19 20:04:41 -0400, Stephen Frost wrote:
> > I agree that we don't want another implementation and that there's a lot
> > that we want to do to improve replay performance. We've already got
> > frontend tools which work with multi
he other hand, maybe the
additional overhead caused by managing the list of files to be fetched
on the client side is negligible. It'd be interesting to see, though,
how busy the server is when running an incremental backup managed by
an external tool like BART or pgbackrest on a cluster wit
Hi,
On 2019-04-19 20:04:41 -0400, Stephen Frost wrote:
> I agree that we don't want another implementation and that there's a lot
> that we want to do to improve replay performance. We've already got
> frontend tools which work with multiple execution threads, so I'm not
> sure I get the "not eas
C and porting it to
Windows (and probably the other platforms PG runs on today), which isn't
very far off now. I'd like to make sure that whatever core ends up with
as an incremental backup solution also matches very closely what we do
with pgBackRest too, but everything that's bee
On 22.04.2019 2:02, Robert Haas wrote:
On Sat, Apr 20, 2019 at 4:32 PM Stephen Frost wrote:
Having been around for a while working on backup-related things, if I
was to implement the protocol for pg_basebackup today, I'd definitely
implement "give me a list" and "give me this file" rather th
On Sat, Apr 20, 2019 at 4:32 PM Stephen Frost wrote:
> Having been around for a while working on backup-related things, if I
> was to implement the protocol for pg_basebackup today, I'd definitely
> implement "give me a list" and "give me this file" rather than the
> tar-based approach, because I'
> 21 апр. 2019 г., в 1:13, Robert Haas написал(а):
>
> On Sat, Apr 20, 2019 at 12:44 PM Andrey Borodin wrote:
>> Incremental backup of 1Tb DB made with distance of few minutes (small change
>> set) is few Gbs. All of this size is made of FSM (no LSN) and VM (hard to
>> use LSN).
>> Sure, th
1 - 100 of 152 matches
Mail list logo
