Re: Copy function for logical replication slots

2019-04-09 Thread Alvaro Herrera
On 2019-Apr-05, Petr Jelinek wrote: > On 05/04/2019 23:16, Alvaro Herrera wrote: > > The mechanism of creating a new slot from the source, then later > > advancing the LSN of the new slot using the updated values from the > > source slot, seems quite clever. I reworded the comment that explained

Re: Re: Copy function for logical replication slots

2019-04-05 Thread Masahiko Sawada
On Sat, Apr 6, 2019 at 6:16 AM Alvaro Herrera wrote: > > Pushed this. Thank you! > > The mechanism of creating a new slot from the source, then later > advancing the LSN of the new slot using the updated values from the > source slot, seems quite clever. I reworded the comment that explained >

Re: Copy function for logical replication slots

2019-04-05 Thread Petr Jelinek
On 05/04/2019 23:16, Alvaro Herrera wrote: > Pushed this. > Thanks! > The mechanism of creating a new slot from the source, then later > advancing the LSN of the new slot using the updated values from the > source slot, seems quite clever. I reworded the comment that explained > how it is suppos

Re: Re: Copy function for logical replication slots

2019-04-05 Thread Alvaro Herrera
Pushed this. The mechanism of creating a new slot from the source, then later advancing the LSN of the new slot using the updated values from the source slot, seems quite clever. I reworded the comment that explained how it is supposed to work; please double-check to ensure I got it right. I ren

Re: Re: Copy function for logical replication slots

2019-03-25 Thread Masahiko Sawada
On Mon, Mar 25, 2019 at 5:26 PM David Steele wrote: > > On 3/11/19 5:16 AM, Masahiko Sawada wrote: > > On Mon, Feb 25, 2019 at 4:50 PM Masahiko Sawada > > wrote: > >> > >> On Wed, Feb 20, 2019 at 1:00 PM Masahiko Sawada > >> wrote: > >>> > >>> BTW, XLogRecPtrIsInvalid(copy_restart_lsn) || cop

Re: Re: Copy function for logical replication slots

2019-03-25 Thread David Steele
On 3/11/19 5:16 AM, Masahiko Sawada wrote: On Mon, Feb 25, 2019 at 4:50 PM Masahiko Sawada wrote: On Wed, Feb 20, 2019 at 1:00 PM Masahiko Sawada wrote: BTW, XLogRecPtrIsInvalid(copy_restart_lsn) || copy_restart_lsn < src_restart_lsn is redundant, the former should be removed. So attach

Re: Copy function for logical replication slots

2019-03-10 Thread Masahiko Sawada
On Mon, Feb 25, 2019 at 4:50 PM Masahiko Sawada wrote: > > On Wed, Feb 20, 2019 at 1:00 PM Masahiko Sawada wrote: > > > > BTW, XLogRecPtrIsInvalid(copy_restart_lsn) || copy_restart_lsn < > > src_restart_lsn is redundant, the former should be removed. > > > > So attached the updated version patch

Re: Copy function for logical replication slots

2019-02-24 Thread Masahiko Sawada
On Wed, Feb 20, 2019 at 1:00 PM Masahiko Sawada wrote: > > BTW, XLogRecPtrIsInvalid(copy_restart_lsn) || copy_restart_lsn < > src_restart_lsn is redundant, the former should be removed. > So attached the updated version patch. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORAT

Re: Copy function for logical replication slots

2019-02-19 Thread Masahiko Sawada
On Wed, Feb 20, 2019 at 12:26 PM Michael Paquier wrote: > > On Tue, Feb 19, 2019 at 05:09:33PM +0900, Masahiko Sawada wrote: > > On Tue, Feb 19, 2019 at 1:28 AM Andres Freund wrote: > >> Well, I'd not thought we'd do it without acquiring the other slot. But > >> that still seems to be easy enough

Re: Copy function for logical replication slots

2019-02-19 Thread Michael Paquier
On Tue, Feb 19, 2019 at 05:09:33PM +0900, Masahiko Sawada wrote: > On Tue, Feb 19, 2019 at 1:28 AM Andres Freund wrote: >> Well, I'd not thought we'd do it without acquiring the other slot. But >> that still seems to be easy enough to address, we just need to recheck >> whether the slot still exis

Re: Copy function for logical replication slots

2019-02-19 Thread Masahiko Sawada
On Tue, Feb 19, 2019 at 1:28 AM Andres Freund wrote: > > Hi, > > On 2019-02-18 16:57:07 +0900, Masahiko Sawada wrote: > > > Stupid question, but couldn't we optimize this to something like: > > > > > > /* > > > * First copy current data of the slot. Then install those in the > > > *

Re: Copy function for logical replication slots

2019-02-18 Thread Andres Freund
Hi, On 2019-02-18 16:57:07 +0900, Masahiko Sawada wrote: > > Stupid question, but couldn't we optimize this to something like: > > > > /* > > * First copy current data of the slot. Then install those in the > > * new slot. The src slot could have progressed while installing, > >

Re: Copy function for logical replication slots

2019-02-17 Thread Masahiko Sawada
On Sat, Feb 16, 2019 at 12:34 PM Andres Freund wrote: > > Hi, > Thank you for your comment. > On 2019-01-15 10:56:04 +0900, Masahiko Sawada wrote: > > > + pg_copy_physical_replication_slot > > + > > + > > pg_copy_physical_replication_slot(src_slot_name > > name, dst_slot

Re: Copy function for logical replication slots

2019-02-15 Thread Andres Freund
Hi, On 2019-01-15 10:56:04 +0900, Masahiko Sawada wrote: > + pg_copy_physical_replication_slot > + > + > pg_copy_physical_replication_slot(src_slot_name > name, dst_slot_name , > temporary > bool) > + > + > +(slot_name name, > lsn pg_lsn) > +

Re: Copy function for logical replication slots

2019-02-15 Thread Petr Jelinek
Hi, On 15/01/2019 02:56, Masahiko Sawada wrote: > On Tue, Nov 27, 2018 at 3:46 AM Petr Jelinek > wrote: >>> + >>> + /* >>> +* The requested wal lsn is no longer available. We don't >>> want to retry >>> +* it, so raise an error. >>> +

Re: Copy function for logical replication slots

2019-01-14 Thread Masahiko Sawada
On Tue, Nov 27, 2018 at 3:46 AM Petr Jelinek wrote: > > Hi, > > On 26/11/2018 01:29, Masahiko Sawada wrote: > > On Sun, Nov 25, 2018 at 12:27 AM Petr Jelinek > > wrote: > >> > >> The more serious thing is: > >> > >>> + if (MyReplicationSlot) > >>> + ReplicationSlotRelease(); > >>>

Re: Copy function for logical replication slots

2018-11-26 Thread Petr Jelinek
Hi, On 26/11/2018 01:29, Masahiko Sawada wrote: > On Sun, Nov 25, 2018 at 12:27 AM Petr Jelinek > wrote: >> >> The more serious thing is: >> >>> + if (MyReplicationSlot) >>> + ReplicationSlotRelease(); >>> + >>> + /* Release the saved slot if exist while preventing double rele

Re: Copy function for logical replication slots

2018-11-25 Thread Masahiko Sawada
On Sun, Nov 25, 2018 at 12:27 AM Petr Jelinek wrote: > > Hi, > > On 31/08/2018 07:03, Masahiko Sawada wrote: > > > > Attached a new version patch incorporated the all comments I got. > > > > This looks pretty reasonable. Thank you for looking at this patch. > > I am personally not big fan of the

Re: Copy function for logical replication slots

2018-11-24 Thread Petr Jelinek
Hi, On 31/08/2018 07:03, Masahiko Sawada wrote: > > Attached a new version patch incorporated the all comments I got. > This looks pretty reasonable. I am personally not big fan of the C wrappers for overloaded functions, but that's what we need to do for opr_sanity to pass so I guess we'll ha

Re: Copy function for logical replication slots

2018-08-30 Thread Masahiko Sawada
On Wed, Aug 29, 2018 at 9:39 AM, Masahiko Sawada wrote: > On Tue, Aug 28, 2018 at 10:34 PM, Michael Paquier wrote: >> On Tue, Aug 28, 2018 at 04:14:04PM +0900, Masahiko Sawada wrote: >>> I think the copying from a slot that already reserved WAL would be >>> helpful for backup cases (maybe you sug

Re: Copy function for logical replication slots

2018-08-28 Thread Masahiko Sawada
On Tue, Aug 28, 2018 at 10:34 PM, Michael Paquier wrote: > On Tue, Aug 28, 2018 at 04:14:04PM +0900, Masahiko Sawada wrote: >> I think the copying from a slot that already reserved WAL would be >> helpful for backup cases (maybe you suggested?). Also, either way we >> need to make a safe logic of

Re: Copy function for logical replication slots

2018-08-28 Thread Michael Paquier
On Tue, Aug 28, 2018 at 04:14:04PM +0900, Masahiko Sawada wrote: > I think the copying from a slot that already reserved WAL would be > helpful for backup cases (maybe you suggested?). Also, either way we > need to make a safe logic of acquring and releasing the source slot > for logical slots case

Re: Copy function for logical replication slots

2018-08-28 Thread Masahiko Sawada
On Mon, Aug 20, 2018 at 2:53 PM, Michael Paquier wrote: > On Tue, Aug 14, 2018 at 01:38:23PM +0900, Masahiko Sawada wrote: >> On Thu, Jul 12, 2018 at 9:28 PM, Masahiko Sawada >> wrote: >>> Attached new version of patch incorporated the all comments I got from >>> Michael-san. >>> >>> To prevent

Re: Copy function for logical replication slots

2018-08-19 Thread Michael Paquier
On Tue, Aug 14, 2018 at 01:38:23PM +0900, Masahiko Sawada wrote: > On Thu, Jul 12, 2018 at 9:28 PM, Masahiko Sawada > wrote: >> Attached new version of patch incorporated the all comments I got from >> Michael-san. >> >> To prevent the WAL segment file of restart_lsn of the origin slot from >> re

Re: Copy function for logical replication slots

2018-08-13 Thread Masahiko Sawada
On Thu, Jul 12, 2018 at 9:28 PM, Masahiko Sawada wrote: > On Mon, Jul 9, 2018 at 10:34 AM, Michael Paquier wrote: >> On Mon, Jul 09, 2018 at 10:06:00AM +0900, Masahiko Sawada wrote: >>> I think that this patch might be splitted but I will be able to send >>> an updated patch in the next week. As

Re: Copy function for logical replication slots

2018-07-12 Thread Masahiko Sawada
On Mon, Jul 9, 2018 at 10:34 AM, Michael Paquier wrote: > On Mon, Jul 09, 2018 at 10:06:00AM +0900, Masahiko Sawada wrote: >> I think that this patch might be splitted but I will be able to send >> an updated patch in the next week. As you suggestion this patch needs >> more careful thoughts. I'll

Re: Copy function for logical replication slots

2018-07-08 Thread Michael Paquier
On Mon, Jul 09, 2018 at 10:06:00AM +0900, Masahiko Sawada wrote: > I think that this patch might be splitted but I will be able to send > an updated patch in the next week. As you suggestion this patch needs > more careful thoughts. I'll move this patch to the next commit fest if > I will not be ab

Re: Copy function for logical replication slots

2018-07-08 Thread Masahiko Sawada
On Fri, Jul 6, 2018 at 9:21 AM, Michael Paquier wrote: > On Thu, Jul 05, 2018 at 05:24:48PM +0900, Masahiko Sawada wrote: >> Yes, you're right. To guarantee that restart LSN of copied slot is >> available, it seems to me that it's better to copy new slot while >> holding the origin slot as you men

Re: Copy function for logical replication slots

2018-07-05 Thread Michael Paquier
On Thu, Jul 05, 2018 at 05:24:48PM +0900, Masahiko Sawada wrote: > Yes, you're right. To guarantee that restart LSN of copied slot is > available, it seems to me that it's better to copy new slot while > holding the origin slot as you mentioned before. Since the replication > slot creation code ass

Re: Copy function for logical replication slots

2018-07-05 Thread Masahiko Sawada
On Thu, Jul 5, 2018 at 4:47 PM, Michael Paquier wrote: > On Tue, Jul 03, 2018 at 05:27:07PM +0900, Masahiko Sawada wrote: >> On Tue, Jul 3, 2018 at 1:01 PM, Michael Paquier wrote: >>> One property which seems important to me is to make sure that the target >>> slot has the same data as the origin

Re: Copy function for logical replication slots

2018-07-05 Thread Michael Paquier
On Tue, Jul 03, 2018 at 05:27:07PM +0900, Masahiko Sawada wrote: > On Tue, Jul 3, 2018 at 1:01 PM, Michael Paquier wrote: >> One property which seems important to me is to make sure that the target >> slot has the same data as the origin slot once the caller knows that the >> copy has completed, a

Re: Copy function for logical replication slots

2018-07-03 Thread Craig Ringer
On 28 June 2018 at 10:51, Masahiko Sawada wrote: > Hi, > > I'd like to propose a copy function for logical replication slots. > Currently when we create a new logical replication slot it starts to > read WAL from an LSN of the current insert. This function copies a > exist

Re: Copy function for logical replication slots

2018-07-03 Thread Masahiko Sawada
On Tue, Jul 3, 2018 at 1:01 PM, Michael Paquier wrote: > On Mon, Jul 02, 2018 at 04:31:32PM +0900, Masahiko Sawada wrote: >> Attached an updated patch including copy function support for logical >> slots as well as physical slots. Please review it. > > I had a look at this patch. Thank you for th

Re: Copy function for logical replication slots

2018-07-02 Thread Michael Paquier
On Mon, Jul 02, 2018 at 04:31:32PM +0900, Masahiko Sawada wrote: > Attached an updated patch including copy function support for logical > slots as well as physical slots. Please review it. I had a look at this patch. As the output plugin can be changed for logical slots, having two functions is

Re: Copy function for logical replication slots

2018-07-02 Thread Masahiko Sawada
On Thu, Jun 28, 2018 at 7:10 PM, Masahiko Sawada wrote: > On Thu, Jun 28, 2018 at 5:37 PM, Peter Eisentraut > wrote: >> On 6/28/18 08:47, Michael Paquier wrote: > There could be some cases where > copying a physical slot also makes sense. I've thought that but I didn't find concrete

Re: Copy function for logical replication slots

2018-06-28 Thread Masahiko Sawada
On Thu, Jun 28, 2018 at 5:37 PM, Peter Eisentraut wrote: > On 6/28/18 08:47, Michael Paquier wrote: There could be some cases where copying a physical slot also makes sense. >>> I've thought that but I didn't find concrete use case. That's why I >>> started with only logical slot. >> Let

Re: Copy function for logical replication slots

2018-06-28 Thread Peter Eisentraut
On 6/28/18 08:47, Michael Paquier wrote: >>> There could be some cases where >>> copying a physical slot also makes sense. >> I've thought that but I didn't find concrete use case. That's why I >> started with only logical slot. > Let's imagine the case of a single base backup which is associated t

Re: Copy function for logical replication slots

2018-06-27 Thread Michael Paquier
On Thu, Jun 28, 2018 at 03:34:00PM +0900, Masahiko Sawada wrote: > On Thu, Jun 28, 2018 at 12:29 PM, Michael Paquier wrote: >> Hm. Shouldn't the original slot copied be owned by the process doing >> the copy with ReplicationSlotAcquire? > > Right, it should do and release it before creating new o

Re: Copy function for logical replication slots

2018-06-27 Thread Masahiko Sawada
On Thu, Jun 28, 2018 at 12:29 PM, Michael Paquier wrote: > On Thu, Jun 28, 2018 at 11:51:20AM +0900, Masahiko Sawada wrote: >> A use case I imagined is for investigations for example. I mean that >> when replication collision occurs on subscriber there is no way to see >> what replicated data is c

Re: Copy function for logical replication slots

2018-06-27 Thread Michael Paquier
On Thu, Jun 28, 2018 at 11:51:20AM +0900, Masahiko Sawada wrote: > A use case I imagined is for investigations for example. I mean that > when replication collision occurs on subscriber there is no way to see > what replicated data is conflicting (perhaps error log helps it but is > not detailed) a

Copy function for logical replication slots

2018-06-27 Thread Masahiko Sawada
Hi, I'd like to propose a copy function for logical replication slots. Currently when we create a new logical replication slot it starts to read WAL from an LSN of the current insert. This function copies a existing logical replication slot while changing output plugin and persistence. Th