Re: Logical insert/update/delete WAL records for custom table AMs

2022-07-20 Thread Jeff Davis
On Mon, 2022-03-21 at 17:43 -0700, Andres Freund wrote: > Currently this doesn't apply: > http://cfbot.cputube.org/patch_37_3394.log Withdrawn for now. With custom WAL resource managers this is less important to me. I still think it has value, and I'm willing to work on it if more use cases come

Re: Logical insert/update/delete WAL records for custom table AMs

2022-03-31 Thread Jeff Davis
On Thu, 2022-03-31 at 09:05 -0400, Robert Haas wrote: > 1. An AM that doesn't care about having anything happening during > recovery, but wants to be able to get logical decoding to do some > work. Maybe the intention of the AM is that data is available only > when the server is not in recovery and

Re: Logical insert/update/delete WAL records for custom table AMs

2022-03-31 Thread Robert Haas
On Wed, Mar 30, 2022 at 2:31 PM Jeff Davis wrote: > Attached a new version. The scope expanded, so this is likely to slip > v15 at this late time. For 15, I'll focus on my extensible rmgr work, > which can serve similar purposes. > > The main purpose of this patch is to be able to emit logical eve

Re: Logical insert/update/delete WAL records for custom table AMs

2022-03-30 Thread Jeff Davis
On Thu, 2022-02-24 at 20:35 +, Simon Riggs wrote: > The approach is perfectly fine, it just needs to be finished and > rebased. Attached a new version. The scope expanded, so this is likely to slip v15 at this late time. For 15, I'll focus on my extensible rmgr work, which can serve similar pu

Re: Logical insert/update/delete WAL records for custom table AMs

2022-03-21 Thread Andres Freund
Hi, On 2022-01-17 01:05:14 -0800, Jeff Davis wrote: > Great, I attached a rebased version. Currently this doesn't apply: http://cfbot.cputube.org/patch_37_3394.log - Andres

Re: Logical insert/update/delete WAL records for custom table AMs

2022-02-24 Thread Simon Riggs
On Mon, 17 Jan 2022 at 09:05, Jeff Davis wrote: > > On Wed, 2022-01-05 at 20:19 +, Simon Riggs wrote: > > I spoke with Jeff in detail about this patch in NYC Dec 21, and I now > > think it is worth pursuing. It seems much more likely that this would > > be acceptable than fully extensible rmgr

Re: Logical insert/update/delete WAL records for custom table AMs

2022-01-17 Thread Jeff Davis
On Wed, 2022-01-05 at 20:19 +, Simon Riggs wrote: > I spoke with Jeff in detail about this patch in NYC Dec 21, and I now > think it is worth pursuing. It seems much more likely that this would > be acceptable than fully extensible rmgr. Thank you. I had some conversations with others who felt

Re: Logical insert/update/delete WAL records for custom table AMs

2022-01-06 Thread Simon Riggs
On Sun, 31 Oct 2021 at 18:10, Jeff Davis wrote: > I'm looking for some review on the approach and structure before I > polish and test it. Repurposing the logical msg rmgr into a general purpose logical rmgr seems right. Structure looks obvious, which is good. Please pursue this and I will rev

Re: Logical insert/update/delete WAL records for custom table AMs

2022-01-05 Thread Simon Riggs
On Wed, 10 Nov 2021 at 03:17, Amit Kapila wrote: > > On Tue, Nov 9, 2021 at 5:12 AM Jeff Davis wrote: > > > > On Fri, 2021-11-05 at 10:00 +0530, Amit Kapila wrote: > > > I am not talking about decoding plugin but rather decoding itself, > > > basically, the work we do in reorderbuffer.c, decode.c

Re: Logical insert/update/delete WAL records for custom table AMs

2021-11-09 Thread Amit Kapila
On Tue, Nov 9, 2021 at 5:12 AM Jeff Davis wrote: > > On Fri, 2021-11-05 at 10:00 +0530, Amit Kapila wrote: > > I am not talking about decoding plugin but rather decoding itself, > > basically, the work we do in reorderbuffer.c, decode.c, etc. The two > > things I remember were tuple format and tra

Re: Logical insert/update/delete WAL records for custom table AMs

2021-11-08 Thread Jeff Davis
On Fri, 2021-11-05 at 10:00 +0530, Amit Kapila wrote: > I am not talking about decoding plugin but rather decoding itself, > basically, the work we do in reorderbuffer.c, decode.c, etc. The two > things I remember were tuple format and transaction ids as mentioned > in my previous email. If it's

Re: Logical insert/update/delete WAL records for custom table AMs

2021-11-04 Thread Amit Kapila
On Fri, Nov 5, 2021 at 4:53 AM Jeff Davis wrote: > > On Thu, 2021-11-04 at 14:33 +0530, Amit Kapila wrote: > > Can't different tableAM's have different representations > > for toast or may be some different concept like speculative > > insertions? > > The decoding plugin should just use the common

Re: Logical insert/update/delete WAL records for custom table AMs

2021-11-04 Thread Jeff Davis
On Thu, 2021-11-04 at 14:33 +0530, Amit Kapila wrote: > Can't different tableAM's have different representations > for toast or may be some different concept like speculative > insertions? The decoding plugin should just use the common interfaces to toast, and if the table AM supports toast, every

Re: Logical insert/update/delete WAL records for custom table AMs

2021-11-04 Thread Amit Kapila
On Thu, Nov 4, 2021 at 7:09 AM Jeff Davis wrote: > > On Wed, 2021-11-03 at 11:25 +0530, Amit Kapila wrote: > > You have modeled DecodeLogicalInsert based on current DecodeInsert > > and > > it generates the same change message, so not sure how exactly these > > new messages will be different from

Re: Logical insert/update/delete WAL records for custom table AMs

2021-11-03 Thread Jeff Davis
On Wed, 2021-11-03 at 11:25 +0530, Amit Kapila wrote: > You have modeled DecodeLogicalInsert based on current DecodeInsert > and > it generates the same change message, so not sure how exactly these > new messages will be different from current heap_insert/update/delete > messages? Is there a rea

Re: Logical insert/update/delete WAL records for custom table AMs

2021-11-02 Thread Amit Kapila
On Sun, Oct 31, 2021 at 11:40 PM Jeff Davis wrote: > > Attached is a WIP patch to add new WAL records to represent a logical > insert, update, or delete. These records do not do anything at REDO > time, they are only processed during logical decoding/replication. > > These are intended to be used

Logical insert/update/delete WAL records for custom table AMs

2021-10-31 Thread Jeff Davis
Attached is a WIP patch to add new WAL records to represent a logical insert, update, or delete. These records do not do anything at REDO time, they are only processed during logical decoding/replication. These are intended to be used by a custom table AM, like my columnar compression extension[0]