Re: Proposal: Conflict log history table for Logical Replication

2025-09-20 Thread Dilip Kumar
On Thu, Sep 11, 2025 at 8:43 AM Amit Kapila wrote: > > On Thu, Sep 11, 2025 at 12:53 AM Bharath Rupireddy > wrote: > > > > On Tue, Aug 5, 2025 at 5:24 AM Dilip Kumar wrote: > > > > > > Currently we log conflicts to the server's log file and updates, this > > > approach has limitations, 1) Diffic

Re: Proposal: Conflict log history table for Logical Replication

2025-09-20 Thread Amit Kapila
On Thu, Sep 18, 2025 at 11:46 PM Masahiko Sawada wrote: > > On Thu, Sep 18, 2025 at 1:33 AM Amit Kapila wrote: > > > > If we compare conflict_history_table with the slot that gets created > > with subscription, one can say the same thing about slots. Users can > > drop the slots and whole replica

Re: Proposal: Conflict log history table for Logical Replication

2025-09-19 Thread Dilip Kumar
On Wed, Sep 10, 2025 at 4:32 PM Alastair Turner wrote: > >> Here we will have to create a built-in type of type table which is I >> think typcategory => 'C' and if we create this type it should be >> supplied with the "typrelid" that means there should be a backing >> catalog table. At least thats

Re: Proposal: Conflict log history table for Logical Replication

2025-09-18 Thread Amit Kapila
On Sun, Sep 14, 2025 at 12:23 PM Dilip Kumar wrote: > > On Sat, Sep 13, 2025 at 6:16 AM Bharath Rupireddy > wrote: > > Thanks for the feedback Bharath > > > On Fri, Sep 12, 2025 at 3:13 AM Dilip Kumar wrote: > > > > > > I was looking into another thread where we provide an error table for > > >

Re: Proposal: Conflict log history table for Logical Replication

2025-09-18 Thread Masahiko Sawada
On Thu, Sep 18, 2025 at 1:33 AM Amit Kapila wrote: > > On Sun, Sep 14, 2025 at 12:23 PM Dilip Kumar wrote: > > > > On Sat, Sep 13, 2025 at 6:16 AM Bharath Rupireddy > > wrote: > > > > Thanks for the feedback Bharath > > > > > On Fri, Sep 12, 2025 at 3:13 AM Dilip Kumar wrote: > > > > > > > > I

Re: Proposal: Conflict log history table for Logical Replication

2025-09-18 Thread Dilip Kumar
On Thu, Sep 18, 2025 at 2:03 PM Amit Kapila wrote: > > On Sun, Sep 14, 2025 at 12:23 PM Dilip Kumar wrote: > > > > On Sat, Sep 13, 2025 at 6:16 AM Bharath Rupireddy > > wrote: > > > > Thanks for the feedback Bharath > > > > > On Fri, Sep 12, 2025 at 3:13 AM Dilip Kumar wrote: > > > > > > > > I

Re: Proposal: Conflict log history table for Logical Replication

2025-09-17 Thread Dilip Kumar
On Sat, Sep 13, 2025 at 6:16 AM Bharath Rupireddy wrote: Thanks for the feedback Bharath > On Fri, Sep 12, 2025 at 3:13 AM Dilip Kumar wrote: > > > > I was looking into another thread where we provide an error table for > > COPY [1], it requires the user to pre-create the error table. And > > i

Re: Proposal: Conflict log history table for Logical Replication

2025-09-12 Thread Bharath Rupireddy
Hi, On Fri, Sep 12, 2025 at 3:13 AM Dilip Kumar wrote: > > I was looking into another thread where we provide an error table for > COPY [1], it requires the user to pre-create the error table. And > inside the COPY command we will validate the table, validation in that > context is a one-time pro

Re: Proposal: Conflict log history table for Logical Replication

2025-09-12 Thread Bharath Rupireddy
Hi, On Wed, Sep 10, 2025 at 8:13 PM Amit Kapila wrote: > > > How about streaming the conflicts in fixed format to a separate log > > file other than regular postgres server log file? > > I would prefer this info to be stored in tables as it would be easy to > query them. If we use separate LOGs t

Re: Proposal: Conflict log history table for Logical Replication

2025-09-11 Thread Amit Kapila
On Mon, Sep 8, 2025 at 12:01 PM Dilip Kumar wrote: > > On Sun, Sep 7, 2025 at 1:42 PM Alastair Turner wrote: > > > > Hi Dilip > > > > Thanks for working on this, I think it will make conflict detection a lot > > more useful. > > Thanks for the suggestions, please find my reply inline. > > > On S

Re: Proposal: Conflict log history table for Logical Replication

2025-09-10 Thread Amit Kapila
On Thu, Sep 11, 2025 at 12:53 AM Bharath Rupireddy wrote: > > On Tue, Aug 5, 2025 at 5:24 AM Dilip Kumar wrote: > > > > Currently we log conflicts to the server's log file and updates, this > > approach has limitations, 1) Difficult to query and analyze, parsing > > plain text log files for confl

Re: Proposal: Conflict log history table for Logical Replication

2025-09-10 Thread Bharath Rupireddy
Hi, On Tue, Aug 5, 2025 at 5:24 AM Dilip Kumar wrote: > > Currently we log conflicts to the server's log file and updates, this > approach has limitations, 1) Difficult to query and analyze, parsing > plain text log files for conflict details is inefficient. 2) Lack of > structured data, key conf

Re: Proposal: Conflict log history table for Logical Replication

2025-09-10 Thread Alastair Turner
On Wed, 10 Sept 2025 at 11:15, Dilip Kumar wrote: > On Wed, Sep 10, 2025 at 3:25 PM Amit Kapila > wrote: > > > ... > > > > How about having this as a built-in type? > > Here we will have to create a built-in type of type table which is I > think typcategory => 'C' and if we create this type it

Re: Proposal: Conflict log history table for Logical Replication

2025-09-10 Thread Dilip Kumar
On Wed, Sep 10, 2025 at 3:25 PM Amit Kapila wrote: > > On Mon, Sep 8, 2025 at 12:01 PM Dilip Kumar wrote: > > > > On Sun, Sep 7, 2025 at 1:42 PM Alastair Turner wrote: > > > > > > Hi Dilip > > > > > > Thanks for working on this, I think it will make conflict detection a lot > > > more useful. >

Re: Proposal: Conflict log history table for Logical Replication

2025-09-07 Thread Dilip Kumar
On Sun, Sep 7, 2025 at 1:42 PM Alastair Turner wrote: > > Hi Dilip > > Thanks for working on this, I think it will make conflict detection a lot > more useful. Thanks for the suggestions, please find my reply inline. > On Sat, 6 Sept 2025, 10:38 Dilip Kumar, wrote: >> >> While working on the p

Re: Proposal: Conflict log history table for Logical Replication

2025-09-07 Thread Alastair Turner
Hi Dilip Thanks for working on this, I think it will make conflict detection a lot more useful. On Sat, 6 Sept 2025, 10:38 Dilip Kumar, wrote: > While working on the patch, I see there are some open questions > > 1. We decided to pass the conflict history table name during > subscription creati

Re: Proposal: Conflict log history table for Logical Replication

2025-09-06 Thread Dilip Kumar
On Thu, Aug 21, 2025 at 9:17 AM Dilip Kumar wrote: > > On Wed, Aug 20, 2025 at 5:46 PM Amit Kapila wrote: > > > > On Wed, Aug 20, 2025 at 11:47 AM Dilip Kumar wrote: > > > > > > On Mon, Aug 18, 2025 at 12:25 PM Amit Kapila > > > wrote: > > > > > > > > > > > One idea to keep things simple for t

Re: Proposal: Conflict log history table for Logical Replication

2025-08-20 Thread Dilip Kumar
On Wed, Aug 20, 2025 at 5:46 PM Amit Kapila wrote: > > On Wed, Aug 20, 2025 at 11:47 AM Dilip Kumar wrote: > > > > On Mon, Aug 18, 2025 at 12:25 PM Amit Kapila > > wrote: > > > > > > > > One idea to keep things simple for the first version is that we allow > > > users to specify the table_name

Re: Proposal: Conflict log history table for Logical Replication

2025-08-20 Thread Amit Kapila
On Wed, Aug 20, 2025 at 11:47 AM Dilip Kumar wrote: > > On Mon, Aug 18, 2025 at 12:25 PM Amit Kapila wrote: > > > > > One idea to keep things simple for the first version is that we allow > > users to specify the table_name for storing conflicts but the table > > should be created internally and

Re: Proposal: Conflict log history table for Logical Replication

2025-08-19 Thread Dilip Kumar
On Mon, Aug 18, 2025 at 12:25 PM Amit Kapila wrote: > > On Fri, Aug 15, 2025 at 2:31 PM Dilip Kumar wrote: > > > > Yet another question is about table names, whether we keep some > > standard name like conflict_log_history_$subid or let users pass the > > name. > > > > It would be good if we can

Re: Proposal: Conflict log history table for Logical Replication

2025-08-17 Thread Amit Kapila
On Fri, Aug 15, 2025 at 2:31 PM Dilip Kumar wrote: > > Yet another question is about table names, whether we keep some > standard name like conflict_log_history_$subid or let users pass the > name. > It would be good if we can let the user specify the table_name and if she didn't specify then use

Re: Proposal: Conflict log history table for Logical Replication

2025-08-15 Thread Dilip Kumar
On Wed, Aug 13, 2025 at 3:39 PM Amit Kapila wrote: > > On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: > > > > On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: > > > > > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: > > > > > > > > So logically for PostgreSQL its an > > > > user tabl

Re: Proposal: Conflict log history table for Logical Replication

2025-08-14 Thread Amit Kapila
On Thu, Aug 14, 2025 at 4:26 PM Alastair Turner wrote: > > On Wed, 13 Aug 2025 at 11:09, Amit Kapila wrote: >> >> On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: >> > >> > On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: >> > > >> > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: >> >

Re: Proposal: Conflict log history table for Logical Replication

2025-08-14 Thread Alastair Turner
On Wed, 13 Aug 2025 at 11:09, Amit Kapila wrote: > On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: > > > > On Fri, Aug 8, 2025 at 8:58 AM shveta malik > wrote: > > > > > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar > wrote: > > > > > > > > So logically for PostgreSQL its an > > > > user tab

Re: Proposal: Conflict log history table for Logical Replication

2025-08-13 Thread Amit Kapila
On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: > > On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: > > > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: > > > > > > So logically for PostgreSQL its an > > > user table but yeah this is created and managed by the extension. > > > > > > >

Re: Proposal: Conflict log history table for Logical Replication

2025-08-08 Thread shveta malik
On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: > > On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: > > > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: > > > > > > So logically for PostgreSQL its an > > > user table but yeah this is created and managed by the extension. > > > > > > >

Re: Proposal: Conflict log history table for Logical Replication

2025-08-07 Thread Dilip Kumar
On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: > > > > So logically for PostgreSQL its an > > user table but yeah this is created and managed by the extension. > > > > Any idea if the user can alter/drop or perform any DML on it? I could >

Re: Proposal: Conflict log history table for Logical Replication

2025-08-07 Thread shveta malik
On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: > > On Thu, Aug 7, 2025 at 1:43 PM shveta malik wrote: > > > > On Thu, Aug 7, 2025 at 12:25 PM shveta malik wrote: > > Thanks Shveta for your opinion on the design. > > > > On Tue, Aug 5, 2025 at 5:54 PM Dilip Kumar wrote: > > > > > > > > > This

Re: Proposal: Conflict log history table for Logical Replication

2025-08-07 Thread Dilip Kumar
On Thu, Aug 7, 2025 at 1:43 PM shveta malik wrote: > > On Thu, Aug 7, 2025 at 12:25 PM shveta malik wrote: Thanks Shveta for your opinion on the design. > > On Tue, Aug 5, 2025 at 5:54 PM Dilip Kumar wrote: > > > > > > This proposal aims to address these limitations by introducing a > > > con

Re: Proposal: Conflict log history table for Logical Replication

2025-08-07 Thread shveta malik
On Thu, Aug 7, 2025 at 12:25 PM shveta malik wrote: > > On Tue, Aug 5, 2025 at 5:54 PM Dilip Kumar wrote: > > > > Currently we log conflicts to the server's log file and updates, this > > approach has limitations, 1) Difficult to query and analyze, parsing > > plain text log files for conflict de

Re: Proposal: Conflict log history table for Logical Replication

2025-08-06 Thread shveta malik
On Tue, Aug 5, 2025 at 5:54 PM Dilip Kumar wrote: > > Currently we log conflicts to the server's log file and updates, this > approach has limitations, 1) Difficult to query and analyze, parsing > plain text log files for conflict details is inefficient. 2) Lack of > structured data, key conflict

Proposal: Conflict log history table for Logical Replication

2025-08-05 Thread Dilip Kumar
Currently we log conflicts to the server's log file and updates, this approach has limitations, 1) Difficult to query and analyze, parsing plain text log files for conflict details is inefficient. 2) Lack of structured data, key conflict attributes (table, operation, old/new data, LSN, etc.) are no