Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-23 Thread Amit Kapila
On Wed, Mar 22, 2023 at 1:39 PM shiy.f...@fujitsu.com wrote: > > On Wed, Mar 22, 2023 2:53 PM Önder Kalacı wrote: > > > > We don't really need to, if you check the first patch, we don't have DROP > > for generated case. I mostly > > wanted to make the test a little more interesting, but it also

RE: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-22 Thread shiy.f...@fujitsu.com
On Wed, Mar 22, 2023 2:53 PM Önder Kalacı wrote: > > We don't really need to, if you check the first patch, we don't have DROP for > generated case. I mostly > wanted to make the test a little more interesting, but it also seems to be a > little confusing. > > Now attaching v2 where we do not

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-21 Thread Önder Kalacı
Hi Shi Yu, > > Is there any reasons why we drop column here? Dropped column case has been > tested on table dropped_cols. The generated column problem can be detected > without dropping columns on my machine. > We don't really need to, if you check the first patch, we don't have DROP for generat

RE: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-21 Thread shiy.f...@fujitsu.com
On Tuesday, March 21, 2023 8:03 PM Önder Kalacı wrote: > > Attached patches again. > Thanks for updating the patch. @@ -408,15 +412,18 @@ $node_subscriber->wait_for_subscription_sync; $node_publisher->safe_psql( 'postgres', qq( ALTER TABLE dropped_cols DROP COLUMN b_dr

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-21 Thread Önder Kalacı
Hi Shi Yu, > > > 1. > $node_publisher->safe_psql( > 'postgres', qq( > ALTER TABLE dropped_cols DROP COLUMN b_drop; > + ALTER TABLE generated_cols DROP COLUMN b_gen; > )); > $node_subscriber->safe_psql( > 'postgres', qq( > ALTER TABLE

RE: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-21 Thread shiy.f...@fujitsu.com
On Tue, Mar 21, 2023 4:51 PM Önder Kalacı wrote: > > Hi Amit, Shi Yu > > Now attaching the similar patches for generated columns. > Thanks for your patches. Here are some comments. 1. $node_publisher->safe_psql( 'postgres', qq( ALTER TABLE dropped_cols DROP COLUMN b_d

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-21 Thread Önder Kalacı
Hi Amit, Shi Yu Now attaching the similar patches for generated columns. Thanks, Onder KALACI Amit Kapila , 21 Mar 2023 Sal, 09:07 tarihinde şunu yazdı: > On Mon, Mar 20, 2023 at 6:28 PM Amit Kapila > wrote: > > > > On Mon, Mar 20, 2023 at 2:58 PM Önder Kalacı > wrote: > > > > > > > > > I a

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-20 Thread Amit Kapila
On Mon, Mar 20, 2023 at 6:28 PM Amit Kapila wrote: > > On Mon, Mar 20, 2023 at 2:58 PM Önder Kalacı wrote: > > > > > > I applied all patches for all the versions, and re-run the subscription > > tests, > > all looks good to me. > > > > LGTM. I'll push this tomorrow unless there are more comments

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-20 Thread Amit Kapila
On Mon, Mar 20, 2023 at 2:58 PM Önder Kalacı wrote: > > > I applied all patches for all the versions, and re-run the subscription tests, > all looks good to me. > LGTM. I'll push this tomorrow unless there are more comments. -- With Regards, Amit Kapila.

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-20 Thread Önder Kalacı
Hi Shi Yu, all Thanks for updating the patches. It seems you forgot to attach the patches > of > dropped columns for HEAD and pg15, I think they are the same as v2. > > Yes, it seems I forgot. And, yes they were the same as v2. > On HEAD, we can re-use clusters in other test cases, which can sav

RE: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-20 Thread shiy.f...@fujitsu.com
On Fri, Mar 17, 2023 11:29 PM Önder Kalacı wrote: > > Thanks for sharing. Fixed > > > This time I was able to run all the tests with all the patches applied. > > Again, the generated column fix also has some minor differences > per version. So, overall we have 6 patches with very minor > diff

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-17 Thread Önder Kalacı
Hi Shi Yu, Thanks for the review, really appreciate it! > I couldn't apply > v2-0001-Ignore-dropped-columns-HEAD-REL_15-REL_14-REL_13.patch > cleanly in v13 and v14. It looks the patch needs some changes in these > versions. > > > ``` > Checking patch src/backend/executor/execReplication.c... >

RE: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-17 Thread shiy.f...@fujitsu.com
On Friday, March 17, 2023 3:38 PM Önder Kalacı wrote: > > Hi Amit, all > > You can first submit the fix for dropped columns with patches till > v10. Once that is committed, then you can send the patches for > generated columns. > > Alright, attaching 2 patches for dropped columns, the names of

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-17 Thread Önder Kalacı
Hi Amit, all > You can first submit the fix for dropped columns with patches till > v10. Once that is committed, then you can send the patches for > generated columns. > > Alright, attaching 2 patches for dropped columns, the names of the files shows which versions the patch can be applied to: v2

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-16 Thread Amit Kapila
On Fri, Mar 17, 2023 at 5:41 AM Tom Lane wrote: > > Amit Kapila writes: > > On Thu, Mar 16, 2023 at 9:33 PM Önder Kalacı wrote: > >>> and backpatch the fix for dropped column to PG10. > > > You can first submit the fix for dropped columns with patches till > > v10. Once that is committed, then y

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-16 Thread Tom Lane
Amit Kapila writes: > On Thu, Mar 16, 2023 at 9:33 PM Önder Kalacı wrote: >>> and backpatch the fix for dropped column to PG10. > You can first submit the fix for dropped columns with patches till > v10. Once that is committed, then you can send the patches for > generated columns. Don't worry

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-16 Thread Amit Kapila
On Thu, Mar 16, 2023 at 9:33 PM Önder Kalacı wrote: > > Hi Amit, Shi Yu, > > > Generated column is introduced in PG12, and I reproduced generated column > > problem > in PG12~PG15. > > For dropped column problem, I reproduced it in PG10~PG15. (Logical > > replication > was introduced in PG10) >

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-16 Thread Önder Kalacı
Hi Amit, Shi Yu, > Generated column is introduced in PG12, and I reproduced generated column problem in PG12~PG15. > For dropped column problem, I reproduced it in PG10~PG15. (Logical replication was introduced in PG10) So, I'm planning to split the changes into two commits. The first one fixes f

RE: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-16 Thread shiy.f...@fujitsu.com
On Thu, Mar 16, 2023 5:23 PM Amit Kapila wrote: > > On Mon, Mar 13, 2023 at 6:26 PM Önder Kalacı > wrote: > > > > Attaching v2 > > > > Can we change the comment to: "Ignore dropped and generated columns as > the publisher doesn't send those."? After your change, att = > TupleDescAttr(slot1->tts

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-16 Thread Amit Kapila
On Mon, Mar 13, 2023 at 6:26 PM Önder Kalacı wrote: > > Attaching v2 > Can we change the comment to: "Ignore dropped and generated columns as the publisher doesn't send those."? After your change, att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum); is done twice in the same function. In te

Re: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-13 Thread Önder Kalacı
Hi Shi Yu, > 1. > @@ -243,6 +243,17 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot > *slot2, > Form_pg_attribute att; > TypeCacheEntry *typentry; > > + > + Form_pg_attribute attr = > TupleDescAttr(slot1->tts_tupleDescriptor, attrnum); > + > > I

RE: Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-13 Thread shiy.f...@fujitsu.com
On Sun, Mar 12, 2023 4:00 AM Önder Kalacı wrote: > > Attaching a patch that could possibly solve the problem. > Thanks for your patch. I tried it and it worked well. Here are some minor comments. 1. @@ -243,6 +243,17 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,

Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL

2023-03-11 Thread Önder Kalacı
Hi all, (cc'ed Amit as he has the context) While working on [1], I realized that on HEAD there is a problem with the $subject. Here is the relevant discussion on the thread [2]. Quoting my own notes on that thread below; I realized that the dropped columns also get into the tuples_equal() > fun