Re: Race condition in recovery?

2021-05-16 Thread Dilip Kumar
On Mon, May 17, 2021 at 10:09 AM Dilip Kumar wrote: > > On Mon, May 17, 2021 at 8:50 AM Kyotaro Horiguchi > wrote: > > > > Before the commit expectedTLEs is always initialized with just one > > entry for the TLI of the last checkpoint record. > > Right > >

Re: pg_dumpall misses --no-toast-compression

2021-05-16 Thread Dilip Kumar
simple but that's perhaps too > late for 14, so I am adding it to the next CF. +1 -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [Patch] ALTER SYSTEM READ ONLY

2021-05-17 Thread Dilip Kumar
On Mon, May 17, 2021 at 11:48 AM Amul Sul wrote: > > On Sat, May 15, 2021 at 3:12 PM Dilip Kumar wrote: > > > > On Thu, May 13, 2021 at 2:56 PM Dilip Kumar wrote: > > > > > > Great thanks. I will review the remaining patch soon. > > > > I hav

Re: "ERROR: deadlock detected" when replicating TRUNCATE

2021-05-17 Thread Dilip Kumar
s scenario. I am not sure this test case is exactly targeting the problematic behavior because that will depends upon the order of execution of the apply workers right? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: "ERROR: deadlock detected" when replicating TRUNCATE

2021-05-17 Thread Dilip Kumar
On Mon, May 17, 2021 at 3:43 PM Amit Kapila wrote: > > On Mon, May 17, 2021 at 3:05 PM Dilip Kumar wrote: > > > > On Mon, May 17, 2021 at 12:30 PM Peter Smith wrote: > > > > > PSA a patch adding a test for this scenario. > > > > I am not sure this t

Re: Race condition in recovery?

2021-05-17 Thread Dilip Kumar
with the recovery target timeline is the right fix. - We still have some differences of opinion about what was the original problem in the base code which was fixed by the commit (ee994272ca50f70b53074f0febaec97e28f83c4e). -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: pg_get_wal_replay_pause_state() should not return 'paused' while a promotion is ongoing.

2021-05-17 Thread Dilip Kumar
then someone can pause again and we will be in loop so better to exit as soon as promotion is requested, see attached patch. Should be applied along with your patch. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com diff --git a/src/backend/access/transam/xlog.c b/src/backend/access

Re: pg_get_wal_replay_pause_state() should not return 'paused' while a promotion is ongoing.

2021-05-18 Thread Dilip Kumar
tion is requested in that case, like when > pg_wal_replay_resume() is executed in that case. Thought? Yeah, you are right, I missed that. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: pg_get_wal_replay_pause_state() should not return 'paused' while a promotion is ongoing.

2021-05-18 Thread Dilip Kumar
ly because > pg_get_wal_replay_pause_state() needs to be executed > before the promotion finishes. Even for #2, we can not ensure that whether it will be 'paused' or 'pause requested'. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: pg_get_wal_replay_pause_state() should not return 'paused' while a promotion is ongoing.

2021-05-19 Thread Dilip Kumar
On Wed, May 19, 2021 at 11:55 AM Kyotaro Horiguchi wrote: > > At Wed, 19 May 2021 11:19:13 +0530, Dilip Kumar wrote > in > > On Wed, May 19, 2021 at 10:16 AM Fujii Masao > > wrote: > > > > > > On 2021/05/18 15:46, Michael Paquier wrote: > > > &

Re: Race condition in recovery?

2021-05-19 Thread Dilip Kumar
TimeLineHistory(recoveryTargetTLI); [1] if (tliOfPointInHistory(ControlFile->checkPoint, expectedTLEs) != ControlFile->checkPointCopy.ThisTimeLineID) { report(FATAL.. } -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-05-20 Thread Dilip Kumar
t; version bump between betas. > > This is still an open item. FWIW, I can get behind the reordering > proposed by Tom for the consistency gained with pg_type, leading to > the attached to reduce the size of FormData_pg_attribute from 116b to > 112b. This makes sense, thanks fo

Re: Race condition in recovery?

2021-05-21 Thread Dilip Kumar
On Thu, May 20, 2021 at 11:19 PM Robert Haas wrote: > > On Tue, May 18, 2021 at 1:33 AM Dilip Kumar wrote: > > Yeah, it will be a fake 1-element list. But just to be clear that > > 1-element can only be "ControlFile->checkPointCopy.ThisTimeLineID" and > &g

Re: Race condition in recovery?

2021-05-21 Thread Dilip Kumar
ll just initialize expectedTLE based on the online entry (ControlFile->checkPointCopy.ThisTimeLineID) and later we will fail to find the checkpoint record on this timeline because the checkpoint LSN is smaller than the start LSN of this timeline. Right? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Race condition in recovery?

2021-05-21 Thread Dilip Kumar
tream with old TL. Now walrecievr is under impression that it has read from the old TL. And, we know the rest of the story that we will set the expectedTLEs based on the old history file and never be able to go to the new TL. Anyways now we understand the issue and there are many ways we can r

Re: Logical Replication - behavior of TRUNCATE ... CASCADE

2021-05-21 Thread Dilip Kumar
On Thu, May 20, 2021 at 5:03 PM Amit Kapila wrote: > > On Fri, May 7, 2021 at 6:06 PM Dilip Kumar wrote: > > > > On Mon, May 3, 2021 at 6:08 PM Bharath Rupireddy > > wrote: > > > > > > Having said that, isn't it good if we can provide a sub

Re: Race condition in recovery?

2021-05-21 Thread Dilip Kumar
On Sat, May 22, 2021 at 10:15 AM Dilip Kumar wrote: > > On Sat, May 22, 2021 at 1:14 AM Robert Haas wrote: > > > > The attached test script, test.sh seems to reliably reproduce this. > > Put that file and the recalcitrant_cp script, also attached, into an > > I have

Re: Race condition in recovery?

2021-05-23 Thread Dilip Kumar
ting fixed after putting the fix we discussed[1] [1] - expectedTLEs = readTimeLineHistory(receiveTLI); + expectedTLEs = readTimeLineHistory(recoveryTargetTLI); -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com test2.sh Description: application/shellscript

Re: Race condition in recovery?

2021-05-23 Thread Dilip Kumar
On Sun, May 23, 2021 at 2:19 PM Dilip Kumar wrote: > > On Sat, May 22, 2021 at 8:33 PM Robert Haas wrote: I have created a tap test based on Robert's test.sh script. It reproduces the issue. I am new with perl so this still needs some cleanup/improvement, but at least it sho

Re: Race condition in recovery?

2021-05-23 Thread Dilip Kumar
On Mon, May 24, 2021 at 10:17 AM Kyotaro Horiguchi wrote: > > At Sun, 23 May 2021 21:37:58 +0530, Dilip Kumar wrote > in > > On Sun, May 23, 2021 at 2:19 PM Dilip Kumar wrote: > > > > > > On Sat, May 22, 2021 at 8:33 PM Robert Haas wrote: > > > >

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-05-23 Thread Dilip Kumar
y in "ATExecAddColumn()" right? But we can not ALTER TABLE ADD COLUMN to matviews right? I agree that even if we don't skip matview it will not create any issue as matview will not reach here. Am I missing something? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-05-23 Thread Dilip Kumar
t need to initialize tup_values[i] with the values[i];, other than that looks fine to me. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-05-24 Thread Dilip Kumar
On Mon, May 24, 2021 at 2:23 PM Michael Paquier wrote: > > On Mon, May 24, 2021 at 11:32:22AM +0530, Dilip Kumar wrote: > > I think you don't need to initialize tup_values[i] with the > > values[i];, other than that looks fine to me. > > You mean because heap_deform

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-05-24 Thread Dilip Kumar
On Tue, 25 May 2021 at 11:16 AM, Michael Paquier wrote: > On Mon, May 24, 2021 at 02:46:11PM +0530, Dilip Kumar wrote: > > Like this. > > if (TupleDescAttr(newTupDesc, i)->attisdropped) > > isnull[i] = true; > > else > > tup_values[i] = v

Re: Different compression methods for FPI

2021-05-24 Thread Dilip Kumar
e same as LZ4_compress_fast with acceleration=1. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Assertion failure while streaming toasted data

2021-05-24 Thread Dilip Kumar
t; transaction, but fails to adjust the buffer size for toast chunks. Maybe we > are missing a call to `ReorderBufferToastReset()` somewhere? > > From what I see, the assertion only triggers when data is inserted via COPY > (multi-insert). > > Let me know if anything else is needed to reproduce this. Thanks, I will look into this and let you know if need some help. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Tue, May 25, 2021 at 12:12 PM Dilip Kumar wrote: > > On Tue, May 25, 2021 at 12:06 PM Pavan Deolasee > wrote: > > > > Hi, > > > > While working on an output plugin that uses streaming protocol, I hit an > > assertion failure. Further investi

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
he > test_decoding plugin, but if you have some other mechanism to test that > change with an actual downstream > node receiving and applying changes, it will be useful to test with that. Okay, I will test that. Thanks. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Tue, May 25, 2021 at 1:59 PM Pavan Deolasee wrote: > > On Tue, May 25, 2021 at 1:49 PM Dilip Kumar wrote: >> >> On Tue, May 25, 2021 at 1:45 PM Pavan Deolasee >> wrote: >> > >> > I am not entirely sure if it works correctly. I'd tried somethi

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Tue, May 25, 2021 at 2:34 PM Dilip Kumar wrote: > > > When the transaction is streamed, I see: > > ``` > > + opening a streamed block for transaction > > + table public.toasted: INSERT: id[integer]:9001 other[text]:'bbb' > > data[text]:'ccc

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
eaming code kicks in very often and even > for small transactions. Thanks for confirming, I will come up with the test and add that to the next version of the patch. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
am unless we get the last tuple of multi insert WAL. How about > changing the code such that when we are clearing the toast flag, we > additionally check 'clear_toast_afterwards' flag? Yes, that can be done, I will fix this in the next version of the patch. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Tue, May 25, 2021 at 5:46 PM Dilip Kumar wrote: > > On Tue, May 25, 2021 at 4:50 PM Amit Kapila wrote: > > > > Your patch will fix the reported scenario but I don't like the way > > multi_insert flag is used to detect incomplete tuple. One problem > > c

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
size 6000 bytes so that is big enough to cross 64kB. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
On Wed, May 26, 2021 at 11:19 AM Amit Kapila wrote: > > On Tue, May 25, 2021 at 6:43 PM Dilip Kumar wrote: > > > > On Tue, May 25, 2021 at 5:46 PM Dilip Kumar wrote: > > > > > > On Tue, May 25, 2021 at 4:50 PM Amit Kapila > > > wrote: > > &g

Re: Assertion failure while streaming toasted data

2021-05-25 Thread Dilip Kumar
> acceptable or not. In the test decoding config, it is already set to a minimum value which is 64k. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Race condition in recovery?

2021-05-25 Thread Dilip Kumar
check whether the cascade standby is able to get that or not, that will guarantee that it is actually following the promoted standby, I have added the test for that so that it matches the comments. > Let's not call the command skip_cp. It's not very descriptive. If you > don't

Re: Assertion failure while streaming toasted data

2021-05-26 Thread Dilip Kumar
On Wed, May 26, 2021 at 11:53 AM Dilip Kumar wrote: > > On Wed, May 26, 2021 at 11:19 AM Amit Kapila wrote: > > > > > I searched and didn't find any similar existing tests. Can we think of > > any other way to test this code path? We already have one copy

Re: Assertion failure while streaming toasted data

2021-05-26 Thread Dilip Kumar
On Wed, May 26, 2021 at 5:28 PM Amit Kapila wrote: > > On Wed, May 26, 2021 at 1:47 PM Dilip Kumar wrote: > > > > On Wed, May 26, 2021 at 11:53 AM Dilip Kumar wrote: > > > > > > On Wed, May 26, 2021 at 11:19 AM Amit Kapila > > > wrote: > > &

Re: Race condition in recovery?

2021-05-26 Thread Dilip Kumar
On Wed, May 26, 2021 at 9:40 PM Robert Haas wrote: > > On Wed, May 26, 2021 at 2:44 AM Dilip Kumar wrote: > > I think we need to create some content on promoted standby and check > > whether the cascade standby is able to get that or not, that will > > guarantee that it i

Re: Race condition in recovery?

2021-05-26 Thread Dilip Kumar
On Wed, 26 May 2021 at 10:06 PM, Robert Haas wrote: > On Wed, May 26, 2021 at 12:26 PM Dilip Kumar > wrote: > > I will check if there is any timing dependency in the test case. > > There is. I explained it in the second part of my email, which you may > have failed to no

Re: Decoding speculative insert with toast leaks memory

2021-05-26 Thread Dilip Kumar
in > ReorderBufferCleanupTXN/ReorderBufferTruncateTXN? It will delay the > clean up of memory till the end of stream or txn but there won't be > any memory leak. Currently, we are ignoring XLH_DELETE_IS_SUPER, so maybe we can do something based on this flag? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Decoding speculative insert with toast leaks memory

2021-05-26 Thread Dilip Kumar
maining, just clean in up, it > * can't have been successful, otherwise we'd gotten a confirmation > * record. > */ > if (specinsert) > { > ReorderBufferReturnChange(rb, specinsert, true); > specinsert = NULL; > } > > But I guess we might miss cleaning it up in case of an error. A > similar problem could be there in the idea where we will try to tie > the clean up with the next change. In error case also we can handle it in the CATCH block no? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Decoding speculative insert with toast leaks memory

2021-05-26 Thread Dilip Kumar
On Thu, May 27, 2021 at 9:47 AM Amit Kapila wrote: > > On Thu, May 27, 2021 at 9:40 AM Dilip Kumar wrote: > > True, but if you do this clean-up in ReorderBufferCleanupTXN then you > don't need to take care at separate places. Also, toast_hash is stored > in txn so it appe

Re: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread Dilip Kumar
lex condition and also it select fewer record say 50%, 40%...10% and see what are the numbers. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Race condition in recovery?

2021-05-26 Thread Dilip Kumar
On Thu, May 27, 2021 at 6:19 AM Kyotaro Horiguchi wrote: > > At Wed, 26 May 2021 22:08:32 +0530, Dilip Kumar wrote > in > > On Wed, 26 May 2021 at 10:06 PM, Robert Haas wrote: > > > > > On Wed, May 26, 2021 at 12:26 PM Dilip Kumar > > > wrote: >

Re: Race condition in recovery?

2021-05-26 Thread Dilip Kumar
t/recovery/t/025_stuck_on_old_timeline.pl > I suggest we rename the test to something a bit more descriptive. Like > instead of 025_timeline_issue.pl, perhaps > 025_stuck_on_old_timeline.pl? Or I'm open to other suggestions, but > "timeline issue" is a bit too vague for my taste. Changed

Re: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread Dilip Kumar
On Thu, 27 May 2021 at 11:32 AM, tsunakawa.ta...@fujitsu.com < tsunakawa.ta...@fujitsu.com> wrote: > From: Dilip Kumar > > I think some other cause of contention on relation extension locks are > > 1. CTAS is using a buffer strategy and due to that, it might need to >

Re: Race condition in recovery?

2021-05-27 Thread Dilip Kumar
On Thu, May 27, 2021 at 12:09 PM Kyotaro Horiguchi wrote: > > At Thu, 27 May 2021 11:44:47 +0530, Dilip Kumar wrote > in > > Maybe we can somehow achieve that without a broken archive command, > > but I am not sure how it is enough to just delete WAL from pg_wal? I >

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-05-27 Thread Dilip Kumar
he user wants to get rid of the dependency on the external library then IMHO, there should be some way to do it by recompressing all the data. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [BUG]Update Toast data failure in logical replication

2021-05-28 Thread Dilip Kumar
or ALTER TABLE toasted_key REPLICA IDENTITY FULL. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Decoding speculative insert with toast leaks memory

2021-05-28 Thread Dilip Kumar
On Fri, May 28, 2021 at 5:16 PM Tomas Vondra wrote: > On 5/27/21 6:36 AM, Dilip Kumar wrote: > > On Thu, May 27, 2021 at 9:47 AM Amit Kapila wrote: > >> > >> On Thu, May 27, 2021 at 9:40 AM Dilip Kumar wrote: > >> > >> True, but if you do this

Re: Decoding speculative insert with toast leaks memory

2021-05-30 Thread Dilip Kumar
mitigation, so that we don't leak the memory > > forever. > > > > Right. > > > So let's get this committed, perhaps with a comment explaining that it > > might be possible to reset earlier if needed. > > > > Okay, I think it would be better if we can test this once for the > streaming case as well. Dilip, would you like to do that and send the > updated patch as per one of the comments by Tomas? I will do that in sometime. > -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [BUG]Update Toast data failure in logical replication

2021-05-30 Thread Dilip Kumar
On Mon, May 31, 2021 at 8:04 AM tanghy.f...@fujitsu.com wrote: > > On Friday, May 28, 2021 6:51 PM, Dilip Kumar wrote: > > Seems you did not set the replica identity for updating the tuple. > > Try this before updating, and it should work. > > Thanks for your reply. I tri

Re: [BUG]Update Toast data failure in logical replication

2021-05-31 Thread Dilip Kumar
On Mon, May 31, 2021 at 12:20 PM Dilip Kumar wrote: > > On Mon, May 31, 2021 at 8:04 AM tanghy.f...@fujitsu.com > wrote: > > > > On Friday, May 28, 2021 6:51 PM, Dilip Kumar wrote: > > > Seems you did not set the replica identity for updating the tuple. > >

Re: [BUG]Update Toast data failure in logical replication

2021-05-31 Thread Dilip Kumar
On Mon, 31 May 2021 at 3:33 PM, tanghy.f...@fujitsu.com < tanghy.f...@fujitsu.com> wrote: > On Mon, May 31, 2021 5:12 PM Dilip Kumar wrote: > > > > The problem is if the key attribute is not changed we don't log it as > > it should get logged along with

Re: Decoding speculative insert with toast leaks memory

2021-05-31 Thread Dilip Kumar
On Mon, May 31, 2021 at 8:50 AM Dilip Kumar wrote: > > On Mon, 31 May 2021 at 8:21 AM, Amit Kapila wrote: >> >> Okay, I think it would be better if we can test this once for the >> streaming case as well. Dilip, would you like to do that and send the >> updated pa

Re: Decoding speculative insert with toast leaks memory

2021-05-31 Thread Dilip Kumar
On Mon, 31 May 2021 at 4:29 PM, Dilip Kumar wrote: > On Mon, May 31, 2021 at 8:50 AM Dilip Kumar wrote: > > > > On Mon, 31 May 2021 at 8:21 AM, Amit Kapila > wrote: > >> > >> Okay, I think it would be better if we can test this once for the > >> str

Re: Decoding speculative insert with toast leaks memory

2021-05-31 Thread Dilip Kumar
On Mon, May 31, 2021 at 6:32 PM Dilip Kumar wrote: > > On Mon, 31 May 2021 at 4:29 PM, Dilip Kumar wrote: >> >> On Mon, May 31, 2021 at 8:50 AM Dilip Kumar wrote: >> > >> > On Mon, 31 May 2021 at 8:21 AM, Amit Kapila >> > wrote: >> >> &

Re: Decoding speculative insert with toast leaks memory

2021-05-31 Thread Dilip Kumar
On Tue, Jun 1, 2021 at 9:53 AM Amit Kapila wrote: > > On Mon, May 31, 2021 at 8:12 PM Dilip Kumar wrote: > > > > On Mon, May 31, 2021 at 6:32 PM Dilip Kumar wrote: > > > > > > I missed to do the test for streaming. I will to that tomorrow and repl

Re: Decoding speculative insert with toast leaks memory

2021-05-31 Thread Dilip Kumar
On Tue, Jun 1, 2021 at 10:21 AM Amit Kapila wrote: > > On Tue, Jun 1, 2021 at 9:59 AM Dilip Kumar wrote: > > > > On Tue, Jun 1, 2021 at 9:53 AM Amit Kapila wrote: > > > > > > On Mon, May 31, 2021 at 8:12 PM Dilip Kumar wrote: > > > > >

Re: Decoding speculative insert with toast leaks memory

2021-05-31 Thread Dilip Kumar
On Tue, Jun 1, 2021 at 11:00 AM Dilip Kumar wrote: > > On Tue, Jun 1, 2021 at 10:21 AM Amit Kapila wrote: > > > > On Tue, Jun 1, 2021 at 9:59 AM Dilip Kumar wrote: > > > > > > On Tue, Jun 1, 2021 at 9:53 AM Amit Kapila > > > wrote: > > >

Re: Exit walsender before confirming remote flush in logical replication

2023-01-22 Thread Dilip Kumar
ed by a shutdown. > ... > ... > > Here, as mentioned in the commit, we are trying to ensure that before > checkpoint writes its shutdown WAL record, we ensure that "if logical > replication is in progress all existing WAL records are processed > followed by a shutdown.". I think even before this commit, we try to > send the entire WAL before shutdown but not completely sure. Yes, I think that there is no change in that behavior. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-22 Thread Dilip Kumar
> > logical_decoding_mode also on the subscriber, as Shveta Malik also > > suggested, probably it's better to rename it so as not to confuse. For > > example, logical_replication_mode or something. > > > > +1. Among the options discussed, this sounds better. Yeah, this looks better option with the parameter name 'logical_replication_mode'. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-23 Thread Dilip Kumar
On Wed, Jan 18, 2023 at 1:47 PM Dilip Kumar wrote: > > On Tue, Jan 17, 2023 at 10:05 AM Peter Geoghegan wrote: My final set of comments for 0002 1. +struct vmsnapshot +{ +/* Target heap rel */ +Relationrel; +/* Scanning strategy used by VACUUM operation */ +vmst

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-23 Thread Dilip Kumar
ty map page. * * index is an offset into the vacrel->dead_items array for the first listed * LP_DEAD item on the page. The return value is the first index immediately * after all LP_DEAD items for the same page in the array. */ This comment needs to be changed as this is referring to the &#

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-29 Thread Dilip Kumar
On Thu, Jan 26, 2023 at 12:39 PM John Naylor wrote: > > > On Tue, Jan 24, 2023 at 1:17 PM Dilip Kumar wrote: > > > > On Mon, Jan 23, 2023 at 6:00 PM John Naylor > > wrote: > > > > > > Attached is a rebase to fix conflicts from recent commits. >

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-02-07 Thread Dilip Kumar
. The original value can always be stored in some temp variable instead of the function argument. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-02-07 Thread Dilip Kumar
n it would be good to have such a test to avoid any future breakage in that logic. But if it is just for hitting the code but no guarantee that whether we can validate as part of the test whether it has hit the WAL buffer or not then I think the existing cases are sufficient. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: DDL result is lost by CREATE DATABASE with WAL_LOG strategy

2023-02-15 Thread Dilip Kumar
s per the attached patch) seems to fix this. > I don't see any reason this _shouldn't_ be true from what I've read so far. > We're reading pg_class, which will probably never be UNLOGGED. Yes, there is no reason to pass this as false, seems like this is passed false by mistake. And your patch fixes the issue. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: DDL result is lost by CREATE DATABASE with WAL_LOG strategy

2023-02-15 Thread Dilip Kumar
d b/src/test/recovery/meson.build index 209118a639..6e9f8a7c7f 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -39,6 +39,7 @@ tests += { 't/031_recovery_conflict.pl', 't/032_relfilenode_reuse.pl', 't/033_replay_tsp_drops.pl', + 't/100_bugs.pl', ], }, } -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: DDL result is lost by CREATE DATABASE with WAL_LOG strategy

2023-02-15 Thread Dilip Kumar
On Thu, Feb 16, 2023 at 12:38 PM Robert Haas wrote: > > On Thu, Feb 16, 2023 at 12:32 PM Dilip Kumar wrote: > > I think the below change is not relevant to this bug right? > > > > diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build > > index

Re: longfin and tamandua aren't too happy but I'm not sure why

2022-09-27 Thread Dilip Kumar
| char_tbl + 16388 || 0 | 11 | float8_tbl + 16391 || 0 | 12 | int2_tbl + 16394 || 0 | 13 | int4_tbl -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: longfin and tamandua aren't too happy but I'm not sure why

2022-09-27 Thread Dilip Kumar
: database system is shut down -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: longfin and tamandua aren't too happy but I'm not sure why

2022-09-27 Thread Dilip Kumar
On Wed, Sep 28, 2022 at 10:45 AM Tom Lane wrote: > > Dilip Kumar writes: > > wrasse is also failing with a bus error, > > Yeah. At this point I think it's time to call for this patch > to get reverted. It should get tested *off line* on some > non-Intel

Re: longfin and tamandua aren't too happy but I'm not sure why

2022-09-28 Thread Dilip Kumar
On Wed, Sep 28, 2022 at 9:40 AM Dilip Kumar wrote: > > On Wed, Sep 28, 2022 at 2:59 AM Tom Lane wrote: > > > > ... also, lapwing's not too happy [1]. The alter_table test > > expects this to yield zero rows, but it doesn't: > > By looking at regression d

Re: making relfilenodes 56 bits

2022-09-28 Thread Dilip Kumar
{ > /* Still not empty, the files must be important then > */ > ereport(ERROR, Thanks, Thomas, these all look fine to me. So far we have committed the patch to make relfilenode 56 bits wide. The Tombstone file removal patch is still pending to be committed, so when I will rebase that patch I will accommodate all these comments in that patch. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: longfin and tamandua aren't too happy but I'm not sure why

2022-09-28 Thread Dilip Kumar
On Wed, Sep 28, 2022 at 11:57 AM Tom Lane wrote: > > Dilip Kumar writes: > > Btw, I think the reason for the bus error on wrasse is the same as > > what is creating failure on longfin[1], I mean this unaligned access > > is causing Bus error during startup, IMHO. > >

Re: problems with making relfilenodes 56-bits

2022-09-30 Thread Dilip Kumar
y small WAL sizes with a very targeted test case we can see a significant 11% increase in WAL size but with pgbench kind of workload the increase in WAL size is very less. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: problems with making relfilenodes 56-bits

2022-09-30 Thread Dilip Kumar
On Sat, Oct 1, 2022 at 5:50 AM Andres Freund wrote: > > Hi, > > On 2022-09-30 15:36:11 +0530, Dilip Kumar wrote: > > I have done some testing around this area to see the impact on WAL > > size especially when WAL sizes are smaller, with a very simple test > > with

Re: problems with making relfilenodes 56-bits

2022-10-10 Thread Dilip Kumar
the WAL generated is 58179.2265625. That means in this particular example this patch is reducing the WAL size by 12% even with the 56 bit relfilenode patch. [1] https://www.postgresql.org/message-id/CAFiTN-uut%2B04AdwvBY_oK_jLvMkwXUpDJj5mXg--nek%2BucApPQ%40mail.gmail.com -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: problems with making relfilenodes 56-bits

2022-10-11 Thread Dilip Kumar
On Mon, Oct 10, 2022 at 5:40 PM Robert Haas wrote: > > On Mon, Oct 10, 2022 at 5:16 AM Dilip Kumar wrote: > > I have also executed my original test after applying these patches on > > top of the 56 bit relfilenode patch. So earlier we saw the WAL size > > increased b

Re: Support logical replication of DDLs

2022-10-11 Thread Dilip Kumar
expanded fully, so that there are no ambiguities even in the + * face of search_path changes. + */ +Datum +ddl_deparse_to_json(PG_FUNCTION_ARGS) +{ It will be nice to have a test case for this utility function. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Perform streaming logical transactions by background workers and parallel apply

2022-10-18 Thread Dilip Kumar
t of each other on the publisher are dependent on each other on the subscriber side because table definition is different on the subscriber. So can't we handle this case in the same way by marking this table unsafe for parallel-apply? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Perform streaming logical transactions by background workers and parallel apply

2022-10-19 Thread Dilip Kumar
B34E1655718F4D1994249%40OS0PR01MB5716.jpnprd01.prod.outlook.com Okay, somehow I missed the latest email. I will look into it soon. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: problems with making relfilenodes 56-bits

2022-10-20 Thread Dilip Kumar
ar more than the cases where it will slightly increase the size. And also the number of bytes we save in winning cases is far bigger than the number of bytes we increase. So IMHO it seems like an overall win at least from the WAL size reduction pov. Do we have some number that how much overhead it has for encoding/decoding? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Segfault on logical replication to partitioned table with foreign children

2022-10-30 Thread Dilip Kumar
ker attempts to insert the data received via > replication into the foreign table. Reproduce with caution, the worker > is going to crash and restart indefinitely. The setup: Yes, this looks like a bug and your fix seems correct to me. It would be nice to add a test case for this scenario

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Dilip Kumar
97 > > 256 550296 334923 > > 512 486207 243679 > > 768 449673 192959 > > 1024410836 157734 > > 204832622482904 > > 409625025232007 > > > > Not perfect with the patch, but not awful either. > >

Re: Segfault on logical replication to partitioned table with foreign children

2022-10-31 Thread Dilip Kumar
On Sun, Oct 30, 2022 at 7:09 PM Tom Lane wrote: > > Dilip Kumar writes: > > Yes, this looks like a bug and your fix seems correct to me. It would > > be nice to add a test case for this scenario. > > A test case doesn't seem that exciting to me. If we were tryi

Re: Code checks for App Devs, using new options for transaction behavior

2022-10-31 Thread Dilip Kumar
t;. They might be changing the same code location but these two are completely different ideas, in fact all these three should be reviewed as three separate threads as you mentioned in the first email in the thread. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Code checks for App Devs, using new options for transaction behavior

2022-10-31 Thread Dilip Kumar
On Mon, Oct 31, 2022 at 4:23 PM Dilip Kumar wrote: > > On Sun, Oct 30, 2022 at 11:32 PM Simon Riggs > wrote: > > > > On Fri, 28 Oct 2022 at 10:33, Simon Riggs > > wrote: > > > > > Thanks for the feedback, I will make all of those corrections in t

Re: Code checks for App Devs, using new options for transaction behavior

2022-10-31 Thread Dilip Kumar
On Mon, Oct 31, 2022 at 5:03 PM Dilip Kumar wrote: > > On Mon, Oct 31, 2022 at 4:23 PM Dilip Kumar wrote: > > > > On Sun, Oct 30, 2022 at 11:32 PM Simon Riggs > > wrote: > > > > > > On Fri, 28 Oct 2022 at 10:33, Simon Riggs > > > wrote: >

Re: SUBTRANS: Minimizing calls to SubTransSetParent()

2022-11-01 Thread Dilip Kumar
t; 88 413714 443343 > 96 251098 442021 > 104 70190 443488 > 112 405507 438866 > 120 177827 443202 > 128 399431 441842 > > From the performance point of view, this patch clearly fixes the > dramatic TPS collapse shown in these tests. I think these are really promising results. Although the perf result shows that the bottleneck on the SLRU is no more there with the patch, I think it would be nice to see the wait event as well. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Code checks for App Devs, using new options for transaction behavior

2022-11-01 Thread Dilip Kumar
ame, NESTED_XACT_NAME) == 0) +found_subxact = true; I think this strcmp(s->name, NESTED_XACT_NAME) is done because there could be other types of internal subtransaction also like savepoints? What will be the behavior if someone declares a savepoint with this name ("_internal_nested_xact"). Will this interfere with this new functionality? Have we tested scenarios like that? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2022-11-03 Thread Dilip Kumar
ured the WAL overhead because of this patch set? maybe these particular patches will not impact but IIUC this is ground work for making xid 64 bit. So each XLOG record size will increase at least by 4 bytes because the XLogRecord contains the xid. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Moving forward with TDE

2022-11-04 Thread Dilip Kumar
gh we are not using those in IV there is no point in saying we are avoiding that requirement. I will review the remaining patches soon. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [BUG]Update Toast data failure in logical replication

2021-06-01 Thread Dilip Kumar
m not sure if > it would bring new problems. Have you checked it? Good observation, basically, my check says that any field in the tuple is toasted then prepare the key tuple, actually, after that, I should recheck whether the key field specifically toasted or not and if it is not then we can continue returning NULL. I will fix this in the next version. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Decoding speculative insert with toast leaks memory

2021-06-01 Thread Dilip Kumar
ation(relation->rd_rel->reltoastrelid); if (!RelationIsValid(toast_rel)) elog(ERROR, "could not open relation with OID %u", relation->rd_rel->reltoastrelid); -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Decoding speculative insert with toast leaks memory

2021-06-01 Thread Dilip Kumar
On Tue, Jun 1, 2021 at 5:22 PM Dilip Kumar wrote: > > On Tue, Jun 1, 2021 at 12:25 PM Amit Kapila wrote: > > > > > > > IMHO, as I stated earlier one way to fix this problem is that we add > > > the spec abort operation (DELETE + XLH_DELETE_IS_SUPER flag) t

Re: Decoding speculative insert with toast leaks memory

2021-06-01 Thread Dilip Kumar
On Wed, Jun 2, 2021 at 11:25 AM Amit Kapila wrote: > > On Tue, Jun 1, 2021 at 5:23 PM Dilip Kumar wrote: > > > > On Tue, Jun 1, 2021 at 12:25 PM Amit Kapila wrote: > > > > > > > > > > IMHO, as I stated earlier one way to fix this problem is that

<    6   7   8   9   10   11   12   13   14   15   >