Re: Is Recovery actually paused?

2021-01-27 Thread Dilip Kumar
On Wed, Jan 27, 2021 at 2:06 PM Yugo NAGATA wrote: > > On Wed, 27 Jan 2021 13:29:23 +0530 > Dilip Kumar wrote: > > > On Wed, Jan 27, 2021 at 12:50 PM Masahiko Sawada > > wrote: > > > > > > On Tue, Jan 26, 2021 at 2:00 AM Robert Haas wrote: >

Re: Support ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION ... syntax

2021-01-27 Thread Dilip Kumar
r new proposed > > syntax user has to write three separate commands. > > > > If we want add and drop some publications, we can use SET PUBLICATION, it > is more convenient than ADD and DROP PUBLICATION, however if we just want to > add (or drop) publication into

Re: Is Recovery actually paused?

2021-01-27 Thread Dilip Kumar
On Wed, Jan 27, 2021 at 2:28 PM Dilip Kumar wrote: > > On Wed, Jan 27, 2021 at 2:06 PM Yugo NAGATA wrote: > > > > On Wed, 27 Jan 2021 13:29:23 +0530 > > Dilip Kumar wrote: > > > > > On Wed, Jan 27, 2021 at 12:50 PM Masahiko Sawada > > > wrote:

Re: [HACKERS] Custom compression methods

2021-01-28 Thread Dilip Kumar
0.857.750.857.7 > 39src/bin/psql/describe.c60.755.160.654.2 > 40contrib/cmzlib/cmzlib.cNANA74.787.5 > Thanks, Neha for testing this, overall coverage looks good to me except compress_pglz.c, compressamapi.c and cmzlib.c. I will analyze this and see if we can improve coverage for these files or not. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Is Recovery actually paused?

2021-01-29 Thread Dilip Kumar
On Fri, Jan 29, 2021 at 3:25 PM Yugo NAGATA wrote: > > On Thu, 28 Jan 2021 09:55:42 +0530 > Dilip Kumar wrote: > > > On Wed, Jan 27, 2021 at 2:28 PM Dilip Kumar wrote: > > > > > > On Wed, Jan 27, 2021 at 2:06 PM Yugo NAGATA wrote: > > >

Re: Is Recovery actually paused?

2021-01-30 Thread Dilip Kumar
On Fri, Jan 29, 2021 at 4:33 PM Dilip Kumar wrote: > > On Fri, Jan 29, 2021 at 3:25 PM Yugo NAGATA wrote: > > > > On Thu, 28 Jan 2021 09:55:42 +0530 > > Dilip Kumar wrote: > > > > > On Wed, Jan 27, 2021 at 2:28 PM Dilip Kumar wrote: > > > >

Re: Faulty HEAP_XMAX_LOCK_ONLY & HEAP_KEYS_UPDATED hintbit combination

2021-01-31 Thread Dilip Kumar
eak; } if (!has_update) bits |= HEAP_XMAX_LOCK_ONLY; Basically, if it is "select for update" then we will mark infomask2 as HEAP_KEYS_UPDATED and the informask as HEAP_XMAX_LOCK_ONLY. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Is Recovery actually paused?

2021-02-01 Thread Dilip Kumar
On Mon, Feb 1, 2021 at 11:59 AM Kyotaro Horiguchi wrote: > > At Sun, 31 Jan 2021 11:24:30 +0530, Dilip Kumar wrote > in > > On Fri, Jan 29, 2021 at 4:33 PM Dilip Kumar wrote: > > > > > > On Fri, Jan 29, 2021 at 3:25 PM Yugo NAGATA wrote: > > > &g

Re: Printing backtrace of postgres processes

2021-02-01 Thread Dilip Kumar
if logging is > enabled, if logging is disabled backtrace will be logged to the > console where the postmaster was started. > > > 13) Not sure, if it's an overkill, but how about pg_print_callstack > > returning a warning/notice along with true, which just says, "See > > <<>

Re: Faulty HEAP_XMAX_LOCK_ONLY & HEAP_KEYS_UPDATED hintbit combination

2021-02-01 Thread Dilip Kumar
On Mon, Feb 1, 2021 at 4:05 PM Julien Rouhaud wrote: > > On Mon, Feb 1, 2021 at 2:05 PM Dilip Kumar wrote: > > > > On Sun, Jan 24, 2021 at 9:31 PM Alvaro Herrera > > wrote: > > > > > > On 2021-Jan-24, Julien Rouhaud wrote: > > > > &g

Re: Faulty HEAP_XMAX_LOCK_ONLY & HEAP_KEYS_UPDATED hintbit combination

2021-02-01 Thread Dilip Kumar
said > UPDATE or a DELETE; as opposed to SELECT FOR NO KEY UPDATE which stands > for a future UPDATE that will only change columns that aren't part of > any keys. Yeah, that makes sense. > So I think that I misspoke earlier in this thread when I said this is a > bug, and that the right fix here is to remove the Assert() and change > amcheck to match. +1 -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-02-02 Thread Dilip Kumar
e VARLENA_RAWSIZE_BITS and VARLENA_RAWSIZE_MASK > and then having toast_internals.h use those constants as well. > > Taken with the email I sent yesterday, I think this is a more or less > complete review of 0001. Although there are a bunch of things to fix > here still, I don't think this is that far from being committable. I > don't at this point see too much in terms of big design problems. > Probably the CompareCompressionMethodAndDecompress() is the closest to > a design-level problem, and certainly something needs to be done about > it, but even that is a fairly localized problem in the context of the > entire patch. Thanks, Robert for the detailed review. I will work on these comments and post the updated patch. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Correct comment in StartupXLOG().

2021-02-03 Thread Dilip Kumar
27;t done anything about it, only replaced the "boolean flag" to > "recovery > state" in the attached patch. I don't think the atomic is correct, it's no more boolean so it is better we get rid of this part of the comment -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Is Recovery actually paused?

2021-02-03 Thread Dilip Kumar
On Mon, Feb 1, 2021 at 1:41 PM Dilip Kumar wrote: > > On Mon, Feb 1, 2021 at 11:59 AM Kyotaro Horiguchi > wrote: > > > > At Sun, 31 Jan 2021 11:24:30 +0530, Dilip Kumar > > wrote in > > > On Fri, Jan 29, 2021 at 4:33 PM Dilip Kumar wrote: > > > &

Re: Correct comment in StartupXLOG().

2021-02-03 Thread Dilip Kumar
On Thu, Feb 4, 2021 at 9:39 AM Amul Sul wrote: > > On Thu, Feb 4, 2021 at 6:18 AM Kyotaro Horiguchi > wrote: > > > > At Wed, 3 Feb 2021 16:36:13 +0530, Amul Sul wrote in > > > On Wed, Feb 3, 2021 at 2:48 PM Dilip Kumar wrote: > > > > > > &g

Re: Is Recovery actually paused?

2021-02-04 Thread Dilip Kumar
On Thu, Feb 4, 2021 at 4:58 PM Bharath Rupireddy wrote: > > On Thu, Feb 4, 2021 at 10:28 AM Dilip Kumar wrote: > > Please find an updated patch which addresses these comments. > > Thanks for the patch. I tested the new function pg_get_wal_replay_pause_state: >

Re: Is Recovery actually paused?

2021-02-04 Thread Dilip Kumar
On Thu, Feb 4, 2021 at 4:58 PM Bharath Rupireddy wrote: > > On Thu, Feb 4, 2021 at 10:28 AM Dilip Kumar wrote: > > Please find an updated patch which addresses these comments. > > Thanks for the patch. I tested the new function pg_get_wal_replay_pause_state: >

Re: Is Recovery actually paused?

2021-02-04 Thread Dilip Kumar
On Thu, Feb 4, 2021 at 6:16 PM Dilip Kumar wrote: > > On Thu, Feb 4, 2021 at 4:58 PM Bharath Rupireddy > wrote: > > > > On Thu, Feb 4, 2021 at 10:28 AM Dilip Kumar wrote: > > > Please find an updated patch which addresses these comments. > > > > Thank

Re: [HACKERS] Custom compression methods

2021-02-04 Thread Dilip Kumar
Okay, so if we take a new slot then we need to set this slot reference in the ScanState also otherwise that might point to the old slot. I haven't yet analyzed where all we might be keeping the reference to that old slot. Or I am missing something. Anyway, I will get a better idea once I try to implement this. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Is Recovery actually paused?

2021-02-04 Thread Dilip Kumar
On Thu, Feb 4, 2021 at 10:19 PM Robert Haas wrote: > > On Thu, Feb 4, 2021 at 7:46 AM Dilip Kumar wrote: > > How can we do that this is not a 1 byte flag this is enum so I don't > > think we can read any atomic state without a spin lock here. > > I think this disc

Re: Is Recovery actually paused?

2021-02-04 Thread Dilip Kumar
On Fri, Feb 5, 2021 at 6:22 AM Bharath Rupireddy wrote: > > On Thu, Feb 4, 2021 at 7:20 PM Dilip Kumar wrote: > > > How can we do that this is not a 1 byte flag this is enum so I don't > > > think we can read any atomic state without a spin lock here. > > >

Re: [HACKERS] Custom compression methods

2021-02-04 Thread Dilip Kumar
On Fri, Feb 5, 2021 at 3:51 AM Robert Haas wrote: > > On Thu, Feb 4, 2021 at 11:39 AM Dilip Kumar wrote: > > Yeah, actually, I thought I would avoid calling slot_getallattrs if > > none of the attributes got decompress. I agree if we call this before > > we can avoid

Re: [HACKERS] Custom compression methods

2021-02-05 Thread Dilip Kumar
d in very many places at all. If we're adding > stuff there that is going to be broadly needed, we're adding it in the > wrong place. Done > varlena.c shouldn't need toast_internals.h either, and if it did, it > should be in alphabetical order. > It was the wrong usage, fixed now. Please refer to the latest patch at https://www.postgresql.org/message-id/CAFiTN-v9Cs1MORnp-3bGZ5QBwr5v3VarSvfaDizHi1acXES5xQ%40mail.gmail.com -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Is Recovery actually paused?

2021-02-07 Thread Dilip Kumar
enever the variable now is used within the for loop in > WaitForWALToBecomeAvailable, it's re-calculated anyways. It's being > used within case XLOG_FROM_STREAM: > > Am I missing something? Yeah, I don't see any reason for doing this, maybe i

Re: Is Recovery actually paused?

2021-02-07 Thread Dilip Kumar
On Mon, 8 Feb 2021 at 6:38 AM, Yugo NAGATA wrote: > Hi, > > On Sun, 7 Feb 2021 19:27:02 +0530 > Dilip Kumar wrote: > > > On Sun, Feb 7, 2021 at 6:44 PM Bharath Rupireddy > > wrote: > > > > > > On Fri, Feb 5, 2021 at 10:14 AM Bharath Rupired

Re: Is Recovery actually paused?

2021-02-07 Thread Dilip Kumar
On Mon, Feb 8, 2021 at 8:18 AM Yugo NAGATA wrote: > > On Mon, 8 Feb 2021 07:51:22 +0530 > Dilip Kumar wrote: > > > On Mon, 8 Feb 2021 at 6:38 AM, Yugo NAGATA wrote: > > > > > Hi, > > > > > > On Sun, 7 Feb 2021 19:27:02 +0530 > > > Dili

Re: Is Recovery actually paused?

2021-02-07 Thread Dilip Kumar
On Mon, Feb 8, 2021 at 9:49 AM Bharath Rupireddy wrote: > > On Mon, Feb 8, 2021 at 9:35 AM Dilip Kumar wrote: > > > > > If a user call pg_wal_replay_pause while waiting in > > > > > RecoveryRequiresIntParameter, > > > > > the state become

Re: Is Recovery actually paused?

2021-02-08 Thread Dilip Kumar
he logical state transition should always be as below NOT PAUSED -> PAUSE REQUESTED or PAUSED (maybe we should always go to request and then paused but there is nothing wrong with going to paused) PAUSE REQUESTED -> NOT PAUSE or PAUSED (either cancel the request or get paused) PAUSED -> NOT PAUSED (from PAUSED we should not go to the PAUSE_REQUESTED without going to NOT PAUSED) -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Is Recovery actually paused?

2021-02-08 Thread Dilip Kumar
On Tue, Feb 9, 2021 at 7:28 AM Kyotaro Horiguchi wrote: > > At Mon, 8 Feb 2021 17:05:52 +0530, Dilip Kumar wrote > in > > On Mon, Feb 8, 2021 at 2:19 PM Yugo NAGATA wrote: > > > > > > On Mon, 08 Feb 2021 17:32:46 +0900 (JST) > > > Kyotaro Horiguchi

Re: Is Recovery actually paused?

2021-02-08 Thread Dilip Kumar
On Tue, Feb 9, 2021 at 8:54 AM Yugo NAGATA wrote: > > On Tue, 09 Feb 2021 10:58:04 +0900 (JST) > Kyotaro Horiguchi wrote: > > > At Mon, 8 Feb 2021 17:05:52 +0530, Dilip Kumar > > wrote in > > > On Mon, Feb 8, 2021 at 2:19 PM Yugo NAGATA wrote: > > &

Re: [HACKERS] Custom compression methods

2021-02-09 Thread Dilip Kumar
On Sun, Feb 7, 2021 at 5:15 PM Dilip Kumar wrote: > > On Fri, Feb 5, 2021 at 8:11 PM Dilip Kumar wrote: > > > > On Wed, Feb 3, 2021 at 2:07 AM Robert Haas wrote: > > > > > > Even more review comments, still looking mostly at 0001: > > > >

Re: [HACKERS] Custom compression methods

2021-02-09 Thread Dilip Kumar
On Tue, Feb 9, 2021 at 2:08 PM Dilip Kumar wrote: > > On Sun, Feb 7, 2021 at 5:15 PM Dilip Kumar wrote: > > > > On Fri, Feb 5, 2021 at 8:11 PM Dilip Kumar wrote: > > > > > > On Wed, Feb 3, 2021 at 2:07 AM Robert Haas wrote: > > > > > >

Re: Is Recovery actually paused?

2021-02-09 Thread Dilip Kumar
ach where we don't wait in any API and just return the recovery pause state is much simpler and more flexible. So I will make the pending changes in that patch and let's see what are the other opinion and based on that we can conclude. Thanks for the patch. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-02-09 Thread Dilip Kumar
On Wed, Feb 10, 2021 at 1:42 AM Robert Haas wrote: > > Please remember to trim unnecessary quoted material. Okay, I will. > On Sun, Feb 7, 2021 at 6:45 AM Dilip Kumar wrote: > > [ a whole lot of quoted stuff ] > > > > Conclusion: > > 1. In most cases lz4 is fa

Re: Is Recovery actually paused?

2021-02-10 Thread Dilip Kumar
On Wed, Feb 10, 2021 at 10:02 AM Dilip Kumar wrote: > > I don't find any problem with this approach as well, but I personally > feel that the other approach where we don't wait in any API and just > return the recovery pause state is much simpler and more flexible. So >

Re: Is Recovery actually paused?

2021-02-11 Thread Dilip Kumar
On Thu, Feb 11, 2021 at 3:20 PM Bharath Rupireddy wrote: > > On Wed, Feb 10, 2021 at 8:39 PM Dilip Kumar wrote: > > > > On Wed, Feb 10, 2021 at 10:02 AM Dilip Kumar wrote: > > > > > > I don't find any problem with this approach as well, but I personally

Re: [HACKERS] Custom compression methods

2021-02-11 Thread Dilip Kumar
o better. Maybe we can add a flag in infomask to detect whether the tuple has any compressed data or not as we have for detecting the external data (HEAP_HASEXTERNAL). So I will do some more analysis in this area and try to come up with a clean solution. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com POC_fix_compression_in_rowtype.patch Description: Binary data

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-02 Thread Dilip Kumar
On Wed, Dec 1, 2021 at 6:04 PM Dilip Kumar wrote: > Thanks a lot for testing this. From the error, it seems like some of > the old buffer w.r.t. the previous tablespace is not dropped after the > movedb. Actually, we are calling DropDatabaseBuffers() after copying > to a new ta

Re: suboverflowed subtransactions concurrency performance optimize

2021-12-02 Thread Dilip Kumar
CHED_SUBXIDS? But the TransactionIdDidCommit(), might need to fetch the parent if the transaction status is TRANSACTION_STATUS_SUB_COMMITTED, so how would we handle that? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: suboverflowed subtransactions concurrency performance optimize

2021-12-03 Thread Dilip Kumar
On Fri, Dec 3, 2021 at 5:00 PM Simon Riggs wrote: > > On Fri, 3 Dec 2021 at 01:27, Dilip Kumar wrote: > > > > On review, I think it is also possible that we update subtrans ONLY if > > > someone uses >PGPROC_MAX_CACHED_SUBXIDS. > > > This would make subtra

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-03 Thread Dilip Kumar
ATE DATABASE in a less cheesy way, * we can avoid this. */ RequestCheckpoint(CHECKPOINT_IMMEDIATE | CHECKPOINT_FORCE | CHECKPOINT_WAIT); -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-05 Thread Dilip Kumar
use_wal = XLogIsNeeded() && > + (relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork); > Since we are creating new relfilenode this is fine, refer "Skipping WAL for New RelFileNode" in src/backend/access/transam/README -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: row filtering for logical replication

2021-12-05 Thread Dilip Kumar
en convert the UPDATE to DELETE OTOH if only new is matching the filter then convert the UPDATE to INSERT. Do you think that even we merge 0001 and 0003 then also there is an open issue regarding which row to select for the filter? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Add sub-transaction overflow status in pg_stat_activity

2021-12-06 Thread Dilip Kumar
-t5BkwdHm1bV8ez64guWZJB_Jjhb7arsQsftxEwpYwObg%40mail.gmail.com -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com From 622e1012667c3cfa0c71f27590e2a49833970e22 Mon Sep 17 00:00:00 2001 From: Dilip Kumar Date: Sun, 5 Dec 2021 17:56:16 +0530 Subject: [PATCH v1] Add subtransaction count and overflow status in

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-07 Thread Dilip Kumar
unt of resource which is true for any other operation e.g. ALTER TABLE SET TABLESPACE so now it is in more sync with the rest of the system, but without the patch, it was a special purpose forced checkpoint only for the createdb. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-07 Thread Dilip Kumar
> > People who want to could use it like: > SELECT * FROM pg_stat_activity psa, pg_backend_get_subxact_status(pid) sub; Yeah, this is a valid point, I will change this accordingly. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-07 Thread Dilip Kumar
On Tue, Dec 7, 2021 at 10:29 AM Nikolay Samokhvalov wrote: > > On Mon, Dec 6, 2021 at 8:16 PM Dilip Kumar wrote: >> >> If the subtransaction cache is overflowed in some of the transactions >> then it will affect all the concurrent queries as they need to access >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-08 Thread Dilip Kumar
patch. Thanks Ahustosh, yeah I have observed the same, earlier we were directly copying the whole directory so this was not an issue, now if some tables of the database are already in the destination tablespace then we should skip them while copying. I will review your patch and merge into the main patch. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-09 Thread Dilip Kumar
than previously raised. I will work on this next week, thanks for testing. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-12 Thread Dilip Kumar
basically, in case of movedb the source and destination dboid are same so we don't need an additional parameter and also readjusted the conditions to avoid nested if. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com From eb27d159bc2aa41011b6352a514c7981a19a64dc Mon Sep 17 00:

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-13 Thread Dilip Kumar
+---+--+ 1 | 43806 |0 | f 2 | 43983 | 64 | t 3 | 43994 |0 | f 4 | 44323 | 22 | f 5 | 43802 |0 | f 6 | 43801 |0 | f 7 | 43804 | 0 | f (7 rows) -- Regards, Dilip Kumar Enter

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-13 Thread Dilip Kumar
my view. If you disagree you can ignore it. > > -- > > + else if (src_dboid == dst_dboid) > + continue; > + else > + dstrnode.spcNode = srcrnode.spcNode;; > > There is an extra semicolon here. Noted. I will fix them in the next version. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Skipping logical replication transactions on subscriber side

2021-12-13 Thread Dilip Kumar
an option to give the list of all such tables. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Skipping logical replication transactions on subscriber side

2021-12-13 Thread Dilip Kumar
whether they wants to skip or not only once we start applying no? I mean only once we start applying the changes we can get some errors and by that time we must be having all the changes for the transaction. So I do not understand the point we are trying to discuss here? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Skipping logical replication transactions on subscriber side

2021-12-14 Thread Dilip Kumar
em worth it because we can anyway skip the changes > while applying as you mentioned in the previous paragraph. Actually, I was trying to understand the use case for skipping while streaming. Actually, during streaming we are not doing any database operation that means this will not generate any e

Re: Skipping logical replication transactions on subscriber side

2021-12-14 Thread Dilip Kumar
hase. I think the patch is skipping it during apply phase. > Sawada-San, please confirm if my understanding is correct? > Got it, thanks for clarifying. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Skipping logical replication transactions on subscriber side

2021-12-14 Thread Dilip Kumar
On Wed, Dec 15, 2021 at 9:46 AM Amit Kapila wrote: > > On Tue, Dec 14, 2021 at 11:40 AM Dilip Kumar wrote: > > That's right and I agree there could be some use case for it and even > specifying the operation but I think we can always extend the existing > feature for it i

Re: generalized conveyor belt storage

2021-12-14 Thread Dilip Kumar
an discuss this on a separate thread when I am ready to post that patch. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com From b6affd2a778b7b4cff5738ad99f34ea21a816562 Mon Sep 17 00:00:00 2001 From: Dilip Kumar Date: Wed, 15 Dec 2021 10:28:49 +0530 Subject: [PATCH v1] Convey

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-16 Thread Dilip Kumar
On Thu, Dec 16, 2021 at 12:15 AM Bruce Momjian wrote: > > On Thu, Dec 2, 2021 at 07:19:50PM +0530, Dilip Kumar wrote: > From the patch: > > > Currently, CREATE DATABASE forces a checkpoint, then copies all the files, > > then forces another checkpoint. The comments in

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-16 Thread Dilip Kumar
On Tue, Dec 14, 2021 at 3:57 AM Bossart, Nathan wrote: > > On 12/13/21, 6:30 AM, "Dilip Kumar" wrote: > > On Tue, Dec 7, 2021 at 11:11 AM Justin Pryzby wrote: > >> Since I think this field is usually not interesting to most users of > >> pg_sta

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-16 Thread Dilip Kumar
transactions in the current session exceeded the > cached > +* subtransaction limit. > +*/ > + bool subxact_overflowed; > > All the variables inside this LocalPgBackendStatus structure are prefixed > with "backend" word. Can we do the same for the newly added variables as well? > > -- > > + * Get the xid and xmin, nsubxid and overflow status of the backend. The > > Should this be put as - "xid, xmin, nsubxid and overflow" instead of "xid and > xmin, nsubxid and overflow"? Thanks, Ashutosh, I will work on your comments and post an updated version by next week. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Checkpointer crashes with "PANIC: could not fsync file "pg_tblspc/.."

2021-12-21 Thread Dilip Kumar
at checkpointer.c:444 Note: This smaller test case is derived from one of the bigger scenarios raised by Neha Sharma [1] [1]https://www.postgresql.org/message-id/CANiYTQs0E8TcB11eU0C4eNN0tUd%3DSQqsqEtL1AVZP1%3DEnD-49A%40mail.gmail.com -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-21 Thread Dilip Kumar
r database test1 set tablespace tab1; > > --cancel the below command > alter database test1 set tablespace pg_default; --press ctrl+c > \c test1 > alter table t set tablespace tab1; > > > Log file attached for reference. Seems like this is an existing issue and I am able

Re: Checkpointer crashes with "PANIC: could not fsync file "pg_tblspc/.."

2021-12-21 Thread Dilip Kumar
lip -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-22 Thread Dilip Kumar
table files w.r.t the old tablespace, but those will get unlinked during the next checkpoint. Although the files must be truncated during commit itself but unlink might not have been processed until the next checkpoint. This is the explanation for the behavior you found during your investigation, but I haven't looked into the issue so I will do it latest by tomorrow and send my analysis. Thanks for working on this. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-22 Thread Dilip Kumar
ating this problem. One solution to this problem could be that, similar to mdpostckpt(), we invent one more function which takes dboid and dsttblspc oid as input and it will unlink all the requests which are w.r.t. the dboid and tablespaceoid, and before doing it we should also do ForgetDatabaseSyncRe

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-24 Thread Dilip Kumar
section which can throw an error and if we start calling the external hook then we lose that control. It should be blocked at the operation level itself e.g. ALTER TABLE READ ONLY, or by some other hook at a little higher level. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-25 Thread Dilip Kumar
On Sun, Dec 26, 2021 at 3:52 AM SATYANARAYANA NARLAPURAM < satyanarlapu...@gmail.com> wrote: > > > On Fri, Dec 24, 2021 at 3:13 AM Dilip Kumar wrote: > >> On Fri, Dec 24, 2021 at 3:27 AM SATYANARAYANA NARLAPURAM < >> satyanarlapu...@gmail.com> wrote: >>

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-25 Thread Dilip Kumar
ook at that level which can make it wait then we would also block any of the read operations needed to read from those buffers. I haven't thought what could be better way to do this but this is certainly not good. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread Dilip Kumar
ed insert WAL, and get committed but after that, it will be throttled. However, if the number of max connections will be very high then even after we detected a lag there a significant amount WAL could be generated, even if we keep long-running transactions aside. But I think still it w

Re: Per-table storage parameters for TableAM/IndexAM extensions

2021-12-29 Thread Dilip Kumar
tead of getting >parameters validated using “heap_reloptions” it will call the registered AM routine. Wrap these long commit message lines at 80 characters. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread Dilip Kumar
the other part that if you throttle after committing then the user can cancel the queries and generate more WAL from those sessions as well. But that is an extreme case where application developers want to bypass the throttling and want to generate more WALs. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-30 Thread Dilip Kumar
this might not be any better than throttling inside the commit. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Fix BUG #17335: Duplicate result rows in Gather node

2021-12-30 Thread Dilip Kumar
e as well if > its child is not parallel aware. > The Gather path will only be created if we have an underlying partial path, so I think if we are generating the append path only from the non-partial paths then we can see if the number of child nodes is just 1 then don't generate the p

Re: Skipping logical replication transactions on subscriber side

2022-01-04 Thread Dilip Kumar
ally, I am in favor of storing it in a catalog as that solution looks cleaner at least from the user pov but if we think there are a lot of complexities from the implementation pov then we might analyze the approach of storing in shmem as well. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Emit "checkpoint skipped because system is idle" message at LOG level if log_checkpoints is set

2022-01-04 Thread Dilip Kumar
> during a certain period of time when DEBUG1 messages aren't captured. > How about emitting the message at LOG level if log_checkpoints is set? > Patch attached. > > Thoughts? +1 to convert to LOG when log_checkpoints is set. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2022-01-05 Thread Dilip Kumar
't happen with important locks held / within a > critical section, but we still delay close to where we went over the maximum > lag. And the overhead should be fairly minimal. +1, this sounds like a really good idea to me. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2022-01-05 Thread Dilip Kumar
rs trying to update the same tuple or the DDL who want to modify the table definition so I don't think we have any issue here because anyway we don't want other writers to continue. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Make relfile tombstone files conditional on WAL level

2022-01-05 Thread Dilip Kumar
les to disappear, because they've got to be > previously-unliked files that Windows is still returning because > they're still opening -- or else they could be a sign of a corrupted > database, but there are no other possibilities. Yes, this approach will solve the problem for the WAL-logged ALTER DATABASE we are facing. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Make relfile tombstone files conditional on WAL level

2022-01-06 Thread Dilip Kumar
On Thu, Jan 6, 2022 at 1:12 PM Dilip Kumar wrote: > > > > > I think this idea is worth more consideration. It seems like 2^56 > > relfilenodes ought to be enough for anyone, recalling that you can > > only ever have 2^64 bytes of WAL. So if we do this, we can eliminate

Re: Add sub-transaction overflow status in pg_stat_activity

2022-01-13 Thread Dilip Kumar
On Fri, Dec 17, 2021 at 9:32 AM Justin Pryzby wrote: > On Fri, Dec 17, 2021 at 09:00:04AM +0530, Dilip Kumar wrote: > > On Tue, Dec 14, 2021 at 3:57 AM Bossart, Nathan > wrote: > > > > > > On 12/13/21, 6:30 AM, "Dilip Kumar" wrote: > > > > On

Re: Add sub-transaction overflow status in pg_stat_activity

2022-01-13 Thread Dilip Kumar
iables as well? Done + * Get the xid and xmin, nsubxid and overflow status of the backend. The Should this be put as - "xid, xmin, nsubxid and overflow" instead of "xid and xmin, nsubxid and overflow"? I missed to fix this one in the last patch so updated again. -- Regar

Re: Add sub-transaction overflow status in pg_stat_activity

2022-01-14 Thread Dilip Kumar
nsaction > overflow, and even with the information available in a monitoring view (I > had > to rely on a quick hackish extension as I couldn't patch postgres) it's not > terribly fun to do this way. On top of that log analyzers like pgBadger > could > h

Re: Error "initial slot snapshot too large" in create replication slot

2022-01-16 Thread Dilip Kumar
On Wed, Jan 12, 2022 at 4:09 PM Julien Rouhaud wrote: > Hi, > > On Tue, Nov 02, 2021 at 04:40:39PM +0530, Dilip Kumar wrote: > > > > I have fixed this, the updated patch is attached. > > The cfbot reports that this patch doesn't compile: > https://cirrus-ci.c

Re: Add sub-transaction overflow status in pg_stat_activity

2022-01-16 Thread Dilip Kumar
change. But I completely agree with you that if we can completely resolve this issue then there is no point in providing any such status or LOG. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-20 Thread Dilip Kumar
ER somecol SET COMPRESSION somealgo works. > > This sounds like a reasonable approach. The attached patch implements that. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com From 40b53e24932b1f9203092a7f6972804af5a7a45b Mon Sep 17 00:00:00 2001 From: Dilip Kumar Date: Sat, 20 Mar

Re: [HACKERS] Custom compression methods

2021-03-20 Thread Dilip Kumar
On Sat, Mar 20, 2021 at 1:22 PM Dilip Kumar wrote: > > On Sat, Mar 20, 2021 at 8:11 AM Robert Haas wrote: > > > > On Fri, Mar 19, 2021 at 8:25 PM Tom Lane wrote: > > > Extrapolating from the way we've dealt with similar issues > > > in the past, I thin

Re: [HACKERS] Custom compression methods

2021-03-20 Thread Dilip Kumar
, so I guess > the only option is using GetDefaultToastCompression(). Perhaps we might > include that into BrinOpcInfo too, in the future. Right, I think for now we can use default compression for this case. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical Replication vs. 2PC

2021-03-20 Thread Dilip Kumar
hat prepared transaction directly on the subscriber. Thoughts? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-20 Thread Dilip Kumar
e? [1] https://www.postgresql.org/message-id/CAFiTN-v7EULPqVJ-6J%3DzH6n0%2BkO%3DYFtgpte%2BFTre%3DWrwcWBBTA%40mail.gmail.com -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods (mac+lz4.h)

2021-03-20 Thread Dilip Kumar
e more robust. Yeah, we need to set the default_toast_compression in the beginning of the test as attached. Also, I see some diffs in the > indirect_toast test, which seems perhaps worthy of investigation. > (The diffs look to be just row ordering, but why?) I will look into this

Re: [HACKERS] Custom compression methods (mac+lz4.h)

2021-03-20 Thread Dilip Kumar
On Sun, Mar 21, 2021 at 9:10 AM Dilip Kumar wrote: > > On Sun, Mar 21, 2021 at 7:03 AM Tom Lane wrote: > > > > BTW, I tried doing "make installcheck" after having adjusted > > default_toast_compression to be "lz4". The compression test > > its

Re: [HACKERS] Custom compression methods (mac+lz4.h)

2021-03-21 Thread Dilip Kumar
n the first page as data are stored inline so the new tuple is stored on the next page and that is what affecting the order. I hope this makes sense. INSERT INTO indtoasttest(descr, f1, f2) VALUES('one-toasted,one-null', NULL, repeat('1234567890',5)); --

Re: [HACKERS] Custom compression methods (mac+lz4.h)

2021-03-22 Thread Dilip Kumar
compression method id stored in the compression header and * decompress the data slice using the appropriate decompression routine. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Dilip Kumar
ut then I found a > use for it. Yeah, it is used in toast_compress_datum, toast_get_compression_id, reform_and_rewrite_tuple and pg_column_compression function. Your patches look fine to me. I agree that v3-0006 also makes sense as it is simplifying the GUC handling. Thanks for fixing these. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods (mac+lz4.h)

2021-03-22 Thread Dilip Kumar
On Mon, Mar 22, 2021 at 8:11 PM Tom Lane wrote: > > Dilip Kumar writes: > > On Mon, Mar 22, 2021 at 5:22 AM Tom Lane wrote: > >> Also, after studying the documentation for LZ4_decompress_safe > >> and LZ4_decompress_safe_partial, I realized that liblz4 is als

Re: [HACKERS] Custom compression methods

2021-03-24 Thread Dilip Kumar
enerate_series(1, 256) g), version()); > """ > > Attached is a backtrace from current HEAD Thanks for reporting this issue. Actually, I missed setting the attcompression for the expression index and that is causing this assert. I will send a patch in some time. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-24 Thread Dilip Kumar
On Wed, Mar 24, 2021 at 1:43 PM Dilip Kumar wrote: > > > > """ > > create table t1 (col1 text, col2 text); > > create unique index on t1 ((col1 || col2)); > > insert into t1 values((select array_agg(md5(g::text))::text from > > generate_series(

Re: [HACKERS] Custom compression methods

2021-03-24 Thread Dilip Kumar
On Wed, Mar 24, 2021 at 2:49 PM Justin Pryzby wrote: > > On Wed, Mar 24, 2021 at 02:24:41PM +0530, Dilip Kumar wrote: > > On Wed, Mar 24, 2021 at 1:43 PM Dilip Kumar wrote: > > > > create table t1 (col1 text, col2 text); > > > > create unique index on t1 ((co

Re: [HACKERS] Custom compression methods

2021-03-24 Thread Dilip Kumar
On Wed, Mar 24, 2021 at 3:10 PM Dilip Kumar wrote: > > On Wed, Mar 24, 2021 at 2:49 PM Justin Pryzby wrote: > > > > On Wed, Mar 24, 2021 at 02:24:41PM +0530, Dilip Kumar wrote: > > > On Wed, Mar 24, 2021 at 1:43 PM Dilip Kumar wrote: > > > >

<    1   2   3   4   5   6   7   8   9   10   >