Re: Parallel heap vacuum

2025-01-17 Thread Dilip Kumar
this patch set? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict detection for update_deleted in logical replication

2025-01-16 Thread Dilip Kumar
On Thu, Jan 16, 2025 at 4:02 PM Amit Kapila wrote: > On Thu, Jan 16, 2025 at 3:45 PM Dilip Kumar wrote: > > > > On Fri, Jan 3, 2025 at 4:31 PM Amit Kapila > wrote: > >> > >> On Thu, Jan 2, 2025 at 2:57 PM vignesh C wrote: > >> > > >>

Re: Conflict detection for update_deleted in logical replication

2025-01-16 Thread Dilip Kumar
E works. At least that's what my opinion about this case. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict detection for update_deleted in logical replication

2024-12-17 Thread Dilip Kumar
On Tue, Dec 17, 2024 at 8:54 AM Zhijie Hou (Fujitsu) wrote: > > On Monday, December 16, 2024 7:21 PM Dilip Kumar > wrote: > > > So IIUC in step 2) we send the message and get the list of all the > > transactions which are in the commit phase? What do you exactly mean by

Re: Conflict detection for update_deleted in logical replication

2024-12-16 Thread Dilip Kumar
ction we were interested on the publisher? At high level the overall idea looks promising to me but wanted to put more thought on lower level details about what transactions exactly we are waiting for and what WAL LSN we are waiting to get flushed. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Skip collecting decoded changes of already-aborted transactions

2024-12-14 Thread Dilip Kumar
he approach you followed in the current patch. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Skip collecting decoded changes of already-aborted transactions

2024-12-11 Thread Dilip Kumar
On Thu, Dec 12, 2024 at 11:08 AM Amit Kapila wrote: > > On Wed, Dec 11, 2024 at 8:21 AM Dilip Kumar wrote: > > > > On Wed, Dec 11, 2024 at 3:18 AM Masahiko Sawada > > wrote: > > > > > > On Mon, Dec 9, 2024 at 10:19 PM Dilip Kumar wrote: > > &

Re: Track the amount of time waiting due to cost_delay

2024-12-10 Thread Dilip Kumar
oo. > However, parallel workers report their sleep time no more frequently > than once per second, so the reported value may be slightly stale. > This description looks good. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Skip collecting decoded changes of already-aborted transactions

2024-12-10 Thread Dilip Kumar
On Wed, Dec 11, 2024 at 3:18 AM Masahiko Sawada wrote: > > On Mon, Dec 9, 2024 at 10:19 PM Dilip Kumar wrote: > > > > > > If the largest transaction is non-streamable, won't the transaction > > > returned by ReorderBufferLargestTXN() in the other case alr

Re: Fix bank selection logic in SLRU

2024-12-10 Thread Dilip Kumar
On Tue, 10 Dec 2024 at 6:32 PM, Andrey M. Borodin wrote: > > > > On 10 Dec 2024, at 15:39, Yura Sokolov wrote: > > > > It is not critical bug, since it doesn't hurt correctness just > performance. In worst case only one bank will be used. > > Ugh... yeah. IMO the problem is that we do not have p

Re: Fix bank selection logic in SLRU

2024-12-10 Thread Dilip Kumar
On Tue, 10 Dec 2024 at 7:30 PM, Robert Haas wrote: > On Tue, Dec 10, 2024 at 8:58 AM Dilip Kumar wrote: > >> Bank selection code assumes that number of buffers is power of 2. > >> If the number of buffers is not power of 2 - only subset of buffers > will be used. In wor

Re: Fix bank selection logic in SLRU

2024-12-10 Thread Dilip Kumar
On Tue, 10 Dec 2024 at 7:24 PM, Andrey M. Borodin wrote: > > > > On 10 Dec 2024, at 16:26, Dilip Kumar wrote: > > > > IIUC, we do check that it should be in multiple of bank size (i.e.) > > which is multiple of 2, right? Am I missing something? > > Bank s

Re: Fix bank selection logic in SLRU

2024-12-10 Thread Dilip Kumar
ck_errdetail("\"%s\" must be a multiple of %d", name, SLRU_BANK_SIZE); return false; } -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Skip collecting decoded changes of already-aborted transactions

2024-12-09 Thread Dilip Kumar
On Tue, Dec 10, 2024 at 11:09 AM Amit Kapila wrote: > > On Tue, Dec 10, 2024 at 10:59 AM Dilip Kumar wrote: > > > > On Tue, Nov 26, 2024 at 3:02 AM Masahiko Sawada > > wrote: > > > > > > > > I've attached a new version patch that incorpo

Re: Skip collecting decoded changes of already-aborted transactions

2024-12-09 Thread Dilip Kumar
we could repeat this process for a few more transactions. In contrast, it might be more efficient to simply discard the large aborted transaction, even if it’s not streamable, to avoid this issue. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Track the amount of time waiting due to cost_delay

2024-12-09 Thread Dilip Kumar
On Mon, Dec 9, 2024 at 10:11 PM Bertrand Drouvot wrote: > > Hi, > > On Mon, Dec 09, 2024 at 08:34:13PM +0530, Dilip Kumar wrote: > > On Mon, Dec 9, 2024 at 6:55 PM Bertrand Drouvot > > > Yeah. I think we could change the wording that way: > > > &

Re: Track the amount of time waiting due to cost_delay

2024-12-09 Thread Dilip Kumar
On Mon, Dec 9, 2024 at 6:55 PM Bertrand Drouvot wrote: > > Hi, > > On Mon, Dec 09, 2024 at 05:18:30PM +0530, Dilip Kumar wrote: > > On Mon, Dec 9, 2024 at 2:51 PM Masahiro Ikeda > > wrote: > > > > > > On 2024-12-06 18:31, Bertrand Drouvot wrote: > &

Re: Track the amount of time waiting due to cost_delay

2024-12-09 Thread Dilip Kumar
in a shared variable instead of local in individual workers and whenever the shared variable crosses WORKER_REPORT_DELAY_INTERVAL we can report this would avoid individual reporting from different workers. Since we are already computing the cost balance in shared variable i.e. VacuumSharedC

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-12-09 Thread Dilip Kumar
h an made a number of stylistic changes > to conform more closely to existing styles (adding back more rs_ > prefixes and typedef'ing BitmapHeapScanDescData * as > BitmapHeapScanDesc [despite how sad it makes me]). > > > BTW would it be a good idea for heapam_scan_bitmap_next_block to check > > the passed "scan" has SO_TYPE_BITMAPSCAN? We haven't been checking that > > so far I think, but we only had a single struct ... > > Good point. I've done this. > > I plan to commit 0002 and 0003 next week. I'm interested if you think > 0001 is correct. > I may also commit 0004-0006 as I feel they are ready too. Are we planning to commit this refactoring? I think this refactoring makes the overall code of BitmapHeapNext() quite clean and readable. I haven't read all patches but 0001-0006 including 0009 makes this code quite clean and readable. I like the refactoring of merging of the shared iterator and the private iterator also. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Question about read_stream_look_ahead()

2024-11-26 Thread Dilip Kumar
Can we merge it with the pending read? */ if (stream->pending_read_nblocks > 0 && stream->pending_read_blocknum + stream->pending_read_nblocks == blocknum) { stream->pending_read_nblocks++; continue; } -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Fix small typo, use InvalidRelFileNumber instead of InvalidOid

2024-11-10 Thread Dilip Kumar
On Sat, Nov 9, 2024 at 12:41 PM Dilip Kumar wrote: > > On Fri, Nov 8, 2024 at 8:36 PM Tom Lane wrote: >> >> Dilip Kumar writes: >> > IIRC, In catalog we intentionally left it as Oid because RelFileNumber is >> > an internal typedef bug, it is not an exposed d

Re: Fix small typo, use InvalidRelFileNumber instead of InvalidOid

2024-11-08 Thread Dilip Kumar
On Fri, Nov 8, 2024 at 8:36 PM Tom Lane wrote: > Dilip Kumar writes: > > IIRC, In catalog we intentionally left it as Oid because RelFileNumber is > > an internal typedef bug, it is not an exposed datatype, so probably we > can > > not use it in catalog. > > We cou

Re: Fix small typo, use InvalidRelFileNumber instead of InvalidOid

2024-11-08 Thread Dilip Kumar
On Fri, Nov 8, 2024 at 4:30 PM Alvaro Herrera wrote: > On 2024-Nov-08, Dilip Kumar wrote: > > > It appears that we are passing InvalidOid instead of InvalidRelFileNumber > > when calling index_create(). While this isn’t technically a bug, since > > InvalidRelFileNumber

Fix small typo, use InvalidRelFileNumber instead of InvalidOid

2024-11-08 Thread Dilip Kumar
It appears that we are passing InvalidOid instead of InvalidRelFileNumber when calling index_create(). While this isn’t technically a bug, since InvalidRelFileNumber is defined as InvalidOid, it’s preferable to use the correct identifier for clarity and consistency. -- Regards, Dilip Kumar

Re: Can rs_cindex be < 0 for bitmap heap scans?

2024-10-28 Thread Dilip Kumar
On Sat, Oct 26, 2024 at 5:30 PM Melanie Plageman wrote: > On Sat, Oct 26, 2024 at 12:17 AM Dilip Kumar > wrote: > > > > On Fri, Oct 25, 2024 at 10:07 PM Melanie Plageman < > melanieplage...@gmail.com> wrote: > >> > >> On Fri, Oct 25,

Re: Can rs_cindex be < 0 for bitmap heap scans?

2024-10-25 Thread Dilip Kumar
ly(scan->rs_inited)) { /* continue from previously returned page/tuple */ page = BufferGetPage(scan->rs_cbuf); lineindex = scan->rs_cindex + dir; if (ScanDirectionIsForward(dir)) --Refer definition of ScanDirection typedef enum ScanDirection { BackwardScanDirection = -1, NoMovemen

Re: Can rs_cindex be < 0 for bitmap heap scans?

2024-10-24 Thread Dilip Kumar
On Thu, Oct 24, 2024 at 7:11 PM Melanie Plageman wrote: > Thanks for the reply, Dilip! > > On Thu, Oct 24, 2024 at 12:50 AM Dilip Kumar > wrote: > > > > On Thu, Oct 24, 2024 at 3:45 AM Melanie Plageman < > melanieplage...@gmail.com> wrote: > >> >

Re: Can rs_cindex be < 0 for bitmap heap scans?

2024-10-24 Thread Dilip Kumar
lock. [1] @@ -378,6 +378,7 @@ initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock) ItemPointerSetInvalid(&scan->rs_ctup.t_self); scan->rs_cbuf = InvalidBuffer; scan->rs_cblock = InvalidBlockNumber; + scan->rs_cindex = 0; -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Make default subscription streaming option as Parallel

2024-10-19 Thread Dilip Kumar
On Fri, 18 Oct 2024 at 5:24 PM, Amit Kapila wrote: > On Fri, Oct 18, 2024 at 9:48 AM Dilip Kumar wrote: > > > > On Tue, Oct 8, 2024 at 3:38 PM Amit Kapila > wrote: > >> > >> On Tue, Oct 8, 2024 at 2:25 PM shveta malik > wrote: > >> > >

Re: Make default subscription streaming option as Parallel

2024-10-17 Thread Dilip Kumar
reasonable new defaults. I think we should make parallel streaming as > default and then wait for some user feedback before changing other > defaults. > > I agree, actually streaming of in progress transactions is a useful feature for performance in case of large transaction

Re: Invalid Assert while validating REPLICA IDENTITY?

2024-09-08 Thread Dilip Kumar
On Fri, Sep 6, 2024 at 4:48 PM vignesh C wrote: > > On Mon, 2 Sept 2024 at 18:22, Dilip Kumar wrote: > > > > On Mon, Sep 2, 2024 at 3:32 PM Amit Kapila wrote: > > > > > > On Mon, Sep 2, 2024 at 11:21 AM Dilip Kumar wrote: > > > > > >

Deadlock due to locking order violation while inserting into a leaf relation

2024-09-08 Thread Dilip Kumar
for query details. LOCATION: DeadLockReport, deadlock.c:1135 -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Invalid Assert while validating REPLICA IDENTITY?

2024-09-02 Thread Dilip Kumar
On Mon, Sep 2, 2024 at 3:32 PM Amit Kapila wrote: > > On Mon, Sep 2, 2024 at 11:21 AM Dilip Kumar wrote: > > > > While working on some other code I noticed that in > > FindReplTupleInLocalRel() there is an assert [1] that seems to be > > passing IndexRelation to Get

Invalid Assert while validating REPLICA IDENTITY?

2024-09-01 Thread Dilip Kumar
idxoid || IsIndexUsableForReplicaIdentityFull(BuildIndexInfo(idxrel), edata->targetRel->attrmap)); index_close(idxrel, AccessShareLock); -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict Detection and Resolution

2024-07-30 Thread Dilip Kumar
On Tue, Jul 30, 2024 at 4:56 PM shveta malik wrote: > > On Tue, Jul 30, 2024 at 4:04 PM Dilip Kumar wrote: > > > > On Fri, Jul 26, 2024 at 9:50 AM Ajin Cherian wrote: > > > > Comment in 0002, > > > > 1) I do not see any test case that set a proper co

Re: Conflict Detection and Resolution

2024-07-30 Thread Dilip Kumar
ode-1. But maybe it would be more difficult to get a consistent value if we are setting up a mess replication topology right? Maybe there I think a more advanced timestamp-based option would work better IMHO. I am doing code level review as well and will share my comments soon on 0003 and 0004

Re: Conflict detection and logging in logical replication

2024-07-30 Thread Dilip Kumar
On Tue, Jul 30, 2024 at 1:49 PM Zhijie Hou (Fujitsu) wrote: > > > On Monday, July 29, 2024 6:59 PM Dilip Kumar > > wrote: > > > > > > On Mon, Jul 29, 2024 at 11:44 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > >

Re: Conflict detection and logging in logical replication

2024-07-29 Thread Dilip Kumar
'ri_onConflictArbiterIndexes' and checking if any of these is a subset of the indexes that is returned by ExecInsertIndexTuples(). Why are we doing that, I think we can directly use the 'recheckIndexes' which is returned by ExecInsertIndexTuples(), and those index

Re: Conflict Detection and Resolution

2024-07-05 Thread Dilip Kumar
On Fri, Jul 5, 2024 at 2:23 PM Amit Kapila wrote: > > On Fri, Jul 5, 2024 at 11:58 AM Dilip Kumar wrote: > > > > On Thu, Jul 4, 2024 at 5:37 PM Amit Kapila wrote: > > > > So, the situation will be the same. We can even > > > > > decide to spill the

Re: Conflict Detection and Resolution

2024-07-04 Thread Dilip Kumar
large size transaction, the local clock should be able to catch up the remote commit_ts and we might not have to wait in most of the cases. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict Detection and Resolution

2024-07-03 Thread Dilip Kumar
On Wed, Jul 3, 2024 at 5:08 PM shveta malik wrote: > > On Wed, Jul 3, 2024 at 4:12 PM Dilip Kumar wrote: > > > > On Wed, Jul 3, 2024 at 4:02 PM shveta malik wrote: > > > > > > On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar wrote: > > > > >

Re: Conflict Detection and Resolution

2024-07-03 Thread Dilip Kumar
On Wed, Jul 3, 2024 at 4:48 PM Amit Kapila wrote: > > On Wed, Jul 3, 2024 at 4:04 PM Dilip Kumar wrote: > > > > On Wed, Jul 3, 2024 at 3:35 PM Amit Kapila wrote: > > > > > > On Wed, Jul 3, 2024 at 2:16 PM Dilip Kumar wrote: > > > > >

Re: Conflict Detection and Resolution

2024-07-03 Thread Dilip Kumar
On Wed, Jul 3, 2024 at 4:02 PM shveta malik wrote: > > On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar wrote: > > > > On Wed, Jul 3, 2024 at 11:00 AM shveta malik wrote: > > > > > > > Yes, I also think it should be independent of CDR. IMHO, it should be >

Re: Conflict Detection and Resolution

2024-07-03 Thread Dilip Kumar
On Wed, Jul 3, 2024 at 3:35 PM Amit Kapila wrote: > > On Wed, Jul 3, 2024 at 2:16 PM Dilip Kumar wrote: > > > > On Wed, Jul 3, 2024 at 12:30 PM Amit Kapila wrote: > > > > > > On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar wrote: > > > > > B

Re: Conflict Detection and Resolution

2024-07-03 Thread Dilip Kumar
On Wed, Jul 3, 2024 at 12:30 PM Amit Kapila wrote: > > On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar wrote: > But waiting after applying the operations and before applying the > commit would mean that we need to wait with the locks held. That could > be a recipe for deadlocks in th

Re: Conflict Detection and Resolution

2024-07-02 Thread Dilip Kumar
have been committed yet. One simple option could be to prevent parallel workers from applying in-progress transactions when CDR is set up. Instead, we could let these transactions spill to files and only apply them once we receive the commit record. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict Detection and Resolution

2024-07-02 Thread Dilip Kumar
On Tue, Jul 2, 2024 at 2:40 PM shveta malik wrote: > > On Wed, Jun 19, 2024 at 1:52 PM Dilip Kumar wrote: > > > > On Tue, Jun 18, 2024 at 3:29 PM shveta malik wrote: > > > On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar > > > wrote: > > > > &

Re: Conflict Detection and Resolution

2024-06-19 Thread Dilip Kumar
On Wed, Jun 19, 2024 at 2:36 PM shveta malik wrote: > > On Wed, Jun 19, 2024 at 1:52 PM Dilip Kumar wrote: > > > > On Tue, Jun 18, 2024 at 3:29 PM shveta malik wrote: > > > On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar > > > wrote: > > > > &

Re: Conflict Detection and Resolution

2024-06-19 Thread Dilip Kumar
On Tue, Jun 18, 2024 at 3:29 PM shveta malik wrote: > On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar wrote: > > I tried to work out a few scenarios with this, where the apply worker > will wait until its local clock hits 'remote_commit_tts - max_skew > permitted'. Please h

Re: Conflict Detection and Resolution

2024-06-18 Thread Dilip Kumar
On Tue, Jun 18, 2024 at 12:11 PM Amit Kapila wrote: > > On Tue, Jun 18, 2024 at 11:54 AM Dilip Kumar wrote: > > > > On Mon, Jun 17, 2024 at 8:51 PM Robert Haas wrote: > > > > > > On Mon, Jun 17, 2024 at 1:42 AM Amit Kapila > > > wrote: > > &

Re: Conflict Detection and Resolution

2024-06-17 Thread Dilip Kumar
result regardless of the order of T2, T3, and Vacuum. Here's how it looks in each case: Case 1: T1, T2, Vacuum, T3 -> Skip the update for a non-existing row -> end result we do not have a row. Case 2: T1, T2, T3 -> Skip the update for a deleted row -> end result

Re: Conflict Detection and Resolution

2024-06-17 Thread Dilip Kumar
On Tue, Jun 18, 2024 at 10:17 AM Dilip Kumar wrote: > > On Mon, Jun 17, 2024 at 3:23 PM Amit Kapila wrote: > > > > On Wed, Jun 12, 2024 at 10:03 AM Dilip Kumar wrote: > > > > > > On Tue, Jun 11, 2024 at 7:44 PM Tomas Vondra > > > wrote: > > &

Re: Conflict Detection and Resolution

2024-06-17 Thread Dilip Kumar
On Mon, Jun 17, 2024 at 3:23 PM Amit Kapila wrote: > > On Wed, Jun 12, 2024 at 10:03 AM Dilip Kumar wrote: > > > > On Tue, Jun 11, 2024 at 7:44 PM Tomas Vondra > > wrote: > > > > > > Yes, that's correct. However, many cases could benefit from the

Re: Conflict Detection and Resolution

2024-06-17 Thread Dilip Kumar
sider > > something beyond commit timestamps to achieve reliable ordering. > > > > I can't really respond to this as there's no suggestion how it would be > implemented in the patch discussed in this thread. > No worries, I'll consider whether finding such a solution is feasible for our situation. Thank you! -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical Replication of sequences

2024-06-12 Thread Dilip Kumar
On Thu, Jun 13, 2024 at 11:53 AM vignesh C wrote: > > On Thu, 13 Jun 2024 at 10:27, Dilip Kumar wrote: > > Thanks for the explanation, but I am still not getting it completely, > > do you mean to say unless all the sequences are not synced any of the > > sequences would

Re: Logical Replication of sequences

2024-06-12 Thread Dilip Kumar
ut I am still not getting it completely, do you mean to say unless all the sequences are not synced any of the sequences would not be marked "ready" in pg_subscription_rel? Is that necessary? I mean why we can not sync the sequences one by one and mark them ready? Why it is necessary to either have all the sequences synced or none of them? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict Detection and Resolution

2024-06-12 Thread Dilip Kumar
A will adjust its counter to X+1. This ensures that if NodeA has seen an operation from NodeB, its next operation will appear to have occurred after NodeB's operation. I admit that I haven't fully thought through how we could design such version tracking in our logical replication protocol or how it would fit into our system. However, my point is that we need to consider something beyond commit timestamps to achieve reliable ordering. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical Replication of sequences

2024-06-12 Thread Dilip Kumar
On Wed, Jun 12, 2024 at 4:08 PM vignesh C wrote: > > On Wed, 12 Jun 2024 at 10:51, Dilip Kumar wrote: > > > > On Tue, Jun 11, 2024 at 4:06 PM vignesh C wrote: > > > > > > Amit and I engaged in an offline discussion regarding the design and > >

Re: Logical Replication of sequences

2024-06-11 Thread Dilip Kumar
. This > will facilitate rollback in case of any failures. I do not understand this point, you mean whenever we are fetching the sequence value from the publisher we need to create a new relfilenode on the subscriber? Why not just update the catalog tuple is sufficient? Or this is for handling the ALTER SEQUENCE case? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict Detection and Resolution

2024-06-11 Thread Dilip Kumar
r conflict resolution. "Vector clocks" allow us to track the causal relationships between changes across nodes, ensuring that we can correctly order events and resolve conflicts in a manner that respects the "happens-before" relationship. This method helps maintain consistency and predictability in the system despite issues like clock skew. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-07 Thread Dilip Kumar
On Fri, Jun 7, 2024 at 2:40 PM Matthias van de Meent wrote: > > On Fri, 7 Jun 2024 at 10:28, Dilip Kumar wrote: > > > > On Fri, Jun 7, 2024 at 11:57 AM Matthias van de Meent > > wrote: > >> > >> On Fri, 7 Jun 2024 at 07:18, Dilip Kumar wrote: &

Re: Compress ReorderBuffer spill files using LZ4

2024-06-07 Thread Dilip Kumar
On Fri, Jun 7, 2024 at 2:39 PM Alvaro Herrera wrote: > > On 2024-Jun-07, Dilip Kumar wrote: > > > I think the compression option should be supported at the CREATE > > SUBSCRIPTION level instead of being controlled by a GUC. This way, we > > can decide on compre

Re: Compress ReorderBuffer spill files using LZ4

2024-06-07 Thread Dilip Kumar
ad of being controlled by a GUC. This way, we can decide on compression for each subscription individually rather than applying it to all subscribers. It makes more sense for the subscriber to control this, especially when we are planning to compress the data sent downstream. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-07 Thread Dilip Kumar
On Fri, Jun 7, 2024 at 11:57 AM Matthias van de Meent wrote: > > On Fri, 7 Jun 2024 at 07:18, Dilip Kumar wrote: > > > > On Wed, Jun 5, 2024 at 10:59 PM Matthias van de Meent > > wrote: > >> > >> On Wed, 5 Jun 2024 at 18:47, Ranier Vilela wro

Re: Vacuum statistics

2024-06-06 Thread Dilip Kumar
ook good to me. -- + /* Tricky turn here: enforce pgstat to think that our database us dbid */ + + MyDatabaseId = dbid; typo /think that our database us dbid/think that our database has dbid Also, remove the blank line between the comment and the next code block that is related to that comment. -- Vacu

Re: How about using dirty snapshots to locate dependent objects?

2024-06-06 Thread Dilip Kumar
On Fri, Jun 7, 2024 at 11:53 AM Ashutosh Sharma wrote: > > On Fri, Jun 7, 2024 at 10:06 AM Dilip Kumar wrote: > > > > On Thu, Jun 6, 2024 at 7:39 PM Ashutosh Sharma > > wrote: > > > > > > On Thu, Jun 6, 2024 at 6:20 PM Dilip Kumar wrote: > > &

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-06 Thread Dilip Kumar
x27; costs > scale approximately linearly with the size of shared buffers, so being > able to skip those checkpoints (with strategy=WAL_LOG) will save a lot > of performance in the systems where this performance impact matters > most. I agree with you that we introduced the WAL_LOG

Re: How about using dirty snapshots to locate dependent objects?

2024-06-06 Thread Dilip Kumar
On Thu, Jun 6, 2024 at 7:39 PM Ashutosh Sharma wrote: > > On Thu, Jun 6, 2024 at 6:20 PM Dilip Kumar wrote: >> >> On Thu, Jun 6, 2024 at 5:59 PM Ashutosh Sharma wrote: >> > >> > Hello everyone, >> > >> > At present, we use MVCC snapshots to

Re: How about using dirty snapshots to locate dependent objects?

2024-06-06 Thread Dilip Kumar
ing from being dropped. Specifically, the create index operation should acquire a shared lock on the Access Method (AM) to ensure it doesn't get dropped concurrently while the transaction is still in progress. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical Replication of sequences

2024-06-06 Thread Dilip Kumar
entraut said here > > https://www.postgresql.org/message-id/42e5cb35-4aeb-4f58-8091-90619c7c3ecc%40eisentraut.org, > > this can be a good starting point to get going. > > > > Agreed. +1 -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Compress ReorderBuffer spill files using LZ4

2024-06-06 Thread Dilip Kumar
ave a very slow disk and a powerful processor, compression could be advantageous. Conversely, if the disk is very fast, the I/O savings might be minimal, and the compression overhead could outweigh the benefits. Additionally, the effectiveness of compression also depends on the compression ratio, which varies with the type of data being compressed. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Remove dependency on VacuumPage(Hit/Miss/Dirty) counters in do_analyze_rel

2024-06-06 Thread Dilip Kumar
or notifying me. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict Detection and Resolution

2024-06-06 Thread Dilip Kumar
On Thu, Jun 6, 2024 at 3:43 PM Amit Kapila wrote: > > On Wed, Jun 5, 2024 at 7:29 PM Dilip Kumar wrote: > > > > On Tue, Jun 4, 2024 at 9:37 AM Amit Kapila wrote: > > > > > > Can you share the use case of "earliest_timestamp_wins" resolution > >

Remove dependency on VacuumPage(Hit/Miss/Dirty) counters in do_analyze_rel

2024-06-06 Thread Dilip Kumar
attached patch addresses the issue by eliminating the need to track VacuumPage(Hit/Miss/Dirty) counters in do_analyze_rel(), making the global counters obsolete. This simplifies the code and improves consistency. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com v1-0001-Remove

Re: Conflict Detection and Resolution

2024-06-05 Thread Dilip Kumar
are doing git push concurrently from multiple clients it is expected that the earliest commit wins. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict Detection and Resolution

2024-06-05 Thread Dilip Kumar
lly considered if this would cause any problems, but users might detect this issue. For instance, a client machine might send a change to node1 first and then, upon confirmation, send it to node2. If the clocks on node1 and node2 are not synchronized, the changes might appear in a different order. Does this seem like a potential problem? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: New committers: Melanie Plageman, Richard Guo

2024-04-26 Thread Dilip Kumar
ss and few reverts! Congratulations to both of you. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Why don't we support external input/output functions for the composite types

2024-04-24 Thread Dilip Kumar
On Thu, Apr 25, 2024 at 10:14 AM Tom Lane wrote: > > Dilip Kumar writes: > > I'm curious about composite types in PostgreSQL. By default, when we > > create a composite type, it utilizes the "record_in" and "record_out" > > functions for inp

Why don't we support external input/output functions for the composite types

2024-04-24 Thread Dilip Kumar
o define specific input/output syntax rather than conforming to what is accepted by the RECORD type. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-04-03 Thread Dilip Kumar
missed acquiring the bank lock w.r.t. intervening pages, thanks for reporting. Your fix looks correct to me. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-14 Thread Dilip Kumar
t; See attached patch, which is the same as previous patch with some extra > assertions. Maybe for now we can just handle this specific case to remove the snapshot serializing for the BitmapHeapScan as you are doing in the patch. After looking into the code your theory seems corre

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-13 Thread Dilip Kumar
On Wed, Mar 13, 2024 at 9:25 PM Robert Haas wrote: > > On Wed, Mar 13, 2024 at 11:39 AM Dilip Kumar wrote: > > > Andres already commented on the snapshot stuff on an earlier patch > > > version, and that's much nicer with this version. However, I don't > >

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-13 Thread Dilip Kumar
ain. Then I checked other low scan nodes like indexscan and seqscan and it seems we are doing the same things there as well. Check for SerializeSnapshot() in table_parallelscan_initialize() and index_parallelscan_initialize() which are being called from ExecSeqScanInitializeDSM() and ExecIndexScanIni

Re: Streaming I/O, vectored I/O (WIP)

2024-03-12 Thread Dilip Kumar
Maybe there is some nice > way to solve that, but I thought this way was the simplest (and if > there is some instruction-cache-locality/tight-loop/perf reason why we > should work harder to find ranges of hits, it could be for later). > Does that make sense? Understood, I think this makes sense. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Streaming I/O, vectored I/O (WIP)

2024-03-11 Thread Dilip Kumar
is not true and pass that blockNum into the smgrprefetch() as a first block right? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PostgreSQL Contributors Updates

2024-03-04 Thread Dilip Kumar
son > * Dean Rasheed > * John Naylor > * Melanie Plageman > * Nathan Bossart > > Thank you and congratulations to all! > Congratulations to all! -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: pgsql: Improve performance of subsystems on top of SLRU

2024-03-03 Thread Dilip Kumar
re comparing that lock against 'MultiXactMemberCtl' related bank lock. This makes code simpler because now in the loop we are sure that we are always holding the lock but I do not like comparing the bank locks for 2 different SLRUs, although there is no problem as there would not be a common lock address, anyway, I do not have any strong objection to what you have done here. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-27 Thread Dilip Kumar
ble_buffers 16 16 - total buffers 88 112 so that is < 200kB of extra memory on a minimally configured system, IMHO this should not matter. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-27 Thread Dilip Kumar
On Mon, Feb 26, 2024 at 9:46 PM Alvaro Herrera wrote: > On 2024-Feb-23, Dilip Kumar wrote: > > + > + For each SLRU area that's part of the core server, > + there is a configuration parameter that controls its size, with the > suffix > + _buffers appended. Fo

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-23 Thread Dilip Kumar
On Fri, Feb 23, 2024 at 1:06 PM Dilip Kumar wrote: > > On Fri, Feb 23, 2024 at 1:48 AM Alvaro Herrera > wrote: > > > > On 2024-Feb-07, Dilip Kumar wrote: > > > > > On Wed, Feb 7, 2024 at 3:49 PM Alvaro Herrera > > > wrote: > > > > >

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-22 Thread Dilip Kumar
On Fri, Feb 23, 2024 at 1:48 AM Alvaro Herrera wrote: > > On 2024-Feb-07, Dilip Kumar wrote: > > > On Wed, Feb 7, 2024 at 3:49 PM Alvaro Herrera > > wrote: > > > > Sure, but is that really what we want? > > > > So your question is do we want these bu

Re: logical decoding and replication of sequences, take 2

2024-02-21 Thread Dilip Kumar
On Wed, Feb 21, 2024 at 2:52 PM Robert Haas wrote: > > On Wed, Feb 21, 2024 at 2:43 PM Dilip Kumar wrote: > > So the problem is that we might consider the transaction change as > > non-transaction and mark this flag as true. > > But it's not "might" righ

Re: logical decoding and replication of sequences, take 2

2024-02-21 Thread Dilip Kumar
On Wed, Feb 21, 2024 at 1:24 PM Robert Haas wrote: > > On Wed, Feb 21, 2024 at 1:06 PM Dilip Kumar wrote: > > > But I am wondering why this flag is always set to true in > > > DecodeTXNNeedSkip() irrespective of the commit or abort. Because the > > > aborted tr

Re: logical decoding and replication of sequences, take 2

2024-02-20 Thread Dilip Kumar
On Tue, Feb 20, 2024 at 4:32 PM Dilip Kumar wrote: > > On Tue, Feb 20, 2024 at 3:38 PM Robert Haas wrote: > > > Let's say fast_forward is true. Then smgr_decode() is going to skip > > recording anything about the relfilenode, so we'll identify all > > seque

Re: logical decoding and replication of sequences, take 2

2024-02-20 Thread Dilip Kumar
commit status. On another thought, can there be a situation where we have identified this flag wrongly as non-transaction and set this flag, and the commit/abort record never appeared in the WAL so never decoded? That can also lead to an incorrect decision during the upgrade. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: logical decoding and replication of sequences, take 2

2024-02-20 Thread Dilip Kumar
d this problem and also [2] where we concluded why this is not creating a real problem. [1] https://www.postgresql.org/message-id/CAFiTN-vAx-Y%2B19ROKOcWnGf7ix2VOTUebpzteaGw9XQyCAeK6g%40mail.gmail.com [2] https://www.postgresql.org/message-id/CAFiTN-sYpyUBabxopJysqH3DAp4OZUCTi6m_qtgt8d32vDcWSA

Re: Synchronizing slots from primary to standby

2024-02-07 Thread Dilip Kumar
anyway I > think we can even extend it after v1 is done. > Okay, I am fine with that. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-07 Thread Dilip Kumar
On Wed, Feb 7, 2024 at 3:49 PM Alvaro Herrera wrote: > > On 2024-Feb-07, Dilip Kumar wrote: > > > On Tue, Feb 6, 2024 at 8:55 PM Alvaro Herrera > > wrote: > > > > > > I made CLOGShmemBuffers, CommitTsShmemBuffers and SUBTRANSShmemBuffers > > > co

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-06 Thread Dilip Kumar
saction_buffers should make sense for the user that it stores transaction-related data in some buffers pool but whether that buffer pool is called SLRU or not doesn't matter much to the user IMHO. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-06 Thread Dilip Kumar
round > * bugs elsewhere in SLRU handling, so we don't care if we read a slightly > * outdated value; therefore we don't add a memory barrier. > */ > > Pushed with those changes. Thank you! Yeah, this looks perfect, thanks. > Now I'll go rebase the rest of the patch on top. Okay, I will review and test after that. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

  1   2   3   4   5   6   7   8   9   10   >