Re: Conflict Detection and Resolution

2024-06-06 Thread Dilip Kumar
On Thu, Jun 6, 2024 at 3:43 PM Amit Kapila wrote: > > On Wed, Jun 5, 2024 at 7:29 PM Dilip Kumar wrote: > > > > On Tue, Jun 4, 2024 at 9:37 AM Amit Kapila wrote: > > > > > > Can you share the use case of "earliest_timestamp_wins" resolution > >

Re: Remove dependency on VacuumPage(Hit/Miss/Dirty) counters in do_analyze_rel

2024-06-06 Thread Dilip Kumar
or notifying me. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Compress ReorderBuffer spill files using LZ4

2024-06-06 Thread Dilip Kumar
ave a very slow disk and a powerful processor, compression could be advantageous. Conversely, if the disk is very fast, the I/O savings might be minimal, and the compression overhead could outweigh the benefits. Additionally, the effectiveness of compression also depends on the compression ratio, which varies with the type of data being compressed. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical Replication of sequences

2024-06-06 Thread Dilip Kumar
entraut said here > > https://www.postgresql.org/message-id/42e5cb35-4aeb-4f58-8091-90619c7c3ecc%40eisentraut.org, > > this can be a good starting point to get going. > > > > Agreed. +1 -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: How about using dirty snapshots to locate dependent objects?

2024-06-06 Thread Dilip Kumar
ing from being dropped. Specifically, the create index operation should acquire a shared lock on the Access Method (AM) to ensure it doesn't get dropped concurrently while the transaction is still in progress. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: How about using dirty snapshots to locate dependent objects?

2024-06-06 Thread Dilip Kumar
On Thu, Jun 6, 2024 at 7:39 PM Ashutosh Sharma wrote: > > On Thu, Jun 6, 2024 at 6:20 PM Dilip Kumar wrote: >> >> On Thu, Jun 6, 2024 at 5:59 PM Ashutosh Sharma wrote: >> > >> > Hello everyone, >> > >> > At present, we use MVCC snapshots to

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-06 Thread Dilip Kumar
x27; costs > scale approximately linearly with the size of shared buffers, so being > able to skip those checkpoints (with strategy=WAL_LOG) will save a lot > of performance in the systems where this performance impact matters > most. I agree with you that we introduced the WAL_LOG

Re: How about using dirty snapshots to locate dependent objects?

2024-06-06 Thread Dilip Kumar
On Fri, Jun 7, 2024 at 11:53 AM Ashutosh Sharma wrote: > > On Fri, Jun 7, 2024 at 10:06 AM Dilip Kumar wrote: > > > > On Thu, Jun 6, 2024 at 7:39 PM Ashutosh Sharma > > wrote: > > > > > > On Thu, Jun 6, 2024 at 6:20 PM Dilip Kumar wrote: > > &

Re: Vacuum statistics

2024-06-06 Thread Dilip Kumar
ook good to me. -- + /* Tricky turn here: enforce pgstat to think that our database us dbid */ + + MyDatabaseId = dbid; typo /think that our database us dbid/think that our database has dbid Also, remove the blank line between the comment and the next code block that is related to that comment. -- Vacu

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-07 Thread Dilip Kumar
On Fri, Jun 7, 2024 at 11:57 AM Matthias van de Meent wrote: > > On Fri, 7 Jun 2024 at 07:18, Dilip Kumar wrote: > > > > On Wed, Jun 5, 2024 at 10:59 PM Matthias van de Meent > > wrote: > >> > >> On Wed, 5 Jun 2024 at 18:47, Ranier Vilela wro

Re: Compress ReorderBuffer spill files using LZ4

2024-06-07 Thread Dilip Kumar
ad of being controlled by a GUC. This way, we can decide on compression for each subscription individually rather than applying it to all subscribers. It makes more sense for the subscriber to control this, especially when we are planning to compress the data sent downstream. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Compress ReorderBuffer spill files using LZ4

2024-06-07 Thread Dilip Kumar
On Fri, Jun 7, 2024 at 2:39 PM Alvaro Herrera wrote: > > On 2024-Jun-07, Dilip Kumar wrote: > > > I think the compression option should be supported at the CREATE > > SUBSCRIPTION level instead of being controlled by a GUC. This way, we > > can decide on compre

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-07 Thread Dilip Kumar
On Fri, Jun 7, 2024 at 2:40 PM Matthias van de Meent wrote: > > On Fri, 7 Jun 2024 at 10:28, Dilip Kumar wrote: > > > > On Fri, Jun 7, 2024 at 11:57 AM Matthias van de Meent > > wrote: > >> > >> On Fri, 7 Jun 2024 at 07:18, Dilip Kumar wrote: &

Re: Conflict Detection and Resolution

2024-06-11 Thread Dilip Kumar
r conflict resolution. "Vector clocks" allow us to track the causal relationships between changes across nodes, ensuring that we can correctly order events and resolve conflicts in a manner that respects the "happens-before" relationship. This method helps maintain consistency and predictability in the system despite issues like clock skew. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical Replication of sequences

2024-06-11 Thread Dilip Kumar
. This > will facilitate rollback in case of any failures. I do not understand this point, you mean whenever we are fetching the sequence value from the publisher we need to create a new relfilenode on the subscriber? Why not just update the catalog tuple is sufficient? Or this is for handling the ALTER SEQUENCE case? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical Replication of sequences

2024-06-12 Thread Dilip Kumar
On Wed, Jun 12, 2024 at 4:08 PM vignesh C wrote: > > On Wed, 12 Jun 2024 at 10:51, Dilip Kumar wrote: > > > > On Tue, Jun 11, 2024 at 4:06 PM vignesh C wrote: > > > > > > Amit and I engaged in an offline discussion regarding the design and > >

Re: Conflict Detection and Resolution

2024-06-12 Thread Dilip Kumar
A will adjust its counter to X+1. This ensures that if NodeA has seen an operation from NodeB, its next operation will appear to have occurred after NodeB's operation. I admit that I haven't fully thought through how we could design such version tracking in our logical replication protocol or how it would fit into our system. However, my point is that we need to consider something beyond commit timestamps to achieve reliable ordering. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical Replication of sequences

2024-06-12 Thread Dilip Kumar
ut I am still not getting it completely, do you mean to say unless all the sequences are not synced any of the sequences would not be marked "ready" in pg_subscription_rel? Is that necessary? I mean why we can not sync the sequences one by one and mark them ready? Why it is necessary to either have all the sequences synced or none of them? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Logical Replication of sequences

2024-06-12 Thread Dilip Kumar
On Thu, Jun 13, 2024 at 11:53 AM vignesh C wrote: > > On Thu, 13 Jun 2024 at 10:27, Dilip Kumar wrote: > > Thanks for the explanation, but I am still not getting it completely, > > do you mean to say unless all the sequences are not synced any of the > > sequences would

Re: Conflict Detection and Resolution

2024-06-17 Thread Dilip Kumar
sider > > something beyond commit timestamps to achieve reliable ordering. > > > > I can't really respond to this as there's no suggestion how it would be > implemented in the patch discussed in this thread. > No worries, I'll consider whether finding such a solution is feasible for our situation. Thank you! -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict Detection and Resolution

2024-06-17 Thread Dilip Kumar
On Mon, Jun 17, 2024 at 3:23 PM Amit Kapila wrote: > > On Wed, Jun 12, 2024 at 10:03 AM Dilip Kumar wrote: > > > > On Tue, Jun 11, 2024 at 7:44 PM Tomas Vondra > > wrote: > > > > > > Yes, that's correct. However, many cases could benefit from the

Re: Conflict Detection and Resolution

2024-06-17 Thread Dilip Kumar
On Tue, Jun 18, 2024 at 10:17 AM Dilip Kumar wrote: > > On Mon, Jun 17, 2024 at 3:23 PM Amit Kapila wrote: > > > > On Wed, Jun 12, 2024 at 10:03 AM Dilip Kumar wrote: > > > > > > On Tue, Jun 11, 2024 at 7:44 PM Tomas Vondra > > > wrote: > > &

Re: Conflict Detection and Resolution

2024-06-17 Thread Dilip Kumar
result regardless of the order of T2, T3, and Vacuum. Here's how it looks in each case: Case 1: T1, T2, Vacuum, T3 -> Skip the update for a non-existing row -> end result we do not have a row. Case 2: T1, T2, T3 -> Skip the update for a deleted row -> end result

Re: Conflict Detection and Resolution

2024-06-18 Thread Dilip Kumar
On Tue, Jun 18, 2024 at 12:11 PM Amit Kapila wrote: > > On Tue, Jun 18, 2024 at 11:54 AM Dilip Kumar wrote: > > > > On Mon, Jun 17, 2024 at 8:51 PM Robert Haas wrote: > > > > > > On Mon, Jun 17, 2024 at 1:42 AM Amit Kapila > > > wrote: > > &

Re: Conflict Detection and Resolution

2024-06-19 Thread Dilip Kumar
On Tue, Jun 18, 2024 at 3:29 PM shveta malik wrote: > On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar wrote: > > I tried to work out a few scenarios with this, where the apply worker > will wait until its local clock hits 'remote_commit_tts - max_skew > permitted'. Please h

Re: Conflict Detection and Resolution

2024-06-19 Thread Dilip Kumar
On Wed, Jun 19, 2024 at 2:36 PM shveta malik wrote: > > On Wed, Jun 19, 2024 at 1:52 PM Dilip Kumar wrote: > > > > On Tue, Jun 18, 2024 at 3:29 PM shveta malik wrote: > > > On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar > > > wrote: > > > > &

Re: Conflict Detection and Resolution

2024-07-02 Thread Dilip Kumar
On Tue, Jul 2, 2024 at 2:40 PM shveta malik wrote: > > On Wed, Jun 19, 2024 at 1:52 PM Dilip Kumar wrote: > > > > On Tue, Jun 18, 2024 at 3:29 PM shveta malik wrote: > > > On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar > > > wrote: > > > > &

Re: Conflict Detection and Resolution

2024-07-02 Thread Dilip Kumar
have been committed yet. One simple option could be to prevent parallel workers from applying in-progress transactions when CDR is set up. Instead, we could let these transactions spill to files and only apply them once we receive the commit record. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict Detection and Resolution

2024-07-03 Thread Dilip Kumar
On Wed, Jul 3, 2024 at 12:30 PM Amit Kapila wrote: > > On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar wrote: > But waiting after applying the operations and before applying the > commit would mean that we need to wait with the locks held. That could > be a recipe for deadlocks in th

Re: Conflict Detection and Resolution

2024-07-03 Thread Dilip Kumar
On Wed, Jul 3, 2024 at 3:35 PM Amit Kapila wrote: > > On Wed, Jul 3, 2024 at 2:16 PM Dilip Kumar wrote: > > > > On Wed, Jul 3, 2024 at 12:30 PM Amit Kapila wrote: > > > > > > On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar wrote: > > > > > B

Re: Conflict Detection and Resolution

2024-07-03 Thread Dilip Kumar
On Wed, Jul 3, 2024 at 4:02 PM shveta malik wrote: > > On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar wrote: > > > > On Wed, Jul 3, 2024 at 11:00 AM shveta malik wrote: > > > > > > > Yes, I also think it should be independent of CDR. IMHO, it should be >

Re: Conflict Detection and Resolution

2024-07-03 Thread Dilip Kumar
On Wed, Jul 3, 2024 at 4:48 PM Amit Kapila wrote: > > On Wed, Jul 3, 2024 at 4:04 PM Dilip Kumar wrote: > > > > On Wed, Jul 3, 2024 at 3:35 PM Amit Kapila wrote: > > > > > > On Wed, Jul 3, 2024 at 2:16 PM Dilip Kumar wrote: > > > > >

Re: Conflict Detection and Resolution

2024-07-03 Thread Dilip Kumar
On Wed, Jul 3, 2024 at 5:08 PM shveta malik wrote: > > On Wed, Jul 3, 2024 at 4:12 PM Dilip Kumar wrote: > > > > On Wed, Jul 3, 2024 at 4:02 PM shveta malik wrote: > > > > > > On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar wrote: > > > > >

Re: Conflict Detection and Resolution

2024-07-04 Thread Dilip Kumar
large size transaction, the local clock should be able to catch up the remote commit_ts and we might not have to wait in most of the cases. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict Detection and Resolution

2024-07-05 Thread Dilip Kumar
On Fri, Jul 5, 2024 at 2:23 PM Amit Kapila wrote: > > On Fri, Jul 5, 2024 at 11:58 AM Dilip Kumar wrote: > > > > On Thu, Jul 4, 2024 at 5:37 PM Amit Kapila wrote: > > > > So, the situation will be the same. We can even > > > > > decide to spill the

Re: 'ERROR: attempted to update invisible tuple' from 'ALTER INDEX ... ATTACH PARTITION' on parent index

2023-07-11 Thread Dilip Kumar
lcache, which be out of date: >newtup = heap_copytuple(partedIdx->rd_indextuple); >((Form_pg_index) GETSTRUCT(newtup))->indisvalid = true; But why the recache entry is outdated, does that mean that in the previous command, we missed registering the invalidation for the re

Re: 'ERROR: attempted to update invisible tuple' from 'ALTER INDEX ... ATTACH PARTITION' on parent index

2023-07-12 Thread Dilip Kumar
On Wed, Jul 12, 2023 at 1:56 PM Michael Paquier wrote: > > On Wed, Jul 12, 2023 at 11:38:05AM +0530, Dilip Kumar wrote: > > On Wed, Jul 12, 2023 at 11:12 AM Michael Paquier > > wrote: > >> > >> On Wed, Jul 12, 2023 at 09:38:41AM +0900, Michael Paquier wro

Re: 'ERROR: attempted to update invisible tuple' from 'ALTER INDEX ... ATTACH PARTITION' on parent index

2023-07-12 Thread Dilip Kumar
ple from the relcache cannot be used to update the catalog tuple? Or do we actually need to update all the tuple header information as well in RelationReloadIndexInfo() in order to fix that invariant so that it can be used for catalog tuple updates as well? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: New WAL record to detect the checkpoint redo location

2023-08-15 Thread Dilip Kumar
f the need for exclusive inserts > here. If we rid of that, we could determine the redoa location based on the > LSN determined by the XLogInsert(). Yeah, good idea, actually we can do this insert outside of the exclusive insert lock and set the LSN of this insert as the checkpoint. redo

Re: New WAL record to detect the checkpoint redo location

2023-08-17 Thread Dilip Kumar
On Thu, Aug 17, 2023 at 10:52 AM Michael Paquier wrote: > > On Tue, Aug 15, 2023 at 02:23:43PM +0530, Dilip Kumar wrote: > > Yeah, good idea, actually we can do this insert outside of the > > exclusive insert lock and set the LSN of this insert as the > > checkpoint. redo

Re: New WAL record to detect the checkpoint redo location

2023-08-17 Thread Dilip Kumar
On Fri, Aug 18, 2023 at 5:24 AM Michael Paquier wrote: > > On Thu, Aug 17, 2023 at 01:11:50PM +0530, Dilip Kumar wrote: > > Yeah right, actually I was confused, I assumed it will return the > > start LSN of the record. And I do not see any easy way to identify > > the Sta

Re: Synchronizing slots from primary to standby

2023-08-23 Thread Dilip Kumar
alrep_worker_attach function. Please check other similar functions and reuse them wherever possible Also, why this function is not registering the cleanup function on shmmem_exit? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: New WAL record to detect the checkpoint redo location

2023-08-24 Thread Dilip Kumar
On Fri, Aug 18, 2023 at 10:12 AM Dilip Kumar wrote: > > On Fri, Aug 18, 2023 at 5:24 AM Michael Paquier wrote: > > > > On Thu, Aug 17, 2023 at 01:11:50PM +0530, Dilip Kumar wrote: > > > Yeah right, actually I was confused, I assumed it will return the > > > st

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-08-27 Thread Dilip Kumar
not sure whether this is a problem or could be just handled by documenting this behavior. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: New WAL record to detect the checkpoint redo location

2023-08-28 Thread Dilip Kumar
On Mon, Aug 28, 2023 at 5:14 AM Michael Paquier wrote: > > On Fri, Aug 25, 2023 at 11:08:25AM +0530, Dilip Kumar wrote: > > Here is the updated version of the patch. > > The concept of the patch looks sound to me. I have a few comments. Thanks for the review > +

Why don't we support external input/output functions for the composite types

2024-04-24 Thread Dilip Kumar
o define specific input/output syntax rather than conforming to what is accepted by the RECORD type. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Why don't we support external input/output functions for the composite types

2024-04-24 Thread Dilip Kumar
On Thu, Apr 25, 2024 at 10:14 AM Tom Lane wrote: > > Dilip Kumar writes: > > I'm curious about composite types in PostgreSQL. By default, when we > > create a composite type, it utilizes the "record_in" and "record_out" > > functions for inp

Re: New committers: Melanie Plageman, Richard Guo

2024-04-26 Thread Dilip Kumar
ss and few reverts! Congratulations to both of you. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict Detection and Resolution

2024-06-05 Thread Dilip Kumar
lly considered if this would cause any problems, but users might detect this issue. For instance, a client machine might send a change to node1 first and then, upon confirmation, send it to node2. If the clocks on node1 and node2 are not synchronized, the changes might appear in a different order. Does this seem like a potential problem? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Conflict Detection and Resolution

2024-06-05 Thread Dilip Kumar
are doing git push concurrently from multiple clients it is expected that the earliest commit wins. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-16 Thread Dilip Kumar
o not see any problem in using PGAlignedBlock so change that. > Although 0002 is formally a performance optimization, I'm inclined to > think that if we're going to commit it, it should also be back-patched > into v15, because letting the code diverge when we're not even out

Re: XLogBeginRead's header comment lies

2022-08-16 Thread Dilip Kumar
e > the LSN I'm using came from the xlogreader machinery. > > Thoughts? Yeah I think it makes sense to make it work as per the comment in XLogBeginRecord(). I think if we modify the Assert as per the comment of XLogBeginRecord() then the remaining code of the XLogDecodeNextRecor

Re: XLogBeginRead's header comment lies

2022-08-16 Thread Dilip Kumar
On Wed, Aug 17, 2022 at 11:18 AM Dilip Kumar wrote: > > On Tue, Aug 16, 2022 at 11:28 PM Robert Haas wrote: > > > > Yeah I think it makes sense to make it work as per the comment in > XLogBeginRecord(). I think if we modify the Assert as per the comment > of XLogBeginRec

Re: XLogBeginRead's header comment lies

2022-08-17 Thread Dilip Kumar
On Wed, Aug 17, 2022 at 11:31 AM Dilip Kumar wrote: > > On Wed, Aug 17, 2022 at 11:18 AM Dilip Kumar wrote: > > > > On Tue, Aug 16, 2022 at 11:28 PM Robert Haas wrote: > > > > > > > Yeah I think it makes sense to make it work as per the comment in >

Re: making relfilenodes 56 bits

2022-08-22 Thread Dilip Kumar
m missing > something. Okay, so this seems exactly the same as your previous proposal but instead of unconditionally rewriting all the system tables we will rewrite only those conflict with the user table or pg_largeobject from the previous cluster. Although it might have additional implementation complexity on the pg upgrade side, it seems cheaper than rewriting everything. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: making relfilenodes 56 bits

2022-08-22 Thread Dilip Kumar
On Tue, Aug 23, 2022 at 8:33 AM Dilip Kumar wrote: > > On Tue, Aug 23, 2022 at 1:46 AM Robert Haas wrote: > > > > On Mon, Aug 22, 2022 at 3:55 AM Amit Kapila wrote: > > > To solve that problem, how about rewriting the system table in the new > > > cluster wh

Re: making relfilenodes 56 bits

2022-08-23 Thread Dilip Kumar
ases? Shouldn't they behave as a normal user table? because before upgrade anyway new cluster can not have any table other than system tables and those tables created by an extension should also be restored as other user table does. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: standby promotion can create unreadable WAL

2022-08-23 Thread Dilip Kumar
OfRecoveryInfo->lastRecTLI && !XLogRecPtrIsInvalid(abortedRecPtr)) { Assert(!XLogRecPtrIsInvalid(missingContrecPtr)); CreateOverwriteContrecordRecord(abortedRecPtr, missingContrecPtr, newTLI); -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: making relfilenodes 56 bits

2022-08-25 Thread Dilip Kumar
On Tue, Aug 23, 2022 at 8:00 PM Robert Haas wrote: > > On Tue, Aug 23, 2022 at 2:06 AM Dilip Kumar wrote: > > OTOH, if we keep the two separate ranges for the user and system table > > then we don't need all this complex logic of conflict checking. > > True. That&#x

Re: Perform streaming logical transactions by background workers and parallel apply

2022-08-25 Thread Dilip Kumar
change the name. So I think "apply parallel worker" might look better and the file name could be "applyparallelworker.c" or just "parallelworker.c". Although "parallelworker.c" file name is a bit generic but we already have worker.c so w.r.t that "parallelworker.c" should just look fine. At least that is what I think. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Handle infinite recursion in logical replication setup

2022-08-25 Thread Dilip Kumar
think that giving an error is very inconvenient because we are already providing a new option "origin=NONE" so along with that lets force the user to choose between copy_data=off or copy_data=force and with that, there is no scope for mistakes. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: making relfilenodes 56 bits

2022-08-26 Thread Dilip Kumar
On Thu, Aug 25, 2022 at 5:26 PM Dilip Kumar wrote: > I agree on this that this system is easy to explain that we just > rewrite anything that conflicts so looks more future-proof. Okay, I > will try this solution and post the patch. While working on this solution I noticed one issue.

Re: standby promotion can create unreadable WAL

2022-08-26 Thread Dilip Kumar
abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; } -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: standby promotion can create unreadable WAL

2022-08-27 Thread Dilip Kumar
eral rule, it's good to > be specific, so let's use that. > > v2 attached. The patch LGTM, this patch will apply on master and v15. PFA patch for back branches. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com fix-contrecord-condition-v2_v14_v96.patch Description: Binary data

Re: standby promotion can create unreadable WAL

2022-08-29 Thread Dilip Kumar
On Fri, Aug 26, 2022 at 6:14 PM Dilip Kumar wrote: > > On Tue, Aug 23, 2022 at 12:06 AM Robert Haas wrote: > > > However, if anything > > did try to look at file #4 it would get confused. Maybe that can > > happen if this is a streaming standby, where we only write an

Re: making relfilenodes 56 bits

2022-08-30 Thread Dilip Kumar
On Fri, Aug 26, 2022 at 9:33 PM Robert Haas wrote: > > On Fri, Aug 26, 2022 at 7:01 AM Dilip Kumar wrote: > > While working on this solution I noticed one issue. Basically, the > > problem is that during binary upgrade when we try to rewrite a heap we >

Re: test_decoding assertion failure for the loss of top-sub transaction relationship

2022-09-01 Thread Dilip Kumar
else if (sub_needs_timetravel) { /* track toplevel txn as well, subxact alone isn't meaningful */ + needs_timetravel = true; SnapBuildAddCommittedTxn(builder, xid); } else if (needs_timetravel) -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: test_decoding assertion failure for the loss of top-sub transaction relationship

2022-09-01 Thread Dilip Kumar
edTxn(builder, xid); > } > - else if (sub_needs_timetravel) > - { > - /* track toplevel txn as well, subxact alone isn't meaningful > */ > - SnapBuildAddCommittedTxn(builder, xid); > - } > else if (needs_timetravel) > { > elog(DEBUG2, "forced transaction %u to do timetravel", xid); Yeah, I am fine with this as well. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-09-02 Thread Dilip Kumar
block by block there could be an issue if someone concurrently modifies the pg_class or there are some tuples that are inserted by the prepared transaction. But in this case, the same backend can not have an open prepared transaction while creating a database and that backend of course can not perform any parallel operation as well. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: making relfilenodes 56 bits

2022-09-03 Thread Dilip Kumar
riting stuff we have done is still intact. The problems I could think of are 1) only for a binary upgrade we will have to change the pg_dump. 2) we will have to use another database name as the reserved database name but what if that name is already in use in the previous cluster? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: making relfilenodes 56 bits

2022-09-03 Thread Dilip Kumar
On Sat, Sep 3, 2022 at 1:50 PM Dilip Kumar wrote: > > On Tue, Aug 30, 2022 at 9:23 PM Robert Haas wrote: > > > Well, that's very awkward. It doesn't seem like it would be very > > difficult to teach pg_upgrade to call pg_restore without --clean and > > just

Re: making relfilenodes 56 bits

2022-09-03 Thread Dilip Kumar
rm drop/create separately via commands (for create, we > need to generate the command as we are generating while generating the > dump in custom format), then rewrite the conflicting tables, and > finally restore the dump. Hmm, you are right. So I think something like this is possible to do, I will explore this more. Thanks for the idea. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Add 64-bit XIDs into PostgreSQL 15

2022-09-03 Thread Dilip Kumar
so consider green cfbot entry important. So PFA rebased > v43. Since we have converted TransactionId to 64-bit, so do we still need the concept of FullTransactionId? I mean it is really confusing to have 3 forms of transaction ids. i.e. Transaction Id, FullTransactionId and ShortTransactionId. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Add 64-bit XIDs into PostgreSQL 15

2022-09-04 Thread Dilip Kumar
On Sun, Sep 4, 2022 at 9:53 AM Dilip Kumar wrote: > > On Mon, Jul 18, 2022 at 2:54 PM Pavel Borisov wrote: > >> > >> > I can agree with you that sending rebased patches too often can be a > >> > little annoying. On the other hand, otherwise, it's ju

Re: making relfilenodes 56 bits

2022-09-06 Thread Dilip Kumar
On Sun, Sep 4, 2022 at 9:27 AM Dilip Kumar wrote: > > On Sat, Sep 3, 2022 at 5:11 PM Amit Kapila wrote: > > Isn't this happening because we are passing "--clean > > --create"/"--create" options to pg_restore in create_new_objects()? If > > so,

Re: SUBTRANS: Minimizing calls to SubTransSetParent()

2022-09-06 Thread Dilip Kumar
ransactionXmin. + */ +if (TransactionIdPrecedes(parentXid, cutoff_xid)) +{ +*xid = previousXid; +return true; +} +parentXid = SubTransGetParent(parentXid); Do we need this while loop if we are directly setting topxid as a parent, so with th

Re: making relfilenodes 56 bits

2022-09-08 Thread Dilip Kumar
On Tue, Sep 6, 2022 at 11:07 PM Robert Haas wrote: > On Tue, Sep 6, 2022 at 4:40 AM Dilip Kumar wrote: > > I have explored this area more and also tried to come up with a > > working prototype, so while working on this I realized that we would > > have almost to execute

Re: making relfilenodes 56 bits

2022-09-09 Thread Dilip Kumar
On Thu, Sep 8, 2022 at 4:10 PM Dilip Kumar wrote: > On a separate note, while reviewing the latest patch I see there is some risk > of using the unflushed relfilenumber in GetNewRelFileNumber() function. > Basically, in the current code, the flushing logic is tightly coupled wi

Re: SI-read predicate locks on materialized views

2022-09-09 Thread Dilip Kumar
record inserted in T2. Yes we do have write skew anomaly. I think the patch looks fine to me. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Error "initial slot snapshot too large" in create replication slot

2022-09-12 Thread Dilip Kumar
ger ->xid array. Yes, we can't do that in CurrentSnapshotData, but IIRC we > need to be in a transactional snapshot anyway, which is copied anyway? Yeah when I first found this issue, I thought that should be the solution. But later it went in a different direction. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Error "initial slot snapshot too large" in create replication slot

2022-09-12 Thread Dilip Kumar
On Tue, Sep 13, 2022 at 11:52 AM Kyotaro Horiguchi wrote: > > Thanks for raizing this up, Robert and the comment, Andres. > > At Tue, 13 Sep 2022 07:00:42 +0530, Dilip Kumar wrote > in > > On Tue, Sep 13, 2022 at 3:22 AM Andres Freund wrote: > > > > > > &

Re: making relfilenodes 56 bits

2022-09-21 Thread Dilip Kumar
On Tue, Sep 20, 2022 at 10:44 PM Robert Haas wrote: Thanks for the review, please see my response inline for some of the comments, rest all are accepted. > On Fri, Sep 9, 2022 at 6:02 AM Dilip Kumar wrote: > > [ new patch ] > > +typedef pg_int64 RelFileNumber; > > This s

Re: making relfilenodes 56 bits

2022-09-22 Thread Dilip Kumar
0. Maybe for this we can introduce RelFileNumberInValidRange() but I am not sure whether it would be cleaner than what we have now, so left as it is for now. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Let's make PostgreSQL multi-threaded

2023-06-06 Thread Dilip Kumar
#x27;t change, right? Even though all memory space is now shared across threads, we can maintain the same rules for modifying critical shared data structures, i.e. modifying such memory should still fall under the CRITICAL SECTION, so I guess the rules for promoting error level to PANIC will remain the same. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Let's make PostgreSQL multi-threaded

2023-06-06 Thread Dilip Kumar
agree in some cases debugging would be hard, but I feel there are cases where the thread model will make the debugging experience better e.g breaking at the entry point of the new parallel worker or other worker is hard with the process model but that would be very smooth with the thread model as pe

Re: Let's make PostgreSQL multi-threaded

2023-06-07 Thread Dilip Kumar
should be forked out of the supervisor process. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Let's make PostgreSQL multi-threaded

2023-06-11 Thread Dilip Kumar
significant > amount of extra RAM usage > -- the discussion did not yet touch various per-backend caches > (pg_catalog cache, statement cache) which are arguably easier to > implement in threaded model > -- TLB reload at each process switch is expensive and would be > mostly avo

Re: index prefetching

2023-06-12 Thread Dilip Kumar
nting this only for a bitmap scan. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

New WAL record to detect the checkpoint redo location

2023-06-15 Thread Dilip Kumar
://www.postgresql.org/message-id/CA%2BTgmoYOYZfMCyOXFyC-P%2B-mdrZqm5pP2N7S-r0z3_402h9rsA%40mail.gmail.com [2] https://www.postgresql.org/message-id/20230614194717.jyuw3okxup4cvtbt%40awork3.anarazel.de -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com v1-0001-New-WAL-record-to

Re: trying again to get incremental backup

2023-06-15 Thread Dilip Kumar
Will do. Adding me to CC: might help, I have a backlog unfortunately :(. Thanks, I have posted it here[1] [1] https://www.postgresql.org/message-id/CAFiTN-s-K%3DmVA%3DHPr_VoU-5bvyLQpNeuzjq1ebPJMEfCJZKFsg%40mail.gmail.com -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Skip collecting decoded changes of already-aborted transactions

2023-06-22 Thread Dilip Kumar
reaming in progress is enabled). Although this check is costly since we are planning only for large transactions then it is worth it if we can occasionally avoid disk or network I/O for the aborted transactions. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Improving btree performance through specializing by key shape, take 2

2023-06-23 Thread Dilip Kumar
ing? @@ -247,13 +256,16 @@ _bt_moveright(Relation rel, { + if (P_IGNORE(opaque) || + _bt_compare(rel, key, page, P_HIKEY, &cmpcol) >= cmpval) + { + *comparecol = 1; } -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Assert while autovacuum was executing

2023-06-23 Thread Dilip Kumar
#x27;page lock' and that's the fact. It holds true irrespective of whether it is being used currently or not. [1] /* * The relation extension or page lock conflict even between the group * members. */ if (LOCK_LOCKTAG(*lock) == LOCKTAG_RELATION_EXTEND || (LOCK_LOCKTAG(*lock) == LOCKTAG_PAGE)) { PROCLOCK_PRINT("LockCheckConflicts: conflicting (group)", proclock); return true; } -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Improving btree performance through specializing by key shape, take 2

2023-06-26 Thread Dilip Kumar
On Fri, Jun 23, 2023 at 8:16 PM Matthias van de Meent wrote: > > On Fri, 23 Jun 2023 at 11:26, Dilip Kumar wrote: > > > and I have one confusion in the > > below hunk in the _bt_moveright function, basically, if the parent > > page's right key is exactly matchi

Re: Improving btree performance through specializing by key shape, take 2

2023-06-26 Thread Dilip Kumar
On Tue, Jun 27, 2023 at 9:42 AM Dilip Kumar wrote: > > On Fri, Jun 23, 2023 at 8:16 PM Matthias van de Meent > wrote: > > > > On Fri, 23 Jun 2023 at 11:26, Dilip Kumar wrote: > > > > > > and I have one confusion in the > > > below hunk in the

Re: Disabling Heap-Only Tuples

2023-07-06 Thread Dilip Kumar
heap, the vacuum can not truncate any page. Is my understanding correct? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Disabling Heap-Only Tuples

2023-07-07 Thread Dilip Kumar
On Fri, Jul 7, 2023 at 3:48 PM Tomas Vondra wrote: > > On 7/7/23 11:55, Matthias van de Meent wrote: > > On Fri, 7 Jul 2023 at 06:53, Dilip Kumar wrote: > >> > >> On Fri, Jul 7, 2023 at 1:48 AM Matthias van de Meent > >> wrote: > >>> &

Re: CHECK Constraint Deferrable

2023-07-07 Thread Dilip Kumar
then deferring the constraint is the simplest way to insert without error. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Invalid Assert while validating REPLICA IDENTITY?

2024-09-01 Thread Dilip Kumar
idxoid || IsIndexUsableForReplicaIdentityFull(BuildIndexInfo(idxrel), edata->targetRel->attrmap)); index_close(idxrel, AccessShareLock); -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Invalid Assert while validating REPLICA IDENTITY?

2024-09-02 Thread Dilip Kumar
On Mon, Sep 2, 2024 at 3:32 PM Amit Kapila wrote: > > On Mon, Sep 2, 2024 at 11:21 AM Dilip Kumar wrote: > > > > While working on some other code I noticed that in > > FindReplTupleInLocalRel() there is an assert [1] that seems to be > > passing IndexRelation to Get

<    9   10   11   12   13   14   15   16   17   18   >