On Thursday, November 20, 2025 10:50 PM Tomas Vondra wrote:
>
> On 11/20/25 14:10, wenhui qiu wrote:
> > Hi
> >> 1) The way the patch determines dependencies seems to be the "writeset"
> >> approach from other replication systems (e.g. MySQL does that). Maybe
> >> we should stick to the same nami
Hi Tomas
> discussing some of this stuff. I'm not saying we should copy all of
> this, but it seems like a good source of inspiration what (not) to do.
I'm not saying we should copy MySQL's implementation. MySQL’s parallel
replication is based on group commit, and PostgreSQL can’t directly adopt
th
On 11/20/25 14:10, wenhui qiu wrote:
> Hi
>> 1) The way the patch determines dependencies seems to be the "writeset"
>> approach from other replication systems (e.g. MySQL does that). Maybe we
>> should stick to the same naming?
>
>> OK, I did not research the design in MySQL in detail but will t
Hi
> 1) The way the patch determines dependencies seems to be the "writeset"
> approach from other replication systems (e.g. MySQL does that). Maybe we
> should stick to the same naming?
> OK, I did not research the design in MySQL in detail but will try to
analyze it.
I have some documents for m
On Thu, Nov 20, 2025 at 3:00 AM Tomas Vondra wrote:
>
> Hello Kuroda-san,
>
> On 11/18/25 12:00, Hayato Kuroda (Fujitsu) wrote:
> > Dear Amit,
> >
> >> It seems you haven't sent the patch that preserves commit order or the
> >> commit message of the attached patch is wrong. I think the first patch
On Thursday, November 20, 2025 5:31 AM Tomas Vondra wrote:
>
> Hello Kuroda-san,
>
> On 11/18/25 12:00, Hayato Kuroda (Fujitsu) wrote:
> > Dear Amit,
> >
> >> It seems you haven't sent the patch that preserves commit order or the
> >> commit message of the attached patch is wrong. I think the fi
Hello Kuroda-san,
On 11/18/25 12:00, Hayato Kuroda (Fujitsu) wrote:
> Dear Amit,
>
>> It seems you haven't sent the patch that preserves commit order or the
>> commit message of the attached patch is wrong. I think the first patch
>> in series should be the one that preserves commit order and the
Dear Amit,
> It seems you haven't sent the patch that preserves commit order or the
> commit message of the attached patch is wrong. I think the first patch
> in series should be the one that preserves commit order and then we
> can build a patch that tracks dependencies and allows parallelization
On Tue, Nov 18, 2025 at 1:46 PM Hayato Kuroda (Fujitsu)
wrote:
>
> Dear hackers,
>
> > I think it is better to enable preserve order by default - for safety
> > reasons.
>
> Per some discussions on -hackers, I implemented the patch which preserves the
> commit ordering on publisher. Let me clarif
Dear hackers,
> I think it is better to enable preserve order by default - for safety reasons.
Per some discussions on -hackers, I implemented the patch which preserves the
commit ordering on publisher. Let me clarify from the beginning.
Background
==
Current patch, say v1, does not pres
Sorry, I missed to attach files.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Dear Hackers,
> I measured the performance data for the shared hash table approach. Based on
> the result,
> local hash table approach seems better.
I did analyze bit more detail for tests. Let me share from the beginning...
Background and current implementation
==
Even if apply worker i
On Sat, Sep 6, 2025 at 10:33 AM Dilip Kumar wrote:
>
> On Wed, Aug 13, 2025 at 4:17 PM Zhijie Hou (Fujitsu)
> wrote:
>
> >
> > Here is the initial POC patch for this idea.
> >
> > The basic implementation is outlined below. Please note that there are
> > several
> > TODO items remaining, which w
On 17/09/2025 8:18 AM, Zhijie Hou (Fujitsu) wrote:
On Wednesday, September 17, 2025 2:40 AM Konstantin Knizhnik
wrote:
I think debug_logical_replication_streaming=immediate differs from real parallel
apply . It wasn't designed to simulate genuine parallel application because it
restricts para
On 17/09/2025 8:18 AM, Zhijie Hou (Fujitsu) wrote:
I think debug_logical_replication_streaming=immediate differs from real parallel
apply . It wasn't designed to simulate genuine parallel application because it
restricts parallelism by requiring the leader to wait for each transaction to
complet
On Wednesday, September 17, 2025 2:40 AM Konstantin Knizhnik
wrote:
> On 11/08/2025 7:45 AM, Amit Kapila wrote:
> > 4. Triggers and Constraints For the initial version, exclude tables with
> > user-defined triggers or constraints from parallel apply due to complexity
> > in
> > dependency detec
On 11/08/2025 7:45 AM, Amit Kapila wrote:
Hi,
4. Triggers and Constraints
For the initial version, exclude tables with user-defined triggers or
constraints from parallel apply due to complexity in dependency
detection. We may need some parallel-apply-safe marking to allow this.
I think that
On Mon, Sep 8, 2025 at 3:10 PM Ashutosh Bapat
wrote:
>
> On Sat, Sep 6, 2025 at 10:33 AM Dilip Kumar wrote:
> > On Wed, Aug 13, 2025 at 4:17 PM Zhijie Hou (Fujitsu)
> > wrote:
> > >
> > > Here is the initial POC patch for this idea.
> > >
> > >
> > > If no parallel apply worker is available, the
On Sat, Sep 13, 2025 at 9:49 PM Abhi Mehta wrote:
>
> Hi Amit,
>
>
> Really interesting proposal! I've been thinking through some of the
> implementation challenges:
>
>
> On the memory side: That hash table tracking RelationId and ReplicaIdentity
> could get pretty hefty under load. Maybe bloom
Hi Amit,
Really interesting proposal! I've been thinking through some of the
implementation challenges:
*On the memory side:* That hash table tracking RelationId and
ReplicaIdentity could get pretty hefty under load. Maybe bloom filters
could help with the initial screening? Also wondering
abo
On Sat, Sep 6, 2025 at 10:33 AM Dilip Kumar wrote:
> On Wed, Aug 13, 2025 at 4:17 PM Zhijie Hou (Fujitsu)
> wrote:
> >
> > Here is the initial POC patch for this idea.
> >
> >
> > If no parallel apply worker is available, the leader will apply the
> > transaction
> > independently.
>
> This type
Hello, Amit!
Amit Kapila :
> So, in such cases as we won't be able to detect
> transaction dependencies, it would be better to allow out-of-order
> commits optionally.
I think it is better to enable preserve order by default - for safety reasons.
I also checked the patch for potential issues lik
On Wed, Aug 13, 2025 at 4:17 PM Zhijie Hou (Fujitsu)
wrote:
>
> Here is the initial POC patch for this idea.
>
> The basic implementation is outlined below. Please note that there are several
> TODO items remaining, which we are actively working on; these are also
> detailed
> further down.
Tha
On Fri, Sep 5, 2025 at 2:59 PM Dilip Kumar wrote:
>
> On Mon, Aug 11, 2025 at 10:16 AM Amit Kapila wrote:
> >
>
> +1 for the idea. So I see we already have the parallel apply workers
> for the large streaming transaction so I am trying to think what
> additional problem we need to solve here. I
On Fri, Sep 5, 2025 at 5:15 PM Mihail Nikalayeu
wrote:
>
> Hello, Amit!
>
> Amit Kapila :
> > So, in such cases as we won't be able to detect
> > transaction dependencies, it would be better to allow out-of-order
> > commits optionally.
>
> I think it is better to enable preserve order by default
On Mon, Aug 11, 2025 at 10:16 AM Amit Kapila wrote:
>
> Hi,
>
> Background and Motivation
> -
> In high-throughput systems, where hundreds of sessions generate data
> on the publisher, the subscriber's apply process often becomes a
> bottleneck due to the single
Hi,
I ran tests to compare the performance of logical synchronous
replication with parallel-apply against physical synchronous
replication.
Highlights
===
On pgHead:(current behavior)
- With synchronous physical replication set to remote_apply, the
Primary’s TPS drops by ~60% (≈2.5x
On Mon, Aug 18, 2025 at 8:20 PM Konstantin Knizhnik wrote:
>
> On 18/08/2025 9:56 AM, Nisha Moond wrote:
> > On Wed, Aug 13, 2025 at 4:17 PM Zhijie Hou (Fujitsu)
> > wrote:
> >> Here is the initial POC patch for this idea.
> >>
> > Thank you Hou-san for the patch.
> >
> > I did some performance b
On 18/08/2025 9:56 AM, Nisha Moond wrote:
On Wed, Aug 13, 2025 at 4:17 PM Zhijie Hou (Fujitsu)
wrote:
Here is the initial POC patch for this idea.
Thank you Hou-san for the patch.
I did some performance benchmarking for the patch and overall, the
results show substantial performance improv
On Wed, Aug 13, 2025 at 4:17 PM Zhijie Hou (Fujitsu)
wrote:
>
> Here is the initial POC patch for this idea.
>
Thank you Hou-san for the patch.
I did some performance benchmarking for the patch and overall, the
results show substantial performance improvements.
Please find the details as follows
On Wed, Aug 13, 2025 at 8:57 PM Bruce Momjian wrote:
>
> On Wed, Aug 13, 2025 at 09:50:27AM +0530, Amit Kapila wrote:
> > On Tue, Aug 12, 2025 at 10:40 PM Bruce Momjian wrote:
> > > > Currently, PostgreSQL supports parallel apply only for large streaming
> > > > transactions (streaming=parallel).
On Wed, Aug 13, 2025 at 09:50:27AM +0530, Amit Kapila wrote:
> On Tue, Aug 12, 2025 at 10:40 PM Bruce Momjian wrote:
> > > Currently, PostgreSQL supports parallel apply only for large streaming
> > > transactions (streaming=parallel). This proposal aims to extend
> > > parallelism to non-streaming
On Monday, August 11, 2025 12:46 PM Amit Kapila wrote:
> Background and Motivation
> -
> In high-throughput systems, where hundreds of sessions generate data
> on the publisher, the subscriber's apply process often becomes a
> bottleneck due to the single apply
On Tue, Aug 12, 2025 at 9:22 PM Константин Книжник wrote:
>
> Hi,
> This is something similar to what I have in mind when starting my experiments
> with LR apply speed improvements. I think that maintaining a full
> (RelationId, ReplicaIdentity) hash may be too expensive - there can be
> hundr
On Tue, Aug 12, 2025 at 10:40 PM Bruce Momjian wrote:
>
> On Mon, Aug 11, 2025 at 10:15:41AM +0530, Amit Kapila wrote:
> > Hi,
> >
> > Background and Motivation
> > -
> > In high-throughput systems, where hundreds of sessions generate data
> > on the publisher,
On Mon, Aug 11, 2025 at 10:15:41AM +0530, Amit Kapila wrote:
> Hi,
>
> Background and Motivation
> -
> In high-throughput systems, where hundreds of sessions generate data
> on the publisher, the subscriber's apply process often becomes a
> bottleneck due to the
On 11.08.2025 7:45 AM, Amit Kapila wrote:
Hi,
Background and Motivation
-
In high-throughput systems, where hundreds of sessions generate data
on the publisher, the subscriber's apply process often becomes a
bottleneck due to the single apply worker model. Whil
On Mon, Aug 11, 2025 at 3:00 PM Kirill Reshke wrote:
>
> On Mon, 11 Aug 2025 at 13:45, Amit Kapila wrote:
> >
> > I am not sure if that is directly applicable because this work
> > proposes to track dependencies based on logical WAL contents. However,
> > if you can point me to README on the over
On Tue, Aug 12, 2025 at 12:04 PM Andrei Lepikhov wrote:
>
> On 11/8/2025 06:45, Amit Kapila wrote:
> > The core idea is that the leader apply worker ensures the following:
> > a. Identifies dependencies between transactions. b. Coordinates
> > parallel workers to apply independent transactions con
On 11/8/2025 06:45, Amit Kapila wrote:
The core idea is that the leader apply worker ensures the following:
a. Identifies dependencies between transactions. b. Coordinates
parallel workers to apply independent transactions concurrently. c.
Ensures correct ordering for dependent transactions.
Depe
On Mon, 11 Aug 2025 at 13:45, Amit Kapila wrote:
>
>
> I am not sure if that is directly applicable because this work
> proposes to track dependencies based on logical WAL contents. However,
> if you can point me to README on the overall design of the work you
> are pointing to then I can check i
On Mon, Aug 11, 2025 at 1:39 PM Kirill Reshke wrote:
>
>
> > Design Overview
> >
> > To safely parallelize non-streaming transactions, we must ensure that
> > transaction dependencies are respected to avoid failures and
> > deadlocks. Consider the following scenarios to un
Hi!
On Mon, 11 Aug 2025 at 09:46, Amit Kapila wrote:
>
> Hi,
>
> Background and Motivation
> -
> In high-throughput systems, where hundreds of sessions generate data
> on the publisher, the subscriber's apply process often becomes a
> bottleneck due to the sing
43 matches
Mail list logo
