On Mon, Nov 24, 2025 at 5:07 PM Amit Kapila <[email protected]> wrote:
> > While it might seem efficient for the producer (leader) to assist > > consumers (workers) when there's a limited number of consumers, I > > believe this isn't the best design. In such scenarios, it's generally > > better to allow the producer to focus solely on its primary task, > > unless there's a severe shortage of processing power. > > > > If computing resources are constrained, allowing producers to join > > consumers in applying changes is acceptable. However, if sufficient > > processing power is available, the producer should ideally be left to > > its own duties. The question then becomes: how do we make this > > decision? > > > > My suggestion is to make this a configurable parameter. Users could > > then decide whether the leader participates in applying changes. > > > > We could do this but another possibility is that the leader does > distribute some threshold of pending transactions (say 5 or 10) to > each of the workers and if none of the workers is still available then > it can perform the task by itself. IMHO making the producer (the leader) join as a consumer (an apply worker) is not the best default behavior for a single-producer, multi-consumer design. This design choice is generally not scalable because the producer is a unique resource no other process can handle its job while multiple parallel workers can act as consumers. By keeping the roles separate, a user always has the option to set up a sufficiently high number of dedicated consumer workers. However, in resource constrained environments where maximum resource utilization is prioritized over the most scalable solution, a configuration parameter could be introduced. This parameter would allow the producer to act as a consumer worker whenever it is free and other consumers are busy. This offers a trade-off between resource efficiency and overall scalability. I think this will avoid the system > performing poorly when the existing workers are waiting on each other > and or backend to finish the current transaction. The core issue is that integrating the producer (sender) as an extra consumer (apply worker) just adds an N+1 worker capacity, but doesn't fundamentally solve the problem of all workers eventually becoming busy or blocked (waiting on transactions) or am I missing something? The possibility remains that all N+1 workers could become busy applying or, more commonly, waiting for transactions to commit or resources to free up. Adding one extra worker doesn't resolve the underlying problem if the workload exceeds the total available processing power or if transactions are frequently waiting. Users already have the ability to address this by configuring N+1 or more dedicated consumer workers based on their resource availability and performance needs. Therefore, relying on the producer as an occasional consumer offers only a minor, temporary capacity gain and doesn't resolve the overall scalability limit or the likelihood of full worker saturation. Having said that, I > think this can be done as a separate optimization patch as well. Yeah we could. -- Regards, Dilip Kumar Google
