Hi Pulsar community, I would like to start a discussion on PIP-491: Prevent Delivery Stalls by Making the Client Return Exactly the Permits Used by the Broker.
PIP PR: https://github.com/apache/pulsar/pull/26336 Background and problem Pulsar consumer flow control depends on a simple relationship: when the broker uses P permits to deliver a CommandMessage, the client must eventually return exactly P permits. However, P is not currently carried in CommandMessage. The broker and client calculate related values independently from batch metadata and ack_set. These calculations normally agree for successful non-batched, complete-batch, and partial-batch delivery, but this agreement is an implementation coincidence rather than an explicit protocol contract. They can disagree in failure and race paths, including: - checksum, metadata, decompression, or batch-deserialization failures; - only part of a batch entering the client message lifecycle; - broker admission changing the final send set after an earlier count was calculated; - asynchronous permit returns racing with consumer recreation, including recreation on the same pooled ClientCnx; - an asynchronous broker write failure leaving permit debt on a consumer that remains registered. Returning too few permits progressively reduces the consumer's effective receiver capacity and can eventually stall Shared dispatch. Returning too many permits, or returning old credit to a replacement consumer, weakens receiver-side backpressure. Proposed design The PIP introduces an explicit per-command permit value P and makes it the common accounting value across the broker, protocol, and Java client: 1. The broker finalizes P once, after it knows which entries will actually be sent. 2. The broker reuses those finalized values for: - per-consumer permit accounting; - aggregate permit accounting in both persistent Shared dispatcher implementations; - CommandMessage serialization. 3. CommandMessage carries P in a new optional message_permits field. 4. The Java client treats P as a command-local permit budget: - permits transferred to successfully created messages are returned through their normal lifecycle; - permits not transferred because of skips or processing failures are returned immediately; - the two parts must add up to exactly P. 5. Returned credit is bound to a local broker-consumer incarnation, rather than only to ClientCnx. Delayed work from an old consumer therefore cannot grant permits to its replacement, even if the physical connection is reused. 6. An asynchronous message-write failure must remove the affected broker consumer, providing a terminal outcome for the failed send. Scope The initial exact guarantee is limited to persistent Shared delivery and the Java native-message path. The following areas are intentionally left for follow-up work: - custom MessagePayloadProcessor output; - encrypted and chunked message processing; - non-Java clients; - absolute permit reset or periodic synchronization; - broad consumer or dispatcher refactoring. These paths are not incompatible with the proposed permit-debt and consumer-incarnation model, but their additional ownership rules are not defined by this PIP. Compatibility The new message_permits field is optional: - an old client ignores the field sent by a new broker; - a new Java client falls back to the existing ack_set and payload-based calculation when connected to an old broker; - the complete guarantee is active when both the broker and Java client support the new contract. The proposal does not change the public consumer API, receiver-queue semantics, acknowledgment semantics, or the meaning of a permit. Implementation plan After the PIP is accepted, the implementation will be kept in one end-to-end PR so the complete invariant can be reviewed and tested together. The commits will remain separated into: 1. protocol and broker changes; 2. Java client changes and end-to-end validation. I would especially appreciate feedback on: - whether the broker should be the single authority for the per-command permit value; - whether removing the consumer is the correct terminal outcome for asynchronous write failure; - whether the broker-consumer incarnation boundary is sufficiently defined; - whether the proposed initial scope is appropriately narrow; - whether any important compatibility or failure scenario is missing. Thanks, void-ptr974
