wzhramc commented on PR #24704:
URL: https://github.com/apache/pulsar/pull/24704#issuecomment-3270531398
> > One potential future improvement could be to add support for transaction
batching. In transaction batching, multiple incoming messages would be
processed within the same transaction due to performance reasons so the number
of transactions can be reduced. There would need to be
`transactionBatchingMaxEntries` ("entry" would refer to an incoming batch
message which could contain multiple messages) and
`transactionBatchingQuietPeriodMs` configuration parameters to control this
type of batching. In most cases, it would be useful to configure
`transactionBatchingMaxEntries` to at least 1 so that batch index
acknowledgement state wouldn't have to be held across transactions. The
`transactionBatchingQuietPeriodMs` would control how long to wait until
"closing" the transaction batch in case the incoming messages don't flow in
continuously and `transactionBatchingMaxEntries` isn't hit.
>
> @wzhramc For high-scale use cases, it might be necessary to implement
"transaction batching" so that unnecessary load isn't added to the broker
(including the transaction coordinator) when enabling transactions. It might be
useful to cover the "transaction batching" aspect directly in PIP-439 if you'd
be ready to do that before proceeding to the vote. It's also possible to handle
that later in another PIP.
I'm not really familiar with batch processing in Pulsar functions in
general, so it would take a bit more time for me to figure it out... I would
eventually need help with that as well. It would be nice if someone could help
me out with this ^^'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
