Hi Richard, I moderated your email onto the list. To participate please subscribe by sending an email to dev-subscr...@pulsar.apache.org and respond to the confirmation email as it instructs.
Regards, Dave Sent from my iPhone > On Feb 27, 2019, at 8:35 PM, Richard Yu <yohan.richard...@gmail.com> wrote: > > Hi all, > > I would like to create a PIP for issue #2664 on Github. The details of the > PIP are below. > I hope we could discuss this thoroughly. > > Cheers, > Richard > > PIP-31: Add support for transactional messaging > > Motivation: Pulsar currently could improve upon their system of sending > packets of data by implementing transactional messaging. This system > enforces eventual consistency within the system, and allows operations to > be performed atomically. > > Proposal: > > As described in the issue, we would implement the following policy in > Producer and Pulsar Broker: > 1. The producer produces the pre-processing transaction message. At this > point, the broker will set the status of this message to unknown. > 2. After the local transaction is successfully executed, the commit message > is sent, otherwise the rollback message is sent. > 3. The broker receives the message. If it is a commit message, it modifies > the transaction status to commit, and then sends an actual message to the > consumer queue. At this time, the consumer can consume the message. > Otherwise, the transaction status is modified to rollback. The message will > be discarded. > 4. If at step 2, the producer is down or abnormal, at this time, the broker > will periodically ask the specific producer for the status of the message, > and update the status according to the producer's response, and process it > according to step 3, the action that comes down. > > Specific concerns: > There are a number of things we will improve upon or add: > - A configuration called ```maxMessageUnknownTime```. Consider this > scenario: the pre-processing transaction message is sent, but the commit or > rollback message is never received, which could mean that the status of a > message would be permanently unknown. To avoid this from happening, we > would need a config which limits the amount of time the status of a message > could be unknown (i.e. ```maxMessageUnknownTime```) After that, the message > would be discarded. > - Logging would be updated to log the status of a message i.e. UNKNOWN, > ROLLBACK, or COMMITTED. This would allow the user to know whether or not a > message had failed or fallen through. > > Possible Additional API: > - We would add a method which allows the user to query the state of the > message i.e. ```getStateOfMessage(long id)```
