Hi all, I would like to create a PIP for issue #2664 on Github. The details of the PIP are below. I hope we could discuss this thoroughly.
Cheers, Richard PIP-31: Add support for transactional messaging Motivation: Pulsar currently could improve upon their system of sending packets of data by implementing transactional messaging. This system enforces eventual consistency within the system, and allows operations to be performed atomically. Proposal: As described in the issue, we would implement the following policy in Producer and Pulsar Broker: 1. The producer produces the pre-processing transaction message. At this point, the broker will set the status of this message to unknown. 2. After the local transaction is successfully executed, the commit message is sent, otherwise the rollback message is sent. 3. The broker receives the message. If it is a commit message, it modifies the transaction status to commit, and then sends an actual message to the consumer queue. At this time, the consumer can consume the message. Otherwise, the transaction status is modified to rollback. The message will be discarded. 4. If at step 2, the producer is down or abnormal, at this time, the broker will periodically ask the specific producer for the status of the message, and update the status according to the producer's response, and process it according to step 3, the action that comes down. Specific concerns: There are a number of things we will improve upon or add: - A configuration called ```maxMessageUnknownTime```. Consider this scenario: the pre-processing transaction message is sent, but the commit or rollback message is never received, which could mean that the status of a message would be permanently unknown. To avoid this from happening, we would need a config which limits the amount of time the status of a message could be unknown (i.e. ```maxMessageUnknownTime```) After that, the message would be discarded. - Logging would be updated to log the status of a message i.e. UNKNOWN, ROLLBACK, or COMMITTED. This would allow the user to know whether or not a message had failed or fallen through. Possible Additional API: - We would add a method which allows the user to query the state of the message i.e. ```getStateOfMessage(long id)```
