Let me try a different approach. Please see the definition of a Pulsar Transaction - https://pulsar.apache.org/docs/3.1.x/transactions/
If messages that are uncommitted are consumed that definition is no longer true. If breaking the definition is going to be allowed to a consumer then the producer who may abort the transaction ought to have given permission. You never know what content is in that transaction and it might be aborted at a user’s choice for any reason. If Kafka follows a different policy with transactions then perhaps you should look into Kafka on Pulsar / Starlight for Kafka protocol handlers. If we want Pulsar to allow for Kafka style then let’s be clear about the implications along with expectations. Best, Dave Sent from my iPhone > On Sep 18, 2023, at 6:41 PM, Xiangying Meng <xiangy...@apache.org> wrote: > > Hi Dave, > > I greatly appreciate your perspective, yet it leaves me with some > uncertainties that I am eager to address. Why would the introduction > of isolation levels constitute an insecure action? > >> I think if this proceeds then the scope needs to be expanded to >> producers/admins needing to proactively allow transactions to be consumed >> uncommitted. > > We are merely presenting an option to the users. The notion of > establishing isolation levels for producers and administrators is, in > my view, devoid of necessity, and I am not inclined to implement it, > for it is devoid of substance. > > Sincerely, > Xiangying > >> On Mon, Sep 18, 2023 at 10:58 PM Dave Fisher <wave4d...@comcast.net> wrote: >> >> Thanks. So, this is to support exfiltration of uncommitted transaction data? >> This is IMO wrong and a security risk. >> >> Pulsar already supports CDC through IO Connectors. >> >> Kafka can be wrong about these isolation levels. >> >> There is really no information in those Paimon issues. How is Paimon’s >> ability to support Pulsar broken by this edge case? >> >> Best, >> Dave >> >> Sent from my iPhone >> >>>> On Sep 18, 2023, at 7:26 AM, Xiangying Meng <xiangy...@apache.org> wrote: >>> >>> Hi Dave, >>> This is an external request. Paimon has added support for Kafka but >>> has not yet incorporated support for Pulsar. Therefore, the Paimon >>> community desires to integrate Pulsar. >>> >>> Furthermore, when integrating Pulsar into Paimon, it is desired to >>> enable the ability to configure isolation levels, similar to Kafka, to >>> support reading uncommitted transaction logs. >>> >>> Additional context can be found in the following link: >>> https://github.com/apache/incubator-paimon/issues/765 >>> >>> Sincerely, >>> Xiangying >>> >>>> On Mon, Sep 18, 2023 at 10:30 AM Dave Fisher <wave4d...@comcast.net> wrote: >>>> >>>> My concern is that this pip allows consumers to change the processing >>>> rules for transactions in ways that a producer might find unexpected. >>>> >>>> I think if this proceeds then the scope needs to be expanded to >>>> producers/admins needing to proactively allow transactions to be consumed >>>> uncommitted. >>>> >>>> I am also interested in the use case that motivates this change. >>>> >>>> Best, >>>> Dave >>>> >>>> Sent from my iPhone >>>> >>>>>> On Sep 17, 2023, at 8:50 AM, hzh0425 <hzh0...@apache.org> wrote: >>>>> >>>>> Hi, all >>>>> >>>>> This PR contributed to pip 298: >>>>> https://github.com/apache/pulsar/pull/21114 >>>>> >>>>> >>>>> >>>>> >>>>> This pip is to implement Read Committed and Read Uncommitted isolation >>>>> levels for Pulsar transactions, allow consumers to configure isolation >>>>> levels during the building process. >>>>> >>>>> For more details, please refer to pip-298.md >>>>> >>>>> >>>>> >>>>> >>>>> I hope everyone can help review and discuss this pip and enter the >>>>> discuss stage. >>>>> >>>>> Thanks >>>>> Zhangheng Huang >>>> >>
