[VOTE] PIP-302 Add new API readAllExistingMessages for TableView
Hi dev, This thread is to start a vote for PIP-302 Add new API readAllExistingMessages for TableView. Discuss thread: https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q PIP: https://github.com/apache/pulsar/pull/21166 BR, Xiangying
Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar
Hi Donglai, Heesung >brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute=60 is the same as brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec=1 So, the "per-min" config can be more granular. I have some doubts about introducing the `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` configuration. Firstly, I also think that a minute is too long. If we want the maximum concurrency per second to be 1, and set the maximum concurrency per minute to 60, then the actual maximum concurrency per second could be up to 60, which is 60 times larger than our expected maximum concurrency. Moreover, if the unload requests are concentrated in the last 10 seconds of the previous minute and the first 10 seconds of the next minute, then the concurrency during this period will exceed our configuration. Such fluctuations are inevitable, but the larger the time span we set, the greater the distortion of the configuration. Secondly, we already have the `maxConcurrentUnloadPerSec` configuration, which is provided to the user in the CLI. Adding a similar `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` configuration might confuse users. When designing configuration parameters, we should try to keep it simple and consistent, and avoid introducing unnecessary complexity. Thanks, Xiangying On Mon, Sep 25, 2023 at 12:14 PM Yubiao Feng wrote: > > Hi Donglai, Mattison > > I agree with @Mattison > > Thanks > Yubiao Feng > > On Mon, Aug 21, 2023 at 8:50 PM wrote: > > > > > Hi, > > > > I agree with this change to improve the stability of the pulsar cluster. > > > > Just one concern. it's better to move this discussion to a new PIP. > > because you wanna introduce a new broker configuration. > > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` > > > > FYI: https://github.com/apache/pulsar/blob/master/pip/README.md > > > > Looking forward this change and thanks for your contribution. :) > > > > Best, > > Mattison > > > > > > > > On 7 Jul 2023 at 15:30 +0800, labuladong , wrote: > > > Thanks you guys. > > > > > > > > > I agree that per-minute is better than per-second, which is more > > flexible. > > > > > > > > > I open an issue here: > > > > > > > > > https://github.com/apache/pulsar/issues/20753 > > > > > > > > > Regards, > > > donglai > >
Re: [VOTE] PIP-302 Add new API readAllExistingMessages for TableView
Hi, Xiangying This PIP is a little confusing to me. This mail title states that we are introducing `readAllExistingMessages`. But actually, we only introduced `refreshAsync` in the PIP: https://github.com/apache/pulsar/pull/21166/files#diff-45c655583d6c0c73d87afd3df3fe67f77caadbf1bd691cf8f8211cc89728a1ceR34-R36 And the PR title doesn't seem relevant. “PIP-302 Add alwaysRefresh Configuration Option for TableView to Read Latest Values” BR, Zike Yang On Mon, Sep 25, 2023 at 3:25 PM Xiangying Meng wrote: > > Hi dev, >This thread is to start a vote for PIP-302 Add new API > readAllExistingMessages for TableView. > Discuss thread: > https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q > PIP: https://github.com/apache/pulsar/pull/21166 > > BR, > Xiangying
Re: [DISCUSS] Release Pulsar 3.1.1
+1 Zike Yang On Mon, Sep 25, 2023 at 2:55 PM Enrico Olivelli wrote: > > +1 > > Enrico > > Il giorno lun 25 set 2023 alle ore 05:08 Yubiao Feng > ha scritto: > > > > +1 > > > > Thanks > > Yubiao Feng > > > > On Mon, Sep 18, 2023 at 1:32 PM guo jiwei wrote: > > > > > Hi all, > > > > > > I would like to propose releasing the Pulsar 3.1.1. > > > > > > It's over one month since the release of 3.1.0 and there are 56 new > > > commits > > > in branch-3.1: > > > https://github.com/apache/pulsar/compare/v3.1.0...branch-3.1 > > > > > > We need to cut a new release. Please let me know if you have any > > > important fixes that need to be included in Pulsar 3.1.1. > > > > > > > > > Regards > > > Jiwei Guo (Tboy) > > >
Re: [VOTE] PIP-302 Add new API readAllExistingMessages for TableView
Thank you for your reminder. In our discussion, there were several changes to the specific plan and method names, which resulted in the PR title not being updated promptly. This was my oversight. The email title for the vote was not modified to match the titles of the discussed emails. Regarding my proposal itself, would you happen to have any other questions? BR, Xiangying On Mon, Sep 25, 2023 at 6:03 PM Zike Yang wrote: > > Hi, Xiangying > > This PIP is a little confusing to me. > This mail title states that we are introducing `readAllExistingMessages`. > But actually, we only introduced `refreshAsync` in the PIP: > https://github.com/apache/pulsar/pull/21166/files#diff-45c655583d6c0c73d87afd3df3fe67f77caadbf1bd691cf8f8211cc89728a1ceR34-R36 > And the PR title doesn't seem relevant. “PIP-302 Add alwaysRefresh > Configuration Option for TableView to Read Latest Values” > > BR, > Zike Yang > > On Mon, Sep 25, 2023 at 3:25 PM Xiangying Meng wrote: > > > > Hi dev, > >This thread is to start a vote for PIP-302 Add new API > > readAllExistingMessages for TableView. > > Discuss thread: > > https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q > > PIP: https://github.com/apache/pulsar/pull/21166 > > > > BR, > > Xiangying
Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar
> If we want the maximum concurrency per second to be 1, and set the maximum concurrency per minute to 60, then the actual maximum concurrency per second could be up to 60, which is 60 times larger than our expected maximum concurrency. I think the RateLimiter can handle it: https://github.com/apache/pulsar/blob/a1405ea006f175b1bd0b9d28b9444d592fb4a010/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L965-L968 > Secondly, we already have the `maxConcurrentUnloadPerSec` configuration, which is provided to the user in the CLI. Adding a similar `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` configuration might confuse users. `maxConcurrentUnloadPerSec ` is for the admin and CLI usage. This proposal is to add `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute ` to the broker configuration. Overall, I'm +1 for this proposal. And I agree that we need a new PIP for this change. BR, Zike Yang On Mon, Sep 25, 2023 at 3:54 PM Xiangying Meng wrote: > > Hi Donglai, Heesung > > >brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute=60 is the same as > brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec=1 So, the "per-min" > config can be more granular. > > I have some doubts about introducing the > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` > configuration. > > Firstly, I also think that a minute is too long. If we want the > maximum concurrency per second to be 1, and set the maximum > concurrency per minute to 60, then the actual maximum concurrency per > second could be up to 60, which is 60 times larger than our expected > maximum concurrency. Moreover, if the unload requests are concentrated > in the last 10 seconds of the previous minute and the first 10 seconds > of the next minute, then the concurrency during this period will > exceed our configuration. Such fluctuations are inevitable, but the > larger the time span we set, the greater the distortion of the > configuration. > > Secondly, we already have the `maxConcurrentUnloadPerSec` > configuration, which is provided to the user in the CLI. Adding a > similar `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` > configuration might confuse users. When designing configuration > parameters, we should try to keep it simple and consistent, and avoid > introducing unnecessary complexity. > > Thanks, > Xiangying > > On Mon, Sep 25, 2023 at 12:14 PM Yubiao Feng > wrote: > > > > Hi Donglai, Mattison > > > > I agree with @Mattison > > > > Thanks > > Yubiao Feng > > > > On Mon, Aug 21, 2023 at 8:50 PM wrote: > > > > > > > > Hi, > > > > > > I agree with this change to improve the stability of the pulsar cluster. > > > > > > Just one concern. it's better to move this discussion to a new PIP. > > > because you wanna introduce a new broker configuration. > > > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` > > > > > > FYI: https://github.com/apache/pulsar/blob/master/pip/README.md > > > > > > Looking forward this change and thanks for your contribution. :) > > > > > > Best, > > > Mattison > > > > > > > > > > > > On 7 Jul 2023 at 15:30 +0800, labuladong , wrote: > > > > Thanks you guys. > > > > > > > > > > > > I agree that per-minute is better than per-second, which is more > > > flexible. > > > > > > > > > > > > I open an issue here: > > > > > > > > > > > > https://github.com/apache/pulsar/issues/20753 > > > > > > > > > > > > Regards, > > > > donglai > > >
Re: [DISCUSS] PIP-300: Add custom dynamic configuration for plugins
Hi, Zixuan Thanks for your proposal. I'm +1 for it. > This is a feature I need. If cherry-pick is not allowed, then it will only be kept in 3.2+. This is a new feature, and I also think that we couldn't cherry-pick it. What about cherry-picking this change to your fork repo and building the pulsar for your own to meet this need? Does it make sense to you? BR, Zike Yang On Mon, Sep 25, 2023 at 12:23 AM mattison chao wrote: > > Hi, Zixuan > > I am afraid I can't support you in cherry-picking this feature for all of the > active branches by the current fact. Because this is a new feature and it's > not a bug fix or security issue. > > We can wait for other contributor's ideas. WDYT? > > Thanks, > Mattison > On 19 Sep 2023 at 10:42 +0800, Zixuan Liu , wrote: > > > 1. When you want to cherry-pick a PR, it should be cherry-picked for all > > > branches after the target branch. Isn't it? > > > > I think we should cherry-pick this PR to Pulsar 2.10, 2.11, 3.0. > > > > > 2. I've got a slight concern about cherry-picking. Do you have any issue > > > or feature request in the community that can demonstrate this feature is > > > required to cherry-pick? > > > > This is a feature I need. If cherry-pick is not allowed, then it will > > only be kept in 3.2+. > > > > Thanks, > > Zixuan > > > > mattison chao 于2023年9月18日周一 09:42写道： > > > > > > > > HI, Zixuan > > > > > > Thanks for your discussion. It's a great feature. But I've got several > > > questions I want to discuss here. > > > > > > 1. When you want to cherry-pick a PR, it should be cherry-picked for all > > > branches after the target branch. Isn't it? > > > 2. I've got a slight concern about cherry-picking. Do you have any issue > > > or feature request in the community that can demonstrate this feature is > > > required to cherry-pick? > > > > > > > > > Best, > > > Mattison > > > On 12 Sep 2023 at 11:25 +0800, Zixuan Liu , wrote: > > > > > BTW, I think we can cherry-pick this feature to the Pulsar 2.10. As > > > > > far as I know, the Pulsar 2.10 is a stable/main version, because many > > > > > users are using it. > > > > > > > > > > Thanks, > > > > > Zixuan > > > > > > > > > > Zixuan Liu 于2023年9月5日周二 17:09写道： > > > > > > > > > > > > > > Hi Pulsar Community, > > > > > > > > > > > > > > Discuss for PIP-300: https://github.com/apache/pulsar/pull/21127 > > > > > > > > > > > > > > Thanks, > > > > > > > Zixuan
Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar
>I think the RateLimiter can handle it: https://github.com/apache/pulsar/blob/a1405ea006f175b1bd0b9d28b9444d592fb4a010/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L965-L968 See here. Do you mean we make `maxConcurrentUnloadPerSec` and `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` work together? What is the meaning of this? https://github.com/apache/pulsar/blob/a1405ea006f175b1bd0b9d28b9444d592fb4a010/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java#L564-L568 >`maxConcurrentUnloadPerSec ` is for the admin and CLI usage. This proposal is to add `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute ` to the broker configuration. Yes, we are adding a broker configuration. That's not an issue, but adding `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` with a minute-based interval and expecting it to work in conjunction with `maxConcurrentUnloadPerSec`, which is only used in the CLI, doesn't make sense. I understand that what we want to achieve is to have a broker configuration that limits the concurrency of unloads even when `maxConcurrentUnloadPerSec` is not set. Instead of adding `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` when `maxConcurrentUnloadPerSec` is already controlling the concurrency Thanks, Xiangying On Mon, Sep 25, 2023 at 6:45 PM Zike Yang wrote: > > > If we want the > maximum concurrency per second to be 1, and set the maximum > concurrency per minute to 60, then the actual maximum concurrency per > second could be up to 60, which is 60 times larger than our expected > maximum concurrency. > > I think the RateLimiter can handle it: > https://github.com/apache/pulsar/blob/a1405ea006f175b1bd0b9d28b9444d592fb4a010/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L965-L968 > > > Secondly, we already have the `maxConcurrentUnloadPerSec` > configuration, which is provided to the user in the CLI. Adding a > similar `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` > configuration might confuse users. > > `maxConcurrentUnloadPerSec ` is for the admin and CLI usage. This > proposal is to add > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute ` to the > broker configuration. > > > Overall, I'm +1 for this proposal. And I agree that we need a new PIP > for this change. > > BR, > Zike Yang > > On Mon, Sep 25, 2023 at 3:54 PM Xiangying Meng wrote: > > > > Hi Donglai, Heesung > > > > >brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute=60 is the same as > > brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec=1 So, the "per-min" > > config can be more granular. > > > > I have some doubts about introducing the > > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` > > configuration. > > > > Firstly, I also think that a minute is too long. If we want the > > maximum concurrency per second to be 1, and set the maximum > > concurrency per minute to 60, then the actual maximum concurrency per > > second could be up to 60, which is 60 times larger than our expected > > maximum concurrency. Moreover, if the unload requests are concentrated > > in the last 10 seconds of the previous minute and the first 10 seconds > > of the next minute, then the concurrency during this period will > > exceed our configuration. Such fluctuations are inevitable, but the > > larger the time span we set, the greater the distortion of the > > configuration. > > > > Secondly, we already have the `maxConcurrentUnloadPerSec` > > configuration, which is provided to the user in the CLI. Adding a > > similar `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` > > configuration might confuse users. When designing configuration > > parameters, we should try to keep it simple and consistent, and avoid > > introducing unnecessary complexity. > > > > Thanks, > > Xiangying > > > > On Mon, Sep 25, 2023 at 12:14 PM Yubiao Feng > > wrote: > > > > > > Hi Donglai, Mattison > > > > > > I agree with @Mattison > > > > > > Thanks > > > Yubiao Feng > > > > > > On Mon, Aug 21, 2023 at 8:50 PM wrote: > > > > > > > > > > > Hi, > > > > > > > > I agree with this change to improve the stability of the pulsar cluster. > > > > > > > > Just one concern. it's better to move this discussion to a new PIP. > > > > because you wanna introduce a new broker configuration. > > > > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` > > > > > > > > FYI: https://github.com/apache/pulsar/blob/master/pip/README.md > > > > > > > > Looking forward this change and thanks for your contribution. :) > > > > > > > > Best, > > > > Mattison > > > > > > > > > > > > > > > > On 7 Jul 2023 at 15:30 +0800, labuladong , > > > > wrote: > > > > > Thanks you guys. > > > > > > > > > > > > > > > I agree that per-minute is better than per-second, which is more > > > > flexible. > > > > > > > > > > > > > > > I open an issue here: > > > > > > > > > > > > > > > https://github.com/apache/pulsar/issues/20753 > > > > > > > > > > >
[VOTE] PIP-298 Consumer supports specifying consumption isolation level
Hi dev, This thread is to start a vote for PIP-298 Consumer supports specifying consumption isolation level Discuss thread: https://lists.apache.org/thread/8ny0qtp7m9qcdbvnfjdvpnkc4c5ssyld https://lists.apache.org/thread/2opqjof83425vry6gzszd5glqgryrv11 PIP: https://github.com/apache/pulsar/pull/21114 BR, hzh
Re: [VOTE] PIP-298 Consumer supports specifying consumption isolation level
-1 (binding) I’m not convinced that breaking transaction isolation is the proper course of action. Regards, Dave > On Sep 25, 2023, at 6:46 AM, hzh0425 wrote: > > Hi dev, > This thread is to start a vote for PIP-298 Consumer supports specifying > consumption isolation level > Discuss thread: > https://lists.apache.org/thread/8ny0qtp7m9qcdbvnfjdvpnkc4c5ssyld > > https://lists.apache.org/thread/2opqjof83425vry6gzszd5glqgryrv11 > > PIP: https://github.com/apache/pulsar/pull/21114 > > BR, > hzh
Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level
> On Sep 20, 2023, at 12:50 AM, Xiangying Meng wrote: > > Hi, all, > > Let's consider another example: > > **System**: Financial Transaction System > > **Operations**: Large volume of deposit and withdrawal operations, a > small number of transfer operations. > > **Roles**: > > - **Client A1** > - **Client A2** > - **User Account B1** > - **User Account B2** > - **Request Topic C** > - **Real-time Monitoring System D** > - **Business Processing System E** > > **Client Operations**: > > - **Withdrawal**: Client A1 decreases the deposit amount from User > Account B1 or B2. > - **Deposit**: Client A1 increases the deposit amount in User Account B1 or > B2. > - **Transfer**: Client A2 decreases the deposit amount from User > Account B1 and increases it in User Account B2. Or vice versa. > > **Real-time Monitoring System D**: Obtains the latest data from > Request Topic C as quickly as possible to monitor transaction data and > changes in bank reserves in real-time. This is necessary for the > timely detection of anomalies and real-time decision-making. > > **Business Processing System E**: Reads data from Request Topic C, > then actually operates User Accounts B1, B2. > > **User Scenario**: Client A1 sends a large number of deposit and > withdrawal requests to Request Topic C. Client A2 writes a small > number of transfer requests to Request Topic C. > > In this case, Business Processing System E needs a read-committed > isolation level to ensure operation consistency and Exactly Once > semantics. The real-time monitoring system does not care if a small > number of transfer requests are incomplete (dirty data). What it > cannot tolerate is a situation where a large number of deposit and > withdrawal requests cannot be presented in real time due to a small > number of transfer requests (the current situation is that uncommitted > transaction messages can block the reading of committed transaction > messages). So you are willing to let uncommitted transactions impact actual users bank accounts? Are you sure that there is not another way to bypass uncommitted records? Letting uncommitted records through is not Exactly once. Are you ready to rewrite Pulsar’s documentation to explain how normal users can avoid allowing this? Best, Dave > > In this case, it is necessary to set different isolation levels for > different consumers/subscriptions. > > Thanks, > Xiangying > > On Tue, Sep 19, 2023 at 11:35 PM 杨国栋 wrote: >> >> Hi Dave and Xiangying, >> Thanks for all your support. >> >> Let me add some background. >> >> Apache Paimon take message queue as External Log Systems and changelog of >> Paimon can also be consumed from message queue. >> By default, change-log of message queue in Paimon are visible to consumers >> only after a snapshot. Snapshot have a same life cycle as message queue >> transactions. >> However, users can immediately consume change-log by read uncommited >> message without waiting for the next snapshot. >> This behavior reduces the latency of changelog, but it relies on reading >> uncommited message in Kafka or other message queue. >> So we hope Pulsar can support Read Uncommitted isolation level. >> >> Put aside the application scenarios of Paimon. Let's discuss Read >> Uncommitted isolation level itself. >> >> Read Uncommitted isolation will bring certain security risks, but will also >> make the message immediately readable. >> Reading submitted data can ensure accuracy, and reading uncommitted data >> can ensure real-time performance (there may be some repeated message or >> dirty message). >> Real-time performance is what users need. How to handle dirty message >> should be considered by the application side. >> >> We can still get complete and accurate data from Read Committed isolation >> level. >> >> Sincerely yours.
Re: [Vote] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager
Hello, I am closing this vote as this PIP received enough +1s for approval. Vote Result: +4 bindings (Matteo, Penghui, Mattison, and Guo) +1 non-bidning (Yubiao) Thank you, Heesung On Tue, Sep 19, 2023 at 6:33 PM Yubiao Feng wrote: > +1 (no binding) > > Thanks > Yubiao Feng > > On Tue, Aug 8, 2023 at 2:55 AM Heesung Sohn > wrote: > > > Hi, > > > > I'd like to start a vote thread for the PIP-281. > > https://github.com/apache/pulsar/pull/20748 > > > > Discussion : > > https://lists.apache.org/thread/bdmx4qhn6hkoxm0xbtf67tq4kt5r8jmy > > > > Best, > > Heesung > > >
Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level
Hi Dave, The uncommitted transactions do not impact actual users' bank accounts. Business Processing System E only reads committed transactional messages and operates users' accounts. It needs Exactly-once semantic. Real-time Monitoring System D reads uncommitted transactional messages. It does not need Exactly-once semantic. They use different subscriptions and choose different isolation levels. One needs transaction, one does not. In general, multiple subscriptions of the same topic do not all require transaction guarantees. Some want low latency without the exact-once semantic guarantee, and some must require the exactly-once guarantee. We just provide a new option for different subscriptions. This should not be a breaking change,right? Looking forward to your reply. Thanks, Xiangying On Tue, Sep 26, 2023 at 4:09 AM Dave Fisher wrote: > > > > > On Sep 20, 2023, at 12:50 AM, Xiangying Meng wrote: > > > > Hi, all, > > > > Let's consider another example: > > > > **System**: Financial Transaction System > > > > **Operations**: Large volume of deposit and withdrawal operations, a > > small number of transfer operations. > > > > **Roles**: > > > > - **Client A1** > > - **Client A2** > > - **User Account B1** > > - **User Account B2** > > - **Request Topic C** > > - **Real-time Monitoring System D** > > - **Business Processing System E** > > > > **Client Operations**: > > > > - **Withdrawal**: Client A1 decreases the deposit amount from User > > Account B1 or B2. > > - **Deposit**: Client A1 increases the deposit amount in User Account B1 or > > B2. > > - **Transfer**: Client A2 decreases the deposit amount from User > > Account B1 and increases it in User Account B2. Or vice versa. > > > > **Real-time Monitoring System D**: Obtains the latest data from > > Request Topic C as quickly as possible to monitor transaction data and > > changes in bank reserves in real-time. This is necessary for the > > timely detection of anomalies and real-time decision-making. > > > > **Business Processing System E**: Reads data from Request Topic C, > > then actually operates User Accounts B1, B2. > > > > **User Scenario**: Client A1 sends a large number of deposit and > > withdrawal requests to Request Topic C. Client A2 writes a small > > number of transfer requests to Request Topic C. > > > > In this case, Business Processing System E needs a read-committed > > isolation level to ensure operation consistency and Exactly Once > > semantics. The real-time monitoring system does not care if a small > > number of transfer requests are incomplete (dirty data). What it > > cannot tolerate is a situation where a large number of deposit and > > withdrawal requests cannot be presented in real time due to a small > > number of transfer requests (the current situation is that uncommitted > > transaction messages can block the reading of committed transaction > > messages). > > So you are willing to let uncommitted transactions impact actual users bank > accounts? Are you sure that there is not another way to bypass uncommitted > records? Letting uncommitted records through is not Exactly once. > > Are you ready to rewrite Pulsar’s documentation to explain how normal users > can avoid allowing this? > > Best, > Dave > > > > > > In this case, it is necessary to set different isolation levels for > > different consumers/subscriptions. > > > > Thanks, > > Xiangying > > > > On Tue, Sep 19, 2023 at 11:35 PM 杨国栋 wrote: > >> > >> Hi Dave and Xiangying, > >> Thanks for all your support. > >> > >> Let me add some background. > >> > >> Apache Paimon take message queue as External Log Systems and changelog of > >> Paimon can also be consumed from message queue. > >> By default, change-log of message queue in Paimon are visible to consumers > >> only after a snapshot. Snapshot have a same life cycle as message queue > >> transactions. > >> However, users can immediately consume change-log by read uncommited > >> message without waiting for the next snapshot. > >> This behavior reduces the latency of changelog, but it relies on reading > >> uncommited message in Kafka or other message queue. > >> So we hope Pulsar can support Read Uncommitted isolation level. > >> > >> Put aside the application scenarios of Paimon. Let's discuss Read > >> Uncommitted isolation level itself. > >> > >> Read Uncommitted isolation will bring certain security risks, but will also > >> make the message immediately readable. > >> Reading submitted data can ensure accuracy, and reading uncommitted data > >> can ensure real-time performance (there may be some repeated message or > >> dirty message). > >> Real-time performance is what users need. How to handle dirty message > >> should be considered by the application side. > >> > >> We can still get complete and accurate data from Read Committed isolation > >> level. > >> > >> Sincerely yours. >
Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level
Hi - OK. I’ll agree, but I think the PIP ought to include documentation. There should also be clear communication about this use case and how to use it. Sent from my iPhone > On Sep 25, 2023, at 6:23 PM, Xiangying Meng wrote: > > Hi Dave, > The uncommitted transactions do not impact actual users' bank accounts. > Business Processing System E only reads committed transactional > messages and operates users' accounts. It needs Exactly-once semantic. > Real-time Monitoring System D reads uncommitted transactional > messages. It does not need Exactly-once semantic. > > They use different subscriptions and choose different isolation > levels. One needs transaction, one does not. > In general, multiple subscriptions of the same topic do not all > require transaction guarantees. > Some want low latency without the exact-once semantic guarantee, and > some must require the exactly-once guarantee. > We just provide a new option for different subscriptions. This should > not be a breaking change,right? Not a breaking change, but it does add to the API. It should be discussed if this PIP is only for master - 3.2, or if may be cherry picked to current versions. > > Looking forward to your reply. Thank you, Dave > > Thanks, > Xiangying > >> On Tue, Sep 26, 2023 at 4:09 AM Dave Fisher wrote: >> >> >> On Sep 20, 2023, at 12:50 AM, Xiangying Meng wrote: >>> >>> Hi, all, >>> >>> Let's consider another example: >>> >>> **System**: Financial Transaction System >>> >>> **Operations**: Large volume of deposit and withdrawal operations, a >>> small number of transfer operations. >>> >>> **Roles**: >>> >>> - **Client A1** >>> - **Client A2** >>> - **User Account B1** >>> - **User Account B2** >>> - **Request Topic C** >>> - **Real-time Monitoring System D** >>> - **Business Processing System E** >>> >>> **Client Operations**: >>> >>> - **Withdrawal**: Client A1 decreases the deposit amount from User >>> Account B1 or B2. >>> - **Deposit**: Client A1 increases the deposit amount in User Account B1 or >>> B2. >>> - **Transfer**: Client A2 decreases the deposit amount from User >>> Account B1 and increases it in User Account B2. Or vice versa. >>> >>> **Real-time Monitoring System D**: Obtains the latest data from >>> Request Topic C as quickly as possible to monitor transaction data and >>> changes in bank reserves in real-time. This is necessary for the >>> timely detection of anomalies and real-time decision-making. >>> >>> **Business Processing System E**: Reads data from Request Topic C, >>> then actually operates User Accounts B1, B2. >>> >>> **User Scenario**: Client A1 sends a large number of deposit and >>> withdrawal requests to Request Topic C. Client A2 writes a small >>> number of transfer requests to Request Topic C. >>> >>> In this case, Business Processing System E needs a read-committed >>> isolation level to ensure operation consistency and Exactly Once >>> semantics. The real-time monitoring system does not care if a small >>> number of transfer requests are incomplete (dirty data). What it >>> cannot tolerate is a situation where a large number of deposit and >>> withdrawal requests cannot be presented in real time due to a small >>> number of transfer requests (the current situation is that uncommitted >>> transaction messages can block the reading of committed transaction >>> messages). >> >> So you are willing to let uncommitted transactions impact actual users bank >> accounts? Are you sure that there is not another way to bypass uncommitted >> records? Letting uncommitted records through is not Exactly once. >> >> Are you ready to rewrite Pulsar’s documentation to explain how normal users >> can avoid allowing this? >> >> Best, >> Dave >> >> >>> >>> In this case, it is necessary to set different isolation levels for >>> different consumers/subscriptions. >>> >>> Thanks, >>> Xiangying >>> >>> On Tue, Sep 19, 2023 at 11:35 PM 杨国栋 wrote: Hi Dave and Xiangying, Thanks for all your support. Let me add some background. Apache Paimon take message queue as External Log Systems and changelog of Paimon can also be consumed from message queue. By default, change-log of message queue in Paimon are visible to consumers only after a snapshot. Snapshot have a same life cycle as message queue transactions. However, users can immediately consume change-log by read uncommited message without waiting for the next snapshot. This behavior reduces the latency of changelog, but it relies on reading uncommited message in Kafka or other message queue. So we hope Pulsar can support Read Uncommitted isolation level. Put aside the application scenarios of Paimon. Let's discuss Read Uncommitted isolation level itself. Read Uncommitted isolation will bring certain security risks, but will also make the message immediately readable. Reading submitted data can ensure acc
Re: [DISCUSS] PIP-300: Add custom dynamic configuration for plugins
Hi Zixuan, This is really a great feature. I support it. Regarding cherry-pick, as far as I know, we have cherry-picked some configuration items and interfaces into branch-2.10. But that should be mentioned in a separate discussion and provide sufficient reasons why we have to do it. Cherry-pick is performed after the vote passes. So, I suggest that feature and cherry-pick can be discussed separately through different emails. Sincerely, Xiangying On Mon, Sep 25, 2023 at 6:51 PM Zike Yang wrote: > > Hi, Zixuan > > Thanks for your proposal. I'm +1 for it. > > > This is a feature I need. If cherry-pick is not allowed, then it will > only be kept in 3.2+. > > This is a new feature, and I also think that we couldn't cherry-pick > it. What about cherry-picking this change to your fork repo and > building the pulsar for your own to meet this need? Does it make sense > to you? > > BR, > Zike Yang > > On Mon, Sep 25, 2023 at 12:23 AM mattison chao wrote: > > > > Hi, Zixuan > > > > I am afraid I can't support you in cherry-picking this feature for all of > > the active branches by the current fact. Because this is a new feature and > > it's not a bug fix or security issue. > > > > We can wait for other contributor's ideas. WDYT? > > > > Thanks, > > Mattison > > On 19 Sep 2023 at 10:42 +0800, Zixuan Liu , wrote: > > > > 1. When you want to cherry-pick a PR, it should be cherry-picked for > > > > all branches after the target branch. Isn't it? > > > > > > I think we should cherry-pick this PR to Pulsar 2.10, 2.11, 3.0. > > > > > > > 2. I've got a slight concern about cherry-picking. Do you have any > > > > issue or feature request in the community that can demonstrate this > > > > feature is required to cherry-pick? > > > > > > This is a feature I need. If cherry-pick is not allowed, then it will > > > only be kept in 3.2+. > > > > > > Thanks, > > > Zixuan > > > > > > mattison chao 于2023年9月18日周一 09:42写道： > > > > > > > > > > > HI, Zixuan > > > > > > > > Thanks for your discussion. It's a great feature. But I've got several > > > > questions I want to discuss here. > > > > > > > > 1. When you want to cherry-pick a PR, it should be cherry-picked for > > > > all branches after the target branch. Isn't it? > > > > 2. I've got a slight concern about cherry-picking. Do you have any > > > > issue or feature request in the community that can demonstrate this > > > > feature is required to cherry-pick? > > > > > > > > > > > > Best, > > > > Mattison > > > > On 12 Sep 2023 at 11:25 +0800, Zixuan Liu , wrote: > > > > > > BTW, I think we can cherry-pick this feature to the Pulsar 2.10. As > > > > > > far as I know, the Pulsar 2.10 is a stable/main version, because > > > > > > many > > > > > > users are using it. > > > > > > > > > > > > Thanks, > > > > > > Zixuan > > > > > > > > > > > > Zixuan Liu 于2023年9月5日周二 17:09写道： > > > > > > > > > > > > > > > > Hi Pulsar Community, > > > > > > > > > > > > > > > > Discuss for PIP-300: https://github.com/apache/pulsar/pull/21127 > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Zixuan
Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level
Hi Dave, Thanks for your support. I also think this should only be for the master branch. Thanks, Xiangying On Tue, Sep 26, 2023 at 9:34 AM Dave Fisher wrote: > > Hi - > > OK. I’ll agree, but I think the PIP ought to include documentation. There > should also be clear communication about this use case and how to use it. > > Sent from my iPhone > > > On Sep 25, 2023, at 6:23 PM, Xiangying Meng wrote: > > > > Hi Dave, > > The uncommitted transactions do not impact actual users' bank accounts. > > Business Processing System E only reads committed transactional > > messages and operates users' accounts. It needs Exactly-once semantic. > > Real-time Monitoring System D reads uncommitted transactional > > messages. It does not need Exactly-once semantic. > > > > They use different subscriptions and choose different isolation > > levels. One needs transaction, one does not. > > In general, multiple subscriptions of the same topic do not all > > require transaction guarantees. > > Some want low latency without the exact-once semantic guarantee, and > > some must require the exactly-once guarantee. > > We just provide a new option for different subscriptions. This should > > not be a breaking change,right? > > Not a breaking change, but it does add to the API. > > It should be discussed if this PIP is only for master - 3.2, or if may be > cherry picked to current versions. > > > > > Looking forward to your reply. > > Thank you, > Dave > > > > Thanks, > > Xiangying > > > >> On Tue, Sep 26, 2023 at 4:09 AM Dave Fisher wrote: > >> > >> > >> > On Sep 20, 2023, at 12:50 AM, Xiangying Meng > wrote: > >>> > >>> Hi, all, > >>> > >>> Let's consider another example: > >>> > >>> **System**: Financial Transaction System > >>> > >>> **Operations**: Large volume of deposit and withdrawal operations, a > >>> small number of transfer operations. > >>> > >>> **Roles**: > >>> > >>> - **Client A1** > >>> - **Client A2** > >>> - **User Account B1** > >>> - **User Account B2** > >>> - **Request Topic C** > >>> - **Real-time Monitoring System D** > >>> - **Business Processing System E** > >>> > >>> **Client Operations**: > >>> > >>> - **Withdrawal**: Client A1 decreases the deposit amount from User > >>> Account B1 or B2. > >>> - **Deposit**: Client A1 increases the deposit amount in User Account B1 > >>> or B2. > >>> - **Transfer**: Client A2 decreases the deposit amount from User > >>> Account B1 and increases it in User Account B2. Or vice versa. > >>> > >>> **Real-time Monitoring System D**: Obtains the latest data from > >>> Request Topic C as quickly as possible to monitor transaction data and > >>> changes in bank reserves in real-time. This is necessary for the > >>> timely detection of anomalies and real-time decision-making. > >>> > >>> **Business Processing System E**: Reads data from Request Topic C, > >>> then actually operates User Accounts B1, B2. > >>> > >>> **User Scenario**: Client A1 sends a large number of deposit and > >>> withdrawal requests to Request Topic C. Client A2 writes a small > >>> number of transfer requests to Request Topic C. > >>> > >>> In this case, Business Processing System E needs a read-committed > >>> isolation level to ensure operation consistency and Exactly Once > >>> semantics. The real-time monitoring system does not care if a small > >>> number of transfer requests are incomplete (dirty data). What it > >>> cannot tolerate is a situation where a large number of deposit and > >>> withdrawal requests cannot be presented in real time due to a small > >>> number of transfer requests (the current situation is that uncommitted > >>> transaction messages can block the reading of committed transaction > >>> messages). > >> > >> So you are willing to let uncommitted transactions impact actual users > >> bank accounts? Are you sure that there is not another way to bypass > >> uncommitted records? Letting uncommitted records through is not Exactly > >> once. > >> > >> Are you ready to rewrite Pulsar’s documentation to explain how normal > >> users can avoid allowing this? > >> > >> Best, > >> Dave > >> > >> > >>> > >>> In this case, it is necessary to set different isolation levels for > >>> different consumers/subscriptions. > >>> > >>> Thanks, > >>> Xiangying > >>> > >>> On Tue, Sep 19, 2023 at 11:35 PM 杨国栋 wrote: > > Hi Dave and Xiangying, > Thanks for all your support. > > Let me add some background. > > Apache Paimon take message queue as External Log Systems and changelog of > Paimon can also be consumed from message queue. > By default, change-log of message queue in Paimon are visible to > consumers > only after a snapshot. Snapshot have a same life cycle as message queue > transactions. > However, users can immediately consume change-log by read uncommited > message without waiting for the next snapshot. > This behavior reduces the latency of changelog, but it relies on reading > uncom
Re: [DISCUSS] PIP-300: Add custom dynamic configuration for plugins
Hi all, Thank you for your feedback. Let's support this feature first! Thanks, Zixuan Xiangying Meng 于2023年9月26日周二 09:39写道： > > Hi Zixuan, > > This is really a great feature. I support it. > > Regarding cherry-pick, as far as I know, we have cherry-picked some > configuration items and interfaces into branch-2.10. > But that should be mentioned in a separate discussion and provide > sufficient reasons why we have to do it. Cherry-pick is performed > after the vote passes. So, I suggest that feature and cherry-pick can > be discussed separately through different emails. > > Sincerely, > Xiangying > > On Mon, Sep 25, 2023 at 6:51 PM Zike Yang wrote: > > > > Hi, Zixuan > > > > Thanks for your proposal. I'm +1 for it. > > > > > This is a feature I need. If cherry-pick is not allowed, then it will > > only be kept in 3.2+. > > > > This is a new feature, and I also think that we couldn't cherry-pick > > it. What about cherry-picking this change to your fork repo and > > building the pulsar for your own to meet this need? Does it make sense > > to you? > > > > BR, > > Zike Yang > > > > On Mon, Sep 25, 2023 at 12:23 AM mattison chao > > wrote: > > > > > > Hi, Zixuan > > > > > > I am afraid I can't support you in cherry-picking this feature for all of > > > the active branches by the current fact. Because this is a new feature > > > and it's not a bug fix or security issue. > > > > > > We can wait for other contributor's ideas. WDYT? > > > > > > Thanks, > > > Mattison > > > On 19 Sep 2023 at 10:42 +0800, Zixuan Liu , wrote: > > > > > 1. When you want to cherry-pick a PR, it should be cherry-picked for > > > > > all branches after the target branch. Isn't it? > > > > > > > > I think we should cherry-pick this PR to Pulsar 2.10, 2.11, 3.0. > > > > > > > > > 2. I've got a slight concern about cherry-picking. Do you have any > > > > > issue or feature request in the community that can demonstrate this > > > > > feature is required to cherry-pick? > > > > > > > > This is a feature I need. If cherry-pick is not allowed, then it will > > > > only be kept in 3.2+. > > > > > > > > Thanks, > > > > Zixuan > > > > > > > > mattison chao 于2023年9月18日周一 09:42写道： > > > > > > > > > > > > > > HI, Zixuan > > > > > > > > > > Thanks for your discussion. It's a great feature. But I've got > > > > > several questions I want to discuss here. > > > > > > > > > > 1. When you want to cherry-pick a PR, it should be cherry-picked for > > > > > all branches after the target branch. Isn't it? > > > > > 2. I've got a slight concern about cherry-picking. Do you have any > > > > > issue or feature request in the community that can demonstrate this > > > > > feature is required to cherry-pick? > > > > > > > > > > > > > > > Best, > > > > > Mattison > > > > > On 12 Sep 2023 at 11:25 +0800, Zixuan Liu , wrote: > > > > > > > BTW, I think we can cherry-pick this feature to the Pulsar 2.10. > > > > > > > As > > > > > > > far as I know, the Pulsar 2.10 is a stable/main version, because > > > > > > > many > > > > > > > users are using it. > > > > > > > > > > > > > > Thanks, > > > > > > > Zixuan > > > > > > > > > > > > > > Zixuan Liu 于2023年9月5日周二 17:09写道： > > > > > > > > > > > > > > > > > > Hi Pulsar Community, > > > > > > > > > > > > > > > > > > Discuss for PIP-300: > > > > > > > > > https://github.com/apache/pulsar/pull/21127 > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Zixuan
[VOTE] PIP-300: Add custom dynamic configuration for plugins
Hi Pulsar Community, Voting for PIP-300: https://github.com/apache/pulsar/pull/21127 Discussion thread: https://lists.apache.org/thread/ysnsnollgy1b6w1dsvmx1t1y2rz1tyd6 Thanks, Zixuan
