Re: [DISCUSS] PIP-291 Improving the usability of the bookie isolation feature

2023-08-06 Thread Yan Zhao
https://github.com/apache/pulsar/pull/12025 already support it, close this session. On 2023/08/04 00:51:12 horizonzy wrote: > Currently, we provide the bookie isolation feature, but its usability is > relatively low. I have proposed PIP-291( > https://github.com/apache/pulsar/pull/20926) to impro

Re: [ANNOUNCE] Zili Chen (tison) as new PMC member in Apache Pulsar

2023-07-21 Thread Yan Zhao
Congrats! On 2023/07/21 17:05:04 Michael Marshall wrote: > Hi Pulsar Community, > > The Apache Pulsar Project Management Committee (PMC) has invited Zili Chen > (https://github.com/tisonkun) as a member of the PMC and we are > pleased to announce that tison has accepted. > > tison is very active

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-03-12 Thread Yan Zhao
> > In the LedgerDeletionService start, it will create a producer for sending > > pending delete ledger. When deleting a ledger, the producer sends > > PendingDeleteLedgerInfo to the system-topic. If the sent succeeds, delete > > the ledger id from the whole ledgers and update the ledgers into the

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-03-04 Thread Yan Zhao
Hi, Asaf, Tune the pip https://github.com/apache/pulsar/issues/16569, please help to review it again, thanks!

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-03-02 Thread Yan Zhao
Hi, Asaf. Tune the pip-186(https://github.com/apache/pulsar/issues/16569), please help to view it again. thanks! On 2023/02/16 13:50:54 Yan Zhao wrote: > > If understood correctly, every broker will have a consumer right? You will > > use a fail-over subscription? The retry-topic is

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-02-16 Thread Yan Zhao
> If understood correctly, every broker will have a consumer right? You will > use a fail-over subscription? The retry-topic is consumed by the same > subscription, same consumer? Yes, you are right, there is the case you mention. The deletion is idempotent, I'm not sure if it's worth making it sy

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-02-14 Thread Yan Zhao
> Shouldn't you specify the changes you are going to make in > `PulsarApi.proto`? We didn't change the proto file, we use the Restful API(AdminAPI). > Also, shouldn't you change the wording in this sentence in the PIP to > clarify you will use AdminAPI / Pulsar RPC? We use AdminAPI, I will tune the

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-02-08 Thread Yan Zhao
On 2023/02/07 18:41:20 Heesung Sohn wrote: > On Sun, Feb 5, 2023 at 2:26 AM Yan Zhao wrote: > > > On 2023/02/03 18:14:53 Heesung Sohn wrote: > > > There are some cases to trigger it. > > > 1. A cursor be removed. > > > 2. Close the current ledger and cre

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-02-06 Thread Yan Zhao
On 2023/02/05 14:40:10 Asaf Mesika wrote: > On Fri, Feb 3, 2023 at 1:48 PM Yan Zhao wrote: > > > > If you persisted the message successfully by the producer and the broker > > > was terminated before being able to delete the ledger from the metadata > > &

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-02-05 Thread Yan Zhao
On 2023/02/03 20:05:59 Heesung Sohn wrote: > For these internal requesters, > 1. A cursor be removed. > 2. Close the current ledger and create a new ledger. > 3. Consumer ack the message, the slowest cursor move forward. > > How do we prevent these callers from requesting duplicate requests for th

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-02-05 Thread Yan Zhao
On 2023/02/03 18:14:53 Heesung Sohn wrote: > There are some cases to trigger it. > 1. A cursor be removed. > 2. Close the current ledger and create a new ledger. > 3. Consumer ack the message, the slowest cursor move forward. > 4. User trigger truncateTopic by admin. > > I see that this pip is for

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-02-03 Thread Yan Zhao
> If we don't want to add more pressure on the metadata store from this > feature as a requirement, > I think we can use a system topic as well. Just to confirm, is this the > direction we are agreeing on? The design is too generic, I'm afraid not it is not suitable for ledger deletion. > > Usin

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-02-03 Thread Yan Zhao
Heesung, thanks. That's a good point for multi steps works. But I'm afraid it will increase the zk pressure and it also needs to handle some corner cases. I prefer to use a system topic to handle it. On 2023/01/31 19:05:34 Heesung Sohn wrote: > On Tue, Jan 31, 2023 at 6:43 AM Ya

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-02-03 Thread Yan Zhao
> If you persisted the message successfully by the producer and the broker > was terminated before being able to delete the ledger from the metadata > service? If the broker is terminated, the consumer won't ack the message, the message will be re-consume later. > I recommend having the logic to

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-01-31 Thread Yan Zhao
> - Have we considered a metadata store to persist and dedup deletion > requests instead of the system topic? Why is the system topic the better > choice than a metadata store for this problem? If we use the metadata store to store the middle step ledger, we need to operate the metadata store afte

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-01-30 Thread Yan Zhao
> Couples of notes: > > 1. > > > In the LedgerDeletionService start, it will create a producer to send > > pending delete ledger. > > When delete a ledger, a PendingDeleteLedgerInfo msg with 1 min delay (the > > delay is for consumer side, if send it immediately, maybe the metadata > > din't c

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2023-01-28 Thread Yan Zhao
We need more eyes and votes. Thanks.

Re: [ANNOUNCE] New Committer: Baodi Shi

2023-01-19 Thread Yan Zhao
congratulation! On 2023/01/18 13:35:41 Yunze Xu wrote: > The Project Management Committee (PMC) for Apache Pulsar has invited > Baodi Shi (https://github.com/shibd) to become a committer and we are > pleased to announce that he has accepted. > > Being a committer enables easier contribution to th

Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar

2023-01-19 Thread Yan Zhao
congratulation! On 2023/01/18 13:50:12 PengHui Li wrote: > Hi all, > > The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong > (https://github.com/congbobo184) as a member of the PMC and we are > pleased to announce that he has accepted. > > He is very active in the community

Re: [ANNOUNCE] Yunze Xu as a new PMC member in Apache Pulsar

2022-12-30 Thread Yan Zhao
Congratulations !

Re: [ANNOUNCE] Jiwei Guo as a new PMC member in Pulsar

2022-08-18 Thread Yan Zhao
Congratulations!

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2022-08-14 Thread Yan Zhao
Good idea. > Good idea. I suggest the naming endings with `Seconds` like > sendDelayOfTwoPhaseDeletionInSeconds,` > reconsumeLaterOfTwoPhaseDeletionInSeconds`. > >private int sendDelaySecondsOfTwoPhaseDeletion = 60; > >private int reconsumeLaterSecondsOfTwoPhaseDeletion = 600;

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2022-08-11 Thread Yan Zhao
> I suggest to include 'topic' in the flag, we have too many entities in > Pulsar Thanks, change it to `topicTwoPhaseDeletionEnabled`.

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2022-07-22 Thread Yan Zhao
Yes, wo define `twoPhaseDeletionEnabled` in the Service Configuration.

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2022-07-20 Thread Yan Zhao
Hi, Enrico. If we bind the the topic with per-tenant, when tenant be deleted or the tenant not be load anymore, we data in the tennat system topic can't be consumed before the tenant next load. On 2022/07/14 15:35:16 Enrico Olivelli wrote: > This is very interesting. > > I have only one concern

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

2022-07-14 Thread Yan Zhao
If we use per-tenant system topic or per-namespace. and there is no active topic in the tenant anymore, how we consume the msg in this tenant. This is the concern I cared about, so I prefer to use system topic.