[DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

[Zike Yang]

https://github.com/apache/pulsar/issues/13408

Pasted below for quoting convenience.

——


## Motivation

If we enable the DLQ when consuming messages. For some messages that can't be 
processed successfully, the messages will be moved to the DLQ, but if we do not 
specify the data retention for the namespace or create a subscription for the 
DLQ to retain the data, the data of the DLQ topic will be removed 
automatically. Therefore, we need to create the initial subscription before 
sending messages to the DLQ.

## Goal

Users can set the initial subscription name in the DeadLetterPolicy. The 
consumer will create the initial subscription before sending messages to the 
DLQ. At this point, subsequent messages produced to the DLQ are not 
automatically deleted unexpectedly.

This PIP needs to be compatible with the previous behavior. The initial 
subscription name in the DeadLetterPolicy is optional. Default value is the 
subscription name of the consumer.

## API Changes

Add `initSubscriptionName` to the `DeadLetterPolicy`

```java
/**
 * Name of the initial subscription name of the dead letter topic.
 * The default value is the subscription name of the consumer.
 */
private String initSubscriptionName;
```

## Implementation

Before the deadLetterProducer is initialized, the consumer first tries to 
create a deadLetterConsumer using the initial subscription name in the 
DeadLetterPolicy. When this subscription already exists, the ConsumerBusy 
exception will occur. In this case, we can ignore that exception and create the 
deadLetterProducer.

Prototype implementation PR: https://github.com/apache/pulsar/pull/13355



Thanks,
Zike Yang

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

[Zike Yang]

Thanks for your review Michael.

On Tue, Dec 21, 2021 at 5:48 AM Michael Marshall  wrote:
>
> Thanks for creating this PIP Zike Yang. This sounds like an important
> improvement. I have a couple thoughts.
>
> 1. Does the `retryLetterTopic` have the same problem with immediate
> message expiration? Based on a quick glance, I don't see anything
> creating a subscription for the retry topic.
>
Yes. I think we also need to handle the retryLetterTopic. I will update the PIP.

> 2. Your prototype exposes a lacking in our existing client
> implementation: the non-admin client cannot directly create
> subscriptions. IIUC, this implementation does not work if
> `allowAutoSubscriptionCreation` is false.
IMO, we won't create the init subscription if `allowAutoSubscriptionCreation`
 is false. Otherwise, it will confuse the user. Users can explicitly create that
subscription to handle this case.

> Further, it is inefficient
> to create and immediately close a consumer just to create a
> subscription. Perhaps we should just update the non-admin client so
> that it can create subscriptions? It could be private so that we don't
> create confusion with the non-admin and the admin clients.
Do you mean to create a private method called `createSubscription` in
the non-admin client? But we don't have a protocol command for
creating subscription.
So in that method, we can send the `CommandSubscribe` and `CommandUnsubscribe`
to the broker to simulate the creation and closure of a consumer. But
I think it is not
straightforward to create a subscription using two commands.
Or the other way is to create a new command `CommandSubscription` in
the protocol
to handle this. But I wonder if there are more use cases for this
command that would
make it worthwhile to use this solution.
What do you think about these two ways?

>
> Thanks,
> Michael
>
>
> On Mon, Dec 20, 2021 at 4:18 AM Zike Yang
>  wrote:
> >
> > https://github.com/apache/pulsar/issues/13408
> >
> > Pasted below for quoting convenience.
> >
> > ——
> >
> >
> > ## Motivation
> >
> > If we enable the DLQ when consuming messages. For some messages that can't 
> > be processed successfully, the messages will be moved to the DLQ, but if we 
> > do not specify the data retention for the namespace or create a 
> > subscription for the DLQ to retain the data, the data of the DLQ topic will 
> > be removed automatically. Therefore, we need to create the initial 
> > subscription before sending messages to the DLQ.
> >
> > ## Goal
> >
> > Users can set the initial subscription name in the DeadLetterPolicy. The 
> > consumer will create the initial subscription before sending messages to 
> > the DLQ. At this point, subsequent messages produced to the DLQ are not 
> > automatically deleted unexpectedly.
> >
> > This PIP needs to be compatible with the previous behavior. The initial 
> > subscription name in the DeadLetterPolicy is optional. Default value is the 
> > subscription name of the consumer.
> >
> > ## API Changes
> >
> > Add `initSubscriptionName` to the `DeadLetterPolicy`
> >
> > ```java
> > /**
> >  * Name of the initial subscription name of the dead letter topic.
> >  * The default value is the subscription name of the consumer.
> >  */
> > private String initSubscriptionName;
> > ```
> >
> > ## Implementation
> >
> > Before the deadLetterProducer is initialized, the consumer first tries to 
> > create a deadLetterConsumer using the initial subscription name in the 
> > DeadLetterPolicy. When this subscription already exists, the ConsumerBusy 
> > exception will occur. In this case, we can ignore that exception and create 
> > the deadLetterProducer.
> >
> > Prototype implementation PR: https://github.com/apache/pulsar/pull/13355
> >
> >
> >
> > Thanks,
> > Zike Yang
> >



Best regards,
Zike Yang

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

[Zike Yang]

> We can avoid confusion by only attempting to create the subscription
> when `initSubscriptionName` is not `null`. As a result, this PIP
> wouldn't change the behavior for current implementations.
I think there is an issue with the current behavior. When the init
subscription
is not created before creating deadLetterProducer, it may result in the
loss of messages.
And this is not the expected behavior. So the purpose of this PIP is to fix
this issue.
IMO, we need to create that subscription by default to fix the current
behavior.

> +1 - I think we should expand the binary protocol to add a
> `CommandSubscription` or possibly `CommandCreateSubscription`
> (`CommandSubscribe` is already taken). Without a new command, we will
> introduce race conditions related to consumer creation and will
> definitely generate unnecessary errors.
+1. `CommandCreateSubscription` seems more reasonable.

> Regarding default authorization, this new command would require the
> same level of permission for the client since the act of creating a
> consumer and the act of creating a subscription both require the same
> `AuthAction` permission: Consume.
+1

Thanks,
Zike Yang

On Wed, Dec 22, 2021 at 2:40 AM Michael Marshall 
wrote:
>
> > IMO, we won't create the init subscription if
`allowAutoSubscriptionCreation`
> > is false. Otherwise, it will confuse the user. Users can explicitly
create that
> > subscription to handle this case.
>
> That is a good point, but I don't think we should attempt to create
> the subscription implicitly via consumer creation because that will
> lead to additional failure, which adds unnecessary load on the broker.
>
> We can avoid confusion by only attempting to create the subscription
> when `initSubscriptionName` is not `null`. As a result, this PIP
> wouldn't change the behavior for current implementations.
>
> > Do you mean to create a private method called `createSubscription` in
> > the non-admin client? But we don't have a protocol command for
> > creating subscription
>
> Yes, that was what I meant, and you're right, we don't have a protocol
> command for this case.
>
> > Or the other way is to create a new command `CommandSubscription` in
> > the protocol to handle this.
>
> +1 - I think we should expand the binary protocol to add a
> `CommandSubscription` or possibly `CommandCreateSubscription`
> (`CommandSubscribe` is already taken). Without a new command, we will
> introduce race conditions related to consumer creation and will
> definitely generate unnecessary errors.
>
> Regarding default authorization, this new command would require the
> same level of permission for the client since the act of creating a
> consumer and the act of creating a subscription both require the same
> `AuthAction` permission: Consume.
>
> Thanks,
> Michael
>
> On Tue, Dec 21, 2021 at 4:57 AM Zike Yang
>
>  wrote:
> >
> > Thanks for your review Michael.
> >
> > On Tue, Dec 21, 2021 at 5:48 AM Michael Marshall 
wrote:
> > >
> > > Thanks for creating this PIP Zike Yang. This sounds like an important
> > > improvement. I have a couple thoughts.
> > >
> > > 1. Does the `retryLetterTopic` have the same problem with immediate
> > > message expiration? Based on a quick glance, I don't see anything
> > > creating a subscription for the retry topic.
> > >
> > Yes. I think we also need to handle the retryLetterTopic. I will update
the PIP.
> >
> > > 2. Your prototype exposes a lacking in our existing client
> > > implementation: the non-admin client cannot directly create
> > > subscriptions. IIUC, this implementation does not work if
> > > `allowAutoSubscriptionCreation` is false.
> > IMO, we won't create the init subscription if
`allowAutoSubscriptionCreation`
> >  is false. Otherwise, it will confuse the user. Users can explicitly
create that
> > subscription to handle this case.
> >
> > > Further, it is inefficient
> > > to create and immediately close a consumer just to create a
> > > subscription. Perhaps we should just update the non-admin client so
> > > that it can create subscriptions? It could be private so that we don't
> > > create confusion with the non-admin and the admin clients.
> > Do you mean to create a private method called `createSubscription` in
> > the non-admin client? But we don't have a protocol command for
> > creating subscription.
> > So in that method, we can send the `CommandSubscribe` and
`CommandUnsubscribe`
> > to the broker to simulate the creation and closure of a consumer. But
> > I think it is not
> &

Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set

[Zike Yang]

+ 1 for skipping the topic level message size check for the chunk message
in the broker.

On Tue, Dec 28, 2021 at 9:14 AM guo jiwei  wrote:

> +1
>
>
> Regards
> Jiwei Guo (Tboy)
>
> On Tue, Dec 28, 2021 at 5:44 AM 陳智弘  wrote:
> >
> > +1
> >
> >  Skip the topic level setting is better
> >
> > Hang Chen  於 2021年12月27日 週一 22:09 寫道：
> >
> > > +1
> > >
> > > We'd better skip the topic level maxMessageSize check for chunk
> messages.
> > >
> > > Best,
> > > Hang
> > >
> > > PengHui Li  于2021年12月27日周一 22:07写道：
> > > >
> > > > +1,
> > > >
> > > > We can only skip the topic level messages size check for the chunk
> > > message.
> > > >
> > > > Regards,
> > > > Penghui
> > > >
> > > > On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang <
> jianghait...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Pulsar Community,
> > > > >
> > > > > I discovered a bug that chunk messages producing fails if topic
> level
> > > > > maxMessageSize is set [1]. The root cause of this issue is because
> > > chunk
> > > > > message is using broker level maxMessageSize as chunk size. And
> topic
> > > level
> > > > > maxMessageSize is always <= broker level maxMessageSize. So once
> it is
> > > set,
> > > > > the on-going chunk message producing fails.
> > > > >
> > > > > ## Proposed changes
> > > > > I would like to fix this by just skipping topic level
> maxMessageSize
> > > check
> > > > > in
> > > > >
> > >
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > > > > level maxMessageSize is introduced in [2], for the purpose of
> "easier
> > > to
> > > > > plan resource quotas for client allocation". And IMO this change
> will
> > > not
> > > > > bring further complex into this.
> > > > >
> > > > > ## Alternative
> > > > > Add a client side topic level maxMessageSize and keep it synced
> with
> > > > > broker.
> > > > >
> > > > > Required changes:
> > > > > - [client] Add a new field
> > > > > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store
> this
> > > > > client side topic level maxMessageSize.
> > > > > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > > > > CommandProducerSuccess for the initial value of
> > > ProducerBase#maxMessageSize
> > > > > - [PulsarApi.proto] Add a new Command like
> > > > > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > > > > ProducerBase#maxMessageSize when topic level maxMessageSize is
> updated.
> > > > > Further more, some other data consistency issues need be handled
> very
> > > > > carefully when maxMessageSize is updated.
> > > > > This alternative is complex but can also solve other topic level
> > > > > maxMessageSize issue [3] when batching is enabled (non-batching
> case is
> > > > > solved with PR [4]).
> > > > >
> > > > > Any suggestions or other use cases of topic level maxMessageSize
> will
> > > be
> > > > > appreciated.
> > > > >
> > > > > Thanks,
> > > > > Haiting Jiang
> > > > >
> > > > > [1] https://github.com/apache/pulsar/issues/13360
> > > > > [2] https://github.com/apache/pulsar/pull/8732
> > > > > [3] https://github.com/apache/pulsar/issues/12958
> > > > > [4] https://github.com/apache/pulsar/pull/13147
> > > > >
> > >
>


-- 
Zike Yang

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

[Zike Yang]

> Oh, that's a very interesting point. I think it'd be easy to add that
> as "internal" feature, though I'm a bit puzzled on how to add that to
> the producer API

I think we can add a field `String initialSubscriptionName` to the
Producer Configuration. And add a new field `optional string
initial_subscription_name` to the `CommnadProducer`.
When the Broker handles the CommandProducer, if it checks that the
initialSubscriptionName is not empty or null, it will use
initialSubscriptionName to create a subscription on that topic. When
creating the deadLetterProducer or retryLetterProducer, we can specify
and create the initial subscription directly through the Producer.
What do you think?

On Thu, Dec 23, 2021 at 7:42 AM Matteo Merli  wrote:
>
> > What if we extended the `CommandProducer` command to add a
> > `create_subscription` field? Then, any time a topic is auto
> > created and this field is true, the broker would auto create a
> > subscription. There are some details to work out, but I think this
> > feature would fulfill the needs of this PIP and would also be broadly
> > useful for many client applications that dynamically create topics.
>
> Oh, that's a very interesting point. I think it'd be easy to add that
> as "internal" feature, though I'm a bit puzzled on how to add that to
> the producer API



-- 
Zike Yang

Re: [VOTE] PIP 131: Resolve produce chunk messages failed when topic level maxMessageSize is set

[Zike Yang]

+1

Thanks,
Zike

On Wed, Dec 29, 2021 at 11:26 AM PengHui Li  wrote:
>
> +1
>
> Thanks,
> Penghui
>
> On Wed, Dec 29, 2021 at 10:29 AM Haiting Jiang 
> wrote:
>
> > This is the voting thread for PIP-131. It will stay open for at least 48h.
> >
> > https://github.com/apache/pulsar/issues/13544
> >
> > The discussion thread is
> > https://lists.apache.org/thread/c63d9s73j9x1m3dkqr3r38gyp8s7cwzf
> >
> > ## Motivation
> >
> > Currently, chunk messages producing fails if topic level maxMessageSize is
> > set [1]. The root cause of this issue is because chunk message is using
> > broker level maxMessageSize as chunk size. And topic level maxMessageSize
> > is always <= broker level maxMessageSize. So once it is set, the on-going
> > chunk message producing fails.
> >
> > ## Goal
> >
> > Resolve topic level maxMessageSize compatibility issue with chunking
> > messages.
> >
> > ## Implementation
> >
> > Current best solution would be just skipping topic level maxMessageSize
> > check in
> > org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.
> > Topic level maxMessageSize is introduced in [2], for the purpose of
> > "easier to plan resource quotas for client allocation". And IMO this change
> > will not bring further complex into this.
> >
> > ## Reject Alternatives
> >
> > Add a client side topic level maxMessageSize and keep it synced with
> > broker.
> >
> > Required changes:
> > - [client] Add a new field
> > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> > client side topic level maxMessageSize.
> > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> > - [PulsarApi.proto] Add a new Command like
> > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> > Further more, some other data consistency issues need be handled very
> > carefully when maxMessageSize is updated.
> > This alternative is complex but can also solve other topic level
> > maxMessageSize issue [3] when batching is enabled (non-batching case is
> > solved with PR [4]).
> >
> > [1] https://github.com/apache/pulsar/issues/13360
> > [2] https://github.com/apache/pulsar/pull/8732
> > [3] https://github.com/apache/pulsar/issues/12958
> > [4] https://github.com/apache/pulsar/pull/13147
> >
> > Thanks,
> > Haiting Jiang
> >



-- 
Zike Yang

Re: [DISCUSSION] PIP-132: Include message header size when check maxMessageSize of non-batch message on the client side.

[Zike Yang]

But how do we handle chunked messages? The chunked message is split
based on the maxMessageSize(max payload size). This would seem to make
`op.getMessageHeaderAndPayloadSize() > ClientCnx.getMaxMessageSize()`
always true.

Thanks,
Zike

On Fri, Dec 31, 2021 at 8:11 PM Haiting Jiang  wrote:
>
> Sorry, it should be PIP-132.
>
> Thanks,
> Haiting Jiang
>
> On 2021/12/31 12:05:54 Haiting Jiang wrote:
> > https://github.com/apache/pulsar/issues/13591
> >
> > Pasted below for quoting convenience.
> >
> > ——
> >
> > ## Motivation
> >
> > Currently, Pulsar client (Java) only checks payload size for max message 
> > size validation.
> >
> > Client throws TimeoutException if we produce a message with too many 
> > properties, see [1].
> > But the root cause is that is trigged TooLongFrameException in broker 
> > server.
> >
> > In this PIP, I propose to include message header size when check 
> > maxMessageSize of non-batch
> > messages, this brings the following benefits.
> > 1. Clients can throw InvalidMessageException immediately if properties 
> > takes too much storage space.
> > 2. This will make the behaviour consistent with topic level max message 
> > size check in broker.
> > 3. Strictly limit the entry size less than maxMessageSize, avoid sending 
> > message to bookkeeper failed.
> >
> > ## Goal
> >
> > Include message header size when check maxMessageSize for non-batch message 
> > on the client side.
> >
> > ## Implementation
> >
> > ```
> > // Add a size check in 
> > org.apache.pulsar.client.impl.ProducerImpl#processOpSendMsg
> > if (op.msg != null // for non-batch messages only
> > && op.getMessageHeaderAndPayloadSize() > ClientCnx.getMaxMessageSize()) 
> > {
> > // finish send op with InvalidMessageException
> > releaseSemaphoreForSendOp(op);
> > op.sendComplete(new PulsarClientException(new InvalidMessageException, 
> > op.sequenceId));
> > }
> >
> >
> > // 
> > org.apache.pulsar.client.impl.ProducerImpl.OpSendMsg#getMessageHeaderAndPayloadSize
> >
> > public int getMessageHeaderAndPayloadSize() {
> > ByteBuf cmdHeader = cmd.getFirst();
> > cmdHeader.markReaderIndex();
> > int totalSize = cmdHeader.readInt();
> > int cmdSize = cmdHeader.readInt();
> > int msgHeadersAndPayloadSize = totalSize - cmdSize - 4;
> > cmdHeader.resetReaderIndex();
> > return msgHeadersAndPayloadSize;
> > }
> > ```
> >
> > ## Reject Alternatives
> > Add a new property like "maxPropertiesSize" or "maxHeaderSize" in 
> > broker.conf and pass it to
> > client like maxMessageSize. But the implementation is much more complex, 
> > and don't have the
> > benefit 2 and 3 mentioned in Motivation.
> >
> > ## Compatibility Issue
> > As a matter of fact, this PIP narrows down the sendable range. Previously, 
> > when maxMessageSize
> > is 1KB, it's ok to send message with 1KB properties and 1KB payload. But 
> > with this PIP, the
> > sending will fail with InvalidMessageException.
> >
> > One conservative way is to add a boolean config "includeHeaderInSizeCheck" 
> > to enable this
> > feature. But I think it's OK to enable this directly as it's more 
> > reasonable, and I don't see good
> > migration plan if we add a config for this.
> >
> > The compatibility issue is worth discussing. And any suggestions are 
> > appreciated.
> >
> > [1] https://github.com/apache/pulsar/issues/13560
> >
> > Thanks,
> > Haiting Jiang
> >



-- 
Zike Yang

Re: [DISCUSSION] PIP-132: Include message header size when check maxMessageSize of non-batch message on the client side.

[Zike Yang]

> as we have the same header size for each chunk message.

The message metadata of different chunks will not be the same. There
are serval fields with different values such as chunk id, publish
time. Because some type of that fields is varint, I think the size of
the header is not always the same.


On Tue, Jan 4, 2022 at 11:51 AM Haiting Jiang  wrote:
>
> Good point,  I think we should shrink chunk size to 
> "ClientCnx.getMaxMessageSize() - chunkMessageHeaderSize", as we have the same 
> header size for each chunk message.
>
> Thanks,
> Haiting Jiang
>
> On 2022/01/04 03:27:00 Zike Yang wrote:
> > But how do we handle chunked messages? The chunked message is split
> > based on the maxMessageSize(max payload size). This would seem to make
> > `op.getMessageHeaderAndPayloadSize() > ClientCnx.getMaxMessageSize()`
> > always true.
> >
> > Thanks,
> > Zike
> >
> > On Fri, Dec 31, 2021 at 8:11 PM Haiting Jiang  
> > wrote:
> > >
> > > Sorry, it should be PIP-132.
> > >
> > > Thanks,
> > > Haiting Jiang
> > >
> > > On 2021/12/31 12:05:54 Haiting Jiang wrote:
> > > > https://github.com/apache/pulsar/issues/13591
> > > >
> > > > Pasted below for quoting convenience.
> > > >
> > > > ——
> > > >
> > > > ## Motivation
> > > >
> > > > Currently, Pulsar client (Java) only checks payload size for max 
> > > > message size validation.
> > > >
> > > > Client throws TimeoutException if we produce a message with too many 
> > > > properties, see [1].
> > > > But the root cause is that is trigged TooLongFrameException in broker 
> > > > server.
> > > >
> > > > In this PIP, I propose to include message header size when check 
> > > > maxMessageSize of non-batch
> > > > messages, this brings the following benefits.
> > > > 1. Clients can throw InvalidMessageException immediately if properties 
> > > > takes too much storage space.
> > > > 2. This will make the behaviour consistent with topic level max message 
> > > > size check in broker.
> > > > 3. Strictly limit the entry size less than maxMessageSize, avoid 
> > > > sending message to bookkeeper failed.
> > > >
> > > > ## Goal
> > > >
> > > > Include message header size when check maxMessageSize for non-batch 
> > > > message on the client side.
> > > >
> > > > ## Implementation
> > > >
> > > > ```
> > > > // Add a size check in 
> > > > org.apache.pulsar.client.impl.ProducerImpl#processOpSendMsg
> > > > if (op.msg != null // for non-batch messages only
> > > > && op.getMessageHeaderAndPayloadSize() > 
> > > > ClientCnx.getMaxMessageSize()) {
> > > > // finish send op with InvalidMessageException
> > > > releaseSemaphoreForSendOp(op);
> > > > op.sendComplete(new PulsarClientException(new 
> > > > InvalidMessageException, op.sequenceId));
> > > > }
> > > >
> > > >
> > > > // 
> > > > org.apache.pulsar.client.impl.ProducerImpl.OpSendMsg#getMessageHeaderAndPayloadSize
> > > >
> > > > public int getMessageHeaderAndPayloadSize() {
> > > > ByteBuf cmdHeader = cmd.getFirst();
> > > > cmdHeader.markReaderIndex();
> > > > int totalSize = cmdHeader.readInt();
> > > > int cmdSize = cmdHeader.readInt();
> > > > int msgHeadersAndPayloadSize = totalSize - cmdSize - 4;
> > > > cmdHeader.resetReaderIndex();
> > > > return msgHeadersAndPayloadSize;
> > > > }
> > > > ```
> > > >
> > > > ## Reject Alternatives
> > > > Add a new property like "maxPropertiesSize" or "maxHeaderSize" in 
> > > > broker.conf and pass it to
> > > > client like maxMessageSize. But the implementation is much more 
> > > > complex, and don't have the
> > > > benefit 2 and 3 mentioned in Motivation.
> > > >
> > > > ## Compatibility Issue
> > > > As a matter of fact, this PIP narrows down the sendable range. 
> > > > Previously, when maxMessageSize
> > > > is 1KB, it's ok to send message with 1KB properties and 1KB payload. 
> > > > But with this PIP, the
> > > > sending will fail with InvalidMessageException.
> > > >
> > > > One conservative way is to add a boolean config 
> > > > "includeHeaderInSizeCheck" to enable this
> > > > feature. But I think it's OK to enable this directly as it's more 
> > > > reasonable, and I don't see good
> > > > migration plan if we add a config for this.
> > > >
> > > > The compatibility issue is worth discussing. And any suggestions are 
> > > > appreciated.
> > > >
> > > > [1] https://github.com/apache/pulsar/issues/13560
> > > >
> > > > Thanks,
> > > > Haiting Jiang
> > > >
> >
> >
> >
> > --
> > Zike Yang
> >



-- 
Zike Yang

Re: [VOTE] PIP-121: Pulsar cluster level auto failover on client side

[Zike Yang]

 > > `pulsar-client-api` package.
> > > > > >
> > > > > > In the `probeAvailable` method, we will probe the Pulsar service 
> > > > > > port,
> > > > > > and check whether the port is open. This probe method has many
> > > > > > disadvantages, such as
> > > > > > We're connecting to a Pulsar proxy, but there are no available 
> > > > > > brokers
> > > > > > Using Istio on the server side, which always accepts the connection
> > > > > > even if the broker is in a bad state
> > > > > > We might have deadlocks in (all) brokers and while the connections 
> > > > > > get
> > > > > > accepted, the brokers are not able to serve them.
> > > > > > In order to solve this problem, we’d better provide a health check
> > > > > > command on the broker side, just like Zookeeper’s `ruok` command.
> > > > > > We can use the probe port method first, and in the next step, we 
> > > > > > will
> > > > > > provide the health check command on the broker side.
> > > > > >
> > > > > >  ControlledClusterFailover
> > > > > > If the users want to control the cluster switch operation, they can
> > > > > > provide the current service URL by a http service. The
> > > > > > `ControlledClusterFailover` will get the newest service url from the
> > > > > > provided http service periodically.
> > > > > > The APIs are listed as follows.
> > > > > > ```Java
> > > > > > public class ControlledClusterFailover implements 
> > > > > > ServiceUrlProvider {
> > > > > >
> > > > > > private ControlledClusterFailover(String defaultServiceUrl, 
> > > > > > String
> > > > > > urlProvider) throws IOException {
> > > > > > }
> > > > > >
> > > > > > @Override
> > > > > > public void initialize(PulsarClient client) {
> > > > > > this.pulsarClient = client;
> > > > > >
> > > > > > // start to check service url every 30 seconds
> > > > > > executor.scheduleAtFixedRate(catchingAndLoggingThrowables(()
> > > > -> {
> > > > > > // probe and switch
> > > > > > }), interval, interval, TimeUnit.MILLISECONDS);
> > > > > >
> > > > > > }
> > > > > >
> > > > > > protected ControlledConfiguration fetchControlledConfiguration()
> > > > > > throws IOException {
> > > > > > // call the service to get controlled configuration
> > > > > > }
> > > > > >
> > > > > > @Override
> > > > > > public String getServiceUrl() {
> > > > > > return this.currentPulsarServiceUrl;
> > > > > > }
> > > > > >
> > > > > > @Override
> > > > > > public void close() {
> > > > > > this.executor.shutdown();
> > > > > > }
> > > > > >
> > > > > > protected static class ControlledConfiguration {
> > > > > > private String serviceUrl;
> > > > > > private String tlsTrustCertsFilePath;
> > > > > >
> > > > > > private String authPluginClassName;
> > > > > > private String authParamsString;
> > > > > >
> > > > > > }
> > > > > > ```
> > > > > > The configuration we get from the third url provider, we define it 
> > > > > > as
> > > > > > java Bean by json format. In the configuration, we provide
> > > > > > authentication-related parameters to support different clusters that
> > > > > > have different authentication configurations. These
> > > > > > authentication-related parameters can support all current
> > > > > > authentication plugin types.
> > > > > >
> > > > > > In order to create an `ControlledClusterFailover` instance, we use 
> > > > > > the
> > > > > > `ControlledClusterFailoverBuilder` interface to build the target
> > > > > > instance. The `ControlledClusterFailoverBuilder` interface is 
> > > > > > located
> > > > > > in the `pulsar-client-api` package.
> > > > > >
> > > > > > ### API Changes
> > > > > > For the current `ServiceUrlProvider` interface, we should add a
> > > > > > `close` method to close an allocated resource, such as a timer 
> > > > > > thread.
> > > > > > ```Java
> > > > > > public interface ServiceUrlProvider {
> > > > > > /**
> > > > > >  * Close the resource that the provider allocated.
> > > > > >  *
> > > > > >  */
> > > > > > default void close() {
> > > > > > // do nothing
> > > > > > }
> > > > > >
> > > > > > /**
> > > > > >  * Update the authentication this client is using.
> > > > > >  *
> > > > > >  * @param authentication
> > > > > >  *
> > > > > >  * @throws IOException
> > > > > >  */
> > > > > > void updateAuthentication(Authentication authentication)
> > > > > > throws IOException;
> > > > > >
> > > > > > /**
> > > > > >  * Update the tlsTrustCertsFilePath this client is using.
> > > > > >  *
> > > > > >  * @param tlsTrustCertsFilePath
> > > > > >  */
> > > > > > void updateTlsTrustCertsFilePath(String tlsTrustCertsFilePath);
> > > > > >
> > > > > > /**
> > > > > >  * Update the tlsTrustStorePath and tlsTrustStorePassword this
> > > > > > client is using.
> > > > > >  *
> > > > > >  * @param tlsTrustStorePath
> > > > > >  * @param tlsTrustStorePassword
> > > > > >  */
> > > > > > void updateTlsTrustStorePathAndPassword(String 
> > > > > > tlsTrustStorePath,
> > > > > > String tlsTrustStorePassword);
> > > > > >
> > > > > > }
> > > > > > ```
> > > > > >
> > > > > > ### Tests
> > > > > > Add tests for the two service provider implementations.
> > > > > >
> > > > > > For `AutoClusterFailover`, when the primary cluster shuts down, it
> > > > > > should switch to the secondary cluster. And then the primary cluster
> > > > > > came back, we should switch back.
> > > > > >
> > > > > > For `ControlledClusterFailover`, when switching the service url on 
> > > > > > the
> > > > > > http service side, it should switch to the newest service url.
> > > > > >
> > > > > > ### Implementation
> > > > > > Prototype implementation PR:
> > > > https://github.com/apache/pulsar/pull/13316
> > > > >
> > > >
> >



-- 
Zike Yang

Re: [VOTE] PIP-132: Include message header size when check maxMessageSize of non-batch message on the client side.

[Zike Yang]

+1 (non-binding)

On Wed, Jan 12, 2022 at 9:58 AM Haiting Jiang  wrote:
>
> This is the voting thread for PIP-132. It will stay open for at least 48 
> hours.
>
> https://github.com/apache/pulsar/issues/13591
>
> Pasted below for quoting convenience.
>
> 
>
> ## Motivation
>
> Currently, Pulsar client (Java) only checks payload size for max message size 
> validation.
>
> Client throws TimeoutException if we produce a message with too many 
> properties, see [1].
> But the root cause is that is trigged TooLongFrameException in broker server.
>
> In this PIP, I propose to include message header size when check 
> maxMessageSize of non-batch
> messages, this brings the following benefits.
> 1. Clients can throw InvalidMessageException immediately if properties takes 
> too much storage space.
> 2. This will make the behaviour consistent with topic level max message size 
> check in broker.
> 3. Strictly limit the entry size less than maxMessageSize, avoid sending 
> message to bookkeeper failed.
>
> ## Goal
>
> Include message header size when check maxMessageSize for non-batch message 
> on the client side.
>
> ## Implementation
>
> ```
> // Add a size check in 
> org.apache.pulsar.client.impl.ProducerImpl#processOpSendMsg
> if (op.msg != null // for non-batch messages only
> && op.getMessageHeaderAndPayloadSize() > ClientCnx.getMaxMessageSize()) {
> // finish send op with InvalidMessageException
> releaseSemaphoreForSendOp(op);
> op.sendComplete(new PulsarClientException(new InvalidMessageException, 
> op.sequenceId));
> }
>
>
> // 
> org.apache.pulsar.client.impl.ProducerImpl.OpSendMsg#getMessageHeaderAndPayloadSize
>
> public int getMessageHeaderAndPayloadSize() {
> ByteBuf cmdHeader = cmd.getFirst();
> cmdHeader.markReaderIndex();
> int totalSize = cmdHeader.readInt();
> int cmdSize = cmdHeader.readInt();
> int msgHeadersAndPayloadSize = totalSize - cmdSize - 4;
> cmdHeader.resetReaderIndex();
> return msgHeadersAndPayloadSize;
> }
> ```
>
> ## Reject Alternatives
> Add a new property like "maxPropertiesSize" or "maxHeaderSize" in broker.conf 
> and pass it to
> client like maxMessageSize. But the implementation is much more complex, and 
> don't have the
> benefit 2 and 3 mentioned in Motivation.
>
> ## Compatibility Issue
> As a matter of fact, this PIP narrows down the sendable range. Previously, 
> when maxMessageSize
> is 1KB, it's ok to send message with 1KB properties and 1KB payload. But with 
> this PIP, the
> sending will fail with InvalidMessageException.
>
> One conservative way is to add a boolean config "includeHeaderInSizeCheck" to 
> enable this
> feature. But I think it's OK to enable this directly as it's more reasonable, 
> and I don't see good
> migration plan if we add a config for this.
>
> [1] https://github.com/apache/pulsar/issues/13560
>
> Thanks,
> Haiting Jiang



-- 
Zike Yang

Re: [Discuss] Create new issues to SDKs in different languages

[Zike Yang]

+1

> I wonder if we can create issue in client repo automatically with bots for 
> PRs labelled"component/client" in pulsar repo.
> This would save the extra effort for the reviewer.

But there are many PRs with "component/client" label that are specific
to java client changes. I think these should not be added to other
clients' repos.



On Wed, Jan 12, 2022 at 4:18 PM Haiting Jiang  wrote:
>
> +1. Great idea.
>
> I wonder if we can create issue in client repo automatically with bots for 
> PRs labelled"component/client" in pulsar repo.
> This would save the extra effort for the reviewer.
>
> Thanks,
> Haiting Jiang
>
> On 2022/01/12 03:45:18 "r...@apache.org" wrote:
> > Hello everyone:
> >
> > At present, all our PIP and related function changes are mainly in the Java
> > language, and all new functions will be merged into the Java SDK first, but
> > for SDKs in other languages, this is completely a black box, they don't
> > know what changes or optimizations have been made on the Java SDK side.
> >
> > The most typical problem is that when users of other languages encounter
> > various problems during use, when the maintainers of other languages want
> > to fix these problems, we do not know that the Java SDK side has made these
> > changes. Therefore, every current solution is to constantly check where the
> > gap of the current Java SDK is, which brings great challenges to the
> > maintainers themselves.
> >
> > So here is an idea, when the committters/PMC responsible for reviewing the
> > Java SDK can do more to help evaluate whether these PIPs or new changes
> > need to support this function in other languages, and then the
> > corresponding issue is created in the corresponding SDK, so that it is
> > convenient for the maintainers of other language SDKs to further evaluate
> > the priority of this function, and it can also attract more contributors
> > who are good at certain languages to claim the corresponding issue and
> > contribute the corresponding function.
> >
> > --
> > Thanks
> > Xiaolong Ran
> >



-- 
Zike Yang

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

[Zike Yang]

+1 for adding init_subscription filed to the metadata of CommandProducer.



On Thu, Jan 20, 2022 at 2:52 PM PengHui Li  wrote:
>
> > What is the justification for avoiding the new protobuf field? If we
> add a structured field to a map of , we are still
> modifying the protocol, even if we aren't modifying the protobuf.
>
> Not all cases need a new field named init_subscription when creating
> producer,
> and users will not touch the metadata of the producer of a DLQ, so I think
> we can use the metadata to achieve the purpose more flexibly.
>
> Yes, it modifies the protocol, but in different ways. It like the message
> properties,
> we also introduce some system message properties such as
> __original_message_id
> in retry letter topic.
>
> Penghui
>
> On Thu, Jan 20, 2022 at 2:05 PM Michael Marshall 
> wrote:
>
> > > I think that, good or bad, the impression that users have that the DLQ
> > > is not a "normal" topic
> >
> > Thanks for your perspective, Matteo. I still prefer my alternative
> > design that bypasses subscription creation, but it seems there
> > is insufficient interest in it, so we should move forward
> > discussing a DLQ specific feature and its implementation.
> >
> > > 1. Instead of modifying the current protocol, we can use producer
> > > metadata to carry the init subscription
> >
> > What is the justification for avoiding the new protobuf field? If we
> > add a structured field to a map of , we are still
> > modifying the protocol, even if we aren't modifying the protobuf.
> >
> > Thanks,
> > Michael
> >
> >
> > On Tue, Jan 18, 2022 at 8:38 AM PengHui Li  wrote:
> > >
> > > +1 for adding the DLQ_init_sub to producer metadata so that we don't
> > > need to introduce a new field in CommandProducer, and the new field
> > > looks a little confusing
> > >
> > > Thanks,
> > > Penghui
> > >
> > > On Mon, Jan 17, 2022 at 10:19 PM Hang Chen  wrote:
> > >
> > > > Thanks for creating this proposal Zike Yang. I have two ideas about it.
> > > > 1. Instead of modifying the current protocol, we can use producer
> > > > metadata to carry the init subscription
> > > > 2. Please add auth for subscription creation when create topic by
> > > > producer, otherwise, it will be easily attacked.
> > > >
> > > > Thanks,
> > > > Hang
> > > >
> > > > Matteo Merli  于2022年1月12日周三 15:13写道：
> > > > >
> > > > > > If we want to hold that the DLQ is not a normal topic, then I can
> > see
> > > > > > why we would have a DLQ specific feature here.
> > > > >
> > > > > I think that, good or bad, the impression that users have that the
> > DLQ
> > > > > is not a "normal" topic comes from 2 factors:
> > > > >  1. The experience with traditional messaging systems JMS and others
> > > > > where the DLQ are handled in slightly different ways, compared to
> > > > > other topics
> > > > >  2. The name "DLQ" which in a way it's implying a "queue"... which
> > can
> > > > > be implemented on topic, using a subscription..
> > > >
> >



--
Zike Yang

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

[Zike Yang]

> This is true of the DLQ producer, but by using the metadata map, won't
we be exposing this feature to end users implicitly because they could
add the field to the metadata map for other producers? (I am okay with
giving users this functionality, but I know that Matteo said in a
community meeting that this feature should be limited to the DLQ producer.)

Yes, but it doesn't seem like a big deal. We just need to implement it
like that `ORIGINAL_MESSAGE_ID` in the retryLetterTopic.

On Sun, Jan 23, 2022 at 2:38 PM Michael Marshall  wrote:
>
> > users will not touch the metadata of the producer of a DLQ
>
> This is true of the DLQ producer, but by using the metadata map, won't
> we be exposing this feature to end users implicitly because they could
> add the field to the metadata map for other producers? (I am okay with
> giving users this functionality, but I know that Matteo said in a
> community meeting that this feature should be limited to the DLQ producer.)
>
> > we also introduce some system message properties such as
> > __original_message_id
> > in retry letter topic.
>
> Thanks for this context. I didn't know we were already doing this.
>
> Thanks,
> Michael
>
>
> On Fri, Jan 21, 2022 at 3:35 AM Zike Yang
>  wrote:
> >
> > +1 for adding init_subscription filed to the metadata of CommandProducer.
> >
> >
> >
> > On Thu, Jan 20, 2022 at 2:52 PM PengHui Li  wrote:
> > >
> > > > What is the justification for avoiding the new protobuf field? If we
> > > add a structured field to a map of , we are still
> > > modifying the protocol, even if we aren't modifying the protobuf.
> > >
> > > Not all cases need a new field named init_subscription when creating
> > > producer,
> > > and users will not touch the metadata of the producer of a DLQ, so I think
> > > we can use the metadata to achieve the purpose more flexibly.
> > >
> > > Yes, it modifies the protocol, but in different ways. It like the message
> > > properties,
> > > we also introduce some system message properties such as
> > > __original_message_id
> > > in retry letter topic.
> > >
> > > Penghui
> > >
> > > On Thu, Jan 20, 2022 at 2:05 PM Michael Marshall 
> > > wrote:
> > >
> > > > > I think that, good or bad, the impression that users have that the DLQ
> > > > > is not a "normal" topic
> > > >
> > > > Thanks for your perspective, Matteo. I still prefer my alternative
> > > > design that bypasses subscription creation, but it seems there
> > > > is insufficient interest in it, so we should move forward
> > > > discussing a DLQ specific feature and its implementation.
> > > >
> > > > > 1. Instead of modifying the current protocol, we can use producer
> > > > > metadata to carry the init subscription
> > > >
> > > > What is the justification for avoiding the new protobuf field? If we
> > > > add a structured field to a map of , we are still
> > > > modifying the protocol, even if we aren't modifying the protobuf.
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > > >
> > > > On Tue, Jan 18, 2022 at 8:38 AM PengHui Li  wrote:
> > > > >
> > > > > +1 for adding the DLQ_init_sub to producer metadata so that we don't
> > > > > need to introduce a new field in CommandProducer, and the new field
> > > > > looks a little confusing
> > > > >
> > > > > Thanks,
> > > > > Penghui
> > > > >
> > > > > On Mon, Jan 17, 2022 at 10:19 PM Hang Chen  
> > > > > wrote:
> > > > >
> > > > > > Thanks for creating this proposal Zike Yang. I have two ideas about 
> > > > > > it.
> > > > > > 1. Instead of modifying the current protocol, we can use producer
> > > > > > metadata to carry the init subscription
> > > > > > 2. Please add auth for subscription creation when create topic by
> > > > > > producer, otherwise, it will be easily attacked.
> > > > > >
> > > > > > Thanks,
> > > > > > Hang
> > > > > >
> > > > > > Matteo Merli  于2022年1月12日周三 15:13写道：
> > > > > > >
> > > > > > > > If we want to hold that the DLQ is not a normal topic, then I 
> > > > > > > > can
> > > > see
> > > > > > > > why we would have a DLQ specific feature here.
> > > > > > >
> > > > > > > I think that, good or bad, the impression that users have that the
> > > > DLQ
> > > > > > > is not a "normal" topic comes from 2 factors:
> > > > > > >  1. The experience with traditional messaging systems JMS and 
> > > > > > > others
> > > > > > > where the DLQ are handled in slightly different ways, compared to
> > > > > > > other topics
> > > > > > >  2. The name "DLQ" which in a way it's implying a "queue"... which
> > > > can
> > > > > > > be implemented on topic, using a subscription..
> > > > > >
> > > >
> >
> >
> >
> > --
> > Zike Yang



-- 
Zike Yang

[VOTE] PIP-124: Create init subscription before sending message to DLQ

[Zike Yang]

e this does not change protobuf, it still implicitly changes the
protocol. This solution has different advantages and disadvantages
than the current one. Finally, we decided to go with the current
solution.

In addition, there is a discussion about whether we need to create the
initial subscription on the retryLetterTopic. The consumer has already
subscribed to the retryLetterTopic and created the subscription before
creating the retryLetterProducer, so we don't need to create the
initial subscription for it.



Prototype implementation PR: https://github.com/apache/pulsar/pull/13355

----

Thanks
Zike Yang

Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ

[Zike Yang]

Thanks for your participation and your great efforts to move this forward!
Close the vote with 3 (+1) bindings and 2 (+1) non-bindings.

Thanks,
Zike

On Wed, Jan 26, 2022 at 3:54 PM PengHui Li  wrote:
>
> +1 (binding)
>
> Regards,
> Penghui
>
> On Wed, Jan 26, 2022 at 12:17 PM Michael Marshall 
> wrote:
>
> > +1 (non binding) - this proposal looks great! Thank you for a good
> > discussion of this feature!
> >
> > Thanks,
> > Michael
> >
> > On Tue, Jan 25, 2022 at 8:20 PM Hang Chen  wrote:
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Hang
> > >
> > > Jia Zhai  于2022年1月26日周三 10:17写道：
> > > >
> > > > +1
> > > >
> > > >
> > > >
> > > > On Tue, Jan 25, 2022 at 8:28 PM mattison chao 
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > > On Jan 25, 2022, at 5:34 PM, Zike Yang  > .INVALID>
> > > > > wrote:
> > > > > >
> > > > > > Hi Pulsar Community,
> > > > > >
> > > > > > This is the voting thread for PIP-124. It will stay open for at
> > least
> > > > > > 48 hours. Pasted below for quoting convenience.
> > > > > >
> > > > > > Here is the issue for this PIP:
> > > > > https://github.com/apache/pulsar/issues/13408
> > > > > >
> > > > > > 
> > > > > >
> > > > > > ## Motivation
> > > > > >
> > > > > > If we enable the DLQ when consuming messages. For some messages
> > that
> > > > > > can't be processed successfully, the messages will be moved to the
> > > > > > DLQ, but if we do not specify the data retention for the namespace
> > or
> > > > > > create a subscription for the DLQ to retain the data, the data of
> > the
> > > > > > DLQ will be removed automatically. Therefore, we need to create the
> > > > > > initial subscription before sending messages to the DLQ.
> > > > > >
> > > > > > ## Goal
> > > > > >
> > > > > > Users can set the initial subscription name in the
> > DeadLetterPolicy.
> > > > > > The consumer will create the initial subscription before sending
> > > > > > messages to the DLQ. At this point, subsequent messages produced to
> > > > > > the DLQ are not automatically deleted unexpectedly. If
> > > > > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the
> > initial
> > > > > > subscription won't be created automatically. Otherwise, it will
> > > > > > confuse the user. Users can explicitly create that subscription to
> > > > > > handle this case.
> > > > > >
> > > > > > This PIP needs to be compatible with the previous behavior. The
> > > > > > initial subscription name in the DeadLetterPolicy is optional. The
> > > > > > default behavior is not to create the initial subscription which is
> > > > > > consistent with the original behavior.
> > > > > >
> > > > > > ## API Changes
> > > > > >
> > > > > > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> > > > > >
> > > > > > ```java
> > > > > > /**
> > > > > > * Name of the initial subscription name of the dead letter topic.
> > > > > > * If this field is not set, the initial subscription will not be
> > created.
> > > > > > */
> > > > > > private String initSubscriptionName;
> > > > > > ```
> > > > > >
> > > > > > * Add a new field to the `CommandProducer`. The broker will use
> > this
> > > > > > field to create the initial subscription.
> > > > > >
> > > > > > ```proto
> > > > > > optional string initial_subscription_name
> > > > > > ```
> > > > > >
> > > > > > * Add a new config to the Producer Configuration. This allows the
> > > > > > consumer to specify the initial subscription when creating the
> > > > > > deadLetterProducer.
> > > > > >
> > > > > > ```java
> > > >

Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ

[Zike Yang]

Hi Michael,

Sorry, I forgot about it.  Sorry for my mistake.
Thanks for your correction. I will stay this vote open.

Thanks,
Zike





On Thu, Jan 27, 2022 at 12:31 AM Michael Marshall  wrote:
>
> Hi Zike,
>
> Based on our current voting rules, I believe we need to leave the vote
> open for at least 17 more hours.
>
> There has been discussion on the dev list about changing this rule,
> but I don't believe we have reached any conclusions yet.
>
> Thanks,
> Michael
>
> On Wed, Jan 26, 2022 at 6:04 AM Zike Yang
>  wrote:
> >
> > Thanks for your participation and your great efforts to move this forward!
> > Close the vote with 3 (+1) bindings and 2 (+1) non-bindings.
> >
> > Thanks,
> > Zike
> >
> > On Wed, Jan 26, 2022 at 3:54 PM PengHui Li  wrote:
> > >
> > > +1 (binding)
> > >
> > > Regards,
> > > Penghui
> > >
> > > On Wed, Jan 26, 2022 at 12:17 PM Michael Marshall 
> > > wrote:
> > >
> > > > +1 (non binding) - this proposal looks great! Thank you for a good
> > > > discussion of this feature!
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > > > On Tue, Jan 25, 2022 at 8:20 PM Hang Chen  wrote:
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Thanks,
> > > > > Hang
> > > > >
> > > > > Jia Zhai  于2022年1月26日周三 10:17写道：
> > > > > >
> > > > > > +1
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 25, 2022 at 8:28 PM mattison chao 
> > > > > > 
> > > > > > wrote:
> > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > > On Jan 25, 2022, at 5:34 PM, Zike Yang  > > > .INVALID>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Pulsar Community,
> > > > > > > >
> > > > > > > > This is the voting thread for PIP-124. It will stay open for at
> > > > least
> > > > > > > > 48 hours. Pasted below for quoting convenience.
> > > > > > > >
> > > > > > > > Here is the issue for this PIP:
> > > > > > > https://github.com/apache/pulsar/issues/13408
> > > > > > > >
> > > > > > > > 
> > > > > > > >
> > > > > > > > ## Motivation
> > > > > > > >
> > > > > > > > If we enable the DLQ when consuming messages. For some messages
> > > > that
> > > > > > > > can't be processed successfully, the messages will be moved to 
> > > > > > > > the
> > > > > > > > DLQ, but if we do not specify the data retention for the 
> > > > > > > > namespace
> > > > or
> > > > > > > > create a subscription for the DLQ to retain the data, the data 
> > > > > > > > of
> > > > the
> > > > > > > > DLQ will be removed automatically. Therefore, we need to create 
> > > > > > > > the
> > > > > > > > initial subscription before sending messages to the DLQ.
> > > > > > > >
> > > > > > > > ## Goal
> > > > > > > >
> > > > > > > > Users can set the initial subscription name in the
> > > > DeadLetterPolicy.
> > > > > > > > The consumer will create the initial subscription before sending
> > > > > > > > messages to the DLQ. At this point, subsequent messages 
> > > > > > > > produced to
> > > > > > > > the DLQ are not automatically deleted unexpectedly. If
> > > > > > > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the
> > > > initial
> > > > > > > > subscription won't be created automatically. Otherwise, it will
> > > > > > > > confuse the user. Users can explicitly create that subscription 
> > > > > > > > to
> > > > > > > > handle this case.
> > > > > > > >
> > > > > > > > This PIP needs to be compatible with the p

Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ

[Zike Yang]

Hi all,

Since there are no objections on this PIP and this vote has been
staying open for 48 hours.
Close the vote with 3 (+1) bindings and 2 (+1) non-bindings.
Thanks for your participation.

Thanks,
Zike

On Thu, Jan 27, 2022 at 10:09 AM Zike Yang  wrote:
>
> Hi Michael,
>
> Sorry, I forgot about it.  Sorry for my mistake.
> Thanks for your correction. I will stay this vote open.
>
> Thanks,
> Zike
>
>
>
>
>
> On Thu, Jan 27, 2022 at 12:31 AM Michael Marshall  
> wrote:
> >
> > Hi Zike,
> >
> > Based on our current voting rules, I believe we need to leave the vote
> > open for at least 17 more hours.
> >
> > There has been discussion on the dev list about changing this rule,
> > but I don't believe we have reached any conclusions yet.
> >
> > Thanks,
> > Michael
> >
> > On Wed, Jan 26, 2022 at 6:04 AM Zike Yang
> >  wrote:
> > >
> > > Thanks for your participation and your great efforts to move this forward!
> > > Close the vote with 3 (+1) bindings and 2 (+1) non-bindings.
> > >
> > > Thanks,
> > > Zike
> > >
> > > On Wed, Jan 26, 2022 at 3:54 PM PengHui Li  wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > Regards,
> > > > Penghui
> > > >
> > > > On Wed, Jan 26, 2022 at 12:17 PM Michael Marshall 
> > > > wrote:
> > > >
> > > > > +1 (non binding) - this proposal looks great! Thank you for a good
> > > > > discussion of this feature!
> > > > >
> > > > > Thanks,
> > > > > Michael
> > > > >
> > > > > On Tue, Jan 25, 2022 at 8:20 PM Hang Chen  wrote:
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Thanks,
> > > > > > Hang
> > > > > >
> > > > > > Jia Zhai  于2022年1月26日周三 10:17写道：
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 25, 2022 at 8:28 PM mattison chao 
> > > > > > > 
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > > On Jan 25, 2022, at 5:34 PM, Zike Yang  > > > > .INVALID>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Pulsar Community,
> > > > > > > > >
> > > > > > > > > This is the voting thread for PIP-124. It will stay open for 
> > > > > > > > > at
> > > > > least
> > > > > > > > > 48 hours. Pasted below for quoting convenience.
> > > > > > > > >
> > > > > > > > > Here is the issue for this PIP:
> > > > > > > > https://github.com/apache/pulsar/issues/13408
> > > > > > > > >
> > > > > > > > > 
> > > > > > > > >
> > > > > > > > > ## Motivation
> > > > > > > > >
> > > > > > > > > If we enable the DLQ when consuming messages. For some 
> > > > > > > > > messages
> > > > > that
> > > > > > > > > can't be processed successfully, the messages will be moved 
> > > > > > > > > to the
> > > > > > > > > DLQ, but if we do not specify the data retention for the 
> > > > > > > > > namespace
> > > > > or
> > > > > > > > > create a subscription for the DLQ to retain the data, the 
> > > > > > > > > data of
> > > > > the
> > > > > > > > > DLQ will be removed automatically. Therefore, we need to 
> > > > > > > > > create the
> > > > > > > > > initial subscription before sending messages to the DLQ.
> > > > > > > > >
> > > > > > > > > ## Goal
> > > > > > > > >
> > > > > > > > > Users can set the initial subscription name in the
> > > > > DeadLetterPolicy.
> > > >

Re: [DISCUSS] PIP-139 : Support Broker send command to real close producer/consumer.

[Zike Yang]

  final long producerId = closeProducer.getProducerId();
> > ProducerImpl producer = producers.get(producerId);
> > if (producer != null) {
> > if (closeProducer.isAllowReconnect) {
> > producer.connectionClosed(this);
> > } else {
> > producer.closeAsync();
> > }
> > } else {
> > log.warn("Producer with id {} not found while closing producer
> > ", producerId);
> > }
> > }
> > ```
> > ### ClientCnx - Consumer:
> >
> > **Before**
> > ```java
> > @Override
> > protected void handleCloseConsumer(CommandCloseConsumer closeConsumer)
> > {
> > log.info("[{}] Broker notification of Closed consumer: {}",
> > remoteAddress, closeConsumer.getConsumerId());
> > final long consumerId = closeConsumer.getConsumerId();
> > ConsumerImpl consumer = consumers.get(consumerId);
> > if (consumer != null) {
> > consumer.connectionClosed(this);
> > } else {
> > log.warn("Consumer with id {} not found while closing consumer
> > ", consumerId);
> > }
> > }
> > ```
> > **After**
> > ```java
> > @Override
> > protected void handleCloseConsumer(CommandCloseConsumer closeConsumer)
> > {
> > log.info("[{}] Broker notification of Closed consumer: {}",
> > remoteAddress, closeConsumer.getConsumerId());
> > final long consumerId = closeConsumer.getConsumerId();
> > ConsumerImpl consumer = consumers.get(consumerId);
> > if (consumer != null) {
> > if (closeConsumer.isAllowReconnect) {
> > consumer.connectionClosed(this);
> > } else {
> > consumer.closeAsync();
> > }
> > } else {
> > log.warn("Consumer with id {} not found while closing consumer
> > ", consumerId);
> > }
> > }
> > ```
> >
> >
> > ## Reject Alternatives
> >
> > none.



-- 
Zike Yang

[DISCUSS] The default value of maxPendingChunkedMessage

[Zike Yang]

Hi, Pulsar community,

We found that there are inconsistencies between the code and the
documentation regarding the default value of maxPendingChunkedMessage.

In the java client code, we use 10 as the default value. [1] But in
the java doc, we use 100 as the default value. [2]
We need to fix this inconsistency. But what should we take as the
default value? From the code or the doc? I would like to hear your
discussions.

[1] 
https://github.com/apache/pulsar/blob/d11147616aa6cc7888420f6325bb71cd7f7ab065/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConsumerConfigurationData.java#L112-L113
[2] 
https://github.com/apache/pulsar/blob/1e2ff8a3941b7cc6d583f528ceedc393b7e607fb/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java#L690

Thanks,
Zike Yang

Re: [DISCUSS] The default value of maxPendingChunkedMessage

[Zike Yang]

Hi, Yunze,

Thanks for your opinion.

> A chunked buffer to contain all chunks could use much memory, for example, if 
> a
> message was split into N chunks, since each chunk is 5MB by default, then 100
> buffers will use N*500 MB. It could reach 1GB if N > 2.
>
> In addition, normally, only if at least 100 producers sent messages to a
> partition would it be meaningful to configure maxPendingChunkedMessages to 
> 100.
> IMO, it's hard to see so many producers on a partition in production.

+1. I agree with you. And keeping the current default value in the
code (10) will not change the default behavior of the current client.
If there are no other objections, I would like to fix this
inconsistency in the java client.

Thanks,
Zike


On Sun, Jan 30, 2022 at 7:26 PM Yunze Xu  wrote:
>
> After thinking for a while, I’d prefer 10 as the default value and I changed
> the default value to 10 in C++ client, see
> https://github.com/apache/pulsar/pull/14070.
>
> A chunked buffer to contain all chunks could use much memory, for example, if 
> a
> message was split into N chunks, since each chunk is 5MB by default, then 100
> buffers will use N*500 MB. It could reach 1GB if N > 2.
>
> In addition, normally, only if at least 100 producers sent messages to a
> partition would it be meaningful to configure maxPendingChunkedMessages to 
> 100.
> IMO, it's hard to see so many producers on a partition in production.
>
> Thanks,
> Yunze Xu
>
> > 2022年1月30日 下午6:32，Zike Yang  写道：
> >
> > Hi, Pulsar community,
> >
> > We found that there are inconsistencies between the code and the
> > documentation regarding the default value of maxPendingChunkedMessage.
> >
> > In the java client code, we use 10 as the default value. [1] But in
> > the java doc, we use 100 as the default value. [2]
> > We need to fix this inconsistency. But what should we take as the
> > default value? From the code or the doc? I would like to hear your
> > discussions.
> >
> > [1] 
> > https://github.com/apache/pulsar/blob/d11147616aa6cc7888420f6325bb71cd7f7ab065/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConsumerConfigurationData.java#L112-L113
> > [2] 
> > https://github.com/apache/pulsar/blob/1e2ff8a3941b7cc6d583f528ceedc393b7e607fb/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java#L690
> >
> > Thanks,
> > Zike Yang
>

Re: [DISCUSS] The default value of maxPendingChunkedMessage

[Zike Yang]

Thanks for all your suggestions.

> For further improvement, I think we can deprecate `maxPendingChunkedMessage` 
> by extending the scope of  `ClientBuilder#memoryLimit` to consumers.

+1

I have created a PR to fix this inconsistency:
https://github.com/apache/pulsar/pull/14144 PTAL.

Thanks,
Zike Yang

On Sat, Feb 5, 2022 at 6:41 PM Haiting Jiang  wrote:
>
> > I agree with updating the Javadoc to align with the actual code. This
> > will lead to fewer surprises,
>
> +1.
>
> For further improvement, I think we can deprecate `maxPendingChunkedMessage` 
> by extending the scope of  `ClientBuilder#memoryLimit` to consumers.
>
> Thanks,
> Haiting
>
> On 2022/02/01 05:23:06 Michael Marshall wrote:
> > > We found that there are inconsistencies between the code and the
> > > documentation regarding the default value of maxPendingChunkedMessage.
> >
> > Great find!
> >
> > > A chunked buffer to contain all chunks could use much memory, for 
> > > example, if a
> > > message was split into N chunks, since each chunk is 5MB by default, then 
> > > 100
> > > buffers will use N*500 MB. It could reach 1GB if N > 2.
> >
> > This is a very good point.
> >
> > I agree with updating the Javadoc to align with the actual code. This
> > will lead to fewer surprises, and as Yunze Xu pointed out, a 10x
> > increase in the default could have dramatic effects on client memory
> > usage.
> >
> > Thanks,
> > Michael
> >
> >
> > On Sun, Jan 30, 2022 at 8:58 PM Zike Yang
> >  wrote:
> > >
> > > Hi, Yunze,
> > >
> > > Thanks for your opinion.
> > >
> > > > A chunked buffer to contain all chunks could use much memory, for 
> > > > example, if a
> > > > message was split into N chunks, since each chunk is 5MB by default, 
> > > > then 100
> > > > buffers will use N*500 MB. It could reach 1GB if N > 2.
> > > >
> > > > In addition, normally, only if at least 100 producers sent messages to a
> > > > partition would it be meaningful to configure maxPendingChunkedMessages 
> > > > to 100.
> > > > IMO, it's hard to see so many producers on a partition in production.
> > >
> > > +1. I agree with you. And keeping the current default value in the
> > > code (10) will not change the default behavior of the current client.
> > > If there are no other objections, I would like to fix this
> > > inconsistency in the java client.
> > >
> > > Thanks,
> > > Zike
> > >
> > >
> > > On Sun, Jan 30, 2022 at 7:26 PM Yunze Xu  
> > > wrote:
> > > >
> > > > After thinking for a while, I’d prefer 10 as the default value and I 
> > > > changed
> > > > the default value to 10 in C++ client, see
> > > > https://github.com/apache/pulsar/pull/14070.
> > > >
> > > > A chunked buffer to contain all chunks could use much memory, for 
> > > > example, if a
> > > > message was split into N chunks, since each chunk is 5MB by default, 
> > > > then 100
> > > > buffers will use N*500 MB. It could reach 1GB if N > 2.
> > > >
> > > > In addition, normally, only if at least 100 producers sent messages to a
> > > > partition would it be meaningful to configure maxPendingChunkedMessages 
> > > > to 100.
> > > > IMO, it's hard to see so many producers on a partition in production.
> > > >
> > > > Thanks,
> > > > Yunze Xu
> > > >
> > > > > 2022年1月30日 下午6:32，Zike Yang  写道：
> > > > >
> > > > > Hi, Pulsar community,
> > > > >
> > > > > We found that there are inconsistencies between the code and the
> > > > > documentation regarding the default value of maxPendingChunkedMessage.
> > > > >
> > > > > In the java client code, we use 10 as the default value. [1] But in
> > > > > the java doc, we use 100 as the default value. [2]
> > > > > We need to fix this inconsistency. But what should we take as the
> > > > > default value? From the code or the doc? I would like to hear your
> > > > > discussions.
> > > > >
> > > > > [1] 
> > > > > https://github.com/apache/pulsar/blob/d11147616aa6cc7888420f6325bb71cd7f7ab065/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConsumerConfigurationData.java#L112-L113
> > > > > [2] 
> > > > > https://github.com/apache/pulsar/blob/1e2ff8a3941b7cc6d583f528ceedc393b7e607fb/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java#L690
> > > > >
> > > > > Thanks,
> > > > > Zike Yang
> > > >
> >

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

[Zike Yang]

Hi, Zixuan
Thanks for creating this PIP.  Here are my thoughts.

> CursorInfo compression format
>
> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + 
> [MANAGED_CURSOR_INFO_PAYLOAD]
>
> MAGIC_NUMBER: Ox4779

Since we change the ManagedCursorInfo data format here. How do we
handle the old data format in the ZK. Could you explain the
compatibility for this PIP?

Thanks,
Zike Yang

On Wed, Mar 2, 2022 at 3:34 PM Zixuan Liu  wrote:
>
> Hi Pulsar Community,
>
> I create a proposal that ManagedCursorInfo compression. The proposal can be 
> found: https://github.com/apache/pulsar/issues/14529 
> <https://github.com/apache/pulsar/issues/14529>
>
> Thanks,
> Zixuan
>
> --
>
> Motivation
>
> The cursor data is managed by ZooKeeper/etcd metadata store. When cursor data 
> becomes more and more, the data size will increase and will take a lot of 
> time to pull the data. Therefore, it is necessary to add compression for the 
> cursor, which can reduce the size of data and reduce the time of pulling data.
>
> Goal
>
> Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the ManagedCursorInfo.
>
> Implementation
>
> CursorInfo compression format
>
> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + 
> [MANAGED_CURSOR_INFO_PAYLOAD]
>
> MAGIC_NUMBER: Ox4779
>
> METADATA
> Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto
>
> message ManagedCursorInfoMetadata {
> required CompressionType compressionType = 1;
> required int32 uncompressedSize = 2;
> }
> CursorInfo compression and decompression design
>
> Currently, these compressions types have been defined and implemented by 
> Pulsar, we only need to deal with compression and decompression of the 
> ManagedCursorInfo data:
>
> Get CursorInfo from the metadata store
> We will check the cursor data header, if it is compressed, we will parse the 
> bytes data by compressed format, otherwise by the original way.
>
> Add/Update CursorInfo to the metadata store
> The default is to use compression if the compression type is specified.
>
> CursorInfo compression type configuration
>
> Add managedCursorInfoCompressionType in 
> org.apache.pulsar.broker.ServiceConfiguration and 
> org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig.
>
>


-- 
Zike Yang

Re: [VOTE][PIP-146] ManagedCursorInfo compression

[Zike Yang]

+1

Thanks,
Zike Yang

On Mon, Mar 14, 2022 at 11:27 AM r...@apache.org
 wrote:
>
> +1
>
> --
> Thanks
> Xiaolong Ran
>
> Hang Chen  于2022年3月14日周一 09:55写道：
>
> > +1
> >
> > Thanks,
> > Hang
> >
> > Haiting Jiang  于2022年3月10日周四 16:21写道：
> > >
> > > +1
> > >
> > > Thanks,
> > > Haiting
> > >
> > > On 2022/03/09 09:18:04 Zixuan Liu wrote:
> > > > Hi Pulsar community,
> > > >
> > > > I want to start this PIP -
> > https://github.com/apache/pulsar/issues/14529
> > > >
> > > > Thanks,
> > > > Zixuan
> > > >
> >



-- 
Zike Yang

Re: [VOTE][PIP-146] ManagedCursorInfo compression

[Zike Yang]

Hi Zixuan,

This proposal has 3 (+1) bindings and 0 (-1) and has stayed open for
at least 48 hours. You can close this vote now and the proposal has
been accepted.

Thanks,
Zike Yang

On Tue, Mar 15, 2022 at 10:50 AM Zixuan Liu  wrote:
>
> Hi Pulsar community,
>
> Thank you for your vote. Is this proposal acceptable?
>
> Thanks,
> Zixuan
>
> Zike Yang  于2022年3月14日周一 14:26写道：
>
> > +1
> >
> > Thanks,
> > Zike Yang
> >
> > On Mon, Mar 14, 2022 at 11:27 AM r...@apache.org
> >  wrote:
> > >
> > > +1
> > >
> > > --
> > > Thanks
> > > Xiaolong Ran
> > >
> > > Hang Chen  于2022年3月14日周一 09:55写道：
> > >
> > > > +1
> > > >
> > > > Thanks,
> > > > Hang
> > > >
> > > > Haiting Jiang  于2022年3月10日周四 16:21写道：
> > > > >
> > > > > +1
> > > > >
> > > > > Thanks,
> > > > > Haiting
> > > > >
> > > > > On 2022/03/09 09:18:04 Zixuan Liu wrote:
> > > > > > Hi Pulsar community,
> > > > > >
> > > > > > I want to start this PIP -
> > > > https://github.com/apache/pulsar/issues/14529
> > > > > >
> > > > > > Thanks,
> > > > > > Zixuan
> > > > > >
> > > >
> >
> >
> >
> > --
> > Zike Yang
> >



-- 
Zike Yang

Re: [DISCUSS] PIP-143 : Standardize Admin REST API

[Zike Yang]

+1

Thanks,
Zike Yang

On Tue, Mar 22, 2022 at 8:32 PM 石宝迪  wrote:
>
> +1
>
>
> > 在 2022年3月22日，19:59，Haiting Jiang  写道：
> >
> > +1
> >
> > Thanks,
> > Haiting
> >
> >> On 2022/03/16 12:43:45 guo jiwei wrote:
> >> Hello community,
> >>
> >>I want to discuss refactoring and standardizing REST API. Users have
> >> encountered several deadlock problems in the API part before, like #13666
> >> <https://github.com/apache/pulsar/pull/13666>, #12590
> >> <https://github.com/apache/pulsar/pull/12590>. After fixing the above two
> >> issues, we check the related part and find that there are still numbers of
> >> asynchronous call synchronous implementations for other functions of the
> >> REST API. To avoid more problems, I would like to discuss the
> >> standardization of API modules and create the PIP-142  #14365
> >> <https://github.com/apache/pulsar/pull/14365> . Feel free to comment and
> >> give suggestions.
> >>
> >>
> >>
> >>
> >> Regards
> >> Jiwei Guo (Tboy)
> >>

Re: [VOTE] PIP-149: Making the REST Admin API fully async

[Zike Yang]

+1,

Zike Yang

On 2022/03/29 09:22:16 guo jiwei wrote:
> +1
> 
> Regards
> Jiwei Guo (Tboy)
> 
> 
> On Tue, Mar 29, 2022 at 4:16 PM Enrico Olivelli  wrote:
> 
> > +1 (binding)
> >
> > Enrico
> >
> > Il giorno mar 29 mar 2022 alle ore 09:55 PengHui Li
> >  ha scritto:
> > >
> > > +1,
> > >
> > > Penghui
> > >
> > > On Fri, Mar 25, 2022 at 10:13 AM mattison chao 
> > > wrote:
> > >
> > > > This is the voting thread for PIP-149. It will stay open for at least
> > 48
> > > > hours.
> > > >
> > > > https://github.com/apache/pulsar/issues/14365
> > > >
> > > > Pasted below for quoting convenience.
> > > >
> > > > -
> > > >
> > > > Motivation
> > > >
> > > > The Rest API was originally designed to be implemented asynchronously,
> > but
> > > > with the iteration of functions, some synchronous implementations were
> > > > added, resulting in many asynchronous methods called synchronous
> > > > implementations. Also, many synchronous calls do not add timeouts. This
> > > > greatly reduces concurrency, user operations, and experience.
> > > > In order to prevent more problems, and improve code readability and
> > > > maintainability, we intend to refactor these synchronous calls and
> > > > standardize the implementation of the API.
> > > >
> > > > Related discussion:
> > > > https://lists.apache.org/thread/pkkz2jgwtzpksp6d4rdm1pyxzb3z6vmg
> > > >
> > > > Goals
> > > >
> > > > Try to avoid synchronous method calls in asynchronous methods.
> > > > Async variable (AsyncResponse) is placed in the first parameter
> > position.
> > > > Async variable (AsyncResponse) cannot be substituted into method
> > > > implementations.
> > > > Add more tests and increase the coverage.
> > > > Modification
> > > > Avoid synchronous method calls in asynchronous methods.
> > > >
> > > > protected void internalDeleteNamespace(boolean authoritative) {
> > > >validateTenantOperation(namespaceName.getTenant(),
> > > > TenantOperation.DELETE_NAMESPACE);
> > > >validatePoliciesReadOnlyAccess();
> > > > }
> > > > Suggest to do like this:
> > > >
> > > > protected CompletableFuture internalDeleteNamespace(boolean
> > > > authoritative) {
> > > > return validateTenantOperationAsync(namespaceName.getTenant(),
> > > > TenantOperation.DELETE_NAMESPACE)
> > > >.thenCompose(__ -> validatePoliciesReadOnlyAccessAsync());
> > > > }
> > > > Async variable (AsyncResponse) is placed in the first parameter
> > position
> > > >
> > > > public void deleteNamespace(@Suspended final AsyncResponse
> > asyncResponse,
> > > >  @PathParam("tenant") String tenant,
> > > >  @PathParam("namespace") String namespace,
> > > >  @QueryParam("force") @DefaultValue("false") boolean force,
> > > >  @QueryParam("authoritative") @DefaultValue("false") boolean
> > > > authoritative) {
> > > >
> > > > Async variable (AsyncResponse) cannot be substituted into method
> > > > implementations
> > > >
> > > > internalCreateNonPartitionedTopicAsync(asyncResponse, authoritative,
> > > > properties);
> > > > Suggest to do like this:
> > > >
> > > > internalCreateNonPartitionedTopicAsync(authoritative, properties)
> > > > .thenAccept(__ ->
> > > > asyncResponse.resume(Response.noContent().build()))
> > > > .exceptionally(ex -> {
> > > > resumeAsyncResponseExceptionally(asyncResponse,
> > ex.getCause());
> > > > return null;
> > > > });
> > > >
> > > > Task tracking
> > > > In order to unify the modification and track the modified part, it's
> > better
> > > > to open an issue to track, like #14353, #14013, #13854.
> > > >
> > > > ---
> > > > Best,
> > > > Mattison
> > > >
> >
> 

Re: [DISSCUSS] [PIP-150] Support read the message of startMessageId position on the broker side

[Zike Yang]

Hi, Zixuan

Thanks for your PIP. I see that you have updated the PIP. Here are my thoughts.

> ### Compatibility
>
> No breaking changes are introduced here, the users can use any client to 
> request any broker.

Do you mean that this is both backward and forward compatibility?

Assume that I have such a consumer below:

```
Consumer consumer = pulsarClient.newConsumer().topic(topicName)
.subscriptionName("my-sub").startMessageIdInclusive()
.subscribe();
```

Does this PIP change the original behavior of this consumer? If it
does, this may introduce the breaking change here.

It would be better if the compatibility part could be described more
specifically.

Thanks,
Zike Yang

On Mon, Mar 28, 2022 at 11:37 AM Zixuan Liu  wrote:
>
> Thanks Enrico,
>
> Good suggestion.
>
> > do you mean that you want to start reading from the latest message?
>
> Yes.
>
> > I suggest to add more messages to the example, in order to make it clearer
> > (like 1,2,3,4,5)
>
> Added.
>
>
> > I am not sure if this will change the semantics of existing programs.
>
> Your worry is necessary. I need to know more people's views.
>
> > Shall we have to add a new configuration option?
>
> I think we should add a new configuration here. Do you mean that you start
> reading from the latest message when using the `latest`? If so, the old
> behavior will be broken.
>
>
>
> Enrico Olivelli  于2022年3月26日周六 05:42写道：
>
> > Il Ven 25 Mar 2022, 17:36 Zixuan Liu  ha scritto:
> >
> > > Hi Pulsar community,
> > >
> > > I create a PIP for support read the message of `startMessageId` position
> > on
> > > the broker side.
> > >
> > > The proposal can be found: https://github.com/apache/pulsar/issues/14883
> > >
> > > -
> > > ## Motivation
> > >
> > > Currently, the Pulsar-client supports setting the `startMessageId` for
> > > Consumer and Reader, and also supports reading the message of
> > > `startMessageId` position.
> > >
> > > Assume, we have two message id 1 and 2:
> > >
> > > - When we set `earliest` as `strartMessageId` value, we can the message
> > of
> > > message id 1
> > > - When we set `latest` as as `strartMessageId` value, we cannot get any
> > > message
> > >
> > > Sometimes we want to read the message id 2 for the first time,
> >
> > do you mean that you want to start reading from the latest message?
> >
> > I suggest to add more messages to the example, in order to make it clearer
> > (like 1,2,3,4,5)
> >
> >
> >
> > we have only
> > > one approach in client:
> > >
> > > ```
> > >  Reader reader = pulsarClient.newReader()
> > > .topic(topicName)
> > > .subscriptionName(subscriptionName)
> > > .startMessageId(MessageId.latest)
> > > .startMessageIdInclusive()
> > > .create();
> > >
> > > reader.hasMessageAvailable();
> > > Message msg = reader.readNext(1, TimeUnit.SECONDS);
> > > ```
> > >
> > > Call `reader.hasMessageAvailable()` before `reader.readNext()`, which can
> > > seek to the message id 2, this approach seems confusing.
> >
> >
> > Agreed
> >
> >
> > If we do this on
> > > the broker side, which will make things easier.
> > >
> > > ## Goal
> > >
> > > This PIP proposes support read the message of `startMessageId` position
> > on
> > > the broker side.
> > >
> > > ## Implementation
> > >
> > > ### Protocol
> > >
> > > Add a `start_message_id_inclusive` field to `CommandSubscribe` for
> > > determine whether to read the message of `startMessageId` position:
> > >
> > > ```
> > > message CommandSubscribe {
> > > // some fields
> > >
> > > // If specified, the subscription will read the message from the
> > start
> > > message id position.
> > > optional bool start_message_id_inclusive = 20 [default = false];
> > > }
> > > ```
> > >
> > > ### ManagedCursorImpl
> > >
> > > Add a check in
> > >
> > >
> > `org.apache.bookkeeper.mledger.impl.ManagedCursorImpl#initializeCursorPosition`.
> > >
> > >
> > > We only need to care that the `startMessageId` is `MessageId.latest` and
> > > the`start_message_id_inclusive` is `true`, we get latest position from
> > > ledger as `readPosition` value, otherwise if
> > > the`start_message_id_inclusive` is `false`,  get next position of the
> > > latest position as `readPosition` value.
> > >
> >
> > I am not sure if this will change the semantics of existing programs.
> >
> > Shall we have to add a new configuration option?
> >
> > Otherwise programs written the way you are describing won't work anymore
> >
> > Enrico
> >
> >
> >
> >
> > > -
> > >
> > > Thanks,
> > > Zixuan
> > >
> >

Re: [ANNOUNCE] New Committer: Zike Yang

[Zike Yang]

Thank you all! Very glad to be the pulsar committer.

Thanks,
Zike Yang

On Fri, Apr 15, 2022 at 10:53 AM Huanli Meng
 wrote:
>
> Good for you, Zike. Congratulations.
>
> BR//Huanli
>
> > On Apr 14, 2022, at 9:28 AM, Yu  wrote:
> >
> > Congrats Zike! Well deserved!
> >
> > On Wed, Apr 13, 2022 at 7:00 PM Enrico Olivelli  wrote:
> >
> >> Congratulations
> >>
> >> Enrico
> >>
> >> Il Mer 13 Apr 2022, 12:38 Hang Chen  ha scritto:
> >>
> >>> Congrats Zike!
> >>>
> >>> Best,
> >>> Hang
> >>>
> >>> Haiting Jiang  于2022年4月13日周三 18:16写道：
> >>>>
> >>>> Congrats!
> >>>>
> >>>> Thanks,
> >>>> Haiting
> >>>>
> >>>> On 2022/04/13 09:34:23 PengHui Li wrote:
> >>>>> The Apache Pulsar Project Management Committee (PMC) has invited Zike
> >>> Yang
> >>>>> https://github.com/RobertIndie to become a committer and we are
> >>> pleased to
> >>>>> announce that he has accepted.
> >>>>>
> >>>>> Welcome and Congratulations, Zike Yang!
> >>>>>
> >>>>> Please join us in congratulating and welcoming Zike Yang onboard!
> >>>>>
> >>>>> Best Regards,
> >>>>> Penghui Li on behalf of the Pulsar PMC
> >>>>>
> >>>
> >>
>

Re: [VOTE] [PIP-152] Support subscription level dispatch rate limiter setting.

[Zike Yang]

+1

Thanks,
Zike

On Wed, Apr 20, 2022 at 9:20 AM Dezhi Liu
 wrote:
>
> +1
>
> Dezhi Liu

Re: [DISCUSS] PIP-155: Drop support for Python2

[Zike Yang]

+1

Thanks,
Zike

On Wed, Apr 20, 2022 at 5:22 PM Christophe Bornet
 wrote:
>
> +1
>
> Le ven. 15 avr. 2022 à 18:06, Matteo Merli  a écrit :
>
> > https://github.com/apache/pulsar/issues/15185
> >
> > -
> >
> > ## Motivation
> >
> > Python 2.x has been deprecated for many years now and it was
> > officially end-of-lifed 2.5 years ago
> > (https://www.python.org/doc/sunset-python-2/).
> >
> > We have well reached the point by which we need to drop Python 2.7
> > compatibility for Pulsar client and for Pulsar functions.
> >
> > ## Goal
> >
> > Support only Python 3.5+ for Pulsar client and for Pulsar functions.
> >
> > ## API Changes
> >
> > No changes at this time, though Pulsar Python client library will be
> > now free to use Python3 specific syntaxes and libraries.
> >
> > ## Changes
> >
> > 1. Switch the CI build to run Python client lib tests with Python3
> > 2. Switch integration tests to use Python3
> > 3. Stop building and distributing wheel files for Python 2.7
> >
> >
> >
> >
> > --
> > Matteo Merli
> > 
> >

Re: [VOTE] PIP-155: Drop support for Python2

[Zike Yang]

+1 (no-binding)

Thanks,
Zike

On Thu, Apr 21, 2022 at 9:24 AM Dezhi Liu
 wrote:
>
> +1
>
> Dezhi Liu

Re: [DISCUSSION] PIP-156: Enable system topic by default

[Zike Yang]

+1

Please update this PIP number. It's duplicated with
https://github.com/apache/pulsar/issues/15207

Zike

On Tue, Apr 26, 2022 at 8:09 PM Enrico Olivelli  wrote:
>
> Which are the downsides of enabling system topics?
>
> Enrico
>
> Il Mar 26 Apr 2022, 14:07 guo jiwei  ha scritto:
>
> > Hi community:
> >In PIP-39
> > ,
> > we have introduced namespace event to support topic level policy. This
> > feature was released in Pulsar 2.6.
> >
> >Then PIP-92
> > <
> > https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
> > >,
> > we support topic policy to replicate across clusters and release in Pulsar
> > 2.10.
> >
> > And with version iteration, the corresponding topic policy CLI and
> > functions are mature and stable, but users need to enable system topic to
> > support these features.  So it's better to enable system topic by default.
> >
> >
> > So I have opened apache/pulsar#15333
> >  to discuss this.
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >

Re: [VOTE] [PIP-150] Support read the message of startMessageId position on the broker side

[Zike Yang]

+1 (non-binding)

Thanks,
Zike

On Wed, Apr 27, 2022 at 10:44 AM Lin Lin  wrote:
>
> +1
>
> Lin Lin

Re: [ANNOUNCE] New Committer: Nicolò Boschi

[Zike Yang]

Congrats, Nicolò!

Best,
Zike

On Wed, Apr 27, 2022 at 2:34 PM Enrico Olivelli  wrote:
>
> Congratulations !
>
> Enrico
>
> Il giorno mer 27 apr 2022 alle ore 05:40 Hang Chen
>  ha scritto:
> >
> > Congrats, Nicolò!
> >
> > Best,
> > Hang
> >
> > ZhangJian He  于2022年4月27日周三 11:35写道：
> > >
> > > Congratulations, Nicolò!
> > >
> > > Thanks
> > > ZhangJian He
> > >
> > > Michael Marshall  于2022年4月27日周三 11:24写道：
> > >
> > > > Congratulations, Nicolò!
> > > >
> > > > - Michael
> > > >
> > > > On Tue, Apr 26, 2022 at 9:39 PM Haiting Jiang 
> > > > wrote:
> > > > >
> > > > > Congrats!
> > > > >
> > > > > BR,
> > > > > Haiting
> > > > >
> > > > > On 2022/04/26 23:48:25 PengHui Li wrote:
> > > > > > The Apache Pulsar Project Management Committee (PMC) has invited 
> > > > > > Nicolò
> > > > > > Boschi
> > > > > > https://github.com/nicoloboschi to become a committer and we are
> > > > pleased to
> > > > > > announce that he has accepted.
> > > > > >
> > > > > > Welcome and Congratulations, Nicolò Boschi
> > > > > >
> > > > > > Please join us in congratulating and welcoming Nicolò Boschi 
> > > > > > onboard!
> > > > > >
> > > > > > Best Regards,
> > > > > > Penghui Li on behalf of the Pulsar PMC
> > > > > >
> > > >

Re: Call for projects and mentors for OSPP 2022

[Zike Yang]

Thanks all, here is my application:

Project Name: Support processing large messages at Go client
Project Description:

Apache Pulsar imposes a size limit on each message sent to the broker.
Currently, the pulsar java client support message chunking which
enables Pulsar to process large payload messages by splitting the
message into chunks at the producer side and combining chunked
messages at the consumer side. But the Pulsar go client doesn't
support this feature. This project aims to add support for message
chunking to process large messages at the go client.

Difficulty Level:
- [ ] Basic
- [x] Advanced
Project Validation Items:
Item 1: Write a proposal to describe your design
Item 2: Implement the message chunking feature at the go client and
push the PR out
Item 3: Add unit tests to verify it
Item 4: Add the documentation for this feature
…
Project Mentor: Zike Yang
Your Name: Zike Yang
Your Email:
Your Apache ID:

On Fri, Apr 29, 2022 at 5:06 PM Yu  wrote:
>
> Thanks all, here are my applications (in English and Chinese).
>
> 
> English version
> 
>
> # Project Name
> Automate Documentation Workflow to Optimize Website User Experience
>
> # Project Description
>
> ## Project Goal
>
> This project aims to improve the user experience on the new Pulsar website
> by optimizing and automating workflow for documentation and website,
> including but not limited to generating documentation websites from code
> automatically, standardizing pull request naming convention using GitHub
> Actions, and more. You can check some related tasks on
> https://github.com/apache/pulsar/issues?q=is%3Aopen+is%3Aissue+label%3Awebsite.
>
>
> ## What You Can Learn from the Project
>
> - Increase your technical knowledge.
> You will get familiar with codebase and learn technical skills about
> front-end and back-end developments of Pulsar. You will be guided by
> veterans who have been doing this for years and who know the tips and
> tricks to help you be a more excellent open-source developer.
>
> - Sharpen your software skills.
> Pulsar adoption engagement skyrocketed over the past years. This success
> depends on its community. By participating in a project in the Pulsar
> community, you can learn the Apache culture, communication skills, and best
> practices to collaborate effectively in the open-source world. This paves
> your way forward for making big contributions.
>
> - Build your network.
> You can interact and share ideas with knowledgeable and passionate
> contributors in the Pulsar community. Moreover, you can make great friends
> from all over the world.
>
> # Difficulty Level
> - [ ] Basic
> - [x] Advanced
>
> # Project Validation Items
> - For the project: set achievable goals and make reasonable plans; complete
> project before deadline and achieve expected outcomes.
> - For the student: acquire new skills and level up your developer portfolio.
>
> # Project Mentor
> Yu Liu
>
> # Your Email
> li...@apache.org
> y...@streamnative.io
>
> # Your Apache ID
> liuyu
>
> 
> Chinese version
> 
>
> # 项目名称
> 文档开发流程自动化 | 优化内容开发体验
>
> # 项目描述
>
> ## 项目目标
> Apache Pulsar
> 近期将发布新版官网，为用户带来全新的内容体验。优质的内容体验离不开高效的内容开发流程，为了提升内容开发效率，该项目旨在于自动化内容开发流程，包括但不限于从代码中自动生成内容网页、通过
> GitHub Actions 规范 Pull Request 内容等。更多相关任务，请参阅
> https://github.com/apache/pulsar/issues?q=is%3Aopen+is%3Aissue+label%3Awebsite。
>
>
> # 你能从这个项目中获得什么收益？
> - 提升硬技能
> Pulsar 是下一代云原生分布式消息流平台，具备广阔的应用前景。通过参与该项目，你将熟悉 Pulsar
> 代码，了解相关技术，获得资深开发人员的专业指导，提高开发能力。
>
> - 增强软实力
> 最近几年，得益于高效的社区管理，Pulsar 发展势如破竹。通过参与该项目，你将了解 Apache 开源社区的协作文化、治理理念与运营模式等。
>
> - 打造高效社交网络
> Pulsar 社区汇集了全球各地的能人志士，你将与他们紧密合作，增进友谊，提升社交价值感和幸福感。
>
> # 项目难度
> - [ ] 初级
> - [x] 高级
>
> # 项目验收标准
> - 对于项目而言，需制定合理的目标和清晰的计划，如期完成并取得预期成果。
> - 对于学生而言，收获有利于人生发展的软硬技能。
>
> # 项目导师
> Yu Liu
>
> # 导师邮箱
> li...@apache.org
> y...@streamnative.io
>
> # 导师 Apache ID
> liuyu
>
>
> On Tue, Apr 26, 2022 at 2:26 PM Hang Chen  wrote:
>
> > Thanks for Penghui and Dianjin.
> >
> > Project Name: Compaction with tiered storage
> >
> > Project Description: (at most 1000 words)
> >
> > Use the tiered storage to store the compacted data to save costs and
> > to avoid the impaction of the BookKeeper cluster.
> >
> > The topic compaction reads all the compacted data along with the
> > additional data after the last compaction. If many topics enabled
> > topic compaction or with a large size compacted data, the compaction
> > task will impact the BookKeeper cluster. Leverage the tiered storag

Re: Call for projects and mentors for OSPP 2022

[Zike Yang]

Hi,

Thank Yu!

Sorry for the missing information:

---

Your Email: z...@apache.org
Your Apache ID: zike

And here is the Chinese version of my application:

---

Project Name: 在Go客户端中支持大消息处理
Project Description:
Apache 
Pulsar对发往broker的消息引入了大小限制。目前，Pulsar的java客户端支持了消息分块的功能，使得Pulsar在生产者中能够将消息分成多个小块并在消费者中将它们组装起来，以支持大消息的处理。但是Pulsar的Go客户端并不支持这个特性。本项目将在Go客户端中添加对大消息处理的支持。

Difficulty Level:
- [ ] Basic
- [x] Advanced

Project Validation Items:
Item 1: 编写一份方案来描述你的设计
Item 2: 在Go客户端上实现消息分块的功能并推送PR
Item 3: 添加单元测试以进行验证
Item 4: 添加关于这个特性的文档
…
Project Mentor: 杨子棵
Your Name: 杨子棵
Your Email: z...@apache.org
Your Apache ID: zike

On Sat, Apr 30, 2022 at 9:51 AM Yu  wrote:
>
> Hi Zike, looks like your email and Apache ID are missing?
>
> On Fri, Apr 29, 2022 at 10:17 PM Zike Yang  wrote:
>
> > Thanks all, here is my application:
> >
> > Project Name: Support processing large messages at Go client
> > Project Description:
> >
> > Apache Pulsar imposes a size limit on each message sent to the broker.
> > Currently, the pulsar java client support message chunking which
> > enables Pulsar to process large payload messages by splitting the
> > message into chunks at the producer side and combining chunked
> > messages at the consumer side. But the Pulsar go client doesn't
> > support this feature. This project aims to add support for message
> > chunking to process large messages at the go client.
> >
> > Difficulty Level:
> > - [ ] Basic
> > - [x] Advanced
> > Project Validation Items:
> > Item 1: Write a proposal to describe your design
> > Item 2: Implement the message chunking feature at the go client and
> > push the PR out
> > Item 3: Add unit tests to verify it
> > Item 4: Add the documentation for this feature
> > …
> > Project Mentor: Zike Yang
> > Your Name: Zike Yang
> > Your Email:
> > Your Apache ID:
> >
> > On Fri, Apr 29, 2022 at 5:06 PM Yu  wrote:
> > >
> > > Thanks all, here are my applications (in English and Chinese).
> > >
> > > 
> > > English version
> > > 
> > >
> > > # Project Name
> > > Automate Documentation Workflow to Optimize Website User Experience
> > >
> > > # Project Description
> > >
> > > ## Project Goal
> > >
> > > This project aims to improve the user experience on the new Pulsar
> > website
> > > by optimizing and automating workflow for documentation and website,
> > > including but not limited to generating documentation websites from code
> > > automatically, standardizing pull request naming convention using GitHub
> > > Actions, and more. You can check some related tasks on
> > >
> > https://github.com/apache/pulsar/issues?q=is%3Aopen+is%3Aissue+label%3Awebsite
> > .
> > >
> > >
> > > ## What You Can Learn from the Project
> > >
> > > - Increase your technical knowledge.
> > > You will get familiar with codebase and learn technical skills about
> > > front-end and back-end developments of Pulsar. You will be guided by
> > > veterans who have been doing this for years and who know the tips and
> > > tricks to help you be a more excellent open-source developer.
> > >
> > > - Sharpen your software skills.
> > > Pulsar adoption engagement skyrocketed over the past years. This success
> > > depends on its community. By participating in a project in the Pulsar
> > > community, you can learn the Apache culture, communication skills, and
> > best
> > > practices to collaborate effectively in the open-source world. This paves
> > > your way forward for making big contributions.
> > >
> > > - Build your network.
> > > You can interact and share ideas with knowledgeable and passionate
> > > contributors in the Pulsar community. Moreover, you can make great
> > friends
> > > from all over the world.
> > >
> > > # Difficulty Level
> > > - [ ] Basic
> > > - [x] Advanced
> > >
> > > # Project Validation Items
> > > - For the project: set achievable goals and make reasonable plans;
> > complete
> > > project before deadline and achieve expected outcomes.
> > > - For the student: acquire new skills and level up your developer
> > portfolio.
> > >
> > > # Project Mentor
> > > Yu Liu
> > >
> > > # Your Email
> > > li...@apache.org
> > > y...@streamnative.io
> > >
> > > # Your Apa

Re: [ANNOUNCE] New Committer: Dezhi Liu

[Zike Yang]

Congratulations!

Best Regards,
Zike Yang

On Tue, Jun 7, 2022 at 3:52 PM ZhangJian He  wrote:

> Congratulations!
>
> Thanks
> ZhangJian He
>
> Haiting Jiang  于2022年6月7日周二 15:46写道：
>
> > Congrats!
> >
> > BR,
> > Haiting
> >
> > On 2022/06/07 06:46:00 Hang Chen wrote:
> > > The Project Management Committee (PMC) for Apache Pulsar has invited
> > > Dezhi Liu (https://github.com/liudezhi2098) to become a committer and
> > > we are pleased to announce that he has accepted.
> > >
> > > Dezhi Liu (with Github id liudezhi2098) contributed many improvements
> > > and bug fixes to Pulsar.
> > >
> > > Being a committer enables easier contribution to the project since
> > > there is no need to go via the patch submission process. This should
> > > enable better productivity.
> > >
> > > Welcome and Congratulations, Dezhi Liu!
> > >
> > > Please join us in congratulating and welcoming Dezhi Liu onboard!
> > >
> > > Best Regards,
> > > Hang Chen on behalf of the Pulsar PMC
> > >
> >
>

Re: [VOTE] PIP-165: Auto release client useless connections

[Zike Yang]

+1


Zike Yang

On Wed, Jun 15, 2022 at 10:28 AM Haiting Jiang 
wrote:

> +1
>
> Haiting
>
> On 2022/06/14 10:28:22 Xiangying Meng wrote:
> > +1
> >
> > On Tue, Jun 14, 2022 at 3:52 PM Gavin Gao 
> wrote:
> >
> > > +1
> > >
> > >
> > > On 2022/06/01 07:11:51 Yubiao Feng wrote:
> > > > Hi Pulsar Community:
> > > >
> > > > There were some mistakes in the last email, so I have corrected them.
> > > >
> > > > I would like to start a VOTE on "Auto release client useless
> connections"
> > > > (PIP-165).
> > > >
> > > > Proposal Link: [PIP-165] Auto release client useless connections ·
> Issue
> > > > #15516 · apache/pulsar (github.com)
> > > > <https://github.com/apache/pulsar/issues/15516>
> > > >
> > > > Discuss Link: [DISCUSS] [PIP-165] Auto release client useless
> > > > connections-Apache Mail Archives
> > > > <https://lists.apache.org/thread/t6h98qs2coc56z06tw38hdlljl67ft4n>
> > > >
> > > > Voting will stay open for at least 48h.
> > > > Thanks, Yubiao Feng
> > > >
> > > > On Wed, Jun 1, 2022 at 2:40 PM Yubiao Feng <
> yubiao.f...@streamnative.io>
> > > > wrote:
> > > >
> > > > > [VOTE] PIP-165: Auto release client useless connectionsHi Pulsar
> > > > > Community, I would like to start a VOTE on "Auto release client
> useless
> > > > > connections" (PIP-165). The proposal can be read at
> > > > > https://github.com/apache/pulsar/issues/15516
> > > > > <https://github.com/apache/pulsar/issues/15254> and the discussion
> > > thead
> > > > > is available at
> > > > > https://lists.apache.org/thread/t6h98qs2coc56z06tw38hdlljl67ft4n
> > > > >   Voting will stay open for at least 48h. Thanks, Yubiao Feng
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] [PIP-179] Support the admin API to check unknown request parameters

[Zike Yang]

+1

Zike Yang

On Wed, Jun 22, 2022 at 11:26 AM PengHui Li  wrote:

> +1
>
> Penghui
>
> On Wed, Jun 22, 2022 at 10:53 AM Yubiao Feng
>  wrote:
>
> > Hi, Pulsar community:
> >
> > I open a pip to discuss "Support the admin API to check unknown request
> > parameters"
> >
> > Proposal Link: https://github.com/apache/pulsar/issues/16135
> >
> > ### Motivation
> >
> > The design of the Admin API is now such that if an incorrect parameter
> name
> > is submitted, this property (if not required) will be ignored, then
> > execution continues, and the response is “204 Success”. This will trick
> the
> > user into thinking the setup succeeded when it didn't correctly as
> expected
> > in some cases, as shown below:
> >
> > User POST request to /{tenant}/{namespace}/{topic}/retention" with
> > incorrect parameter:
> > ```json
> > {"retention_size_in_mb":-1,"retention_time_in_minutes":40320}
> > ```
> >
> > Which should have been this:
> >
> > ```json
> > {"retentionSizeInMB":-1,"retentionTimeInMinutes":40320}
> > ```
> >
> > Response:
> >
> > ```http
> > HTTP/1.1 204 No Content
> > Date: Mon, 20 Jun 2022 02:54:25 GMT
> > broker-address: 127.0.0.1
> > Server: Jetty(9.4.44.v20210927)
> > ```
> >
> > We can provide an optional mechanism: "fail (HTTP status 400 bad
> requests)
> > on unknown request parameters".
> >
> > ## Goal
> >
> > - scope:
> >   - ~~Path variables~~(no need for change):  This represents the domain.
> > The current API has been validated, so no additional modifications are
> > required.
> >   - ~~Query params~~(no support on this proposal):  I haven't found an
> > elegant way to do it yet, so this proposal does not include Query Param
> > validation.
> >   - *Entity properties*:  This proposal only handles requests whose
> > Content-type is "application/json" (in fact, this is the only type in our
> > project).
> > - Configurable(Support dynamic switching).
> >
> >
> > ## Approach
> >
> > When parsing the request body, any unknown property is considered a bad
> > request. The [Jackson unknown property rule](
> >
> >
> https://github.com/FasterXML/jackson-databind/blob/de3d0ecbc1fd0a1a6b061e62a198b3ba0d0d163e/src/main/java/com/fasterxml/jackson/databind/DeserializationFeature.java#L121
> > )
> > is adopted:
> >
> > - Case sensitive.
> > - Special characters are not ignored.
> > - Do not trim Spaces.
> >
> > If the check fails,  return a text/plain response with 400 code. Like
> this:
> >
> > ```http
> > HTTP/1.1 400 Bad Request
> > Date: Mon, 20 Jun 2022 03:52:10 GMT
> > broker-address: 127.0.0.1
> > Content-Type: text/plain
> > Content-Length: 432
> > Server: Jetty(9.4.44.v20210927)
> >
> > Unrecognized field "retention_size_in_mb" (class
> > org.apache.pulsar.common.policies.data.RetentionPolicies known
> properties:
> > "retentionSizeInMB", "retentionTimeInMinutes"])
> > ```
> >
> > ## Configuration Changes
> >
> > broker.conf
> >
> > ```properties
> > # Admin API fail on unknown request parameter in request-body. see
> PIP-178.
> > Setting this to blank means that this feature is turned off.
> > httpRequestsFailOnUnknownPropertiesEnabled=false
> > ```
> >
> > ## Dynamic switching
> > Enabling this feature affects all of the broker's HTTP services,
> including
> > the following:
> >
> > - /status.html (no post-entity request)
> > - /admin [v2,v3]
> > - /lookup (no post-entity request)
> > - /topics (http client)
> > - /metrics (no post-entity request)
> >
> > Because of the number of apis involved, we provide dynamic configuration.
> > When a user discovers any problem, it can be turned on and off
> dynamically
> > using the Admin API(without restarting Broker), which can reduce impact.
> >
> > Note: Since admin api v1 is no longer maintained, this feature does not
> > affect this part of the functionality.
> >
> > ```shell
> > pulsar-admin brokers update-dynamic-config --config
> > httpRequestsFailOnUnknownPropertiesEnabled --value [boolean]
> > ```
> >
> > Thanks
> > Yubiao Feng
> >
>

Re: [VOTE] [PIP-179] Support the admin API to check unknown request parameters

[Zike Yang]

+1

Zike Yang

On Thu, Jun 30, 2022 at 12:38 PM mattison chao 
wrote:

> +1(non-binding)
>
> Best,
> Mattison
>
> On Thu, 30 Jun 2022 at 12:35, Michael Marshall 
> wrote:
>
> > +1
> >
> > - Michael
> >
> > On Wed, Jun 29, 2022 at 9:29 PM Qiang Huang 
> > wrote:
> > >
> > > +1
> > >
> > > Yubiao Feng  于2022年6月30日周四
> 00:40写道：
> > >
> > > > Hi Pulsar Community
> > > >
> > > > I would like to start a VOTE on "Support the admin API to check
> unknown
> > > > request parameters" (PIP-179).
> > > >
> > > > The proposal can be read at
> > https://github.com/apache/pulsar/issues/16135
> > > >
> > > > and the discussion thread is available at
> > > > https://lists.apache.org/thread/m8vkxl46njm7sh0r1mqsn25jggq9v8kb
> > > >
> > > > Voting will stay open for at least 48h.
> > > >
> > > > Thanks
> > > > Yubiao Feng
> > > >
> > >
> > >
> > > --
> > > BR,
> > > Qiang Huang
> >
>

Re: [ANNOUNCE] New Committer: Zixuan Liu

[Zike Yang]

Congratulations! Zixuan

Zike Yang

On Fri, Jul 8, 2022 at 10:47 PM PengHui Li  wrote:

> Congratulations! Zixuan
>
> Penghui
>
> On Fri, Jul 8, 2022 at 9:56 PM Qiang Huang 
> wrote:
>
> > Congratulations!!! Zixuan.
> >
> > Max Xu  于2022年7月8日周五 21:36写道：
> >
> > > Congratulations! Zixuan
> > >
> > > Best
> > > Max Xu
> > >
> > >
> > >
> > > On Thu, Jul 7, 2022 at 8:55 PM Nicolò Boschi 
> > wrote:
> > >
> > > > Congrats!
> > > >
> > > > Nicolò Boschi
> > > >
> > > >
> > > > Il giorno gio 7 lug 2022 alle ore 13:28 Haiting Jiang <
> > > > jianghait...@apache.org> ha scritto:
> > > >
> > > > > Congratulates, Zixuan!
> > > > >
> > > > > BR,
> > > > > Haiting
> > > > >
> > > > > On 2022/07/07 10:03:36 Yu wrote:
> > > > > > Hi team,
> > > > > >
> > > > > > The Project Management Committee (PMC) for Apache Pulsar has
> > invited
> > > > > > Zixuan Liu (https://github.com/nodece) to become a committer
> > > > > > and we are pleased to announce that he has accepted.
> > > > > >
> > > > > > Being a committer enables easier contribution to the
> > > > > > project since there is no need to go via the patch
> > > > > > submission process. This should enable better productivity.
> > > > > >
> > > > > > Welcome and congratulations, Zixuan Liu!
> > > > > >
> > > > > > Please join us in congratulating and welcoming Zixuan Liu
> onboard!
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu on behalf of the Pulsar PMC
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > BR,
> > Qiang Huang
> >
>

Re: [DISCUSS] PIP-184: Topic specific consumer priorityLevel

[Zike Yang]

Hi, Dave

Thanks for your proposals. Overall looks good to me. Just a minor comment:

What about using the Pattern type to store the compiled Pattern in the
TopicConsumerConfigurationData? Like below:
```
public class TopicConsumerConfigurationData implements Serializable {
private static final long serialVersionUID = 1L;

private Pattern topicsPattern;
private int priorityLevel;
}
```
Just like what we did for the topicsPattern in the
ConsumerConfigurationData. [0]


[0] 
https://github.com/apache/pulsar/blob/bbf2a47867ff54327c1f2940e72f08a44a5dc5f7/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConsumerConfigurationData.java#L58

Zike Yang



Zike Yang


On Sat, Jul 9, 2022 at 12:10 AM Dave Maughan
 wrote:
>
> Hi Enrico,
>
> Why can't you create multiple independent Consumers ?
> > They will share the same resources (memory pools, thread pools)
>
>
> You can do that but it means the user has to manage the consumption from
> each of the consumer instances manually. This is already solved in the
> existing multi topic consumer. This change adds user convenience - to
> continue consuming from multiple topics in a single consumer instance, but
> allowing them to customise the configuration.
>
> - Dave
>
> On Fri, Jul 8, 2022 at 4:26 PM Enrico Olivelli  wrote:
>
> > Dave,
> > Why can't you create multiple independent Consumers ?
> > They will share the same resources (memory pools, thread pools)
> >
> >
> >
> >
> > Enrico
> >
> > Il giorno ven 8 lug 2022 alle ore 15:00 Dave Maughan
> >  ha scritto:
> > >
> > > Hi Pulsar community,
> > >
> > > I've opened a PIP to discuss a new Java client option to allow setting
> > > topic specific consumer priorityLevel.
> > >
> > > Proposal Link: https://github.com/apache/pulsar/issues/16481
> > >
> > > ---
> > >
> > > ## Motivation
> > >
> > > The Pulsar Java consumer supports setting a priority level for priority
> > > message
> > > dispatch in shared subscription consumers and priority assignment in
> > > failover
> > > subscription consumers. See the [ConsumerBuilder.html#priorityLevel(int)
> > > Javadoc](
> > >
> > https://javadoc.io/static/org.apache.pulsar/pulsar-client-api/2.10.1/org/apache/pulsar/client/api/ConsumerBuilder.html#priorityLevel(int)
> > > )
> > > for a detailed functional description. The Pulsar Java consumer also
> > > supports
> > > consuming from multiple topics. However, it is not possible to set a
> > > different
> > > priority level for different topics in the same Consumer instance.
> > >
> > > This behaviour is desirable in some use cases. For example, a consumer
> > > processing region specific topics might wish to configure region
> > stickiness
> > > - A
> > > multi-region application might be consuming from topics events-us-east-1
> > and
> > > events-eu-west-1. Consumers in all regions should be configured to
> > consume
> > > all
> > > topics to ensure data completeness. However, to ensure low latency, the
> > > us-east-1 consumer would need to set a higher priority level for the
> > > us-east-1
> > > topic. Similarly, the eu-west-1 consumer would need to set a higher
> > priority
> > > level for the eu-west-1 topic.
> > >
> > > ## Goal
> > >
> > > Update the Java client API to allow the configuration of different
> > priority
> > > levels for different topics.
> > >
> > > Do so in such a way that supports the addition of other topic specific
> > > configuration options or overrides in the future.
> > >
> > > Issues will be created to track feature parity in the other client
> > > implementations for this PIP.
> > >
> > > ## API Changes
> > >
> > > In pulsar-client-api, update `ConsumerBuilder` to include two new
> > methods:
> > >
> > > ```java
> > > public interface ConsumerBuilder extends Cloneable {
> > > ...
> > >
> > > TopicConsumerBuilder topicConfiguration(String
> > topicNameOrPattern);
> > >
> > > ConsumerBuilder topicConfiguration(String topicNameOrPattern,
> > > java.util.function.Consumer>
> > > builderConsumer);
> > > }
> > > ```
> > >
> > > Create a new interface:
> > >
> > > ```java
> > > public interface TopicConsumerBuilder {
> > > TopicConsumerBuilder priority

Re: CI - remove cmd line test retries?

[Zike Yang]

+1

On Tue, Jul 12, 2022 at 3:36 PM Enrico Olivelli  wrote:
>
> +1
>
> Enrico
>
> Il giorno mar 12 lug 2022 alle ore 09:24 PengHui Li
>  ha scritto:
> >
> > +1
> >
> > Penghui
> >
> > On Tue, Jul 12, 2022 at 1:47 PM Michael Marshall 
> > wrote:
> >
> > > +1 I support removing the retries for the many reasons already mentioned.
> > >
> > > - Michael
> > >
> > > On Tue, Jul 12, 2022 at 12:35 AM Matteo Merli  wrote:
> > > >
> > > > +1
> > > >
> > > > The retry that repeats the whole maven job is often hiding the real
> > > > test failures that were in the 1st failed job.
> > > > --
> > > > Matteo Merli
> > > > 
> > > >
> > > > On Mon, Jul 11, 2022 at 9:06 PM Anon Hxy  wrote:
> > > > >
> > > > > Hi Boschi
> > > > >
> > > > > I support this plan and I think we need take some effort to make the
> > > Pulsar
> > > > > CI more stable.
> > > > >
> > > > > Thanks,
> > > > > Xiaoyu Hou
> > > > >
> > > > > Nicolò Boschi  于2022年7月11日周一 22:14写道：
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to start a discussion about the stability of Pulsar CI.
> > > > > >
> > > > > > It is common that some tests suite in our CI times out. This is
> > > because
> > > > > > when a test fails the entire suite is retried from the beginning
> > > (max 3
> > > > > > times). (example:
> > > > > >
> > > https://github.com/apache/pulsar/runs/7281063499?check_suite_focus=true)
> > > > > >
> > > > > > The command-line retries may sound helpful in making the CI green
> > > for a
> > > > > > given pull but they actually hide test failures (that may be flakies
> > > or
> > > > > > real issues!!).
> > > > > >
> > > > > > Another issue is that you can't easily see the failed test and most
> > > of the
> > > > > > time the quickest solution is just to blindly restart the failed
> > > jobs. This
> > > > > > is not the correct behaviour and it will make the CI less stable
> > > over time.
> > > > > >
> > > > > > The plan would be:
> > > > > > - Remove the retries (see
> > > https://github.com/apache/pulsar/pull/16524)
> > > > > > - Create issue for flaky tests
> > > > > > - Fix them / move to quarantine
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Thanks,
> > > > > > Nicolò Boschi
> > > > > >
> > >

-- 
Zike Yang

Re: [VOTE] Enable GitHub Discussion for user-facing discussion

[Zike Yang]

+1

On Wed, Jul 13, 2022 at 3:21 PM tison  wrote:
>
> Thanks for your feedback. To be clear, I'll conclude the result after 72
> hours from the first email sent if there's no objections.
>
> Best,
> tison.
>
>
> Qiang Huang  于2022年7月13日周三 14:10写道：
>
> > +1
> >
> > guo jiwei  于2022年7月13日周三 10:11写道：
> >
> > > +1
> > >
> > >
> > > Regards
> > > Jiwei Guo (Tboy)
> > >
> > >
> > > On Tue, Jul 12, 2022 at 9:53 PM Enrico Olivelli 
> > > wrote:
> > >
> > > > +1
> > > >
> > > >
> > > > Enrico
> > > >
> > > > Il giorno mar 12 lug 2022 alle ore 13:05 Yu  ha
> > > scritto:
> > > > >
> > > > > Hi tison, thanks for creating this vote!
> > > > > I thought we reached a lay consensus since there was no objection
> > since
> > > > the
> > > > > last discussion [1]
> > > > >
> > > > > +1 for this proposal.
> > > > > I've explained the reasons and showed the benefits previously [2]
> > > > >
> > > > > [1] https://lists.apache.org/thread/1y7zbchlbokwnpd0jv2tt5jtzg6px6yn
> > > > > [2] https://lists.apache.org/thread/83pst643h9cqcryo3zsjd240jmqzvn73
> > > > >
> > > > > On Tue, Jul 12, 2022 at 6:01 PM tison  wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > In the previous email[1] I started a discussion about enabling
> > GitHub
> > > > > > Discussions in apache/pulsar repository and we meet a consensus to
> > > > avoid
> > > > > > making development discussions truth happen on the sources. It's
> > > also a
> > > > > > requirement of INFRA team[2]
> > > > > >
> > > > > > Apart of it, for user-facing discussions it's conventions for
> > Pulsar
> > > > users
> > > > > > who have a GitHub account but are unfamiliar with mailing list to
> > > throw
> > > > > > their use case and questions on GitHub discussion.
> > > > > >
> > > > > > Dave has opened and issue[2][3] and prepare the patch[4] for
> > turning
> > > > on the
> > > > > > discussion option but obviously we still need an explicit consensus
> > > > among
> > > > > > the community and it's also required before INFRA team takes
> > action.
> > > > > >
> > > > > > I'd like to start this voting thread for "Enable GitHub Discussion
> > > for
> > > > > > user-facing discussion", please reply with your opinion:
> > > > > >
> > > > > > [ ] +1 approve
> > > > > > [ ] +0 no opinion
> > > > > > [ ] -1 disapprove (and reason why)
> > > > > >
> > > > > > Best,
> > > > > > tison.
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/1y7zbchlbokwnpd0jv2tt5jtzg6px6yn
> > > > > > [2] https://issues.apache.org/jira/browse/INFRA-23477
> > > > > > [3] https://github.com/apache/pulsar/issues/16275
> > > > > > [4] https://github.com/apache/pulsar/pull/16528
> > > > > >
> > > >
> > >
> >
> >
> > --
> > BR,
> > Qiang Huang
> >

Re: [VOTE] PIP-184: Topic specific consumer priorityLevel

[Zike Yang]

+1

Thanks


On Fri, Jul 15, 2022 at 3:58 PM Dave Maughan
 wrote:
>
> Hi Pulsar Community
>
> I would like to start a VOTE on "PIP-184: Topic specific consumer
> priorityLevel".
>
> The proposal can be read at https://github.com/apache/pulsar/issues/16481
>
> and the discussion thread is available at
> https://lists.apache.org/thread/5zs4gd3r0rtzz16nv62o8ntygg01qjhq
>
> Voting will stay open for at least 48h.
>
> Thanks,
> Dave

Re: [DISCUSS] PIP 194 : Pulsar client: seek command add epoch

[Zike Yang]

Hi, Qiang Huang.
This is a good proposal to solve the seek issue of readers. Overall
looks good to me. Left some comments here. Thanks.

> > - stage 1: Check the current cursor status when handling flowPermits from
> > the server side.

Could you explain more details on this step? It looks like there is
not much described above. What kind of status needs to be checked, and
what kind of behavior will the broker take?

> > 1. Consumer reconnect need reset epoch.

Why do we need to reset the epoch when the consumer reconnects?

Thanks!

Zike Yang

On Tue, Jul 26, 2022 at 11:51 AM Anon Hxy  wrote:
>
> +1, Good work.
>
> Thanks,
> Xiaoyu Hou
>
> Qiang Huang  于2022年7月24日周日 22:25写道：
>
> > Hi Pulsar community:
> > I open a pip to discuss "Pulsar client: seek command add epoch"
> > Proposal Link:
> >
> >- issue link: https://github.com/apache/pulsar/issues/16757
> >
> > --
> > ## Motivation
> > `Reader` belongs to exclusive subscription type, and it uses `nonDurable`
> > cursor. After receiving messages, `Reader` will ack cumulatively
> > immediately.
> > The `flowPermits` are triggered in multiple scenarios from the client side
> > and it is isolated from `seek` of `Consumer`. Therefore, it is possibile
> > that `flowPermits` will execute after `seek` from the client side, like the
> > following flow chart.
> >
> > [image: image.png]
> >
> > When `handleSeek` processing is delay from the server side, the
> > `MarkDelete position` is modified in a wrong way.
> > The expected result is that `Reader`can re-consume messages from `mark
> > delete:(1,1)` after `seek`. But it doesn't work.
> >
> > Pulsar read message and seek position is not a synchronous operation, the
> > seek request can't prevent an in-process entry reading operation. The
> > client-side also has an opportunity to receive messages after the seek
> > position.
> >
> > Pulsar client make read messages operation and seek position operation
> > synchronized so add an epoch into server and client consumer.  After client
> > reader consumer invoke `seek` , the epoch increase 1 and send `seek`
> >  command carry the epoch and then server consumer will update the epoch.
> > When dispatcher messages to client will carry the epoch which the cursor
> > read at the time. Client consumer will filter the send messages command
> > which is smaller than current epoch.
> > In this way, after the client consumer send `seek` command successfully,
> > because it has passed the epoch filtering, the consumer will not receive a
> > message with a messageID greater than the user previously seek position.
> >
> >
> > ### Current implementation details
> >  CommandSeek Protocal
> > ```proto
> > // Reset an existing consumer to a particular message id
> > message CommandSeek {
> > required uint64 consumer_id = 1;
> > required uint64 request_id  = 2;
> >
> > optional MessageIdData message_id = 3;
> > optional uint64 message_publish_time = 4;
> > }
> > ```
> > ### CommandMessage
> > ```proto
> > message CommandMessage {
> > required uint64 consumer_id   = 1;
> > required MessageIdData message_id = 2;
> > optional uint32 redelivery_count  = 3 [default = 0];
> > repeated int64 ack_set = 4;
> > optional uint64 epoch = 5 [default = 0];
> > }
> > ```
> > `CommandMessage` already add epoch by [PIP-84](
> > https://github.com/apache/pulsar/wiki/PIP-84-:-Pulsar-client:-Redeliver-command-add-epoch)
> > , when client receive `CommandMessage` will compare the command epoch and
> > local epoch to handle this command.
> >
> > ## Goal
> > Add epoch into seek command.
> >
> > ## API Changes
> > ### Protocal change: CommandSeek
> > ```proto
> > // Reset an existing consumer to a particular message id
> > message CommandSeek {
> > required uint64 consumer_id = 1;
> > required uint64 request_id  = 2;
> >
> > optional MessageIdData message_id = 3;
> > optional uint64 message_publish_time = 4;
> > optional uint64 consumer_epoch = 5;
> > }
> > ```
> > `CommandSeek` command add epoch field, when client send seek command to
> > server successfully, the server will change the server consumer epoch to
> > the command epoch. The epoch only can bigger than the old epoch in server.
> > Now the client can filter out the message which contains less consumer
> > epoch.
> >
> > ## Implementation
> > - stage 1: Check the current cursor status when handling flowPermits from
> > the server side.
> > - stage 2: Add epoch into seek command, and server update the consumer
> > epoch. It can prevent an in-process entry reading operation after the seek
> > request.
> >
> > ## Reject Alternatives
> > None yet.
> >
> > ## Note
> > 1. Consumer reconnect need reset epoch.
> >
> > --
> > BR,
> > Qiang Huang
> >

Re: [VOTE]PIP-189: No batching if only one message in batch.

[Zike Yang]

Hi, Anon

So are you going to add the configuration `batchingSingleMessage ` to
the producer? I saw that it was still in the PIP:

> So this may cause  ((BatchMessageIdImpl) messageId) throw ClassCastException. 
>  we need to add a switch for the producer to enable or disable this feature
> ProducerBuilder batchingSingleMessage(boolean batchingSingleMessage);  // 
> default value is true




Zike Yang

On Tue, Jul 26, 2022 at 6:05 PM Qiang Huang  wrote:
>
> +1(non-binding)
>
> BR,
> Qiang Huang
>
> mattison chao  于2022年7月25日周一 13:17写道：
>
> > +1(non-binding)
> >
> > Best,
> > Mattison
> >
> > On Mon, 25 Jul 2022 at 10:35, Haiting Jiang 
> > wrote:
> > >
> > > +1
> > > Thanks,
> > > Haiting
> > >
> > > On 2022/07/25 02:23:20 Anon Hxy wrote:
> > > > Dear Community,
> > > >
> > > > I would like to start a VOTE on "PIP-189: No batching if only one
> > message
> > > > in batch."
> > > >
> > > > The proposal can be read at [0] and the discussion thread is available
> > at
> > > > [1] and the PR link is available at [2]
> > > >
> > > > Voting will stay open for at least 48h.
> > > >
> > > > [0] https://github.com/apache/pulsar/issues/16619
> > > > [1] https://lists.apache.org/thread/dbq1lrv03bhtk0lr5nwm5txo9ndjplv0
> > > > [2] https://github.com/apache/pulsar/pull/16605
> > > >
> > > > Thanks,
> > > > Xiaoyu
> > > >
> >
>
>
> --
> BR,
> Qiang Huang

Re: [VOTE]PIP-189: No batching if only one message in batch.

[Zike Yang]

+1 (non-binding)
Great work!

Thanks
Zike Yang

On Tue, Jul 26, 2022 at 7:31 PM Enrico Olivelli  wrote:
>
> +1 (binding)
>
> This is a small improvement but a great impact !!
>
> Enrico
>
> Il giorno mar 26 lug 2022 alle ore 12:48 Anon Hxy
>  ha scritto:
> >
> > Hi Zike
> >
> > Thanks for your reminding.  I just forgot to update the issue and have
> > updated it now.  I am not going to add the configuration
> > `batchingSingleMessage ` to the producer.
> >
> > Thanks,
> > Xiaoyu Hou
> >
> > Zike Yang  于2022年7月26日周二 18:16写道：
> >
> > > Hi, Anon
> > >
> > > So are you going to add the configuration `batchingSingleMessage ` to
> > > the producer? I saw that it was still in the PIP:
> > >
> > > > So this may cause  ((BatchMessageIdImpl) messageId) throw
> > > ClassCastException.  we need to add a switch for the producer to enable or
> > > disable this feature
> > > > ProducerBuilder batchingSingleMessage(boolean
> > > batchingSingleMessage);  // default value is true
> > >
> > >
> > >
> > >
> > > Zike Yang
> > >
> > > On Tue, Jul 26, 2022 at 6:05 PM Qiang Huang 
> > > wrote:
> > > >
> > > > +1(non-binding)
> > > >
> > > > BR,
> > > > Qiang Huang
> > > >
> > > > mattison chao  于2022年7月25日周一 13:17写道：
> > > >
> > > > > +1(non-binding)
> > > > >
> > > > > Best,
> > > > > Mattison
> > > > >
> > > > > On Mon, 25 Jul 2022 at 10:35, Haiting Jiang 
> > > > > wrote:
> > > > > >
> > > > > > +1
> > > > > > Thanks,
> > > > > > Haiting
> > > > > >
> > > > > > On 2022/07/25 02:23:20 Anon Hxy wrote:
> > > > > > > Dear Community,
> > > > > > >
> > > > > > > I would like to start a VOTE on "PIP-189: No batching if only one
> > > > > message
> > > > > > > in batch."
> > > > > > >
> > > > > > > The proposal can be read at [0] and the discussion thread is
> > > available
> > > > > at
> > > > > > > [1] and the PR link is available at [2]
> > > > > > >
> > > > > > > Voting will stay open for at least 48h.
> > > > > > >
> > > > > > > [0] https://github.com/apache/pulsar/issues/16619
> > > > > > > [1]
> > > https://lists.apache.org/thread/dbq1lrv03bhtk0lr5nwm5txo9ndjplv0
> > > > > > > [2] https://github.com/apache/pulsar/pull/16605
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Xiaoyu
> > > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > BR,
> > > > Qiang Huang
> > >

Re: [DISCUSS] Restart Release Pulsar 2.7.5

[Zike Yang]

+1

Thanks
Zike Yang

On Tue, Jul 26, 2022 at 7:46 PM Hang Chen  wrote:
>
> +1
>
> Thanks,
> Hang
>
> Enrico Olivelli  于2022年7月26日周二 19:21写道：
> >
> > +1
> >
> > thank you
> >
> > Enrico
> >
> > Il giorno mar 26 lug 2022 alle ore 13:19 Haiting Jiang
> >  ha scritto:
> > >
> > > Hello Pulsar Community,
> > >
> > > Previously in early February, we had a discussion about 2.7.5 release in 
> > > [1].
> > > But unfortunately, there seems to be a test regression in Candidate 1 and 
> > > the
> > > release process is blocked since [2]. I would like to pick up the work and
> > > volunteer to be the release manager.
> > >
> > > Here [3] you can find the list of 52 commits to branch-2.7 since the 2.7.4
> > > release. The commits number is 41 more than RC1. Since 2.7.5 should be the
> > > last release of branch 2.7, I plan to include all existing commit this 
> > > time.
> > > After that branch 2.7 should not accept any more commits.
> > >
> > > There are 4 closed PRs targeting 2.7.5 that have not yet been 
> > > cherry-picked
> > > [4]. I'll follow up on these PRs and solve the test regression issue and 
> > > start
> > > preparing branch-2.7 RC2 for the release.
> > >
> > > BR,
> > > Haiting Jiang
> > >
> > > [1] https://lists.apache.org/thread/wxl5z67nnzv05r5vldjq3c6sm7fxnrcc
> > > [2] https://lists.apache.org/thread/yvrmwldtzbo9lbs0jqjfj31lxtd792jq
> > > [3] https://github.com/apache/pulsar/compare/v2.7.4...branch-2.7
> > > [4] 
> > > https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.7.5+-label%3Acherry-picked%2Fbranch-2.7

Re: [ANNOUNCE] Micheal Marshall as a new PMC member in Pulsar

[Zike Yang]

Congratulations, Michael!

BR
Zike Yang

On Wed, Jul 27, 2022 at 10:04 AM guo jiwei  wrote:
>
> Congrats Michael!
>
>
> Regards
> Jiwei Guo (Tboy)
>
>
> On Wed, Jul 27, 2022 at 9:56 AM Haiting Jiang 
> wrote:
>
> > Congratulations, Michael!
> >
> > BR,
> > Haiting
> >
> > On 2022/07/26 15:21:59 Enrico Olivelli wrote:
> > > I am glad to announce that the Apache Pulsar PMC invited Micheal to
> > > join the PMC and he accepted.
> > >
> > > Micheal is doing a great job in stewarding our community
> > >
> > > Please join me and celebrate !
> > >
> > > Enrico Olivelli
> > >
> >

Re: [VOTE] PIP-190: Simplify Pulsar documentation release and maintenance strategy

[Zike Yang]

+1 (non-binding)
This will greatly reduce the amount of work that contributors have to
do to update and maintain the documentation.

Thanks
Zike Yang

On Thu, Jul 28, 2022 at 4:04 PM Max Xu  wrote:
>
> +1
>
>
> Best,
>
> Max Xu
>
>
> On Thu, Jul 28, 2022 at 9:35 AM Yu  wrote:
>
> > +1 since it improves the efficiency of managing docs.
> >
> > On Thu, Jul 28, 2022 at 8:16 AM M Jun  wrote:
> >
> > > Hi, Pulsar community,
> > >
> > > I'd like to start a vote on PIP-190: Simplify Pulsar documentation
> > release
> > > and maintenance strategy.
> > >
> > > You can find the proposal at
> > https://github.com/apache/pulsar/issues/16637
> > > and the discussion thread at
> > > https://lists.apache.org/thread/g5b8lcwtn10ox7vd67n31om9w1rxn4l7.
> > >
> > > The vote will stay open for at least 48 hours.
> > >
> > >
> > > Cheers,
> > > momo-jun
> > >
> >

Re: [DISCUSS] PIP 194 : Pulsar client: seek command add epoch

[Zike Yang]

Hi, Qiang

> It is necessary to check the current cursor status when handling flowPermits
> request from the server side. If the server is handling seek request, it
> should ignore flowPermits request because the request is illegal.

Thanks for your explanation. I think it's better to add this
explanation to the PIP.

> The reconnected consumer can regard as a new consumer with new epoch.

The consumer will reconnect to the broker during the seek operation.
And this will change the existing behavior. It doesn't seem to make
sense. Please correct me if I have misunderstood.

Thanks,
Zike Yang

On Wed, Jul 27, 2022 at 8:06 PM Qiang Huang  wrote:
>
> Thanks Zike.
> > > - stage 1: Check the current cursor status when handling flowPermits
> from
> > > the server side.
>
> > > Could you explain more details on this step? It looks like there is
> not much described above. What kind of status needs to be checked, and
> what kind of behavior will the broker take?
> It is necessary to check the current cursor status when handling flowPermits
> request from the server side. If the server is handling seek request, it
> should ignore flowPermits request because the request is illegal.
>
>
> > > 1. Consumer reconnect need reset epoch.
> >> Why do we need to reset the epoch when the consumer reconnects?
> The reconnected consumer can regard as a new consumer with new epoch.

Re: [Vote] PIP 198: Standardize PR Naming Convention using GitHub Actions

[Zike Yang]

+1 (non-binding)

Thanks
Zike Yang

On Mon, Aug 8, 2022 at 1:06 PM Xiangying Meng  wrote:
>
> +1(non-binding)
>
> yours sincerely,
> xiangying Meng
>
> On Thu, Aug 4, 2022 at 4:13 PM Yu  wrote:
>
> > Hi team,
> >
> > It has been 4 months since we discussed the [Guideline] Pulsar PR Naming
> > Convention [1].
> >
> > Nowadays, when reading the PR list [2], you’ll find more and more people
> > follow and get used to this rule.
> >
> > It improves collaboration efficiency, that is great!
> >
> > This makes us think about moving the rule forward, that is, standardizing
> > PR title naming using GitHub Actions, which is a more efficient way.
> >
> > So we'd like to start a vote on PIP 198: Standardize PR Naming Convention
> > using GitHub Actions [3].
> >
> >
> > This proposal contains:
> >
> > - Why do this?
> >
> > - How do this?
> >
> > - Pre-discussions and other thoughts
> >
> > Feel free to comment, thank you!
> >
> > [1] https://lists.apache.org/thread/sk9ops3t94jmzc5tndk08y9khf7pj6so
> >
> > [2] https://github.com/apache/pulsar/pulls
> >
> > [3]
> >
> > https://docs.google.com/document/d/1sJlUNAHnYAbvu9UtEgCrn_oVTnVc1M5nHC19x1bFab4/edit?pli=1#
> >
> >
> > Yu, Max, mangoGoForward
> >

[DISCUSS] Skip unnecessary tests when there are only cpp/python related changes

[Zike Yang]

Hi, Pulsar community

Currently, Java tests consume significant CI resources. And it is not
necessary to run all the tests for changes that are only on the C++ or
python parts of the code. I have created a PR [0] to improve the CI by
skipping unnecessary tests when there are only CPP/Python changes.
This can significantly increase the efficiency of CI when testing the
C++/Python part of the code.

After this PR gets merged, we will skip java unit tests, integration
tests(the part only for java codes), and go function tests when there
are only cpp/python changes. But the system test is not skipped
because there are some python function codes in that test. Perhaps in
the future, we can further optimize the system test to skip
unnecessary matrix tests for PRs with only C++ changes.

I have created a test PR in a separate repo to verify this PR. [1]
And more detail in [2].

Please take a look and feel free to comment on it.

Regarding the current Pulsar CI, I have a question. Why do we need to
add doc_only check at each step when skipping code tests instead of
just skipping the whole job for PR with only doc changes? [3] Is there
any concern?

Please let me know what you think. Thanks!


[0] https://github.com/apache/pulsar/pull/16988
[1] https://github.com/RobertIndie/pulsar-ci-test/pull/1
[2] https://github.com/RobertIndie/pulsar-ci-test/actions/runs/2829525510
[3] 
https://github.com/apache/pulsar/blob/master/.github/workflows/pulsar-ci.yaml#L380

Best,
Zike Yang

Re: [Discussion] PIP 198 - How to define [type] and [scope]?

[Zike Yang]

LGTM.

> - Cherry pick changes [4]
> Choice A: [fix][broker][branch-2.9] xxx
> Choice B: [fix][broker] xxx. And add "cherry pick xxx to branch-2.9" in the
> PR description.

I prefer A.

Thanks,
Zike Yang
Zike Yang

On Wed, Aug 17, 2022 at 7:41 PM Qiang Huang  wrote:
>
> Great work. I prefer "Choice A".
> > Cherry pick changes [4]
> > Choice A: [fix][broker][branch-2.9] xxx
>
> Yunze Xu  于2022年8月17日周三 18:32写道：
>
> > LGTM.
> >
> > Thanks,
> > Yunze
> >
> >
> >
> >
> > > 2022年8月17日 11:15，Yu  写道：
> > >
> > > Hi team,
> > >
> > > For PIP 198: Standardize PR Naming Convention using GitHub Actions [1]
> > >
> > > How to define [type] and [scope]? Do these abbreviations LGTY?
> > >
> > > *[Guide] Pulsar Pull Request Naming Convention* [2] contains everything
> > > about the definition. Feel free to check and comment!
> > >
> > > ~~
> > >
> > > TL;DR
> > >
> > > PR title format: [type][scope] Summary [3]
> > >
> > > ~~
> > >
> > > [type]
> > >
> > > 1. Definition: what actions do you take?
> > >
> > > 2. It must be one of the following:
> > > - feat (abbr for "feature")
> > > - improve
> > > - fix
> > > - cleanup
> > > - refactor
> > > - revert
> > >
> > > ~~
> > >
> > > [scope]
> > >
> > > 1. Definition: where do you make changes?
> > >
> > > 2. It must be one of the following:
> > > - admin (changes to pulsar-admin, REST API, Java admin API)
> > > - broker
> > > - io
> > > - deploy
> > > - dep (abbr for dependency)
> > > - fcn (abbr for function)
> > > - monitor
> > > - pkg (abbr for package)
> > > - proxy
> > > - schema
> > > - sec (abbr for security)
> > > - sql
> > > - ts (abbr for tiered storage)
> > > - tool
> > > - txn (abbr for transaction)
> > >
> > > - java (changes to Java client)
> > > - cpp (changes to C++ client)
> > > - py (changes to Python client)
> > > - ws (changes to WebSocket)
> > > - rest (changes to REST)
> > >
> > > - test
> > > - ci
> > > - workflow
> > > - build
> > > - misc (abbr for miscellaneous)
> > >
> > > - doc
> > > - blog
> > > - site (abbr for website)
> > >
> > > ~~
> > >
> > > Besides, many developers have different opinions on the following
> > aspects.
> > > What's your writing preference?
> > >
> > > - Submit breaking changes
> > > [feat][broker]! Support xx
> > >
> > > - Submit PIP changes
> > > [feat][broker] PIP-198: Support xx
> > >
> > > - Cherry pick changes [4]
> > > Choice A: [fix][broker][branch-2.9] xxx
> > > Choice B: [fix][broker] xxx. And add "cherry pick xxx to branch-2.9" in
> > the
> > > PR description.
> > >
> > > ~~
> > >
> > > Feel free to comment and make your voice heard. Go vote! Thank you!
> > >
> > > [1]
> > >
> > https://docs.google.com/document/d/1sJlUNAHnYAbvu9UtEgCrn_oVTnVc1M5nHC19x1bFab4/edit
> > > [2] https://lists.apache.org/thread/90rcjf1dv0fbkb5hm31kmgr65fj0nfnn
> > > [3]
> > >
> > https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.y8943h392zno
> > > [4]
> > >
> > https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=kix.849jztd92uk7
> > >
> > > Yu
> >
> >
>
> --
> BR,
> Qiang Huang

[DISCUSS] Use Github project to track the multi clients development

[Zike Yang]

Hi, all

Currently, many features or bug fixes of some clients like the C++
client and Python client have not caught up with the Java client. For
better tracking of these features or bug fixes, I have created a
Github project called `Apache Pulsar / Multi Clients`[0] under the
pulsar main repo. It can be removed if we don't like it this way.

There are six columns in this Github project:
* Backlog: Low priority issues or issues(or cards) that have not been
initially investigated.
* Todo: Newly added issues and issues suitable for contributors to pick first.
* In progress: Work in progress issues or PRs.
* Review in progress: The PRs or issues that are waiting for review.
* Reviewer approved: The PRs or issues that are waiting to merge.
* Done: Merged PRs and Solved issues.

This not only helps to track the status of other clients' features and
bug fixes but also helps contributors who want to participate in other
language client feature development to pick the task.

This Project is currently only for clients maintained by the main
pulsar repository, such as C++ and Python clients.

I have created a PR [1] to add C++ and Python client issues and PRs to
the Multi Clients project.

Please let me know what you think. Thanks!

[0] https://github.com/apache/pulsar/projects/12
[1] https://github.com/apache/pulsar/pull/17201

Best,
Zike Yang

Re: [DISCUSS] Enable message deduplication for replicator by default

[Zike Yang]

+1

Thanks,
Zike Yang

On Mon, Aug 22, 2022 at 3:16 PM mattison chao  wrote:
>
> +1
>
> Best,
> Mattison
>
> On Fri, 19 Aug 2022 at 01:40, Enrico Olivelli  wrote:
>
> > I agree
> >
> > Enrico
> >
> > Il Gio 18 Ago 2022, 18:23 PengHui Li  ha scritto:
> >
> > > Hi all,
> > >
> > > When I tried to fix a problem related to replicator
> > > https://github.com/apache/pulsar/pull/17154
> > > It surprised me that the message deduplication will not work by default
> > > with the replicator.
> > > I always thought it was enabled for replicators by default. Details to
> > see
> > > [0].
> > >
> > > I think we should enable the deduplication for the replicator. Otherwise,
> > > we will see duplicated
> > > messages on the remote cluster. And the producer of the replicator always
> > > has a fixed producer
> > > name, this will make the message deduplication work properly.
> > >
> > > The test introduced in https://github.com/apache/pulsar/pull/17154 will
> > > check the message
> > > replication ordering. Without the message deduplication enabled, the test
> > > is flaky with received
> > > duplicated messages. After enabling, everything is fine.
> > >
> > > Best,
> > > Penghui
> > >
> > > [0] https://github.com/apache/pulsar/pull/17154#discussion_r948736894
> > >
> >

Re: [DISCUSS] Use Github project to track the multi clients development

[Zike Yang]

> I think that the main problem is that in our community we have plenty
of Java developers and there are a few committers that review
the patches on non-Java client.

Yes. The primary purpose of this project is to keep track of many
uncaught-up features of other languages clients. And there will be a
lot of features that we need to develop.

If there is no objection, please help review and merge this PR:
https://github.com/apache/pulsar/pull/17201 Thanks.


Best,
Zike Yang

On Wed, Aug 24, 2022 at 9:20 PM Enrico Olivelli  wrote:
>
> I think that the main problem is that in our community we have plenty
> of Java developers and there are a few committers that review
> the patches on non-Java client.
> The same problem is also for other projects like the Helm Chart.
> Also, the Java client is used internally by the broker so it is
> usually more used and it needs more work
>
> That said I agree that with the proposal in this thread
>
> Enrico
>
> Il giorno mer 24 ago 2022 alle ore 14:32 Qiang Huang
>  ha scritto:
> >
> > It makes sense to me. I'm thinking, if an issue is encountered in a Java
> > client, it is possible to be in other clients. If so, the Java client board
> > is also needed.
> >
> > mattison chao  于2022年8月22日周一 16:00写道：
> >
> > > +1
> > >
> > > Best,
> > > Mattison
> > >
> > > On Mon, 22 Aug 2022 at 15:52, Baodi Shi 
> > > wrote:
> > >
> > > > Nice idea!
> > > >
> > > > Thanks,
> > > > Baodi Shi
> > > >
> > > > > On Aug 22, 2022, at 12:4250, Zike Yang  wrote:
> > > > >
> > > > > Hi, all
> > > > >
> > > > > Currently, many features or bug fixes of some clients like the C++
> > > > > client and Python client have not caught up with the Java client. For
> > > > > better tracking of these features or bug fixes, I have created a
> > > > > Github project called `Apache Pulsar / Multi Clients`[0] under the
> > > > > pulsar main repo. It can be removed if we don't like it this way.
> > > > >
> > > > > There are six columns in this Github project:
> > > > > * Backlog: Low priority issues or issues(or cards) that have not been
> > > > > initially investigated.
> > > > > * Todo: Newly added issues and issues suitable for contributors to 
> > > > > pick
> > > > first.
> > > > > * In progress: Work in progress issues or PRs.
> > > > > * Review in progress: The PRs or issues that are waiting for review.
> > > > > * Reviewer approved: The PRs or issues that are waiting to merge.
> > > > > * Done: Merged PRs and Solved issues.
> > > > >
> > > > > This not only helps to track the status of other clients' features and
> > > > > bug fixes but also helps contributors who want to participate in other
> > > > > language client feature development to pick the task.
> > > > >
> > > > > This Project is currently only for clients maintained by the main
> > > > > pulsar repository, such as C++ and Python clients.
> > > > >
> > > > > I have created a PR [1] to add C++ and Python client issues and PRs to
> > > > > the Multi Clients project.
> > > > >
> > > > > Please let me know what you think. Thanks!
> > > > >
> > > > > [0] https://github.com/apache/pulsar/projects/12
> > > > > [1] https://github.com/apache/pulsar/pull/17201
> > > > >
> > > > > Best,
> > > > > Zike Yang
> > > >
> > > >
> > >
> >
> >
> > --
> > BR,
> > Qiang Huang

Re: [DISCUSS] Use Github project to track the multi clients development

[Zike Yang]

Thanks for your information.
I will mark this PR as a draft until the improvement of
`add-to-project` is done.


Zike Yang

On Mon, Aug 29, 2022 at 9:36 AM PengHui Li  wrote:
>
> More context can be found here
> https://lists.apache.org/thread/q49r2blh1wmvnr0mqx2lxxkhd8c306bm
>
> Thanks,
> Penghui
>
> On Sat, Aug 27, 2022 at 6:55 PM Max Xu  wrote:
>
> > Reviewed and left a comment: the actions/add-to-project says itself does
> > not support the Classic GitHub Project. [1]
> >
> > We may need to be aware of this.
> >
> > [1] https://github.com/actions/add-to-project#actionsadd-to-project
> >
> >
> > Best,
> > Max Xu
> >
> >
> > On Thu, Aug 25, 2022 at 7:43 PM Zike Yang  wrote:
> >
> > > > I think that the main problem is that in our community we have plenty
> > > of Java developers and there are a few committers that review
> > > the patches on non-Java client.
> > >
> > > Yes. The primary purpose of this project is to keep track of many
> > > uncaught-up features of other languages clients. And there will be a
> > > lot of features that we need to develop.
> > >
> > > If there is no objection, please help review and merge this PR:
> > > https://github.com/apache/pulsar/pull/17201 Thanks.
> > >
> > >
> > > Best,
> > > Zike Yang
> > >
> > > On Wed, Aug 24, 2022 at 9:20 PM Enrico Olivelli 
> > > wrote:
> > > >
> > > > I think that the main problem is that in our community we have plenty
> > > > of Java developers and there are a few committers that review
> > > > the patches on non-Java client.
> > > > The same problem is also for other projects like the Helm Chart.
> > > > Also, the Java client is used internally by the broker so it is
> > > > usually more used and it needs more work
> > > >
> > > > That said I agree that with the proposal in this thread
> > > >
> > > > Enrico
> > > >
> > > > Il giorno mer 24 ago 2022 alle ore 14:32 Qiang Huang
> > > >  ha scritto:
> > > > >
> > > > > It makes sense to me. I'm thinking, if an issue is encountered in a
> > > Java
> > > > > client, it is possible to be in other clients. If so, the Java client
> > > board
> > > > > is also needed.
> > > > >
> > > > > mattison chao  于2022年8月22日周一 16:00写道：
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Best,
> > > > > > Mattison
> > > > > >
> > > > > > On Mon, 22 Aug 2022 at 15:52, Baodi Shi
> >  > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Nice idea!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Baodi Shi
> > > > > > >
> > > > > > > > On Aug 22, 2022, at 12:4250, Zike Yang 
> > wrote:
> > > > > > > >
> > > > > > > > Hi, all
> > > > > > > >
> > > > > > > > Currently, many features or bug fixes of some clients like the
> > > C++
> > > > > > > > client and Python client have not caught up with the Java
> > > client. For
> > > > > > > > better tracking of these features or bug fixes, I have created
> > a
> > > > > > > > Github project called `Apache Pulsar / Multi Clients`[0] under
> > > the
> > > > > > > > pulsar main repo. It can be removed if we don't like it this
> > way.
> > > > > > > >
> > > > > > > > There are six columns in this Github project:
> > > > > > > > * Backlog: Low priority issues or issues(or cards) that have
> > not
> > > been
> > > > > > > > initially investigated.
> > > > > > > > * Todo: Newly added issues and issues suitable for contributors
> > > to pick
> > > > > > > first.
> > > > > > > > * In progress: Work in progress issues or PRs.
> > > > > > > > * Review in progress: The PRs or issues that are waiting for
> > > review.
> > > > > > > > * Reviewer approved: The PRs or issues that are waiting to
> > merge.
> > > > > > > > * Done: Merged PRs and Solved issues.
> > > > > > > >
> > > > > > > > This not only helps to track the status of other clients'
> > > features and
> > > > > > > > bug fixes but also helps contributors who want to participate
> > in
> > > other
> > > > > > > > language client feature development to pick the task.
> > > > > > > >
> > > > > > > > This Project is currently only for clients maintained by the
> > main
> > > > > > > > pulsar repository, such as C++ and Python clients.
> > > > > > > >
> > > > > > > > I have created a PR [1] to add C++ and Python client issues and
> > > PRs to
> > > > > > > > the Multi Clients project.
> > > > > > > >
> > > > > > > > Please let me know what you think. Thanks!
> > > > > > > >
> > > > > > > > [0] https://github.com/apache/pulsar/projects/12
> > > > > > > > [1] https://github.com/apache/pulsar/pull/17201
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Zike Yang
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > BR,
> > > > > Qiang Huang
> > >
> >

[DISCUSS] Return the message ID of the first chunk when sending chunked messages

[Zike Yang]

Hi Pulsar Community,


Currently, when we send chunked messages, the producer returns the
message-id of the last chunk. This can cause some problems. For example,
when we use this message-id to seek, it will cause the consumer to consume
from the position of the last chunk, and the consumer will mistakenly think
that the previous chunks are lost and choose to skip the current message.
If we use the inclusive seek, the consumer may skip the first message,
which brings the wrong behavior.


Here is the simple code used to demonstrate the problem.

```

var msgId = producer.send(...); // eg. return 0:1:-1

var otherMsg = producer.send(...); // return 0:2:-1

consumer.seek(msgId); // inclusive seek

var receiveMsgId = consumer.receive().getMessageId(); // it may skip the
first message and return like 0:2:-1

Assert.assertEquals(msgId, receiveMsgId); // fail

```


To fix this, I think we could return the message ID of the first chunk when
sending chunked messages. I would like to know if this solution will bring
other problems. Any ideas on this?


Thanks
-- 
Zike Yang

Re: [DISCUSS] Return the message ID of the first chunk when sending chunked messages

[Zike Yang]

Hi, Rajan. 

I think we can make changes only on the client side. Probably only the part of 
sendMessages of the producer. 

When the producer sends a chunked message, the message-id of the first chunk 
is stored temporarily in the producer until all chunks have been sent, and then 
it is 
returned to the user after all chunks has been sent successfully. 

 I think this solution may be simpler to implement and will not cause any 
changes 
on the server side, nor will it change the seek part of the consumer. What do 
you think?

Thanks,
Zike

> On Sep 22, 2021, at 3:58 PM, Rajan Dhabalia  wrote:
> 
> Hi,
> 
> Though chunked messages are sequential for a specific producer, it's not
> guaranteed that they will be contiguous when the broker receives them and
> writes them to a ledger. So, it will be a little tricky to find out the
> first message-id of any chunked-message at any given time unless broker
> tags chunked messages while persisting them at server side. But such
> tagging and updating message metadata might not be straightforward and may
> not scale when the topic has a large number of producers and chunked
> messages are being published from all different producers.
> 
> Consumer::seek(messageId) also doesn't work if the user provides an
> incorrect messageId. So, if the user points to an incomplete chunk then
> it's expected that the consumer can't receive the same chunked-message but
> then the consumer should be able to receive and consume the next complete
> chunk and deliver it to the application. This behavior should not require
> any server side change but should expect the client to consume the next
> correct chunked message after the given messageId.
> 
> Thanks,
> Rajan
> 
> 
> 
> On Tue, Sep 21, 2021 at 8:56 PM Zike Yang 
> wrote:
> 
>> Hi Pulsar Community,
>> 
>> 
>> Currently, when we send chunked messages, the producer returns the
>> message-id of the last chunk. This can cause some problems. For example,
>> when we use this message-id to seek, it will cause the consumer to consume
>> from the position of the last chunk, and the consumer will mistakenly think
>> that the previous chunks are lost and choose to skip the current message.
>> If we use the inclusive seek, the consumer may skip the first message,
>> which brings the wrong behavior.
>> 
>> 
>> Here is the simple code used to demonstrate the problem.
>> 
>> ```
>> 
>> var msgId = producer.send(...); // eg. return 0:1:-1
>> 
>> var otherMsg = producer.send(...); // return 0:2:-1
>> 
>> consumer.seek(msgId); // inclusive seek
>> 
>> var receiveMsgId = consumer.receive().getMessageId(); // it may skip the
>> first message and return like 0:2:-1
>> 
>> Assert.assertEquals(msgId, receiveMsgId); // fail
>> 
>> ```
>> 
>> 
>> To fix this, I think we could return the message ID of the first chunk when
>> sending chunked messages. I would like to know if this solution will bring
>> other problems. Any ideas on this?
>> 
>> 
>> Thanks
>> --
>> Zike Yang
>> 

RE: [DISCUSS] Return the message ID of the first chunk when sending chunked messages

[Zike Yang]

I submitted a PR to demonstrate the 
solution:https://github.com/apache/pulsar/pull/12171 
<https://github.com/apache/pulsar/pull/12171>.
If you have any suggestions, please feel free to comment on it.

Thanks,
Zike Yang

On 2021/09/22 03:55:44 Zike Yang wrote:
> Hi Pulsar Community,
> 
> 
> Currently, when we send chunked messages, the producer returns the
> message-id of the last chunk. This can cause some problems. For example,
> when we use this message-id to seek, it will cause the consumer to consume
> from the position of the last chunk, and the consumer will mistakenly think
> that the previous chunks are lost and choose to skip the current message.
> If we use the inclusive seek, the consumer may skip the first message,
> which brings the wrong behavior.
> 
> 
> Here is the simple code used to demonstrate the problem.
> 
> ```
> 
> var msgId = producer.send(...); // eg. return 0:1:-1
> 
> var otherMsg = producer.send(...); // return 0:2:-1
> 
> consumer.seek(msgId); // inclusive seek
> 
> var receiveMsgId = consumer.receive().getMessageId(); // it may skip the
> first message and return like 0:2:-1
> 
> Assert.assertEquals(msgId, receiveMsgId); // fail
> 
> ```
> 
> 
> To fix this, I think we could return the message ID of the first chunk when
> sending chunked messages. I would like to know if this solution will bring
> other problems. Any ideas on this?
> 
> 
> Thanks
> -- 
> Zike Yang
> 

[VOTE] PIP 107: Introduce the chunk message ID

[Zike Yang]

Hi Pulsar Community,
 
I would like to start a VOTE for PIP 107: Introduce the chunk message ID.

The issue for this PIP is here: 
https://github.com/apache/pulsar/issues/12402 

Please VOTE within 72 hours. 

Thanks,
Zike Yang

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Zike Yang]

Hi, xiangying,

Thanks for your PIP.

IIUC, this may change the existing behavior and may introduce inconsistencies.
Suppose that we have a large message with 3 chunks. But the producer
crashes and resends the message after sending the chunk-1. It will
send a total of 5 messages to the Pulsar topic:

1. SequenceID: 0, ChunkID: 0
2. SequenceID: 0, ChunkID: 1
3. SequenceID: 0, ChunkID: 0   -> This message will be dropped
4. SequenceID: 0, ChunkID: 1-> Will also be dropped
5. SequenceID: 0, ChunkID: 2-> The last chunk of the message

For the existing behavior, the consumer assembles messages 3,4,5 into
the original large message. But the changes brought about by this PIP
will cause the consumer to use messages 1,2,5 for assembly. There is
no guarantee that the producer will split the message in the same way
twice before and after. For example, the producer's maxMessageSize may
be different. This may cause the consumer to receive a corrupt
message.

Also, this PIP increases the complexity of handling chunks on the
broker side. Brokers should, in general, treat the chunk as a normal
message.

I think a simple better approach is to only check the deduplication
for the last chunk of the large message. The consumer only gets the
whole message after receiving the last chunk. We don't need to check
the deduplication for all previous chunks. Also by doing this we only
need bug fixes, we don't need to introduce a new PIP.

BR,
Zike Yang

On Fri, Aug 18, 2023 at 7:54 PM Xiangying Meng  wrote:
>
> Dear Community,
>
> I hope this email finds you well. I'd like to address an important
> issue related to Apache Pulsar and discuss a solution I've proposed on
> GitHub. The problem pertains to the handling of Chunk Messages after
> enabling deduplication.
>
> In the current version of Apache Pulsar, all chunks of a Chunk Message
> share the same sequence ID. However, enabling the depublication
> feature results in an inability to send Chunk Messages. To tackle this
> problem, I've proposed a solution [1] that ensures messages are not
> duplicated throughout end-to-end delivery. While this fix addresses
> the duplication issue for end-to-end messages, there remains a
> possibility of duplicate chunks within topics.
>
> To address this concern, I believe we should introduce a "Chunk ID
> map" at the Broker level, similar to the existing "sequence ID map",
> to facilitate effective filtering. However, implementing this has led
> to a challenge: a producer requires storage for two Long values
> simultaneously (sequence ID and chunk ID). Because the snapshot of the
> sequence ID map is stored through the properties of the cursor
> (Map), so in order to satisfy the storage of two Longs
> (sequence ID, chunk ID) corresponding to one producer, we hope to add
> a mark DeleteProperties (Map) String, String>) to
> replace the properties (Map) field. To resolve this,
> I've proposed an alternative proposal [2] involving the introduction
> of a "mark DeleteProperties" (Map) to replace the
> current properties (Map) field.
>
> I'd appreciate it if you carefully review both PRs and share your
> valuable feedback and insights. Thank you immensely for your time and
> attention. I eagerly anticipate your valuable opinions and
> recommendations.
>
> Warm regards,
> Xiangying
>
> [1] https://github.com/apache/pulsar/pull/20948
> [2] https://github.com/apache/pulsar/pull/21027

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Zike Yang]

Hi, xiangying

> it will find that the message
is out of order and rewind the cursor. Loop this operation, and
discard this message after it expires instead of assembling 3, 4, 5
into a message.

Could you point out where the implementation for this?  From my
understanding, there should not be any rewind operation for the
chunking feature. You can check more detail here:
https://streamnative.io/blog/deep-dive-into-message-chunking-in-pulsar#how-message-chunking-is-implemented

> This solution also cannot solve the out-of-order messages inside the
chunks. For example, the above five messages will still be persisted.

The consumer already handles this case. The above 5 messages will all
be persisted but the consumer will skip message 1 and 2.
For messages 3, 4, and 5. The producer can guarantee these chunks are in order.

BR,
Zike Yang

On Thu, Aug 24, 2023 at 11:48 AM Yubiao Feng
 wrote:
>
> > 1. SequenceID: 0, ChunkID: 0
> > 2. SequenceID: 0, ChunkID: 1
> > 3. SequenceID: 0, ChunkID: 0
> > 4. SequenceID: 0, ChunkID: 1
> > 5. SequenceID: 0, ChunkID: 2
> > For the existing behavior, the consumer assembles
> > messages 3,4,5 into
> > the original large message. But the changes brought
> > about by this PIP
> > will cause the consumer to use messages 1,2,5 for
> > assembly. There is
> > no guarantee that the producer will split the message
> > in the same way
> > twice before and after. For example, the producer's
> > maxMessageSize may
> > be different. This may cause the consumer to
> > receive a corrupt
> > message.
>
> Good point.
>
>
> Thanks
> Yubiao Feng
>
> On Wed, Aug 23, 2023 at 12:34 PM Zike Yang  wrote:
>
> > Hi, xiangying,
> >
> > Thanks for your PIP.
> >
> > IIUC, this may change the existing behavior and may introduce
> > inconsistencies.
> > Suppose that we have a large message with 3 chunks. But the producer
> > crashes and resends the message after sending the chunk-1. It will
> > send a total of 5 messages to the Pulsar topic:
> >
> > 1. SequenceID: 0, ChunkID: 0
> > 2. SequenceID: 0, ChunkID: 1
> > 3. SequenceID: 0, ChunkID: 0   -> This message will be dropped
> > 4. SequenceID: 0, ChunkID: 1-> Will also be dropped
> > 5. SequenceID: 0, ChunkID: 2-> The last chunk of the message
> >
> > For the existing behavior, the consumer assembles messages 3,4,5 into
> > the original large message. But the changes brought about by this PIP
> > will cause the consumer to use messages 1,2,5 for assembly. There is
> > no guarantee that the producer will split the message in the same way
> > twice before and after. For example, the producer's maxMessageSize may
> > be different. This may cause the consumer to receive a corrupt
> > message.
> >
> > Also, this PIP increases the complexity of handling chunks on the
> > broker side. Brokers should, in general, treat the chunk as a normal
> > message.
> >
> > I think a simple better approach is to only check the deduplication
> > for the last chunk of the large message. The consumer only gets the
> > whole message after receiving the last chunk. We don't need to check
> > the deduplication for all previous chunks. Also by doing this we only
> > need bug fixes, we don't need to introduce a new PIP.
> >
> > BR,
> > Zike Yang
> >
> > On Fri, Aug 18, 2023 at 7:54 PM Xiangying Meng 
> > wrote:
> > >
> > > Dear Community,
> > >
> > > I hope this email finds you well. I'd like to address an important
> > > issue related to Apache Pulsar and discuss a solution I've proposed on
> > > GitHub. The problem pertains to the handling of Chunk Messages after
> > > enabling deduplication.
> > >
> > > In the current version of Apache Pulsar, all chunks of a Chunk Message
> > > share the same sequence ID. However, enabling the depublication
> > > feature results in an inability to send Chunk Messages. To tackle this
> > > problem, I've proposed a solution [1] that ensures messages are not
> > > duplicated throughout end-to-end delivery. While this fix addresses
> > > the duplication issue for end-to-end messages, there remains a
> > > possibility of duplicate chunks within topics.
> > >
> > > To address this concern, I believe we should introduce a "Chunk ID
> > > map" at the Broker level, similar to the existing "sequence ID map",
> > > to facilitate effective filtering. However, implementing this has led
> > > to a challenge: a producer requires storage f

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Zike Yang]

Hi Heesung,

> I believe in this PIP "similar to the existing "sequence ID map",
to facilitate effective filtering" actually means tracking the last
chunkId(not all chunk ids) on the broker side.

With this simple solution, I think we don't need to track the
(sequenceID, chunkID) on the broker side at all. The broker just needs
to apply the deduplication logic to the last chunk instead of all
previous chunks. This PIP actually could do that, but it will
introduce a new data format and compatibility issue.

> This is still a behavior change(deduping chunk messages on the broker),
and I believe we need to discuss this addition as a PIP.

Actually, we didn't specifically state the deduping chunk message
behavior before. The chunked message should be equally applicable to
the de-duplication logic as a regular message. Therefore, I think it
should be considered as a bug fix. But if this FIX is worth discussing
in depth. I have no objection to it being a new PIP.

> I think brokers can track the last chunkMaxMessageSize for
each producer.

Using different chunkMaxMessageSize is just one of the aspects. In
PIP-132 [0], we have included the message metadata size when checking
maxMessageSize.
The message metadata can be changed after splitting the chunks. We are
still uncertain about the way the chunked message is split, even using
the same ss chunkMaxMessageSize.

> then the brokers can assume that the producer is resending the chunks from
the beginning with a different scheme(restarted with a different
chunkMaxMessageSize) and accept those new chunks from the beginning.

Regarding this, it seems like we are implementing dynamic
configuration for the chunkMaxMessageSize. I'm afraid that this would
change the expected behavior and introduce more complexity to this
configuration.


[0] https://github.com/apache/pulsar/pull/14007

BR,
Zike Yang

On Thu, Aug 24, 2023 at 2:21 PM Zike Yang  wrote:
>
> Hi, xiangying
>
> > it will find that the message
> is out of order and rewind the cursor. Loop this operation, and
> discard this message after it expires instead of assembling 3, 4, 5
> into a message.
>
> Could you point out where the implementation for this?  From my
> understanding, there should not be any rewind operation for the
> chunking feature. You can check more detail here:
> https://streamnative.io/blog/deep-dive-into-message-chunking-in-pulsar#how-message-chunking-is-implemented
>
> > This solution also cannot solve the out-of-order messages inside the
> chunks. For example, the above five messages will still be persisted.
>
> The consumer already handles this case. The above 5 messages will all
> be persisted but the consumer will skip message 1 and 2.
> For messages 3, 4, and 5. The producer can guarantee these chunks are in 
> order.
>
> BR,
> Zike Yang
>
> On Thu, Aug 24, 2023 at 11:48 AM Yubiao Feng
>  wrote:
> >
> > > 1. SequenceID: 0, ChunkID: 0
> > > 2. SequenceID: 0, ChunkID: 1
> > > 3. SequenceID: 0, ChunkID: 0
> > > 4. SequenceID: 0, ChunkID: 1
> > > 5. SequenceID: 0, ChunkID: 2
> > > For the existing behavior, the consumer assembles
> > > messages 3,4,5 into
> > > the original large message. But the changes brought
> > > about by this PIP
> > > will cause the consumer to use messages 1,2,5 for
> > > assembly. There is
> > > no guarantee that the producer will split the message
> > > in the same way
> > > twice before and after. For example, the producer's
> > > maxMessageSize may
> > > be different. This may cause the consumer to
> > > receive a corrupt
> > > message.
> >
> > Good point.
> >
> >
> > Thanks
> > Yubiao Feng
> >
> > On Wed, Aug 23, 2023 at 12:34 PM Zike Yang  wrote:
> >
> > > Hi, xiangying,
> > >
> > > Thanks for your PIP.
> > >
> > > IIUC, this may change the existing behavior and may introduce
> > > inconsistencies.
> > > Suppose that we have a large message with 3 chunks. But the producer
> > > crashes and resends the message after sending the chunk-1. It will
> > > send a total of 5 messages to the Pulsar topic:
> > >
> > > 1. SequenceID: 0, ChunkID: 0
> > > 2. SequenceID: 0, ChunkID: 1
> > > 3. SequenceID: 0, ChunkID: 0   -> This message will be dropped
> > > 4. SequenceID: 0, ChunkID: 1-> Will also be dropped
> > > 5. SequenceID: 0, ChunkID: 2-> The last chunk of the message
> > >
> > > For the existing behavior, the consumer assembles messages 3,4,5 into
> > > the original large message. But the changes brought about by this PIP
> > > w

Re: [DISCUSS] PIP-293: Add configuration for batching/chunking for retry letter and dead letter producers

[Zike Yang]

Hi, Rin

Thanks for your PIP!

Chunking/batching is the feature for the producer. It would be
confusing if we introduce these concepts to the DeadLetterPolicy.
Regarding the issue mentioned in this PIP. I have created a PR to try
to fix it: https://github.com/apache/pulsar/pull/21048
I'm trying to enable chunking and disable batching for retry and DLQ
producers. PTAL. Thanks.

BR,
Zike Yang

On Sat, Aug 19, 2023 at 5:28 AM Rin Davis
 wrote:
>
> Hi Pulsar Community, This is the discussion thread for PIP
> https://github.com/apache/pulsar/pull/20959
> This PIP will allow users to specify that chunking/batching is enabled on
> retry letter and dead letter producers associated with a consumer.
>
> Thank you,
> Rin

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Zike Yang]

HI xiangying

> The rewind operation is seen in the test log.

That seems weird. Not sure if this rewind is related to the chunk consuming.

> 1. SequenceID: 0, ChunkID: 0
2. SequenceID: 0, ChunkID: 1
3. SequenceID: 0, ChunkID: 1
4. SequenceID: 0, ChunkID: 2
Such four chunks cannot be processed correctly by the consumer.

How would this happen to get two duplicated and consecutive ChunkID-1
messages? The producer should guarantee to retry the whole chunked
messages instead of some parts of the chunks.

> But chunks 1, 2, 3, and 4 are still persisted in the topic.

I think it's OK to persist them all in the topic. Is there any issue
with doing that?

> There is another point. The resend of the chunk message has a bug that
I shared with you, and you fixed in PR [0]. It will make this case
happen in another way.

If the user sets the sequence ID manually, the case could be reproduced.

BR,
Zike Yang

On Thu, Aug 24, 2023 at 8:48 PM Xiangying Meng  wrote:
>
> >IIUC, this may change the existing behavior and may introduce 
> >inconsistencies.
> >Suppose that we have a large message with 3 chunks. But the producer
> >crashes and resends the message after sending the chunk-1. It will
> >send a total of 5 messages to the Pulsar topic:
> >
> >1. SequenceID: 0, ChunkID: 0
> >2. SequenceID: 0, ChunkID: 1
> >3. SequenceID: 0, ChunkID: 0   -> This message will be dropped
> >4. SequenceID: 0, ChunkID: 1-> Will also be dropped
> >5. SequenceID: 0, ChunkID: 2-> The last chunk of the message
>
> Hi Zike
> There is another point. The resend of the chunk message has a bug that
> I shared with you, and you fixed in PR [0]. It will make this case
> happen in another way.
> Sample description for the  bug:
> Because the chunk message uses the same message metadata, if the chunk
> is not sent out immediately. Then, when resending, all chunks of the
> same chunk message use the chunk ID of the last chunk.
> In this case, It should happen as:
> 1. SequenceID: 0, ChunkID: 0 (Put op1 into `pendingMessages` and send)
> 2. SequenceID: 0, ChunkID: 1 (Put op2 into `pendingMessages` and send)
> 3. SequenceID: 0, ChunkID: 2   -> (Put op3 into `pendingMessages`)
> 4. SequenceID: 0, ChunkID: 2   -> (Resend op1)
> 5. SequenceID: 0, ChunkID: 2   -> (Resend op2)
> 6. SequenceID: 0, ChunkID: 2   -> (Send op3)
>
>
> BR,
> Xiangying
>
> [0] - https://github.com/apache/pulsar/pull/21048
>
> On Thu, Aug 24, 2023 at 8:09 PM Xiangying Meng  wrote:
> >
> > >> This solution also cannot solve the out-of-order messages inside the
> > >>chunks. For example, the above five messages will still be persisted.
> > >The consumer already handles this case. The above 5 messages will all
> > >be persisted but the consumer will skip message 1 and 2.
> > >For messages 3, 4, and 5. The producer can guarantee these chunks are in 
> > >order.
> >
> > The rewind operation is seen in the test log. Every time an incorrect
> > chunk message is received, it will rewind, and the code has yet to be
> > studied in depth.
> > If it does not call rewind, then this case is considered a workable
> > case. Let's look at another case.
> > 1. SequenceID: 0, ChunkID: 0
> > 2. SequenceID: 0, ChunkID: 1
> > 3. SequenceID: 0, ChunkID: 1
> > 4. SequenceID: 0, ChunkID: 2
> > Such four chunks cannot be processed correctly by the consumer.
> >
> > In fact, this solution is my original idea. The PR I mentioned in the
> > first email above uses a similar solution and modifies the logic on
> > the consumer side.
> > Also, as I mentioned in the first email, this solution can only solve
> > the problem of end-to-end duplication. But chunks 1, 2, 3, and 4 are
> > still persisted in the topic.
> >
> > On Thu, Aug 24, 2023 at 3:00 PM Zike Yang  wrote:
> > >
> > > Hi Heesung,
> > >
> > > > I believe in this PIP "similar to the existing "sequence ID map",
> > > to facilitate effective filtering" actually means tracking the last
> > > chunkId(not all chunk ids) on the broker side.
> > >
> > > With this simple solution, I think we don't need to track the
> > > (sequenceID, chunkID) on the broker side at all. The broker just needs
> > > to apply the deduplication logic to the last chunk instead of all
> > > previous chunks. This PIP actually could do that, but it will
> > > introduce a new data format and compatibility issue.
> > >
> > > > This is still a behavior change(deduping chunk messages on the broker),
> > > and I believe we need to discuss this addition as a P

Re: [VOTE] PIP 296: Introduce the `getLastMessageIds` API to Reader

[Zike Yang]

+1 (non-binding)

Thanks,
Zike Yang

On Fri, Aug 25, 2023 at 2:52 PM Xiangying Meng  wrote:
>
> Hi Pulsar Community,
>
> This is the vote thread for PIP 296:
> https://github.com/apache/pulsar/pull/21052
>
> This PIP will help to improve the flexibility of Reader usage.
>
> Thanks,
> Xiangying

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Zike Yang]

> Consumer receive:
1. SequenceID: 0, ChunkID: 0
2. SequenceID: 0, ChunkID: 1
3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
chunk and recycle its `chunkedMsgCtx`.
4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.

I think this case is wrong. For the current implementation, the
message 3,4,5 will be assembled as a original large message.

HI, Heesung


> I think brokers probably need to track map to dedup

I propose a simpler solution in this mail thread earlier, which
doesn't need to introduce map :

> I think a simple better approach is to only check the deduplication
for the last chunk of the large message. The consumer only gets the
whole message after receiving the last chunk. We don't need to check
the deduplication for all previous chunks. Also by doing this we only
need bug fixes, we don't need to introduce a new PIP.

Could you explain or show a case in what cases would lead to this
simpler solution not working?

Thanks,
Zike Yang

On Sat, Aug 26, 2023 at 1:38 PM Heesung Sohn
 wrote:
>
> > In this case, the consumer only can receive m1.
>
> Regarding this comment, can you explain how the consumer only receives m1?
> Here, m1's and m2's uuid and msgId will be different(if we suffix with a
> chunkingSessionId), although the sequence id is the same.
>
> > If we throw an exception when users use the same sequence to send the
> message.
> Do You mean `If "producers" throw an exception when users use the same
> sequence to send the message.`.
> Again, when the producers restart, they lose the last sequence id sent.
>
>
> > If we do not throw an exception when users use the same sequence to
> send the message.
>
> For this logic, how do we handle if the producer suddenly resends the
> chunked message with a different chunking scheme(e.g. maxMessageSize) ?
> uuid=1, sid=0, cid=0
> uuid=1, sid=0, cid=1
> uuid=2, sid=0, cid=0
> uuid=2, sid=0, cid=1
>
> We could refine what to track and algo logic on the broker side more, but
> do we agree that the broker chunk dedup logic is needed?
>
> I will continue to think more next week. Have a nice weekend.
>
>
>
>
> On Fri, Aug 25, 2023 at 9:14 PM Xiangying Meng  wrote:
>
> > Hi Heesung,
> >
> > Maybe we only need to maintain the last chunk ID in a map.
> > Map map1.
> > And we already have a map maintaining the last sequence ID.
> > Map map2.
> >
> > If we do not throw an exception when users use the same sequence to
> > send the message.
> >
> > For any incoming msg, m :
> > chunk ID = -1;
> > If m is a chunk message:
> > chunk ID = m.chunkID.
> >   If m.currentSeqid < LastSeqId, dedup.
> >   If m.currentSeqid > LastSeqId && m.chunk ID = 0, nodedup
> > if chunk ID exists in the map.
> >Update it and log an error. This means there is an
> > incomplete chunk message.
> > If chunk ID does not exist in the map.
> >Put it on the map.
> >   If m.currentSeqid == LastSeqId,
> >1. if m.chunk ID == -1 || m.chunk ID == 0. dedup.
> >2. If 1 <= m.chunkID <= total chunk.
> >   1. If chunk ID does not exist in the map. dedup.
> >   2. If chunk ID exists in the map. dedup. Check the order
> > of the chunkID to determine whether dedup;
> >3. If m.chunkID == total chunk, persistent the chunk and
> > remove the chunkID in the map.
> >
> >
> > If we throw an exception when users use the same sequence to send the
> > message.
> >
> > For any incoming msg, m :
> > chunk ID = 0;
> > If m is a chunk message:
> > chunk ID = m.chunkID.
> >If m.currentSeqid < LastSeqId, dedup.
> >If m.currentSeqid == LastSeqId.
> >If chunkID > 0, Check the last chunkID to determine whether to
> > dedup.
> > If chunkID == 1, put chunkID into the map if absent.
> >IF chunkID = 0, dedup.
> >
> > BR,
> > xiangying
> >
> > On Sat, Aug 26, 2023 at 11:53 AM Heesung Sohn
> >  wrote:
> > >
> > > However, what If the producer jvm gets restarted after the broker
> > persists
> > > the m1 (but before updating their sequence id in their persistence
> > > storage), and the producer is trying to resend the same msg(so m2) with
> > the
> > > same sequence id after restarting?
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Aug 25, 2023 at 8:22 PM Xiangying Meng 
&g

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Zike Yang]

> Hi Zike
You can see the processMessageChunk method of the ConsumerImpl.

Ah. That seems like a regression bug introduced by
https://github.com/apache/pulsar/pull/18511. I have pushed a PR to fix
it: https://github.com/apache/pulsar/pull/21070

For the behavior before Pulsar 3.0.0. The consumer should assemble the
message using 3,4,5.

Thanks for pointing this out.

BR,
Zike Yang

On Sat, Aug 26, 2023 at 3:58 PM Xiangying Meng  wrote:
>
> >> Consumer receive:
> >1. SequenceID: 0, ChunkID: 0
> >2. SequenceID: 0, ChunkID: 1
> >3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> >chunk and recycle its `chunkedMsgCtx`.
> >4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
> >5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.
> >
> >I think this case is wrong. For the current implementation, the
> >message 3,4,5 will be assembled as a original large message.
>
> Hi Zike
> You can see the processMessageChunk method of the ConsumerImpl.
>
> ```
>
> ChunkedMessageCtx chunkedMsgCtx = 
> chunkedMessagesMap.get(msgMetadata.getUuid());
>
> if (msgMetadata.getChunkId() == 0 && chunkedMsgCtx == null) {
> //assemble a chunkedMsgCtx and put into
> pendingChunkedMessageUuidQueue and chunkedMessagesMap.
> }
>
> if (chunkedMsgCtx == null || chunkedMsgCtx.chunkedMsgBuffer == null
> || msgMetadata.getChunkId() !=
> (chunkedMsgCtx.lastChunkedMessageId + 1)) {
> if (chunkedMsgCtx != null) {
> if (chunkedMsgCtx.chunkedMsgBuffer != null) {
> ReferenceCountUtil.safeRelease(chunkedMsgCtx.chunkedMsgBuffer);
> }
> chunkedMsgCtx.recycle();
> }
> chunkedMessagesMap.remove(msgMetadata.getUuid());
> compressedPayload.release();
> increaseAvailablePermits(cnx);
> }
>
> ```
>
> On Sat, Aug 26, 2023 at 3:48 PM Zike Yang  wrote:
> >
> > > Consumer receive:
> > 1. SequenceID: 0, ChunkID: 0
> > 2. SequenceID: 0, ChunkID: 1
> > 3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> > chunk and recycle its `chunkedMsgCtx`.
> > 4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
> > 5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.
> >
> > I think this case is wrong. For the current implementation, the
> > message 3,4,5 will be assembled as a original large message.
> >
> > HI, Heesung
> >
> >
> > > I think brokers probably need to track map to dedup
> >
> > I propose a simpler solution in this mail thread earlier, which
> > doesn't need to introduce map :
> >
> > > I think a simple better approach is to only check the deduplication
> > for the last chunk of the large message. The consumer only gets the
> > whole message after receiving the last chunk. We don't need to check
> > the deduplication for all previous chunks. Also by doing this we only
> > need bug fixes, we don't need to introduce a new PIP.
> >
> > Could you explain or show a case in what cases would lead to this
> > simpler solution not working?
> >
> > Thanks,
> > Zike Yang
> >
> > On Sat, Aug 26, 2023 at 1:38 PM Heesung Sohn
> >  wrote:
> > >
> > > > In this case, the consumer only can receive m1.
> > >
> > > Regarding this comment, can you explain how the consumer only receives m1?
> > > Here, m1's and m2's uuid and msgId will be different(if we suffix with a
> > > chunkingSessionId), although the sequence id is the same.
> > >
> > > > If we throw an exception when users use the same sequence to send the
> > > message.
> > > Do You mean `If "producers" throw an exception when users use the same
> > > sequence to send the message.`.
> > > Again, when the producers restart, they lose the last sequence id sent.
> > >
> > >
> > > > If we do not throw an exception when users use the same sequence to
> > > send the message.
> > >
> > > For this logic, how do we handle if the producer suddenly resends the
> > > chunked message with a different chunking scheme(e.g. maxMessageSize) ?
> > > uuid=1, sid=0, cid=0
> > > uuid=1, sid=0, cid=1
> > > uuid=2, sid=0, cid=0
> > > uuid=2, sid=0, cid=1
> > >
> > > We could refine what to track and algo logic on the broker side more, but
> > > do we agree that the broker chunk dedup logic is needed?
> > >
> > > I will continue to think more next week. Have a nice weekend.
> > >
> > >
&g

Re: [DISCUSS] Release DotPulsar 3.0.0

[Zike Yang]

+1

I am looking forward to this new release.

Thanks,
Zike Yang

On Mon, Aug 28, 2023 at 9:15 AM PengHui Li  wrote:
>
> +1
>
> Thanks for driving the release.
>
> Penghui
>
> On Fri, Aug 25, 2023 at 9:48 PM David Jensen  wrote:
>
> > Dear Apache PMC and Committers
> >
> > Me and Daniel Blankensteiner (blankensteiner) would like to announce we are
> > soon ready to release DotPulsar 3.0.0.
> >
> > The release contains breaking changes, therefor we bump to a new major
> > version.
> >
> > Changelog
> >
> > ### Added
> > - Added partitioned topic support for the Consumer and Reader (was
> > already implemented for the Producer)
> > - MessageId now includes an extra field for the topic
> > - A TryParse method is added to MessageId. Therefore, it is now
> > possible to parse a string into a MessageId object
> > - Support for `ProducerAccessMode` to prevent multiple producers on a
> > single topic
> > - A new `Fenced` state for producers which is a final state
> > - The ability to explicitly set compression information on an outgoing
> > message using `MessageMetadata` (for sending pre-compressed messages)
> >
> > ### Changed
> > - The DelayedStateMonitor extension method now invokes onStateLeft
> > when the initial state change is to a final state
> >
> > ### Fixed
> > - Issue preventing readers from correctly going into the `Faulted` state
> > - Calling `await Send(...)` on a producer did not correctly terminate
> > with an exception when a send operation failed (e.g. because the
> > producer faulted)
> > - The 'Partition' in 'MessageId' will now be set to the correct
> > partition when producing to partitioned topics
> > - The OnStateChangeFrom extension method with delay functionality
> > returned the inputted state but should return the current state
> > - The DelayedStateMonitor extension method invoked onStateLeft with
> > the inputted state but should have invoked it with the current state
> >
> > ### Deprecated
> > - GetLastMessageId of the Consumer and Reader is deprecated, and soon
> > to be removed. Please use GetLastMessageIds instead.
> >
> > Repo: https://github.com/apache/pulsar-dotpulsar
> >
> > Greetings
> > David Jensen
> >

[DISCUSS] PIP-297: Support raising exceptions using Function & Connector context

[Zike Yang]

Hi, all

I opened a new PIP: https://github.com/apache/pulsar/pull/21079

This PIP is to improve the current function exception handler. It will
be applied to both Pulsar Function and Pulsar IO Connector.

Please feel free to give your feedback. Thanks!

BR,
Zike Yang

Re: [DISCUSS] PIP-297: Support raising exceptions using Function & Connector context

[Zike Yang]

Hi, Enrico

Thanks for your review. I have replied under the PR.

BR,
Zike Yang

On Mon, Aug 28, 2023 at 6:39 PM Enrico Olivelli  wrote:
>
> Zike,
>
> Il giorno lun 28 ago 2023 alle ore 11:53 Zike Yang  ha 
> scritto:
> >
> > Hi, all
> >
> > I opened a new PIP: https://github.com/apache/pulsar/pull/21079
> >
> > This PIP is to improve the current function exception handler. It will
> > be applied to both Pulsar Function and Pulsar IO Connector.
> >
> > Please feel free to give your feedback. Thanks!
>
> I have added feedback ont he GH issue.
>
> Summary:
> - let's have two methods: raiseTemporaryError and raiseFatalError
> - please clarify what happens in the method is called on a thread that
> is not the main thread
>
> Enrico
>
> >
> > BR,
> > Zike Yang

[VOTE] PIP-297: Support terminating Function & Connector with the fatal exception

[Zike Yang]

Hi, all

Since there are no other concerns in the discussion, I'm delighted to
start the voting process for the PIP-297.

Here is the link to the PIP: https://github.com/apache/pulsar/pull/21079

BR,
Zike Yang

Re: [VOTE] Pulsar DotPulsar Release 3.0.0 Candidate 1

[Zike Yang]

+1 (non-binding)

- Verified signature and checksums
- Install the nupkg file
- Build the client from the source
- Run examples

But I find that the dotpulsar 3.0.0 has already been published to the
nuget.org: https://www.nuget.org/packages/DotPulsar/3.0.0

BR,
Zike Yang

On Sun, Sep 3, 2023 at 11:48 AM tison  wrote:
>
> Hi everyone,
>
> This is the first release candidate for Apache DotPulsar, version 3.0.0.
>
> It fixes the following issues:
> https://github.com/apache/pulsar-dotpulsar/compare/2.11.1...3.0.0
>
> Please download the source files and review this release candidate:
> - Download the source package, verify shasum and asc
> - Follow the README.md to build and run the DotPulsar.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Source files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-dotpulsar-3.0.0-candidate-1/
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://downloads.apache.org/pulsar/KEYS
>
> The tag to be voted upon:
> v3.0.0
> https://github.com/apache/pulsar-dotpulsar/releases/tag/3.0.0
>
> Please review and vote on the release candidate #1 for the version 3.0.0,
> as follows:
>
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> Best,
> tison.

Re: [VOTE] PIP-297: Support terminating Function & Connector with the fatal exception

[Zike Yang]

Thank you all.

Close this vote by 3 binding +1s:
- Enrico
- Penghui Li
- Mattison

And 1 non-binding +1: Baodi Shi

Thanks,
ZIke Yang

On Tue, Sep 5, 2023 at 9:25 AM  wrote:
>
> +1 (binding)
>
> Best,
> Mattison
> On 5 Sep 2023 at 09:07 +0800, PengHui Li , wrote:
> > +1 (binding)
> >
> > Thanks,
> > Penghui
> >
> > On Tue, Sep 5, 2023 at 8:40 AM Baodi Shi  wrote:
> >
> > > +1(non-binding)
> > >
> > >
> > > Thanks,
> > > Baodi Shi
> > >
> > >
> > > On Sep 5, 2023 at 05:23:38, Enrico Olivelli  wrote:
> > >
> > > > > +1 (binding)
> > > > >
> > > > > Enrico
> > > > >
> > > > > Il giorno lun 4 set 2023 alle ore 12:32 Zike Yang  ha
> > > > > scritto:
> > > > >
> > > > >
> > > > > Hi, all
> > > > >
> > > > >
> > > > > Since there are no other concerns in the discussion, I'm delighted to
> > > > >
> > > > > start the voting process for the PIP-297.
> > > > >
> > > > >
> > > > > Here is the link to the PIP: 
> > > > > https://github.com/apache/pulsar/pull/21079
> > > > >
> > > > >
> > > > > BR,
> > > > >
> > > > > Zike Yang
> > > > >
> > > > >
> > >

Re: [VOTE] PIP-277: Add `current` option in the Clusters list cmd

[Zike Yang]

+1 (non-binding)

BR,
Zike Yang

On Wed, Sep 6, 2023 at 11:47 PM tison  wrote:
>
> +1 (binding)
>
> Trivial with low risk.
>
> Best,
> tison.
>
>
> mattison chao  于2023年9月6日周三 22:50写道：
>
> > +1 (binding)
> >
> > Best,
> > Mattison
> > On 6 Sep 2023 at 22:37 +0800, PengHui Li , wrote:
> > > +1 (binding)
> > >
> > > The proposal just added a new option to show which cluster is the current
> > > cluster accessed. It will not break any existing behavior.
> > >
> > > Regards,
> > > Penghui
> > >
> > > On Wed, Sep 6, 2023 at 4:21 PM guo jiwei  wrote:
> > >
> > > > Hi dev,
> > > > This thread is to start a vote for PIP-277: Add `current` option in the
> > > > Clusters list cmd
> > > >
> > > > Discuss thread :
> > > > https://lists.apache.org/thread/800r6ld5wg7bttbywmk38m1qx12hs6nl
> > > > PIP: https://github.com/apache/pulsar/pull/20614
> > > >
> > > >
> > > >
> > > > Regards
> > > > Jiwei Guo (Tboy)
> > > >
> >

[VOTE] Pulsar Client Go Release 0.11.1 Candidate 1

[Zike Yang]

Hi everyone,
Please review and vote on the release candidate #1 for the version
0.11.1, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

This is the first release candidate for Apache Pulsar Go client, version 0.11.1.

It fixes the following issues:
https://github.com/apache/pulsar-client-go/compare/v0.11.0...v0.11.1-candidate-1

Pulsar Client Go's KEYS file contains PGP keys we used to sign this release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Please download these packages and review this release candidate:
- Review release notes: https://github.com/apache/pulsar-client-go/pull/1092
- Download the source package (verify shasum, and asc) and follow the
README.md to build and run the pulsar-client-go.

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Source file:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.11.1-candidate-1/

The tag to be voted upon:
v0.11.1-candidate-1
https://github.com/apache/pulsar-client-go/tree/v0.11.1-candidate-1

SHA-512 checksums:
d2209c652918acee8d2c77d52a0a556af16ff7fc3e30ad96d05e01285b83a61d1a1f0d32bace184f830e2dd2e4dd20910e9ce5ae23aac4a40eb3d19885cb0182
 apache-pulsar-client-go-0.11.1-src.tar.gz

Re: [DISCUSS] PIP-286: Add findNewestPosition method in the TopicCompactionService API

[Zike Yang]

+1 for me.

Thanks,
Zike Yang

On Mon, Sep 11, 2023 at 11:13 AM Cong Zhao  wrote:
>
> Hi, guys
>
> I noticed that the only globally ordered fields in the metadata are 
> 'publishTime' and 'entryIndex' (offset), so I split the previous method into 
> `findEntryByPublishTime` and `findEntryByEntryIndex`,
> this will make method definitions more precise.
>
> The method returns the entry directly instead of RawEntryMetadata to avoid 
> introducing a new interface, it's important to note that the caller needs to 
> release it manually.
>
> I already updated the proposal, please help review it.
>
> Thanks,
> Cong Zhao
>
> On 2023/07/25 04:12:09 Cong Zhao wrote:
> > Hi Pulsar Community, I am writing to start the discussion on PIP 286  PR
> > with PIP contents: https://github.com/apache/pulsar/pull/20867  Thanks,
> > Cong Zhao
> >

Re: [VOTE] Pulsar DotPulsar Release 3.0.0 Candidate 1

[Zike Yang]

> We may publish an X.Y.Z-rcN version on NuGet as a staged one for 
> verification. And publish
an X.Y.Z version on NuGet as a formal one when the vote is passed.

+1. Installing the NuGet package from a local package file is
complicated and not recommended. Publishing to the NuGet would make
the release validation easier.

Thanks,
Zike Yang

On Tue, Sep 12, 2023 at 9:30 PM tison  wrote:
>
> Thanks for your participants!
>
> The release is moved to
> https://dist.apache.org/repos/dist/release/pulsar/pulsar-dotpulsar-3.0.0.
>
> Here are several follow-ups for the next release:
>
> * NOTICE file - Fixed by https://github.com/apache/pulsar-dotpulsar/pull/174
> .
> * Write a RELEASE guide - I'll summarize to actions I did in this release.
> * Automaze releases - https://github.com/apache/pulsar-dotpulsar/issues/117
> * Binary release - @blankenstei...@apache.org 
> told me that releasing nuget package to SVN is meaningless. We may publish
> an X.Y.Z-rcN version on NuGet as a staged one for verification. And publish
> an X.Y.Z version on NuGet as a formal one when the vote is passed.
>
> Best,
> tison.
>
>
> mattison chao  于2023年9月11日周一 13:01写道：
>
> > +1 (binding)
> >
> > • Checked signature and checksums
> > • Built the client from the source
> > • Ran basic example.
> >
> >
> > Best,
> > Mattison
> > On 8 Sep 2023 at 17:57 +0800, tison , wrote:
> > > +1 (binding)
> > >
> > > I checked
> > >
> > > - Signature and checksums match
> > > - Build the client from the source
> > > - Run examples
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Yunze Xu  于2023年9月8日周五 01:15写道：
> > >
> > > > +1 (binding)
> > > >
> > > > - Verified signature and checksums
> > > > - Build from source with dotnet 7.0.400 on Windows 11
> > > > - Run the example by adding the dotpulsar 3.0.0 dependency
> > > >
> > > > But I think there are some points to improve with the document. I'm
> > > > new to the dotnet CLI though I have some experiences of C# a few years
> > > > ago. I created the project via Visual Studio. However, when I tried to
> > > > add the dependency via `dotnet add` command, I found it failed with a
> > > > project template created by Visual Studio. Eventually I have to create
> > > > and run a project via dotnet CLI:
> > > >
> > > > ```
> > > > cd project-dir
> > > > dotnet new console
> > > > dotnet add package DotPulsar --version 3.0.0
> > > > # Edit the Program.cs
> > > > dotnet build
> > > > dotnet run
> > > > ```
> > > >
> > > > It would be helpful to provide more guide, even links to existings
> > > > tutorial from MSDN.
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Tue, Sep 5, 2023 at 3:51 PM Zike Yang  wrote:
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > - Verified signature and checksums
> > > > > > - Install the nupkg file
> > > > > > - Build the client from the source
> > > > > > - Run examples
> > > > > >
> > > > > > But I find that the dotpulsar 3.0.0 has already been published to
> > the
> > > > > > nuget.org: https://www.nuget.org/packages/DotPulsar/3.0.0
> > > > > >
> > > > > > BR,
> > > > > > Zike Yang
> > > > > >
> > > > > > On Sun, Sep 3, 2023 at 11:48 AM tison 
> > wrote:
> > > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > This is the first release candidate for Apache DotPulsar,
> > version
> > > > 3.0.0.
> > > > > > > >
> > > > > > > > It fixes the following issues:
> > > > > > > >
> > https://github.com/apache/pulsar-dotpulsar/compare/2.11.1...3.0.0
> > > > > > > >
> > > > > > > > Please download the source files and review this release
> > candidate:
> > > > > > > > - Download the source package, verify shasum and asc
> > > > > > > > - Follow the README.md to build and run the DotPulsar.
> > > > > > > >
> > > > > > > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > > >
> > > > > > > > Source files:
> > > > > > > >
> > > >
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-dotpulsar-3.0.0-candidate-1/
> > > > > > > >
> > > > > > > > Pulsar's KEYS file containing PGP keys we use to sign the
> > release:
> > > > > > > > https://downloads.apache.org/pulsar/KEYS
> > > > > > > >
> > > > > > > > The tag to be voted upon:
> > > > > > > > v3.0.0
> > > > > > > > https://github.com/apache/pulsar-dotpulsar/releases/tag/3.0.0
> > > > > > > >
> > > > > > > > Please review and vote on the release candidate #1 for the
> > version
> > > > 3.0.0,
> > > > > > > > as follows:
> > > > > > > >
> > > > > > > > [ ] +1, Approve the release
> > > > > > > > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > tison.
> > > >
> >

Re: [VOTE] PIP-286: Make the TopicCompactionService to support find entry based on publishTime or index

[Zike Yang]

+1 (non-binding)

BR,
Zike Yang

On Mon, Sep 18, 2023 at 9:35 AM mattison chao  wrote:
>
> +1 (binding)
>
> Best,
> Mattison
> On 18 Sep 2023 at 09:19 +0800, PengHui Li , wrote:
> > +1 (binding)
> >
> > Thanks,
> > Penghui
> >
> > On Thu, Sep 14, 2023 at 11:23 AM Cong Zhao  wrote:
> >
> > > Hi, all
> > >
> > > Since there are no other concerns in the discussion, I'm delighted to
> > > start the voting process for the PIP-286.
> > >
> > > Here is the link to the PIP: https://github.com/apache/pulsar/pull/20867
> > >
> > > Thanks,
> > > Cong Zhao
> > >

Re: [DISCUSS] Release Pulsar 3.0.2

[Zike Yang]

+1

BR,
Zike Yang

On Sun, Sep 24, 2023 at 11:58 PM Dave Fisher  wrote:
>
> Please use the correct name of the project - Apache Pulsar.
>
> Thanks,
> Dave
>
> Sent from my iPhone
>
> > On Sep 24, 2023, at 7:08 AM, Yubiao Feng 
> >  wrote:
> >
> > Hi all
> >
> > I would like to propose releasing the Pulsar 3.0.2
> >
> > It's over one month since the release of 3.1.0, and there are 82 new
> > commits.
> >
> > in branch-3.0:
> > https://github.com/apache/pulsar/compare/v3.0.1...branch-3.0
> >
> > We need to cut a new release. Please let me know if you have any
> > important fixes that need to be included in Pulsar 3.0.2
> >
> > Thanks
> > Yubiao Feng
>

Re: [VOTE] PIP-302 Add new API readAllExistingMessages for TableView

[Zike Yang]

Hi, Xiangying

This PIP is a little confusing to me.
This mail title states that we are introducing `readAllExistingMessages`.
But actually, we only introduced `refreshAsync` in the PIP:
https://github.com/apache/pulsar/pull/21166/files#diff-45c655583d6c0c73d87afd3df3fe67f77caadbf1bd691cf8f8211cc89728a1ceR34-R36
And the PR title doesn't seem relevant. “PIP-302 Add alwaysRefresh
Configuration Option for TableView to Read Latest Values”

BR,
Zike Yang

On Mon, Sep 25, 2023 at 3:25 PM Xiangying Meng  wrote:
>
> Hi dev,
>This thread is to start a vote for PIP-302 Add new API
> readAllExistingMessages for TableView.
> Discuss thread: 
> https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q
> PIP: https://github.com/apache/pulsar/pull/21166
>
> BR,
> Xiangying

Re: [DISCUSS] Release Pulsar 3.1.1

[Zike Yang]

+1

Zike Yang

On Mon, Sep 25, 2023 at 2:55 PM Enrico Olivelli  wrote:
>
> +1
>
> Enrico
>
> Il giorno lun 25 set 2023 alle ore 05:08 Yubiao Feng
>  ha scritto:
> >
> > +1
> >
> > Thanks
> > Yubiao Feng
> >
> > On Mon, Sep 18, 2023 at 1:32 PM guo jiwei  wrote:
> >
> > > Hi all,
> > >
> > > I would like to propose releasing the Pulsar 3.1.1.
> > >
> > > It's over one month since the release of 3.1.0 and there are 56 new 
> > > commits
> > > in branch-3.1:
> > > https://github.com/apache/pulsar/compare/v3.1.0...branch-3.1
> > >
> > > We need to cut a new release. Please let me know if you have any
> > > important fixes that need to be included in Pulsar 3.1.1.
> > >
> > >
> > > Regards
> > > Jiwei Guo (Tboy)
> > >

Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar

[Zike Yang]

> If we want the
maximum concurrency per second to be 1, and set the maximum
concurrency per minute to 60, then the actual maximum concurrency per
second could be up to 60, which is 60 times larger than our expected
maximum concurrency.

I think the RateLimiter can handle it:
https://github.com/apache/pulsar/blob/a1405ea006f175b1bd0b9d28b9444d592fb4a010/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L965-L968

> Secondly, we already have the `maxConcurrentUnloadPerSec`
configuration, which is provided to the user in the CLI. Adding a
similar `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute`
configuration might confuse users.

`maxConcurrentUnloadPerSec ` is for the admin and CLI usage. This
proposal is to add
`brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute ` to the
broker configuration.


Overall, I'm +1 for this proposal. And I agree that we need a new PIP
for this change.

BR,
Zike Yang

On Mon, Sep 25, 2023 at 3:54 PM Xiangying Meng  wrote:
>
> Hi Donglai, Heesung
>
> >brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute=60 is the same as
> brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec=1 So, the "per-min"
> config can be more granular.
>
> I have some doubts about introducing the
> `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute`
> configuration.
>
> Firstly, I also think that a minute is too long. If we want the
> maximum concurrency per second to be 1, and set the maximum
> concurrency per minute to 60, then the actual maximum concurrency per
> second could be up to 60, which is 60 times larger than our expected
> maximum concurrency. Moreover, if the unload requests are concentrated
> in the last 10 seconds of the previous minute and the first 10 seconds
> of the next minute, then the concurrency during this period will
> exceed our configuration. Such fluctuations are inevitable, but the
> larger the time span we set, the greater the distortion of the
> configuration.
>
> Secondly, we already have the `maxConcurrentUnloadPerSec`
> configuration, which is provided to the user in the CLI. Adding a
> similar `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute`
> configuration might confuse users. When designing configuration
> parameters, we should try to keep it simple and consistent, and avoid
> introducing unnecessary complexity.
>
> Thanks,
> Xiangying
>
> On Mon, Sep 25, 2023 at 12:14 PM Yubiao Feng
>  wrote:
> >
> > Hi Donglai, Mattison
> >
> > I agree with @Mattison
> >
> > Thanks
> > Yubiao Feng
> >
> > On Mon, Aug 21, 2023 at 8:50 PM  wrote:
> >
> > >
> > > Hi,
> > >
> > > I agree with this change to improve the stability of the pulsar cluster.
> > >
> > > Just one concern. it's better to move this discussion to a new PIP.
> > > because you wanna introduce a new broker configuration.
> > > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute`
> > >
> > > FYI: https://github.com/apache/pulsar/blob/master/pip/README.md
> > >
> > > Looking forward this change and thanks for your contribution. :)
> > >
> > > Best,
> > > Mattison
> > >
> > >
> > >
> > > On 7 Jul 2023 at 15:30 +0800, labuladong , wrote:
> > > > Thanks you guys.
> > > >
> > > >
> > > > I agree that per-minute is better than per-second, which is more
> > > flexible. 
> > > >
> > > >
> > > > I open an issue here:
> > > >
> > > >
> > > > https://github.com/apache/pulsar/issues/20753
> > > >
> > > >
> > > > Regards,
> > > > donglai
> > >

Re: [DISCUSS] PIP-300: Add custom dynamic configuration for plugins

[Zike Yang]

Hi, Zixuan

Thanks for your proposal. I'm +1 for it.

>  This is a feature I need. If cherry-pick is not allowed, then it will
only be kept in 3.2+.

This is a new feature, and I also think that we couldn't cherry-pick
it. What about cherry-picking this change to your fork repo and
building the pulsar for your own to meet this need? Does it make sense
to you?

BR,
Zike Yang

On Mon, Sep 25, 2023 at 12:23 AM mattison chao  wrote:
>
> Hi, Zixuan
>
> I am afraid I can't support you in cherry-picking this feature for all of the 
> active branches by the current fact. Because this is a new feature and it's 
> not a bug fix or security issue.
>
> We can wait for other contributor's ideas.  WDYT?
>
> Thanks,
> Mattison
> On 19 Sep 2023 at 10:42 +0800, Zixuan Liu , wrote:
> > > 1. When you want to cherry-pick a PR, it should be cherry-picked for all 
> > > branches after the target branch. Isn't it?
> >
> > I think we should cherry-pick this PR to Pulsar 2.10, 2.11, 3.0.
> >
> > > 2. I've got a slight concern about cherry-picking. Do you have any issue 
> > > or feature request in the community that can demonstrate this feature is 
> > > required to cherry-pick?
> >
> > This is a feature I need. If cherry-pick is not allowed, then it will
> > only be kept in 3.2+.
> >
> > Thanks,
> > Zixuan
> >
> > mattison chao  于2023年9月18日周一 09:42写道：
> >
> > >
> > > HI, Zixuan
> > >
> > > Thanks for your discussion. It's a great feature. But I've got several 
> > > questions I want to discuss here.
> > >
> > > 1. When you want to cherry-pick a PR, it should be cherry-picked for all 
> > > branches after the target branch. Isn't it?
> > > 2. I've got a slight concern about cherry-picking. Do you have any issue 
> > > or feature request in the community that can demonstrate this feature is 
> > > required to cherry-pick?
> > >
> > >
> > > Best,
> > > Mattison
> > > On 12 Sep 2023 at 11:25 +0800, Zixuan Liu , wrote:
> > > > > BTW, I think we can cherry-pick this feature to the Pulsar 2.10. As
> > > > > far as I know, the Pulsar 2.10 is a stable/main version, because many
> > > > > users are using it.
> > > > >
> > > > > Thanks,
> > > > > Zixuan
> > > > >
> > > > > Zixuan Liu  于2023年9月5日周二 17:09写道：
> > > > > > >
> > > > > > > Hi Pulsar Community,
> > > > > > >
> > > > > > > Discuss for PIP-300: https://github.com/apache/pulsar/pull/21127
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Zixuan

Re: [VOTE] PIP-302 Introduce refreshAsync API for TableView

[Zike Yang]

+1

BR,
Zike Yang

On Wed, Sep 27, 2023 at 3:05 PM Xiangying Meng  wrote:
>
> Hi dev,
>This thread is to start a vote for PIP-302 Add new API
> refreshAsync for TableView.
> Discuss thread: 
> https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q
> PIP: https://github.com/apache/pulsar/pull/21166
>
> BR,
> Xiangying

Re: [Discuss] Release Pulsar C++ Client 3.4.0

[Zike Yang]

+1,

BR,
Zike Yang

On Mon, Oct 23, 2023 at 1:10 PM PengHui Li  wrote:
>
> Thanks for driving the release,
>
> +1
>
> Penghui
>
> On Mon, Oct 23, 2023 at 11:00 AM Yunze Xu  wrote:
>
> > I would like to propose releasing the Pulsar C++ Client 3.4.0. It has
> > been about 3 months since the last release. There have been many new
> > features and bug fixes since then.
> >
> > Besides, from my own perspective, it's better to let Python and
> > Node.js clients depend on this new version of C++ client. Especially I
> > observed that the topic name was not shown correctly many times
> > recently, which are fixed by
> > https://github.com/apache/pulsar-client-cpp/pull/331 and
> > https://github.com/apache/pulsar-client-cpp/pull/329.
> >
> > So it's time to release a new version. Please let me know if you have
> > any PRs that need to be included in 3.4.0
> >
> > Thanks,
> > Yunze
> >

Re: [DISCUSS] Auto release Nuget artifacts for DotPulsar

[Zike Yang]

Hi, David

Thanks for initializing this discussion.
I'm +1 for making the auto-release process for the DotPulsar. I have
left some comments in the PR.

Thanks,
Zike Yang

On Thu, Oct 26, 2023 at 5:39 PM David Jensen  wrote:
>
> Hi everyone,
> I have worked with Tison to make the release process smoother for DotPulsar
> <https://github.com/apache/pulsar-dotpulsar>. See GitHub Issue #184 Release
> process and GitHub Action
> <https://github.com/apache/pulsar-dotpulsar/pull/184> and the linked issues.
>
> In summary, the document presents a suggested approach for creating release
> notes and optimizing the release workflow of the Pulsar C# Client. This
> approach involves working closely with team members, adhering to ASF
> guidelines, participating in Apache release voting, implementing automation
> via GitHub Actions, and managing versioning through git tags and GitHub
> releases.
>
> Sincerely
> David Jensen

Re: Question about Pulsar gRPC client(s)

[Zike Yang]

Another point is that there are many features implemented on the
client side, including batching, chunking, DLQ, etc. This makes it
hard to replace the existing pulsar clients completely.

Zike Yang


On Wed, Nov 1, 2023 at 4:43 AM Christophe Bornet  wrote:
>
> Hi Kiryl,
>
> Thanks for mentioning pulsar-grpc.
> Indeed, using gRPC simplifies the implementation of the networking logic
> (keep-alive, reconnection, flow control,…). On the other hand, the Java
> gRPC implementation makes a lot of buffer copies to cleanly separate the
> network and app layers but that takes a toll on performance. Compared to
> that, the broker Pulsar protocol impl is optimized to not do copies between
> the consumer/producer endpoints and the bookkeeper client.
> So I think we could not replace completely the Pulsar protocol by gRPC.
> We could maybe support both but it’s a lot of work to maintain both
> protocols. (I kind of gave up maintaining pulsar-grpc because of the amount
> of work compared to the number of users, but if there’s interest I can
> reconsider).
> Another possibility would be to do a proxy instead of a low-level protocol
> handler. A bit like the WebSocket proxy. This would be far less work to
> maintain as it would use the Pulsar protocol to communicate with the
> brokers. It could be done as a Proxy extension. Compared to the WS proxy,
> this would provide type safety, discovery, and so on…
> As for the Admin, it’s a bit the same. It would be a bunch of work to
> support both gRPC and REST. You have some kind of type hinting with the
> OpenAPI spec that you can use to generate client SDKs (eg. with
> openapi-generator.
> I wonder what others have to say.
>
> Christophe
>
>
> Le mar. 31 oct. 2023 à 19:57, Kiryl Valkovich 
> a écrit :
>
> > Hi! Am I understanding it right, that if this project
> > https://github.com/cbornet/pulsar-grpc is merged to the apache/pulsar
> > repo, we could easily cover non-mainstream platforms that are supported by
> > gRPC, but don't have ready-to-use Pulsar clients?
> >
> > https://github.com/apache/pulsar/wiki/PIP-59:-gPRC-Protocol-Handler
> >
> > Similar to already supported language-agnostic client interfaces - REST
> > and WebSocket.
> >
> > Actively maintained gRPC libraries I found (19, or 15 if considering JVM
> > languages and web as duplicates):
> > - [C# / .NET](https://grpc.io/docs/languages/csharp/)
> > - [C++](https://grpc.io/docs/languages/cpp/)
> > - [Dart](https://grpc.io/docs/languages/dart/)
> > - [Go](https://grpc.io/docs/languages/go/)
> > - [Java](https://grpc.io/docs/languages/java/)
> > - [Kotlin](https://grpc.io/docs/languages/kotlin/)
> > - [Node](https://grpc.io/docs/languages/node/)
> > - [Objective-C](https://grpc.io/docs/languages/objective-c/)
> > - [PHP](https://grpc.io/docs/languages/php/)
> > - [Python](https://grpc.io/docs/languages/python/)
> > - [Ruby](https://grpc.io/docs/languages/ruby/)
> > - [OCaml](https://github.com/dialohq/ocaml-grpc)
> > - [Haskell](https://github.com/awakesecurity/gRPC-haskell)
> > - [Elixir](https://github.com/elixir-grpc/grpc)
> > - [Rust](https://github.com/hyperium/tonic)
> > - [Scala](https://scalapb.github.io/)
> > - [Swift](https://github.com/grpc/grpc-swift)
> > - Web client: https://github.com/grpc/grpc-web
> > - Web client 2: https://connectrpc.com/docs/web/getting-started
> >
> > Actively maintained Pulsar libraries (8):
> > - Java
> > - C++
> > - Python
> > - Go
> > - Node.js
> > - C#
> > - PHP
> > - Rust
> >
> > Is there any reason for not merging it to the apache/pulsar?
> >
> > I would definitely prefer to work with a statically typed gRPC client
> > instead of REST or WebSocket.
> >
> > By the way, the same can work for the Pulsar Admin API. Implement the gRPC
> > server once in Java, and we have full-featured native statically typed
> > (where applicable :)) Pulsar Admin clients for any platform.
> >
> > Best,
> > Kiryl
> >

Re: [VOTE] Pulsar Client Go Release 0.11.1 Candidate 1

[Zike Yang]

> The KEYS file link is dead, I think it should be
https://downloads.apache.org/pulsar/KEYS

Thanks for your reminder. Yes, that should be the correct link. I have
also pushed a PR to fix the release process:
https://github.com/apache/pulsar-client-go/pull/1127 PTAL. Thanks!

BR,
Zike Yang

On Tue, Nov 7, 2023 at 4:00 PM mattison chao  wrote:
>
> +1 (binding)
>
> - Built from the source
> - Ran the test on pulsar 3.0.1
>
> Best,
> Mattison
>
> > On Sep 11, 2023, at 18:07, Zike Yang  wrote:
> >
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the version
> > 0.11.1, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > This is the first release candidate for Apache Pulsar Go client, version 
> > 0.11.1.
> >
> > It fixes the following issues:
> > https://github.com/apache/pulsar-client-go/compare/v0.11.0...v0.11.1-candidate-1
> >
> > Pulsar Client Go's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Please download these packages and review this release candidate:
> > - Review release notes: https://github.com/apache/pulsar-client-go/pull/1092
> > - Download the source package (verify shasum, and asc) and follow the
> > README.md to build and run the pulsar-client-go.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Source file:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.11.1-candidate-1/
> >
> > The tag to be voted upon:
> > v0.11.1-candidate-1
> > https://github.com/apache/pulsar-client-go/tree/v0.11.1-candidate-1
> >
> > SHA-512 checksums:
> > d2209c652918acee8d2c77d52a0a556af16ff7fc3e30ad96d05e01285b83a61d1a1f0d32bace184f830e2dd2e4dd20910e9ce5ae23aac4a40eb3d19885cb0182
> > apache-pulsar-client-go-0.11.1-src.tar.gz
>

Re: [VOTE] Pulsar Client Go Release 0.11.1 Candidate 1

[Zike Yang]

Close this vote by 3 binding +1:

- Yunze
- Penghui
- Mattison

BR,
Zike Yang

On Tue, Nov 7, 2023 at 4:43 PM Zike Yang  wrote:
>
> > The KEYS file link is dead, I think it should be
> https://downloads.apache.org/pulsar/KEYS
>
> Thanks for your reminder. Yes, that should be the correct link. I have
> also pushed a PR to fix the release process:
> https://github.com/apache/pulsar-client-go/pull/1127 PTAL. Thanks!
>
> BR,
> Zike Yang
>
> On Tue, Nov 7, 2023 at 4:00 PM mattison chao  wrote:
> >
> > +1 (binding)
> >
> > - Built from the source
> > - Ran the test on pulsar 3.0.1
> >
> > Best,
> > Mattison
> >
> > > On Sep 11, 2023, at 18:07, Zike Yang  wrote:
> > >
> > > Hi everyone,
> > > Please review and vote on the release candidate #1 for the version
> > > 0.11.1, as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > This is the first release candidate for Apache Pulsar Go client, version 
> > > 0.11.1.
> > >
> > > It fixes the following issues:
> > > https://github.com/apache/pulsar-client-go/compare/v0.11.0...v0.11.1-candidate-1
> > >
> > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this 
> > > release:
> > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > >
> > > Please download these packages and review this release candidate:
> > > - Review release notes: 
> > > https://github.com/apache/pulsar-client-go/pull/1092
> > > - Download the source package (verify shasum, and asc) and follow the
> > > README.md to build and run the pulsar-client-go.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Source file:
> > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.11.1-candidate-1/
> > >
> > > The tag to be voted upon:
> > > v0.11.1-candidate-1
> > > https://github.com/apache/pulsar-client-go/tree/v0.11.1-candidate-1
> > >
> > > SHA-512 checksums:
> > > d2209c652918acee8d2c77d52a0a556af16ff7fc3e30ad96d05e01285b83a61d1a1f0d32bace184f830e2dd2e4dd20910e9ce5ae23aac4a40eb3d19885cb0182
> > > apache-pulsar-client-go-0.11.1-src.tar.gz
> >

[ANNOUNCE] Apache Pulsar Go Client 0.11.1 released

[Zike Yang]

The Apache Pulsar team is proud to announce Apache Pulsar Go Client
version 0.11.1.

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, automatic cursor management for
subscribers, and cross-datacenter replication.

For Pulsar release details and downloads, visit:
https://github.com/apache/pulsar-client-go/releases/tag/v0.11.1

Release Notes are at:
https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG.md

We would like to thank the contributors that made the release possible.

Regards,

The Pulsar Team

Re: [ANNOUNCE] Yubiao Feng as new PMC member in Apache Pulsar

[Zike Yang]

Congratulations!

BR,
Zike Yang

On Tue, Nov 14, 2023 at 3:34 AM Heesung Sohn  wrote:
>
> Congrats!
> Heesung
>
> On Mon, Nov 13, 2023 at 6:57 AM Yubiao Feng
>  wrote:
>
> > Thank you all
> >
> > Yubiao Feng
> >
> >
> >
> > On Mon, Nov 13, 2023 at 3:36 PM mattison chao 
> > wrote:
> >
> > > Dear Community,
> > >
> > > We are thrilled to announce that Yubiao Feng
> > > https://github.com/poorbarcode has been invited and has accepted the
> > role
> > > of a member of the Apache Pulsar Project Management Committee (PMC).
> > >
> > > Yubiao Feng has proven to be an invaluable asset to our community,
> > > consistently showcasing dedication and active engagement through
> > > substantial contributions. Beyond his noteworthy technical input, Yubiao
> > > plays a crucial role in meticulously reviewing pull requests, thereby
> > > ensuring the overall excellence of our project. We eagerly anticipate and
> > > appreciate his ongoing contributions. On behalf of the Pulsar PMC, we
> > > extend a heartfelt welcome and congratulations to Yubiao Feng.
> > >
> > > Sincerely,
> > > Mattison
> >

Re: [VOTE] PIP-312 Use StateStoreProvider to manage state in Pulsar Functions endpoints

[Zike Yang]

+1(no-binding)

Thanks,
Zike Yang

On Wed, Nov 15, 2023 at 4:01 PM Baodi Shi  wrote:
>
> +1(non-binding)
>
>
> Thanks,
> Baodi Shi
>
>
> On Nov 15, 2023 at 11:49:24, Neng Lu  wrote:
>
> > +1
> >
> > On 2023/11/15 03:39:42 Pengcheng Jiang wrote:
> >
> > Hi Pulsar Community,
> >
> >
> > This thread is to start a vote for PIP-312: Use StateStoreProvider to
> >
> > manage state in Pulsar Functions endpoints.
> >
> >
> > I start the voting process since there are some approves for the PIP PR.
> >
> >
> > PR: https://github.com/apache/pulsar/pull/21438
> >
> > Discussion thread:
> >
> > https://lists.apache.org/thread/0rz29wotonmdck76pdscwbqo19t3rbds
> >
> >
> > Sincerely,
> >
> > Pengcheng Jiang
> >
> >
> >

Re: [Discuss] Release Pulsar C++ Client 3.4.1

[Zike Yang]

+1
Thanks for your fixes.

BR,
Zike Yang

On Thu, Nov 16, 2023 at 5:48 PM Yunze Xu  wrote:
>
> There is another critical issue introduced by 3.4.0:
> https://github.com/apache/pulsar-client-python/issues/162
>
> I will open another fix for it.
>
> Thanks,
> Yunze
>
> On Thu, Nov 16, 2023 at 12:45 PM Yunze Xu  wrote:
> >
> > In the latest C++ Client 3.4.0 release, we observed a regression
> > https://github.com/apache/pulsar-client-cpp/issues/346. It's a
> > critical bug that a crash could happen for each reconnection.
> >
> > I pushed a fix (https://github.com/apache/pulsar-client-cpp/pull/347).
> > After it's merged, I will start the release process for 3.4.1.
> >
> > Please let me know if you have any questions.
> >
> > Thanks,
> > Yunze

Re: [VOTE] Pulsar Release 3.0.2 Candidate 4

[Zike Yang]

Hi Yubiao,

I see that there is only arm64 arch for the docker image. We need to
follow this guide to build the image:
https://github.com/apache/pulsar-site/pull/595.

The current release process is incorrect for building the docker
image. This PR(https://github.com/apache/pulsar-site/pull/595) has
been blocked for some time. We need to move this PR forward.

BR,
Zike Yang

On Mon, Nov 20, 2023 at 5:07 PM Yubiao Feng
 wrote:
>
> Sorry, the SHA-512 checksums are wrong by mistake. Correct it below:
>
> apache-pulsar-3.0.2-bin.tar.gz
>
> b3421eb57233638e4d8305b040b1ffd1374b9ddfa55ea1a1043dc9be78ed89f72604cf249fcf5e663f1e3e1be86daf36991ee069ea1eb146f384c374484da0ff
>
> apache-pulsar-3.0.2-src.tar.gz
>
> 02154a2c190ac2ef7e1ff3fce5e90eab896f392317c28b28e9a6ab912a5ca3b3a38dc358b5925153ba1b0dde340cafd1e6aa72349dd9c9ae5dd99d0bdef587fb
>
>
> Thanks
> Yubiao Feng
>
> 233638e4d8305b040b1ffd1374b9ddfa55ea1a1043dc9be78ed89f72604cf249fcf5e663f1e3e1be86daf36991ee069ea1eb146f384c374484da0ff
> ./apache-pulsar-3.0.2-bin.tar.gz
>
> On Mon, Nov 20, 2023 at 10:24 AM Yubiao Feng 
> wrote:
>
> > This is the first release candidate for Apache Pulsar version 3.0.4.
> >
> > It fixes the following issues:
> >
> > https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Amerged+label%3Arelease%2F3.0.2+label%3Acherry-picked%2Fbranch-3.0+
> >
> > *** Please download, test and vote on this release. This vote will
> > stay open for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided
> > for convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.0.2-candidate-4/
> >
> > SHA-512 checksums:
> >
> >
> > 9bd5b4030de47c10a906d6271731ca7bcfb8801984cdd625789bc0c5047138f0f2eb1aaceaec1c8941e2ccfa639bbb2458496518e87fb29abb9ebc518e2da60d
> >
> > apache-pulsar-3.0.2-bin.tar.gz
> >
> >
> > 6854d1776de08c9079228a5a5d2f670f92b1b905f0d92c50ee35c5bc9c53be5225626a6d84fc3bbb209b8b12ff3b142685b5c9ea33609e1bb934e9679402e0b8
> >
> > apache-pulsar-3.0.2-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1249
> >
> > The tag to verify:
> > v3.0.2-candidate-4 (12c92fed7847965e3bc3769a91c866b2f0ec2e44)
> > https://github.com/apache/pulsar/releases/tag/v3.0.2-candidate-4
> >
> > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Docker images:
> >
> > pulsar images:
> > https://hub.docker.com/repository/docker/9947090/pulsar-all
> >
> > pulsar-all images:
> > https://hub.docker.com/repository/docker/9947090/pulsar
> >
> > Please download the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> >
> > Regards
> > Yubiao Feng(poorbarcode)
> >

Re: Pulsar 3.0.2 docker image only supports arm64 platform

[Zike Yang]

We need to use the following commands to push the images:
```sh
regctl image copy 9947090/pulsar:3.0.2-12c92fe apachepulsar/pulsar:3.0.2
regctl image copy 9947090/pulsar-all:3.0.2-12c92fe apachepulsar/pulsar-all:3.0.2
```

It's missing in the release doc. I will update it later.

Thanks,
Zike Yang

On Mon, Dec 4, 2023 at 10:31 AM Yunze Xu  wrote:
>
> I found Matteo has updated 3.1.1 as the latest image:
> https://hub.docker.com/layers/apachepulsar/pulsar/latest/images/sha256-21e8bf1571e4ab559a51b3f6e524d725cffabe3c6836101f9d7ea7eb1e2bf62c?context=explore
>
> But the 3.0.2 still lacks the image for the amd64 platform.
>
> Thanks,
> Yunze
>
> On Mon, Dec 4, 2023 at 12:51 AM Yunze Xu  wrote:
> >
> > Hi all,
> >
> > When I ran unit tests for the pulsar-client-cpp repo, which uses
> > `apachepulsar/pulsar:latest` image to start a standalone, I found the
> > tests failed and the following error [1]:
> >
> > ```
> > standalone The requested image's platform (linux/arm64) does not match
> > the detected host platform (linux/amd64/v3) and no specific platform
> > was requested
> > ```
> >
> > It seems the 3.0.2 image [2] was only built for the arm64 platform. I
> > think there is something wrong with the steps to build the image. It
> > breaks the CI for pulsar-client-cpp repo, as well as any other repo
> > that depends on the latest image. We should fix the image and push it
> > again.
> >
> > [1] 
> > https://github.com/apache/pulsar-client-cpp/actions/runs/7078043224/job/19263093582?pr=360
> > [2] 
> > https://hub.docker.com/layers/apachepulsar/pulsar/3.0.2/images/sha256-5af72da140f3901bccc0a5b56de8764cd60f0d351011fd1b90c484a30889fbf8?context=explore
> >
> > Thanks,
> > Yunze

[DISCUSS] Release Pulsar Python clieng 3.4.0

[Zike Yang]

Hi all,

I would like to propose releasing the Pulsar Python client 3.4.0.

It's over three months since the release of 3.3.0, and there are 46 new commits:
https://github.com/apache/pulsar-client-python/compare/v3.0.0...main

The Pulsar C++ client 3.4.1 has been released. I have upgraded the C++
client for the Python client to 3.4.1:
https://github.com/apache/pulsar-client-python/pull/167

We need to cut a new release. Please let me know if you have any
important fixes that need to be included in Pulsar Python client 3.4.0.

Thanks,
Zike Yang