Re: [VOTE] PIP-168: Support zero-copy of NIC to NIC on Proxy

[Cong Zhao]

ok, I have added the new configuration to the proposal.

On 2022/05/31 05:14:13 PengHui Li wrote:
> +1
> 
> Could you please also add the new configuration to the proposal?
> I see you added in the PR https://github.com/apache/pulsar/pull/15678/files
> but not present in the proposal
> 
> Thanks,
> Penghui
> 
> On Thu, May 26, 2022 at 9:40 PM zhaocong  wrote:
> 
> > Hi Pulsar Community,
> >
> >
> > I would like to start a VOTE on "Support zero-copy of NIC to NIC on Proxy"
> > (PIP-168).
> >
> >
> > The proposal can be read at https://github.com/apache/pulsar/issues/15631
> >
> > and the discussion thead is available at
> >
> > https://lists.apache.org/thread/gjys9tvbd5hy28mbkbcq7wkqfldycn7v
> >
> >
> > Voting will stay open for at least 48h.
> >
> >
> > Thanks,
> >
> > Cong Zhao
> >
> 

Re: [VOTE] PIP-168: Support zero-copy of NIC to NIC on Proxy

[Enrico Olivelli]

+1 (binding)

Il giorno mar 31 mag 2022 alle ore 09:58 Cong Zhao
 ha scritto:
>
> ok, I have added the new configuration to the proposal.
>
> On 2022/05/31 05:14:13 PengHui Li wrote:
> > +1
> >
> > Could you please also add the new configuration to the proposal?
> > I see you added in the PR https://github.com/apache/pulsar/pull/15678/files
> > but not present in the proposal
> >
> > Thanks,
> > Penghui
> >
> > On Thu, May 26, 2022 at 9:40 PM zhaocong  wrote:
> >
> > > Hi Pulsar Community,
> > >
> > >
> > > I would like to start a VOTE on "Support zero-copy of NIC to NIC on Proxy"
> > > (PIP-168).
> > >
> > >
> > > The proposal can be read at https://github.com/apache/pulsar/issues/15631
> > >
> > > and the discussion thead is available at
> > >
> > > https://lists.apache.org/thread/gjys9tvbd5hy28mbkbcq7wkqfldycn7v
> > >
> > >
> > > Voting will stay open for at least 48h.
> > >
> > >
> > > Thanks,
> > >
> > > Cong Zhao
> > >
> >

RE: [VOTE] PIP-168: Support zero-copy of NIC to NIC on Proxy

[Zixuan Liu]

+1 (non-binding)

On 2022/05/26 13:40:30 zhaocong wrote:
> Hi Pulsar Community,
> 
> 
> I would like to start a VOTE on "Support zero-copy of NIC to NIC on Proxy"
> (PIP-168).
> 
> 
> The proposal can be read at https://github.com/apache/pulsar/issues/15631
> 
> and the discussion thead is available at
> 
> https://lists.apache.org/thread/gjys9tvbd5hy28mbkbcq7wkqfldycn7v
> 
> 
> Voting will stay open for at least 48h.
> 
> 
> Thanks,
> 
> Cong Zhao
> 

Re: [DISCUSS] PIP-166: Function add NONE delivery semantics

[Baozi]

Hi, Asaf.

Thanks review. 

>> I'm not entirely sure that is accurate. The Effectively-Once as I
> understand it is achieved using transactions, thus the consumption of that
> message and production of any messages, as a result, are considered one
> atomic unit - either message acknowledged and messages produced or neither.


Not using transactions now, I understand: EFFECTIVELY_ONCE = ATLEAST_ONCE + 
Message Deduplication.

@Neng Lu @Rui Fu Can help make sure?

>> I would issue a WARN when reading configuring the function (thus emitted
> once) when the user actively configured autoAck=false and warn them that
> this configuration is deprecated and they should switch to the MANUAL
> ProcessingGuarantee configuration option.


Added to API Change(2)

>> suggest you clarify what existing behavior remains for backward
> compatibility with the appropriate comment that this is deprecated and will
> be removed.

Yes, I have rewritten it, please see Implementation(1)

> 5. Regarding Test Plan
> * I would add: Validate the test of autoAck=false still works (you haven't
> broken anything)
> * I would add: Validate existing ProcessingGuarantee test for AtMostOnce,
> AtLeastOnce, ExactlyOnce still works (when autoAck=true)


Nice, I added to PIP.


Thanks,
Baodi Shi

> 2022年5月30日 22:0011，Asaf Mesika  写道：
> 
> Thanks for applying the fixes.
> 
> 1. Regarding
> 
>> 
>>   - EFFECTIVELY_ONCE: The message is acknowledged *after* the function
>>   finished execution. Depends on pulsar deduplication, and provides
>>   end-to-end effectively once processing.
>> 
>> I'm not entirely sure that is accurate. The Effectively-Once as I
> understand it is achieved using transactions, thus the consumption of that
> message and production of any messages, as a result, are considered one
> atomic unit - either message acknowledged and messages produced or neither.
> 
> 2. Regarding
> 
>> 
>>   1. Indication of autoAck is deprecated, and not use it in the code.
>>   (and also Function.proto)
>> 
>> * I would issue a WARN when reading configuring the function (thus emitted
> once) when the user actively configured autoAck=false and warn them that
> this configuration is deprecated and they should switch to the MANUAL
> ProcessingGuarantee configuration option.
> 
> 3. Regarding
> 
>> 
>>   1. When the delivery semantic is ATMOST_ONCE, the message will be
>>   acked immediately after receiving the message, no longer affected by the
>>   autoAck configuration.
>> 
>> I suggest you clarify what existing behavior remains for backward
> compatibility with the appropriate comment that this is deprecated and will
> be removed.
> 
> 4. Regarding
> 
>> 
>>   1.
>> 
>>   When user call record.ack() in function, just ProcessingGuarantees ==
>>   MANUAL can be work. In turn, when ProcessingGuarantees != MANUAL, user
>>   call record.ack() is invalid(print warn log).
>> 
>> That might blast WARN messages to the user. Perhaps save the fact that you
> have printed a WARN message once and only print when the variable is not
> set?
> 
> 5. Regarding Test Plan
> * I would add: Validate the test of autoAck=false still works (you haven't
> broken anything)
> * I would add: Validate existing ProcessingGuarantee test for AtMostOnce,
> AtLeastOnce, ExactlyOnce still works (when autoAck=true)
> 
> 
> 
> On Mon, May 30, 2022 at 4:09 PM Baozi 
> wrote:
> 
>> Hi, Mesika.
>> 
>> Thanks review.
>> 
 2. I suggest calling it `MANUAL` `ProcessingGuarantee` instead of NONE.
>> As
 you carefully explained, ProcessingGuarantee comes does to the fact that
 the function executor calls acknowledge, in specific timing:
>> 
>> 
>> Added, Refer to the latest pip.
>> https://github.com/apache/pulsar/issues/15560
>> 
 3. Removing autoAck from Function Config breaks backward compatibility,
 thus shouldn't this be strongly reflected in the PIP - how does Pulsar
 release handle breaking change?
>> 
>> As suggested by @neng, They will be marked as deprecated first and clearly
>> stated in the documentation. Remove it after 2~3 release.
>> 
 4. Regarding Implementation (1), isn't the class itself
 PulsarSinkAtLeastOnceProcessor encodes what to do? I'm not sure I
 understand how you use that enum value *inside* the class/
>> 
>> I changed PIP, add new PulsarSinkManualProcessor. Refer to the latest PIP
>> API Changes（3）
>> 
>> Thanks,
>> Baodi Shi
>> 
>>> 2022年5月30日 12:5128，Rui Fu  写道：
>>> 
>>> Hi Baodi,
>>> 
>>> Nice work. Put some suggestions below, ptal.
>>> 
>>> 1. API changes should also contain the changes of `Function.proto`,
>> including new `ProcessingGuarantees` option and `autoAck`.
>>> 2. Please be sure the other language runtimes (like Python, Golang) do
>> support similar `record.ack()` function from the context, if no, it might
>> have some API changes for different runtime we well.
>>> 
>>> 
>>> Best,
>>> 
>>> Rui Fu
>>> 在 2022年5月29日 +0800 22:18，Asaf Mesika ，写道：
 1. "Added NONE delivery semantics and delete autoAck config."
 -

Re: [DISCUSS] PIP-166: Function add NONE delivery semantics

[石宝迪]

Only commiters can modify wikis, we can discuss that first.


Thanks,
Baodi Shi

> 2022年5月30日 22:5828，Asaf Mesika  写道：
> 
> Still new to the PIP process: Shouldn't this PIP be added here
> ?
> 
> On Thu, May 12, 2022 at 11:52 AM Baozi 
> wrote:
> 
>> Hi Pulsar community,
>> 
>> I open a https://github.com/apache/pulsar/issues/15560 for Function add
>> NONE delivery semantics
>> 
>> Let me know what you think.
>> 
>> 
>> Thanks,
>> Baodi Shi
>> 
>> 
>> ## Motivation
>> 
>> Currently Function supports three delivery semantics, and also provides
>> autoAck to control whether to automatically ack.
>> Because autoAck affects the delivery semantics of Function, it can be
>> confusing for users to understand the relationship between these two
>> parameters.
>> 
>> For example, when the user configures `Guarantees == ATMOST_ONCE` and
>> `autoAck == false`, then the framework will not help the user to ack
>> messages, and the processing semantics may become `ATLEAST_ONCE`.
>> 
>> The delivery semantics provided by Function should be clear. When the user
>> sets the guarantees, the framework should ensure point-to-point semantic
>> processing and cannot be affected by other parameters.
>> 
>> ## Goal
>> 
>> Added `NONE` delivery semantics and delete `autoAck` config.
>> 
>> The original intention of `autoAck` semantics is that users want to
>> control the timing of ack by themselves. When autoAck == false, the
>> processing semantics provided by the framework should be invalid. Then we
>> can add `NONE` processing semantics to replace the autoAck == false
>> scenario.
>> 
>> When the user configuration `ProcessingGuarantees == NONE`, the framework
>> does not help the user to do any ack operations, and the ack is left to the
>> user to handle. In other cases, the framework guarantees processing
>> semantics.
>> 
>> ## API Changes
>> 1. Add `NONE` type to ProcessingGuarantees
>> ``` java
>> public enum ProcessingGuarantees {
>>  ATLEAST_ONCE,
>>  ATMOST_ONCE,
>>  EFFECTIVELY_ONCE,
>>  NONE
>> }
>> ```
>> 
>> 2. Delete autoAck config in FunctionConfig
>> ``` java
>> public class FunctionConfig {
>> -private Boolean autoAck;
>> }
>> ```
>> 
>> ## Implementation
>> 
>> 1. In `PulsarSinkAtLeastOnceProcessor` and
>> `PulsarSinkEffectivelyOnceProcessor`, when `ProcessingGuarantees != NONE`
>> can be ack.
>> 
>> <
>> https://github.com/apache/pulsar/blob/c49a977de4b0b525ec80e5070bc90eddcc7cddad/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java#L274-L276
>>> 
>> 
>> 2. When the delivery semantic is `ATMOST_ONCE`, the message will be acked
>> immediately after receiving the message, no longer affected by the autoAck
>> configuration.
>> 
>> 
>> https://github.com/apache/pulsar/blob/c49a977de4b0b525ec80e5070bc90eddcc7cddad/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java#L271-L276
>> 
>> 3. When user call `record.ack()` in function, just  `ProcessingGuarantees
>> == NONE` can be work.
>> 
>> ## Plan test
>> The main test and assert is that when ProcessingGuarantees == NONE, the
>> function framework will not do any ack operations for the user.
>> 
>> ## Compatibility
>> 1. This change will invalidate the user's setting of autoAck, which should
>> be explained in the documentation and provide parameter verification to
>> remind the user.
>> 2. Runtimes of other languages ​​need to maintain consistent processing
>> logic (python, go).
>> 
>> 
>> 

Re: [VOTE] PIP-168: Support zero-copy of NIC to NIC on Proxy

[Christophe Bornet]

+1

Le mar. 31 mai 2022 à 10:33, Zixuan Liu  a écrit :

> +1 (non-binding)
>
> On 2022/05/26 13:40:30 zhaocong wrote:
> > Hi Pulsar Community,
> >
> >
> > I would like to start a VOTE on "Support zero-copy of NIC to NIC on
> Proxy"
> > (PIP-168).
> >
> >
> > The proposal can be read at
> https://github.com/apache/pulsar/issues/15631
> >
> > and the discussion thead is available at
> >
> > https://lists.apache.org/thread/gjys9tvbd5hy28mbkbcq7wkqfldycn7v
> >
> >
> > Voting will stay open for at least 48h.
> >
> >
> > Thanks,
> >
> > Cong Zhao
> >
>

Re: [DISCUSS] [PIP-165] Auto release client useless connections

[Haiting Jiang]

Hi Yubiao,

Thanks for your explanation. I get your point. It's a nice optimization, and
I think we can do this in follow up PRs.

Thanks,
Haiting.

On 2022/05/27 14:00:16 Yubiao Feng wrote:
> Hi Haiting
> 
> > Currently, the client picks connection id randomly within [0 -
> `maxConnectionsPerHosts`) and creates the connection if it's absent.
> How to reserve just one connection for lookup?
> 
> I guess I didn't express it cleanly enough. For example, we built a
> cluster: 192.168.0.1, 192.168.0.2, 192.168.0.3 (owner), client set
> service-url: 192.168.0.1. After the release task executed once, only one
> connection 192.168.0.3 (owner) was still connected. When starting another
> producer, the connection-192.168.0.1(service-URL) will be reconnected for
> command-lookup. This will cause connection-192.168.0.1 to be created and
> released repeatedly. We can declare a policy to either command lookup using
> 192.168.0.3(already created connection) or not release the service-URL
> connection.
> 
> Yubiao Feng
> Thanks
> 
> On Fri, May 27, 2022 at 6:31 PM Haiting Jiang 
> wrote:
> 
> > Hi Yubiao,
> >
> > > We can make sure there is at least one connection in the pool for lookup,
> > Currently, the client picks connection id randomly within [0 -
> > `maxConnectionsPerHosts`) and creates the connection if it's absent.
> > How to reserve just one connection for lookup?
> >
> > > but we can't reduce the time it takes to reconnect to ownership.
> > Yes, it has nothing to do with these released connections.
> >
> > Thanks,
> > Haiting
> >
> > On 2022/05/27 05:39:24 Yubiao Feng wrote:
> > > We can make sure there is at least one connection in the pool for lookup,
> > > but we can't reduce the time it takes to reconnect to ownership. Do you
> > > have a good idea.
> > >
> > > On Fri, May 27, 2022 at 10:15 AM Haiting Jiang 
> > > wrote:
> > >
> > > > Hi Yubiao,
> > > >
> > > > It seems one possible downside of this pip is increasing produce
> > latency
> > > > when topic ownership changed, any way to avoid that?
> > > >
> > > > Thanks,
> > > > Haiting
> > > >
> > > >
> > > > On 2022/05/26 06:31:37 Yubiao Feng wrote:
> > > > > I open a pip to discuss Auto release client useless connections,
> > could
> > > > you
> > > > > help me review
> > > > >
> > > > >
> > > > > ## Motivation
> > > > > Currently, the Pulsar client keeps the connection even if no
> > producers or
> > > > > consumers use this connection.
> > > > > If a client produces messages to topic A and we have 3 brokers 1, 2,
> > 3.
> > > > Due
> > > > > to the bundle unloading(load manager)
> > > > > topic ownership will change from A to B and finally to C. For now,
> > the
> > > > > client-side will keep 3 connections to all 3 brokers.
> > > > > We can optimize this part to reduce the broker side connections, the
> > > > client
> > > > > should close the unused connections.
> > > > >
> > > > > So a mechanism needs to be added to release unwanted connections.
> > > > >
> > > > > ### Why are there idle connections?
> > > > >
> > > > > 1.When configuration `maxConnectionsPerHosts ` is not set to 0, the
> > > > > connection is not closed at all.
> > > > > The design is to hold a fixed number of connections per Host,
> > avoiding
> > > > > frequent closing and creation.
> > > > >
> > > > >
> > > >
> > https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java#L325-L335
> > > > >
> > > > > 2-1. When clients receive `command-close`, will reconnect
> > immediately.
> > > > > It's designed to make it possible to reconnect, rebalance, and
> > unload.
> > > > >
> > > > >
> > > >
> > https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java#L122-L141
> > > > >
> > > > > 2-2. The broker will close client connections before writing
> > ownership
> > > > info
> > > > > to the ZK. Then clients will get deprecated broker address when it
> > tries
> > > > > lookup.
> > > > >
> > > > >
> > > >
> > https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1282-L1293
> > > > >
> > > > > ## Goal
> > > > > Automatically release connections that are no longer used.
> > > > >
> > > > > - Scope
> > > > >   - **Pulsar client**
> > > > > Contains connections used by consumers, Producers, and Transactions.
> > > > >
> > > > >   - **Pulsar proxy**
> > > > > Contains only the connection between Proxy and broker
> > > > >
> > > > > ## Approach
> > > > > Periodically check for idle connections and close them.
> > > > >
> > > > > ## Changes
> > > > >
> > > > > ### API changes
> > > > > **ClientCnx** added an idle check method to mark idle time.
> > > > >
> > > > > ```java
> > > > > /** Create time. **/
> > > > > private final long createTime;
> > > > > /** The time when marks the connec

Re: [VOTE] PIP-168: Support zero-copy of NIC to NIC on Proxy

[Haiting Jiang]

+1

Thanks,
Haiting

On 2022/05/26 13:40:30 zhaocong wrote:
> Hi Pulsar Community,
> 
> 
> I would like to start a VOTE on "Support zero-copy of NIC to NIC on Proxy"
> (PIP-168).
> 
> 
> The proposal can be read at https://github.com/apache/pulsar/issues/15631
> 
> and the discussion thead is available at
> 
> https://lists.apache.org/thread/gjys9tvbd5hy28mbkbcq7wkqfldycn7v
> 
> 
> Voting will stay open for at least 48h.
> 
> 
> Thanks,
> 
> Cong Zhao
> 

Re: [DISCUSS] PIP-166: Function add NONE delivery semantics

[石宝迪]

>> This I don't understand fully. When you introduce MANUAL
> ProcessingGuarantee configuration, isn't configuration universal to all
> languages? When someone uses pulsar-admin to create a function, regardless
> of the language, they can specify Processing Guarantee. If we're adding
> this ability and documenting it, isn't supporting all clients mandatory? I
> definitely think it can be separate pull requests, but it seems important
> IMO to deliver consistent developer experience to the person developing
> functions.

Yes, to support all clients. I didn't express it clearly before. Since our 
changes are backward compatible, before the change is released, we can open 
multiple PR  to iteratively implement the runtime of each language. When all 
languages ​​are supported, publish documentation to inform users.


Thanks,
Baodi Shi

> 2022年5月30日 22:0613，Asaf Mesika  写道：
> 
> On Mon, May 30, 2022 at 4:24 PM Baozi 
> wrote:
> 
>> Hi, Rui.
>> 
>> Thanks review.
>> 
>>> 1. API changes should also contain the changes of `Function.proto`,
>> including new `ProcessingGuarantees` option and `autoAck`.
>> 
>> 
>> I added to pip.
>> 
>>> 2. Please be sure the other language runtimes (like Python, Golang) do
>> support similar `record.ack()` function from the context, if no, it might
>> have some API changes for different runtime we well.
>> 
>> 
>> I added to Compatibility. The goal of this PIP is to keep other language
>> runtimes consistent with java, but it needs to be iterated slowly. We will
>> support java runtimes first.
>> 
>> This I don't understand fully. When you introduce MANUAL
> ProcessingGuarantee configuration, isn't configuration universal to all
> languages? When someone uses pulsar-admin to create a function, regardless
> of the language, they can specify Processing Guarantee. If we're adding
> this ability and documenting it, isn't supporting all clients mandatory? I
> definitely think it can be separate pull requests, but it seems important
> IMO to deliver consistent developer experience to the person developing
> functions.
> 
> 
> 
>> Thanks,
>> Baodi Shi
>> 
>>> 2022年5月30日 12:5128，Rui Fu  写道：
>>> 
>>> Hi Baodi,
>>> 
>>> Nice work. Put some suggestions below, ptal.
>>> 
>>> 1. API changes should also contain the changes of `Function.proto`,
>> including new `ProcessingGuarantees` option and `autoAck`.
>>> 2. Please be sure the other language runtimes (like Python, Golang) do
>> support similar `record.ack()` function from the context, if no, it might
>> have some API changes for different runtime we well.
>>> 
>>> 
>>> Best,
>>> 
>>> Rui Fu
>>> 在 2022年5月29日 +0800 22:18，Asaf Mesika ，写道：
 1. "Added NONE delivery semantics and delete autoAck config."
 - Added --> add
 
 2. I suggest calling it `MANUAL` `ProcessingGuarantee` instead of NONE.
>> As
 you carefully explained, ProcessingGuarantee comes does to the fact that
 the function executor calls acknowledge, in specific timing:
 - `AT_MOST_ONCE` - When the message is read by the client, it is
 immediately acknowledged and only then the function is executed, thus
 guaranteeing it will not run more than once
 - `AT_LEAST_ONCE` - Message is acknowledged *after* the function
>> finished
 execution, thus it will be run at least once.
 - `MANUAL` - Signals to the user that it is up to them to acknowledge
>> the
 message, inside the function.
 
 I think if you couple that change with adding the explanation I wrote
 above to the documentation it will become crystal clear (hopefully)
>> what is
 a Processing Guarantee exactly and what each value signifies.
 
 3. Removing autoAck from Function Config breaks backward compatibility,
 thus shouldn't this be strongly reflected in the PIP - how does Pulsar
 release handle breaking change?
 
 4. Regarding Implementation (1), isn't the class itself
 PulsarSinkAtLeastOnceProcessor encodes what to do? I'm not sure I
 understand how you use that enum value *inside* the class/
 
 
 On Fri, May 27, 2022 at 1:08 AM Neng Lu  wrote:
 
> Some suggestions:
> 
> 1. Instead of deleting the `autoAck`, keep it but not use it in the
>> code.
> And documented clearly it's deprecated for the following 2~3 release.
>> And
> then delete it.
> 2. For `PulsarSinkAtLeastOnceProcessor` and
> `PulsarSinkEffectivelyOnceProcessor`, if `NONE` is configured, it
>> defaults
> to ATLEAST_ONCE.
> 3. Need to let users know the behavior when they call `record.ack()`
>> inside
> the function implementation.
> 
> On Thu, May 12, 2022 at 1:52 AM Baozi > .invalid>
> wrote:
> 
>> Hi Pulsar community,
>> 
>> I open a https://github.com/apache/pulsar/issues/15560 for Function
>> add
>> NONE delivery semantics
>> 
>> Let me know what you think.
>> 
>> 
>> Thanks,
>> Baodi Shi
>> 
>> 
>> ## Motivation
>> 
>> Currently Functi

Re: [DISCUSS] PIP-166: Function add NONE delivery semantics

[Asaf Mesika]

Hi Baodi,

Regarding

>
>1. When the delivery semantic is ATMOST_ONCE, add verify autoAck must
>be true. If the validation fails, let the function fail to start (This
>temporarily resolves the configuration ambiguity). When autoAck is
>subsequently removed, the message will be acked immediately after receiving
>the message.
>
>
> If you fail to start the function, you immediately break people's
functions when they upgrade to this version. How about notifying them once
via logger (WARN)?

Regarding

>
>1.
>
>
>When user call record.ack() in function, just ProcessingGuarantees ==
>MANUAL can be work. In turn, when ProcessingGuarantees != MANUAL, user
>call record.ack() is invalid(print warn log).
>
> It will flood their logs if they used it wrong. Maybe write to log once?

On Tue, May 31, 2022 at 12:24 PM Baozi 
wrote:

> Hi, Asaf.
>
> Thanks review.
>
> >> I'm not entirely sure that is accurate. The Effectively-Once as I
> > understand it is achieved using transactions, thus the consumption of
> that
> > message and production of any messages, as a result, are considered one
> > atomic unit - either message acknowledged and messages produced or
> neither.
>
>
> Not using transactions now, I understand: EFFECTIVELY_ONCE = ATLEAST_ONCE
> + Message Deduplication.
>
> @Neng Lu @Rui Fu Can help make sure?
>
> >> I would issue a WARN when reading configuring the function (thus emitted
> > once) when the user actively configured autoAck=false and warn them that
> > this configuration is deprecated and they should switch to the MANUAL
> > ProcessingGuarantee configuration option.
>
>
> Added to API Change(2)
>
> >> suggest you clarify what existing behavior remains for backward
> > compatibility with the appropriate comment that this is deprecated and
> will
> > be removed.
>
> Yes, I have rewritten it, please see Implementation(1)
>
> > 5. Regarding Test Plan
> > * I would add: Validate the test of autoAck=false still works (you
> haven't
> > broken anything)
> > * I would add: Validate existing ProcessingGuarantee test for AtMostOnce,
> > AtLeastOnce, ExactlyOnce still works (when autoAck=true)
>
>
> Nice, I added to PIP.
>
>
> Thanks,
> Baodi Shi
>
> > 2022年5月30日 22:0011，Asaf Mesika  写道：
> >
> > Thanks for applying the fixes.
> >
> > 1. Regarding
> >
> >>
> >>   - EFFECTIVELY_ONCE: The message is acknowledged *after* the function
> >>   finished execution. Depends on pulsar deduplication, and provides
> >>   end-to-end effectively once processing.
> >>
> >> I'm not entirely sure that is accurate. The Effectively-Once as I
> > understand it is achieved using transactions, thus the consumption of
> that
> > message and production of any messages, as a result, are considered one
> > atomic unit - either message acknowledged and messages produced or
> neither.
> >
> > 2. Regarding
> >
> >>
> >>   1. Indication of autoAck is deprecated, and not use it in the code.
> >>   (and also Function.proto)
> >>
> >> * I would issue a WARN when reading configuring the function (thus
> emitted
> > once) when the user actively configured autoAck=false and warn them that
> > this configuration is deprecated and they should switch to the MANUAL
> > ProcessingGuarantee configuration option.
> >
> > 3. Regarding
> >
> >>
> >>   1. When the delivery semantic is ATMOST_ONCE, the message will be
> >>   acked immediately after receiving the message, no longer affected by
> the
> >>   autoAck configuration.
> >>
> >> I suggest you clarify what existing behavior remains for backward
> > compatibility with the appropriate comment that this is deprecated and
> will
> > be removed.
> >
> > 4. Regarding
> >
> >>
> >>   1.
> >>
> >>   When user call record.ack() in function, just ProcessingGuarantees ==
> >>   MANUAL can be work. In turn, when ProcessingGuarantees != MANUAL, user
> >>   call record.ack() is invalid(print warn log).
> >>
> >> That might blast WARN messages to the user. Perhaps save the fact that
> you
> > have printed a WARN message once and only print when the variable is not
> > set?
> >
> > 5. Regarding Test Plan
> > * I would add: Validate the test of autoAck=false still works (you
> haven't
> > broken anything)
> > * I would add: Validate existing ProcessingGuarantee test for AtMostOnce,
> > AtLeastOnce, ExactlyOnce still works (when autoAck=true)
> >
> >
> >
> > On Mon, May 30, 2022 at 4:09 PM Baozi  .invalid>
> > wrote:
> >
> >> Hi, Mesika.
> >>
> >> Thanks review.
> >>
>  2. I suggest calling it `MANUAL` `ProcessingGuarantee` instead of
> NONE.
> >> As
>  you carefully explained, ProcessingGuarantee comes does to the fact
> that
>  the function executor calls acknowledge, in specific timing:
> >>
> >>
> >> Added, Refer to the latest pip.
> >> https://github.com/apache/pulsar/issues/15560
> >>
>  3. Removing autoAck from Function Config breaks backward
> compatibility,
>  thus shouldn't this be strongly reflected in the PIP - how does Pulsar
>  release handle bre

Re: [DISCUSS] PIP-166: Function add NONE delivery semantics

[Asaf Mesika]

Regarding
>
>
>1. I would add: Validate existing ProcessingGuarantee test for
>AtMostOnce, AtLeastOnce, ExactlyOnce still works (when autoAck=true)
>
> Remove "I would add:"

On Tue, May 31, 2022 at 12:24 PM Baozi 
wrote:

> Hi, Asaf.
>
> Thanks review.
>
> >> I'm not entirely sure that is accurate. The Effectively-Once as I
> > understand it is achieved using transactions, thus the consumption of
> that
> > message and production of any messages, as a result, are considered one
> > atomic unit - either message acknowledged and messages produced or
> neither.
>
>
> Not using transactions now, I understand: EFFECTIVELY_ONCE = ATLEAST_ONCE
> + Message Deduplication.
>
> @Neng Lu @Rui Fu Can help make sure?
>
> >> I would issue a WARN when reading configuring the function (thus emitted
> > once) when the user actively configured autoAck=false and warn them that
> > this configuration is deprecated and they should switch to the MANUAL
> > ProcessingGuarantee configuration option.
>
>
> Added to API Change(2)
>
> >> suggest you clarify what existing behavior remains for backward
> > compatibility with the appropriate comment that this is deprecated and
> will
> > be removed.
>
> Yes, I have rewritten it, please see Implementation(1)
>
> > 5. Regarding Test Plan
> > * I would add: Validate the test of autoAck=false still works (you
> haven't
> > broken anything)
> > * I would add: Validate existing ProcessingGuarantee test for AtMostOnce,
> > AtLeastOnce, ExactlyOnce still works (when autoAck=true)
>
>
> Nice, I added to PIP.
>
>
> Thanks,
> Baodi Shi
>
> > 2022年5月30日 22:0011，Asaf Mesika  写道：
> >
> > Thanks for applying the fixes.
> >
> > 1. Regarding
> >
> >>
> >>   - EFFECTIVELY_ONCE: The message is acknowledged *after* the function
> >>   finished execution. Depends on pulsar deduplication, and provides
> >>   end-to-end effectively once processing.
> >>
> >> I'm not entirely sure that is accurate. The Effectively-Once as I
> > understand it is achieved using transactions, thus the consumption of
> that
> > message and production of any messages, as a result, are considered one
> > atomic unit - either message acknowledged and messages produced or
> neither.
> >
> > 2. Regarding
> >
> >>
> >>   1. Indication of autoAck is deprecated, and not use it in the code.
> >>   (and also Function.proto)
> >>
> >> * I would issue a WARN when reading configuring the function (thus
> emitted
> > once) when the user actively configured autoAck=false and warn them that
> > this configuration is deprecated and they should switch to the MANUAL
> > ProcessingGuarantee configuration option.
> >
> > 3. Regarding
> >
> >>
> >>   1. When the delivery semantic is ATMOST_ONCE, the message will be
> >>   acked immediately after receiving the message, no longer affected by
> the
> >>   autoAck configuration.
> >>
> >> I suggest you clarify what existing behavior remains for backward
> > compatibility with the appropriate comment that this is deprecated and
> will
> > be removed.
> >
> > 4. Regarding
> >
> >>
> >>   1.
> >>
> >>   When user call record.ack() in function, just ProcessingGuarantees ==
> >>   MANUAL can be work. In turn, when ProcessingGuarantees != MANUAL, user
> >>   call record.ack() is invalid(print warn log).
> >>
> >> That might blast WARN messages to the user. Perhaps save the fact that
> you
> > have printed a WARN message once and only print when the variable is not
> > set?
> >
> > 5. Regarding Test Plan
> > * I would add: Validate the test of autoAck=false still works (you
> haven't
> > broken anything)
> > * I would add: Validate existing ProcessingGuarantee test for AtMostOnce,
> > AtLeastOnce, ExactlyOnce still works (when autoAck=true)
> >
> >
> >
> > On Mon, May 30, 2022 at 4:09 PM Baozi  .invalid>
> > wrote:
> >
> >> Hi, Mesika.
> >>
> >> Thanks review.
> >>
>  2. I suggest calling it `MANUAL` `ProcessingGuarantee` instead of
> NONE.
> >> As
>  you carefully explained, ProcessingGuarantee comes does to the fact
> that
>  the function executor calls acknowledge, in specific timing:
> >>
> >>
> >> Added, Refer to the latest pip.
> >> https://github.com/apache/pulsar/issues/15560
> >>
>  3. Removing autoAck from Function Config breaks backward
> compatibility,
>  thus shouldn't this be strongly reflected in the PIP - how does Pulsar
>  release handle breaking change?
> >>
> >> As suggested by @neng, They will be marked as deprecated first and
> clearly
> >> stated in the documentation. Remove it after 2~3 release.
> >>
>  4. Regarding Implementation (1), isn't the class itself
>  PulsarSinkAtLeastOnceProcessor encodes what to do? I'm not sure I
>  understand how you use that enum value *inside* the class/
> >>
> >> I changed PIP, add new PulsarSinkManualProcessor. Refer to the latest
> PIP
> >> API Changes（3）
> >>
> >> Thanks,
> >> Baodi Shi
> >>
> >>> 2022年5月30日 12:5128，Rui Fu  写道：
> >>>
> >>> Hi Baodi,
> >>>
> >>> Nice work. Put some suggestions below, ptal.
> >>

Re: [DISCUSS] PIP-166: Function add NONE delivery semantics

[石宝迪]

>> If you fail to start the function, you immediately break people's
> functions when they upgrade to this version. How about notifying them once
> via logger (WARN)?


I tend to fail. Although this breaks the current logic. but the current 
implementation can be considered is a bug.

> It will flood their logs if they used it wrong. Maybe write to log once?


Agree, I changed PIP.

Thanks,
Baodi Shi

> 2022年5月31日 23:5720，Asaf Mesika  写道：
> 
> Hi Baodi,
> 
> Regarding
> 
>> 
>>   1. When the delivery semantic is ATMOST_ONCE, add verify autoAck must
>>   be true. If the validation fails, let the function fail to start (This
>>   temporarily resolves the configuration ambiguity). When autoAck is
>>   subsequently removed, the message will be acked immediately after receiving
>>   the message.
>> 
>> 
>> If you fail to start the function, you immediately break people's
> functions when they upgrade to this version. How about notifying them once
> via logger (WARN)?
> 
> Regarding
> 
>> 
>>   1.
>> 
>> 
>>   When user call record.ack() in function, just ProcessingGuarantees ==
>>   MANUAL can be work. In turn, when ProcessingGuarantees != MANUAL, user
>>   call record.ack() is invalid(print warn log).
>> 
>> It will flood their logs if they used it wrong. Maybe write to log once?
> 
> On Tue, May 31, 2022 at 12:24 PM Baozi 
> wrote:
> 
>> Hi, Asaf.
>> 
>> Thanks review.
>> 
 I'm not entirely sure that is accurate. The Effectively-Once as I
>>> understand it is achieved using transactions, thus the consumption of
>> that
>>> message and production of any messages, as a result, are considered one
>>> atomic unit - either message acknowledged and messages produced or
>> neither.
>> 
>> 
>> Not using transactions now, I understand: EFFECTIVELY_ONCE = ATLEAST_ONCE
>> + Message Deduplication.
>> 
>> @Neng Lu @Rui Fu Can help make sure?
>> 
 I would issue a WARN when reading configuring the function (thus emitted
>>> once) when the user actively configured autoAck=false and warn them that
>>> this configuration is deprecated and they should switch to the MANUAL
>>> ProcessingGuarantee configuration option.
>> 
>> 
>> Added to API Change(2)
>> 
 suggest you clarify what existing behavior remains for backward
>>> compatibility with the appropriate comment that this is deprecated and
>> will
>>> be removed.
>> 
>> Yes, I have rewritten it, please see Implementation(1)
>> 
>>> 5. Regarding Test Plan
>>> * I would add: Validate the test of autoAck=false still works (you
>> haven't
>>> broken anything)
>>> * I would add: Validate existing ProcessingGuarantee test for AtMostOnce,
>>> AtLeastOnce, ExactlyOnce still works (when autoAck=true)
>> 
>> 
>> Nice, I added to PIP.
>> 
>> 
>> Thanks,
>> Baodi Shi
>> 
>>> 2022年5月30日 22:0011，Asaf Mesika  写道：
>>> 
>>> Thanks for applying the fixes.
>>> 
>>> 1. Regarding
>>> 
 
  - EFFECTIVELY_ONCE: The message is acknowledged *after* the function
  finished execution. Depends on pulsar deduplication, and provides
  end-to-end effectively once processing.
 
 I'm not entirely sure that is accurate. The Effectively-Once as I
>>> understand it is achieved using transactions, thus the consumption of
>> that
>>> message and production of any messages, as a result, are considered one
>>> atomic unit - either message acknowledged and messages produced or
>> neither.
>>> 
>>> 2. Regarding
>>> 
 
  1. Indication of autoAck is deprecated, and not use it in the code.
  (and also Function.proto)
 
 * I would issue a WARN when reading configuring the function (thus
>> emitted
>>> once) when the user actively configured autoAck=false and warn them that
>>> this configuration is deprecated and they should switch to the MANUAL
>>> ProcessingGuarantee configuration option.
>>> 
>>> 3. Regarding
>>> 
 
  1. When the delivery semantic is ATMOST_ONCE, the message will be
  acked immediately after receiving the message, no longer affected by
>> the
  autoAck configuration.
 
 I suggest you clarify what existing behavior remains for backward
>>> compatibility with the appropriate comment that this is deprecated and
>> will
>>> be removed.
>>> 
>>> 4. Regarding
>>> 
 
  1.
 
  When user call record.ack() in function, just ProcessingGuarantees ==
  MANUAL can be work. In turn, when ProcessingGuarantees != MANUAL, user
  call record.ack() is invalid(print warn log).
 
 That might blast WARN messages to the user. Perhaps save the fact that
>> you
>>> have printed a WARN message once and only print when the variable is not
>>> set?
>>> 
>>> 5. Regarding Test Plan
>>> * I would add: Validate the test of autoAck=false still works (you
>> haven't
>>> broken anything)
>>> * I would add: Validate existing ProcessingGuarantee test for AtMostOnce,
>>> AtLeastOnce, ExactlyOnce still works (when autoAck=true)
>>> 
>>> 
>>> 
>>> On Mon, May 30, 2022 at 4:09 PM Baozi > .invalid>
>>> wrote:
>>> 
 Hi, Mesika.
>>>

Re: [VOTE] PIP-168: Support zero-copy of NIC to NIC on Proxy

[Neng Lu]

+1 (non-binding)

On Tue, May 31, 2022 at 6:21 AM Haiting Jiang 
wrote:

> +1
>
> Thanks,
> Haiting
>
> On 2022/05/26 13:40:30 zhaocong wrote:
> > Hi Pulsar Community,
> >
> >
> > I would like to start a VOTE on "Support zero-copy of NIC to NIC on
> Proxy"
> > (PIP-168).
> >
> >
> > The proposal can be read at
> https://github.com/apache/pulsar/issues/15631
> >
> > and the discussion thead is available at
> >
> > https://lists.apache.org/thread/gjys9tvbd5hy28mbkbcq7wkqfldycn7v
> >
> >
> > Voting will stay open for at least 48h.
> >
> >
> > Thanks,
> >
> > Cong Zhao
> >
>


-- 
Best Regards,
Neng

Re: [Discuss] Update Helm Chart to Support 2.10 Docker Image

[Neng Lu]

Hi Michael,

Thanks for the detailed explanation.

On Thu, May 26, 2022 at 11:08 PM Michael Marshall 
wrote:

> Hi Neng Lu,
>
> I put together a doc [0] that includes some tips for troubleshooting a
> non-root docker image. Some of the details depend on how you're
> deploying Pulsar.
>
> If you can ssh to the host as the root user, you can run `docker exec
> --user 0 ...` to get a shell in the container as the root user.
>
> When running on Kubernetes, you might be able to utilize [1] to gain
> root access to the host node for the pod, and then you can exec into
> the container as the root user, as described in the doc [0]. Or, if
> you don't have any pod security policies, you can set the pod's
> securityContext so that the container runs as the root user.
>
> The final option is to build a custom image with additional tooling.
>
> If you find other helpful resources, feel free to update that doc or
> send a note here, and I'll update the doc.
>
> - Michael
>
> [0]
> https://github.com/apache/pulsar/blob/master/docker/README.md#troubleshooting-non-root-containers
> [1] https://github.com/kvaps/kubectl-node-shell
>
> On Thu, May 26, 2022 at 5:24 PM Neng Lu  wrote:
> >
> > Hi All,
> >
> > I'm curious to learn once the image is run as non-root, how can we debug
> or
> > investigate production issues inside a running cluster?
> >
> > On Thu, May 19, 2022 at 12:14 PM Michael Marshall 
> > wrote:
> >
> > > Hello Pulsar Community,
> > >
> > > With the 2.10.0 release, our Pulsar Docker images default to run as a
> > > non-root user. In order to use the 2.10.0 Docker image with the Apache
> > > Pulsar Helm Chart, we need to merge this PR [0]. If you're able,
> > > please review it. Once merged, I propose that we follow up with a
> > > release so that users wanting to upgrade to 2.10.0 have an upgrade
> > > path.
> > >
> > > Thanks,
> > > Michael
> > >
> > > [0] https://github.com/apache/pulsar-helm-chart/pull/266
> > >
> >
> >
> > --
> > Best Regards,
> > Neng
>


-- 
Best Regards,
Neng

Re: [DISCUSS] PIP-166: Function add NONE delivery semantics

[Neng Lu]

Hi Baodi,

Thanks for the reply and update of the PIP.

1. Pulsar Functions currently isn't integrated with the Transaction feature
yet, so there's no EXACTLY_ONCE support.

2. And Yes, "EFFECTIVELY_ONCE = ATLEAST_ONCE + Message Deduplication"



On Tue, May 31, 2022 at 9:16 AM 石宝迪 
wrote:

> >> If you fail to start the function, you immediately break people's
> > functions when they upgrade to this version. How about notifying them
> once
> > via logger (WARN)?
>
>
> I tend to fail. Although this breaks the current logic. but the current
> implementation can be considered is a bug.
>
> > It will flood their logs if they used it wrong. Maybe write to log once?
>
>
> Agree, I changed PIP.
>
> Thanks,
> Baodi Shi
>
> > 2022年5月31日 23:5720，Asaf Mesika  写道：
> >
> > Hi Baodi,
> >
> > Regarding
> >
> >>
> >>   1. When the delivery semantic is ATMOST_ONCE, add verify autoAck must
> >>   be true. If the validation fails, let the function fail to start (This
> >>   temporarily resolves the configuration ambiguity). When autoAck is
> >>   subsequently removed, the message will be acked immediately after
> receiving
> >>   the message.
> >>
> >>
> >> If you fail to start the function, you immediately break people's
> > functions when they upgrade to this version. How about notifying them
> once
> > via logger (WARN)?
> >
> > Regarding
> >
> >>
> >>   1.
> >>
> >>
> >>   When user call record.ack() in function, just ProcessingGuarantees ==
> >>   MANUAL can be work. In turn, when ProcessingGuarantees != MANUAL, user
> >>   call record.ack() is invalid(print warn log).
> >>
> >> It will flood their logs if they used it wrong. Maybe write to log once?
> >
> > On Tue, May 31, 2022 at 12:24 PM Baozi  .invalid>
> > wrote:
> >
> >> Hi, Asaf.
> >>
> >> Thanks review.
> >>
>  I'm not entirely sure that is accurate. The Effectively-Once as I
> >>> understand it is achieved using transactions, thus the consumption of
> >> that
> >>> message and production of any messages, as a result, are considered one
> >>> atomic unit - either message acknowledged and messages produced or
> >> neither.
> >>
> >>
> >> Not using transactions now, I understand: EFFECTIVELY_ONCE =
> ATLEAST_ONCE
> >> + Message Deduplication.
> >>
> >> @Neng Lu @Rui Fu Can help make sure?
> >>
>  I would issue a WARN when reading configuring the function (thus
> emitted
> >>> once) when the user actively configured autoAck=false and warn them
> that
> >>> this configuration is deprecated and they should switch to the MANUAL
> >>> ProcessingGuarantee configuration option.
> >>
> >>
> >> Added to API Change(2)
> >>
>  suggest you clarify what existing behavior remains for backward
> >>> compatibility with the appropriate comment that this is deprecated and
> >> will
> >>> be removed.
> >>
> >> Yes, I have rewritten it, please see Implementation(1)
> >>
> >>> 5. Regarding Test Plan
> >>> * I would add: Validate the test of autoAck=false still works (you
> >> haven't
> >>> broken anything)
> >>> * I would add: Validate existing ProcessingGuarantee test for
> AtMostOnce,
> >>> AtLeastOnce, ExactlyOnce still works (when autoAck=true)
> >>
> >>
> >> Nice, I added to PIP.
> >>
> >>
> >> Thanks,
> >> Baodi Shi
> >>
> >>> 2022年5月30日 22:0011，Asaf Mesika  写道：
> >>>
> >>> Thanks for applying the fixes.
> >>>
> >>> 1. Regarding
> >>>
> 
>   - EFFECTIVELY_ONCE: The message is acknowledged *after* the function
>   finished execution. Depends on pulsar deduplication, and provides
>   end-to-end effectively once processing.
> 
>  I'm not entirely sure that is accurate. The Effectively-Once as I
> >>> understand it is achieved using transactions, thus the consumption of
> >> that
> >>> message and production of any messages, as a result, are considered one
> >>> atomic unit - either message acknowledged and messages produced or
> >> neither.
> >>>
> >>> 2. Regarding
> >>>
> 
>   1. Indication of autoAck is deprecated, and not use it in the code.
>   (and also Function.proto)
> 
>  * I would issue a WARN when reading configuring the function (thus
> >> emitted
> >>> once) when the user actively configured autoAck=false and warn them
> that
> >>> this configuration is deprecated and they should switch to the MANUAL
> >>> ProcessingGuarantee configuration option.
> >>>
> >>> 3. Regarding
> >>>
> 
>   1. When the delivery semantic is ATMOST_ONCE, the message will be
>   acked immediately after receiving the message, no longer affected by
> >> the
>   autoAck configuration.
> 
>  I suggest you clarify what existing behavior remains for backward
> >>> compatibility with the appropriate comment that this is deprecated and
> >> will
> >>> be removed.
> >>>
> >>> 4. Regarding
> >>>
> 
>   1.
> 
>   When user call record.ack() in function, just ProcessingGuarantees ==
>   MANUAL can be work. In turn, when ProcessingGuarantees != MANUAL,
> user
>   call record.ack() is invalid(print warn log

[GitHub] [pulsar-client-node] massakam merged pull request #216: Bump grunt from 1.5.2 to 1.5.3

[GitBox]

massakam merged PR #216:
URL: https://github.com/apache/pulsar-client-node/pull/216


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

About 2.10.1, 2.9.2, 2.8.4 has this been cherry picked yet? [fix][java-client] Fix performance regression with message listener

[Dave Fisher]

This is a fix for a severe performance regression and I wanted to be sure that 
it gets cherry picked.

> On May 31, 2022, at 6:48 PM, GitBox  wrote:
> 
> 
> dave2wave commented on PR #15162:
> URL: https://github.com/apache/pulsar/pull/15162#issuecomment-1143028123
> 
>   Has this been cherry picked to 2.10, 2.9, and 2.8 yet?
> 
> 
> -- 
> This is an automated message from the Apache Git Service.
> To respond to the message, please log on to GitHub and use the
> URL above to go to the specific comment.
> 
> To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
> 
> For queries about this service, please contact Infrastructure at:
> us...@infra.apache.org
> 

[GitHub] [pulsar-site] urfreespace opened a new pull request, #101: update style for navbar

[GitBox]

urfreespace opened a new pull request, #101:
URL: https://github.com/apache/pulsar-site/pull/101

   https://user-images.githubusercontent.com/76024046/171316581-03ab4f1a-0c4b-4e6b-8ae9-619ab5fb9afe.png";>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] urfreespace merged pull request #101: update style for navbar

[GitBox]

urfreespace merged PR #101:
URL: https://github.com/apache/pulsar-site/pull/101


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: About 2.10.1, 2.9.2, 2.8.4 has this been cherry picked yet? [fix][java-client] Fix performance regression with message listener

[Haiting Jiang]

Hi Dave,

Yes, it's cherry-picked on Apr 20. Here are the commits:
https://github.com/apache/pulsar/commit/dcecd86225e2ca681edb0cbf922919539c7174f9
https://github.com/apache/pulsar/commit/0aad9a12339de434ce149cd57d524a3b45b1acc8
https://github.com/apache/pulsar/commit/c7124415b118bc37ee917f4b4c532eda9df24440

Thanks,
Haiting

On 2022/06/01 01:57:40 Dave Fisher wrote:
> This is a fix for a severe performance regression and I wanted to be sure 
> that it gets cherry picked.
> 
> > On May 31, 2022, at 6:48 PM, GitBox  wrote:
> > 
> > 
> > dave2wave commented on PR #15162:
> > URL: https://github.com/apache/pulsar/pull/15162#issuecomment-1143028123
> > 
> >   Has this been cherry picked to 2.10, 2.9, and 2.8 yet?
> > 
> > 
> > -- 
> > This is an automated message from the Apache Git Service.
> > To respond to the message, please log on to GitHub and use the
> > URL above to go to the specific comment.
> > 
> > To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
> > 
> > For queries about this service, please contact Infrastructure at:
> > us...@infra.apache.org
> > 
> 
> 

Re: About 2.10.1, 2.9.2, 2.8.4 has this been cherry picked yet? [fix][java-client] Fix performance regression with message listener

[Yunze Xu]

We use the `cherry-picked` label to mark the PR has been cherry-picked into
the specific branch. For example, the `cherry-picked/branch-2.10` label means
the PR has been cherry-picked to branch-2.10.

Thanks,
Yunze




> 2022年6月1日 09:57，Dave Fisher  写道：
> 
> This is a fix for a severe performance regression and I wanted to be sure 
> that it gets cherry picked.
> 
>> On May 31, 2022, at 6:48 PM, GitBox  wrote:
>> 
>> 
>> dave2wave commented on PR #15162:
>> URL: https://github.com/apache/pulsar/pull/15162#issuecomment-1143028123
>> 
>>  Has this been cherry picked to 2.10, 2.9, and 2.8 yet?
>> 
>> 
>> -- 
>> This is an automated message from the Apache Git Service.
>> To respond to the message, please log on to GitHub and use the
>> URL above to go to the specific comment.
>> 
>> To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
>> 
>> For queries about this service, please contact Infrastructure at:
>> us...@infra.apache.org
>> 
> 

[GitHub] [pulsar-site] urfreespace merged pull request #102: update scripts

[GitBox]

urfreespace merged PR #102:
URL: https://github.com/apache/pulsar-site/pull/102


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] PIP-158: Split client TLS transport encryption from authentication

[Michael Marshall]

I know it's a bit late to respond, but just want to confirm that
Zixuan is correct that we cannot deprecate the `AuthenticationTls`
class because we rely on that class to set the authentication mode in
the pulsar protocol. That mode is then used by the ServerCnx for
authentication and authorization.

Is it possible to deprecate the option to configure the certificates
in the AuthenticationTls class? I think it should be since the certs
are now configured as top level configuration, and they need only be
configured once.

Thanks,
Michael

On Sat, May 14, 2022 at 3:29 AM Zixuan Liu  wrote:
>
> Hi Michael, It's not the same here.
>
> If you use AuthenticationTLS, which means you enable TLS authentication and
> transport.
>
> ```
> PulsarClient client = PulsarClient.builder()
> .serviceUrl("pulsar://my-host:6651")
> .tlsTrustCertsFilePath("/path/to/cacert.pem")
> .tlsKeyFilePath("/path/to/client-key.pem")
> .tlsCertificateFilePath("/path/to/client-cert.pem")
> .authentication(AuthenticationTls.class.getName()) //
> AuthenticationTls will uses the above certificate.
> .build();
> ```
>
> If you remove AuthenticationTLS, means we only use TLS transport.
>
> Thanks,
> Zixuan
>
>
>
>
> Michael Marshall  于2022年5月14日周六 13:27写道：
>
> > Thanks for your responses, Zixuan.
> >
> > I think it might make sense to eventually deprecate the
> > AuthenticationTLS class, if only because I think it can be confusing
> > to give users two ways to configure the same thing. However, that is a
> > minor detail. For now, we'll need to support both.
> >
> > Thanks,
> > Michael
> >
> > On Thu, May 12, 2022 at 4:43 AM Zixuan Liu  wrote:
> > >
> > > You can see the code in the implementation part, this will be consistent
> > > with the actual document.
> > >
> > > Zixuan Liu  于2022年5月12日周四 17:03写道：
> > >
> > > > Hi Michael,
> > > >
> > > > Thanks for your feedback!
> > > >
> > > > >  I notice that the PIP doesn't
> > > > mention documentation. Since we're adding another way to configure
> > > > mTLS, please make sure to document the recommended way that users
> > > > should take advantage of this feature and how this feature relates to
> > the
> > > > existing AuthenticationTLS feature.
> > > >
> > > > Good idea, let me add a simple document that how to use TLS transport
> > and
> > > > TLS authentication.
> > > >
> > > > > We are removing the client's need to use the AuthenticationTLS class
> > > > to perform TLS authentication of clients by the server.
> > > >
> > > > We don't remove the use of the AuthenticationTLS.
> > > >
> > > > > If a user wants to use TLS certificates for authorization, they can
> > > > still put
> > > > roles in their client certificates and continue to use the
> > > > AuthenticationProviderTLS class to map a TLS certificate to a role on
> > > > the server side.
> > > >
> > > > You are right, the users still can use the AuthenticationTLS to perform
> > > > the TLS transport and TLS authentication.
> > > >
> > > > Currently, the AuthenticationTLS includes TLS transport and TLS
> > > > authentication, if the user only uses the TLS transport, not use the
> > TLS
> > > > authentication, it is confusing, so I want to add a TLS transport
> > config in
> > > > `ClientBuilder`.
> > > >
> > > > Thanks,
> > > > Zixuan
> > > >
> > > >
> > > > Michael Marshall  于2022年5月12日周四 01:51写道：
> > > >
> > > >> I agree that the current state of this feature is a bit confusing, and
> > > >> I think the proposed changes make sense. I notice that the PIP doesn't
> > > >> mention documentation. Since we're adding another way to configure
> > > >> mTLS, please make sure to document the recommended way that users
> > > >> should take advantage of this feature and how this feature relates to
> > the
> > > >> existing AuthenticationTLS feature.
> > > >>
> > > >> In order to make sure I understand the feature correctly, can you
> > > >> confirm that the following is correct?
> > > >>
> > > >> We are removing the client's need to use the AuthenticationTLS class
> > > >> to perform TLS authentication of clients by the server. If a user
> > > >> wants to use TLS certificates for authorization, they can still put
> > > >> roles in their client certificates and continue to use the
> > > >> AuthenticationProviderTLS class to map a TLS certificate to a role on
> > > >> the server side.
> > > >>
> > > >> Thanks,
> > > >> Michael
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Mon, May 9, 2022 at 12:58 AM Yunze Xu  > >
> > > >> wrote:
> > > >> >
> > > >> > Thanks for your clarification. Let’s continue maintaining these
> > configs
> > > >> in
> > > >> > `ClientBuilder`.
> > > >> >
> > > >> > Thanks,
> > > >> > Yunze
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > > 2022年5月9日 13:54，Zixuan Liu  写道：
> > > >> > >
> > > >> > > Hi Yunze,
> > > >> > >
> > > >> > > Thanks for your suggestion, your idea is great, but we have the
> > > >> > > `tlsProtocols()` and `tlsCiphers()` in `ClientBui

Pulsar Community Meeting Notes 2022/05/26, (8:30 AM PST)

[Michael Marshall]

Hi Pulsar Community,

Here are the meeting notes from last Thursday's community meeting.
Thanks to all who participated!

Disclaimer: If something is misattributed or misrepresented, please
send a correction to this list.

Source google doc:
https://docs.google.com/document/d/19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE

Thanks,
Michael

2022/05/26, (8:30 AM PST)
-   Attendees:
-   Matteo Merli
-   Michael Marshall
-   Heesung Sohn
-   Dave Fisher
-   Andrey Yegorov
-   Chris Bartholomew
-   Hang Chen

-   Discussions

-   Matteo: the thread from Lari on the mailing list. One of the
points from Lari’s email is about full consistency, and that would
certainly be nice, but the reality is that it is pretty much
impossible to achieve that with a distributed cache because it is
based on the assumption that each broker can do its own caching.
Michael: I believe there is supposed to be an owner for each piece of
metadata, so there isn’t the same caching. Matteo: for example, create
a namespace on one broker and then call a get operation on another
broker, it might not know about the new namespace. This would only
work without caching by going back through zookeeper. Dave: I’ve seen
this behavior with OMB. What if you return right away but signal
eventual consistency? Matteo: either you wait for all brokers to
respond, or you don’t. If you do, you have an availability problem.
Dave: what if you have a guarantee of about a second? Matteo:
everything should be distributed within a second, but there are always
edge cases. Michael: I am not able to represent the full argument, so
we’ll need to defer to another meeting to discuss further with Lari.
Matteo: acknowledges that there is cost to the persistent watches, but
believes it is better in the new implementation than having many
smaller watches, though it’d be worth doing additional testing to
verify the assumption.

-   Matteo: Heesung is on the call and he will be looking at improving
the load manager. Heesung: I’m starting by documenting the current
flow, including sequence diagrams. Michael: that sounds great!

-   PIPs

-   Michael: Asks for review on PIP 167
https://github.com/apache/pulsar/issues/15597. Discussed with Matteo.
There was confusion about how the current subscription level
authorization actually works because it is not additive, but rather
negative. Matteo proposed changing the current implementation to make
subscription permission additive and add a configuration to allow for
users that rely on the current implementation to have backwards
compatibility. Michael: that would certainly work, too. The main goal
is to improve the subscription level auth, and it’d certainly be
easier to understand in an additive model. Michael: also note that the
current feature is essentially undocumented. Matteo: that likely means
that only a handful of users are actually using the feature, which
makes it even more reasonable to take the time to change it to the
“right” implementation. Matteo to review the PIP, discuss the
historical implementation with Rajan, and then respond on the mailing
list.

Re: [DISCUSS] PIP-158: Split client TLS transport encryption from authentication

[Zixuan Liu]

> Is it possible to deprecate the option to configure the certificates
in the AuthenticationTls class? I think it should be since the certs
are now configured as top level configuration, and they need only be
configured once.

We can't do that deprecate the option to configure the certificates in the
AuthenticationTls class, if do this, we need to deal with the
AuthenticationDataProvider, it is complex.

When using `AuthenticationTls` and TLS config of `ClientBuilder`, we only
use the `AuthenticationTls`, which can override the top level
configuration, it is easier.

I submitted https://github.com/apache/pulsar/pull/15634 for this PIP.

---
PIP has some changes:

~For `AuthenticationTls`, we need to check the authParams, when the
authParams is empty, we need to read TLS config from `ClientBuilder`,
otherwise read from the authParams, the authParams can override the config
from `ClientBuilder`.~

When using `AuthenticationTls` and TLS config of `ClientBuilder`, we only
use the `AuthenticationTls`, which can override the top level
configuration.

**Sort priority:** `AuthenticationTls` > TLS config of `ClientBuilder`.
---

Thanks,
Zixuan


Michael Marshall  于2022年6月1日周三 12:54写道：

> I know it's a bit late to respond, but just want to confirm that
> Zixuan is correct that we cannot deprecate the `AuthenticationTls`
> class because we rely on that class to set the authentication mode in
> the pulsar protocol. That mode is then used by the ServerCnx for
> authentication and authorization.
>
> Is it possible to deprecate the option to configure the certificates
> in the AuthenticationTls class? I think it should be since the certs
> are now configured as top level configuration, and they need only be
> configured once.
>
> Thanks,
> Michael
>
> On Sat, May 14, 2022 at 3:29 AM Zixuan Liu  wrote:
> >
> > Hi Michael, It's not the same here.
> >
> > If you use AuthenticationTLS, which means you enable TLS authentication
> and
> > transport.
> >
> > ```
> > PulsarClient client = PulsarClient.builder()
> > .serviceUrl("pulsar://my-host:6651")
> > .tlsTrustCertsFilePath("/path/to/cacert.pem")
> > .tlsKeyFilePath("/path/to/client-key.pem")
> > .tlsCertificateFilePath("/path/to/client-cert.pem")
> > .authentication(AuthenticationTls.class.getName()) //
> > AuthenticationTls will uses the above certificate.
> > .build();
> > ```
> >
> > If you remove AuthenticationTLS, means we only use TLS transport.
> >
> > Thanks,
> > Zixuan
> >
> >
> >
> >
> > Michael Marshall  于2022年5月14日周六 13:27写道：
> >
> > > Thanks for your responses, Zixuan.
> > >
> > > I think it might make sense to eventually deprecate the
> > > AuthenticationTLS class, if only because I think it can be confusing
> > > to give users two ways to configure the same thing. However, that is a
> > > minor detail. For now, we'll need to support both.
> > >
> > > Thanks,
> > > Michael
> > >
> > > On Thu, May 12, 2022 at 4:43 AM Zixuan Liu  wrote:
> > > >
> > > > You can see the code in the implementation part, this will be
> consistent
> > > > with the actual document.
> > > >
> > > > Zixuan Liu  于2022年5月12日周四 17:03写道：
> > > >
> > > > > Hi Michael,
> > > > >
> > > > > Thanks for your feedback!
> > > > >
> > > > > >  I notice that the PIP doesn't
> > > > > mention documentation. Since we're adding another way to configure
> > > > > mTLS, please make sure to document the recommended way that users
> > > > > should take advantage of this feature and how this feature relates
> to
> > > the
> > > > > existing AuthenticationTLS feature.
> > > > >
> > > > > Good idea, let me add a simple document that how to use TLS
> transport
> > > and
> > > > > TLS authentication.
> > > > >
> > > > > > We are removing the client's need to use the AuthenticationTLS
> class
> > > > > to perform TLS authentication of clients by the server.
> > > > >
> > > > > We don't remove the use of the AuthenticationTLS.
> > > > >
> > > > > > If a user wants to use TLS certificates for authorization, they
> can
> > > > > still put
> > > > > roles in their client certificates and continue to use the
> > > > > AuthenticationProviderTLS class to map a TLS certificate to a role
> on
> > > > > the server side.
> > > > >
> > > > > You are right, the users still can use the AuthenticationTLS to
> perform
> > > > > the TLS transport and TLS authentication.
> > > > >
> > > > > Currently, the AuthenticationTLS includes TLS transport and TLS
> > > > > authentication, if the user only uses the TLS transport, not use
> the
> > > TLS
> > > > > authentication, it is confusing, so I want to add a TLS transport
> > > config in
> > > > > `ClientBuilder`.
> > > > >
> > > > > Thanks,
> > > > > Zixuan
> > > > >
> > > > >
> > > > > Michael Marshall  于2022年5月12日周四 01:51写道：
> > > > >
> > > > >> I agree that the current state of this feature is a bit
> confusing, and
> > > > >> I think the proposed changes make sense. I notice that the PIP
> doesn't
> > > > >> mention documentation. Si

[GitHub] [pulsar-helm-chart] michaeljmarshall commented on pull request #266: Add bk, zk securityContext to support upgrade to non-root docker image

[GitBox]

michaeljmarshall commented on PR #266:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/266#issuecomment-1143149449

   > Thanks @michaeljmarshall shouldn't the helm chart version change?
   
   @frankjkelly good question. I know our recent release procedure has tied 
individual PRs with version bumps, but it seems more appropriate to me that we 
should separate releases and features into separate PRs.
   
   Regarding releases, I think the Pulsar Community needs to revisit Helm Chart 
release procedures. We're operating in a gray area by performing releases that 
are not first voted upon on the dev mailing list. By my understanding, all 
Apache project releases are supposed to have a vote.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[VOTE] PIP-165: Auto release client useless connections

[Yubiao Feng]

[VOTE] PIP-165: Auto release client useless connectionsHi Pulsar Community,
I would like to start a VOTE on "Auto release client useless connections"
(PIP-165). The proposal can be read at
https://github.com/apache/pulsar/issues/15516
 and the discussion thead is
available at
https://lists.apache.org/thread/t6h98qs2coc56z06tw38hdlljl67ft4n
  Voting will stay open for at least 48h. Thanks, Yubiao Feng