[GitHub] [pulsar-manager] urfreespace opened a new pull request, #460: fix: launch error with log lib conflict

[GitBox]

urfreespace opened a new pull request, #460:
URL: https://github.com/apache/pulsar-manager/pull/460

   fix launch error due to log lib conflict


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[DISCUSS] Cancel the configuration of autoAck in Function framework

[Baozi]

Hi, guys:

I found out that autoAck configuration in function framework now affects 
Delivery semantics, and make it difficult for users to understand. Refer to the 
following two scenarios.

1. If the user understands that the semantics of Guarantees shall prevail

If the user set Guarantees == ATMOST_ONCE and autoAck == false. Then the 
processing semantics of the actual Function will become ATLEAST_ONCE. Refer to 
the following code, this scenario will not immediately ack.

JavaInstanceRunnable#run():Line273  

if (instanceConfig.getFunctionDetails().getProcessingGuarantees() == 
org.apache.pulsar.functions
.proto.Function.ProcessingGuarantees.ATMOST_ONCE) {
if (instanceConfig.getFunctionDetails().getAutoAck()) { // just when 
autoAck == true to auto ack
currentRecord.ack();
}
}

2. If the user thinks that the framework doesn’t auto ack when autoAck == false

According to the following code, the framework is still automatically acked.

PulsarSinkAtLeastOnceProcessor#sendOutputMessage():Line275 

PulsarSinkEffectivelyOnceProcessor#sendOutputMessage():Line325 


public void sendOutputMessage(TypedMessageBuilder msg, SinkRecord record) 
{
msg.sendAsync()
.thenAccept(messageId -> record.ack()) 
.exceptionally(getPublishErrorHandler(record, true));
}

To sum up, users may be confused when configuring Guarantees and autoAck, and 
cannot judge their correct expected behavior.

I would like to discuss whether it is possible to cancel the autoAck 
configuration and add a CUSTOM type for Guarantees.

switch (processingGuarantees) {
case Guarantees.ATMOST_ONCE: After the framework consumes the message, 
it immediately acks
case Guarantees.ATLEAST_ONCE: After processing on the source side, 
perform ack again
case Guarantees.EFFECTIVELY_ONCE: After processing on the source side, 
perform ack again
case  Guarantees.CUSTOM: The function framework does not help users 
with any ack operations and semantic guarantees
}

If you have any ideas, welcome to discuss. If everyone agrees with this idea, I 
will mention a PIP to promote implementation.

Thanks,
Baodi Shi

Re: [VOTE] PIP-156: Build and Run Pulsar Server on Java 17

[Lari Hotari]

PIP-156 PR https://github.com/apache/pulsar/pull/15264 has been merged to 
master branch.

Please notice that Java 17 is now required for building Pulsar master branch.

btw. https://sdkman.io/ is handy for managing multiple JDK versions in local 
development environments.

-Lari


On 2022/04/20 16:37:21 Heesung Sohn wrote:
> Dear Pulsar Community,
> 
> Please review and vote on this PIP.
> 
> PIP link : https://github.com/apache/pulsar/issues/15207
> 
> Thank you,
> --
> 
> 
> 
> Heesung Sohn
> 
> Platform Engineer
> 
> e: heesung.s...@streamnative.io
> 
> streamnative.io
> 
> 
> 
> 
> 

Build Pulsar Server on Java 17- too strict ?

[Enrico Olivelli]

I am sorry,
I have missed this thread.

I believe that requiring JDK17 to build and especially to RUN the
Pulsar broker is not a good idea currently.
Many enterprises, especially the bigger, or banks, insurance
companieshave strict requirements on some components and they are
very slow to accept bleeding edge tecnologie.

I believe that it is good to run CI on JDK17 and also to build the
docker images on JDK17.
But I know a few companies who won't be able to switch to JDK17 very quickly.

I think it is better to require JDK11 at this moment, and not JDK17,
otherwise users will be stuck with Pulsar 2.10 for a long time.

Requiring JDK17 would be justified only if there is some required new
feature, but this is not the case.

So I propose to change the required JDK version to build and run to
JDK11 for the server part and JDK8 for the client.

Enrico

Il giorno lun 9 mag 2022 alle ore 12:03 Lari Hotari
 ha scritto:
>
> PIP-156 PR https://github.com/apache/pulsar/pull/15264 has been merged to 
> master branch.
>
> Please notice that Java 17 is now required for building Pulsar master branch.
>
> btw. https://sdkman.io/ is handy for managing multiple JDK versions in local 
> development environments.
>
> -Lari
>
>
> On 2022/04/20 16:37:21 Heesung Sohn wrote:
> > Dear Pulsar Community,
> >
> > Please review and vote on this PIP.
> >
> > PIP link : https://github.com/apache/pulsar/issues/15207
> >
> > Thank you,
> > --
> >
> > 
> >
> > Heesung Sohn
> >
> > Platform Engineer
> >
> > e: heesung.s...@streamnative.io
> >
> > streamnative.io
> >
> > 
> > 
> > 
> >

Discuss about creating partitioned topic in V1

[guo jiwei]

Hi, community:
   I found that in the admin API V1, it is possible to successfully create
a partitioned topic without a namespace.
However, in V1, when getting the partitioned topic list
(getPartitionedTopicList) and when creating a non-partitioned topic, the
existence of the namespace is checked.
In V2, we all need to check whether the namespace exists.
For consistency, I think creating a partitioned topic in V1 also needs to
check for the existence of the namespace.
So, I push out patch #14858 .
If the patch is passed, the user's existing topics will not be affected,
the newly created partitioned topic will fail, and a namespace needs to be
created first.

Above, I hope you can give some other ideas.


Regards
Jiwei Guo (Tboy)

[DISCUSS] [PIP-160] ManagedLedger decorator for batch append enties

[Yubiao Feng]

Hi Pulsar community:



I open a pip to discuss ManagedLedger Decorator for batch append enties.



link: https://github.com/apache/pulsar/issues/15370



Thanks, Feng

[GitHub] [pulsar-client-node] Matt-Esch opened a new pull request, #215: prevent double callback segfault in client close

[GitBox]

Matt-Esch opened a new pull request, #215:
URL: https://github.com/apache/pulsar-client-node/pull/215

   There appears to be a race condition that causes the close callback to 
execute twice on close. I believe that a similar issue existed with the old 
synchronous api call in a async worker, it just materialised slightly 
differently. There is a proposed fix for it here 
https://github.com/apache/pulsar/pull/15508
   
   This PR temporarily fixes the problem, though I think a better solution 
should be decided on. For example
   
 - I'm writing to `std::cerr`, I'm not convinced this is generally 
acceptable, but I cannot guarantee the lifetime of the logger
 - The behaviour for old pulsar clients should be slightly different to the 
behaviour once cpp client is fixed, so there probably needs to be a version 
check. We don't want older clients to segfault, so either we bump the min 
version supported or we need to actively check the pulsar client version before 
deciding how to handle the close callback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Build Pulsar Server on Java 17- too strict ?

[Neng Lu]

+1 for requiring JDK11 and prepare for JDK17

On 2022/05/09 11:03:27 Enrico Olivelli wrote:
> I am sorry,
> I have missed this thread.
> 
> I believe that requiring JDK17 to build and especially to RUN the
> Pulsar broker is not a good idea currently.
> Many enterprises, especially the bigger, or banks, insurance
> companieshave strict requirements on some components and they are
> very slow to accept bleeding edge tecnologie.
> 
> I believe that it is good to run CI on JDK17 and also to build the
> docker images on JDK17.
> But I know a few companies who won't be able to switch to JDK17 very quickly.
> 
> I think it is better to require JDK11 at this moment, and not JDK17,
> otherwise users will be stuck with Pulsar 2.10 for a long time.
> 
> Requiring JDK17 would be justified only if there is some required new
> feature, but this is not the case.
> 
> So I propose to change the required JDK version to build and run to
> JDK11 for the server part and JDK8 for the client.
> 
> Enrico
> 
> Il giorno lun 9 mag 2022 alle ore 12:03 Lari Hotari
>  ha scritto:
> >
> > PIP-156 PR https://github.com/apache/pulsar/pull/15264 has been merged to 
> > master branch.
> >
> > Please notice that Java 17 is now required for building Pulsar master 
> > branch.
> >
> > btw. https://sdkman.io/ is handy for managing multiple JDK versions in 
> > local development environments.
> >
> > -Lari
> >
> >
> > On 2022/04/20 16:37:21 Heesung Sohn wrote:
> > > Dear Pulsar Community,
> > >
> > > Please review and vote on this PIP.
> > >
> > > PIP link : https://github.com/apache/pulsar/issues/15207
> > >
> > > Thank you,
> > > --
> > >
> > > 
> > >
> > > Heesung Sohn
> > >
> > > Platform Engineer
> > >
> > > e: heesung.s...@streamnative.io
> > >
> > > streamnative.io
> > >
> > > 
> > > 
> > > 
> > >
> 

Re: [DISCUSS] Cancel the configuration of autoAck in Function framework

[Neng Lu]

Thanks for this detailed discussion about processing guarantee and ack.
These two settings are together affecting the behavior of a running function.

One thing I want to clarify is: 
AUTO_ACK setting means if the function runtime will ack messages or not. 
("function runtime" here specifically refers to the JavaInstanceRunnable. If 
the ack happens inside a sink's implemented write method, it's not auto-ack). 

If AUTO_ACK is TRUE, then the JavaInstanceRunnable will be acking messages.
If AUTO_ACK is  FALSE, then the acking will be done by Sink implementation.

Now with this context, let's review your two scenarios:

> 1.If the user set Guarantees == ATMOST_ONCE and autoAck == false.
To be precise, the processing semantics is not ATLEAST_ONCE. It's actually left 
to the implemented Sink to decide which semantics it is. It can be ATMOST_ONCE, 
ATLEAST_ONCE and probably EFFECTIVELLY_ONCE.

> 2. If the user thinks that the framework doesn’t auto ack when autoAck == 
> false
This behavior is actually correct based on our previous context.

A real problematic scenario here is when USER sets 
ATLEAST_ONCE/EFFECTIVELY_ONCE and AUTO_ACK=true. I don't think the 
JavaInstanceRunnable can ack for use under these cases. So there should be some 
check to ban user submit function with such configs.



On 2022/05/09 09:02:12 Baozi wrote:
> Hi, guys:
> 
> I found out that autoAck configuration in function framework now affects 
> Delivery semantics, and make it difficult for users to understand. Refer to 
> the following two scenarios.
> 
> 1. If the user understands that the semantics of Guarantees shall prevail
> 
> If the user set Guarantees == ATMOST_ONCE and autoAck == false. Then the 
> processing semantics of the actual Function will become ATLEAST_ONCE. Refer 
> to the following code, this scenario will not immediately ack.
> 
> JavaInstanceRunnable#run():Line273  
> 
> if (instanceConfig.getFunctionDetails().getProcessingGuarantees() == 
> org.apache.pulsar.functions
> .proto.Function.ProcessingGuarantees.ATMOST_ONCE) {
> if (instanceConfig.getFunctionDetails().getAutoAck()) { // just when 
> autoAck == true to auto ack
> currentRecord.ack();
> }
> }
> 
> 2. If the user thinks that the framework doesn’t auto ack when autoAck == 
> false
> 
> According to the following code, the framework is still automatically acked.
> 
> PulsarSinkAtLeastOnceProcessor#sendOutputMessage():Line275 
> 
> PulsarSinkEffectivelyOnceProcessor#sendOutputMessage():Line325 
> 
> 
> public void sendOutputMessage(TypedMessageBuilder msg, SinkRecord 
> record) {
> msg.sendAsync()
> .thenAccept(messageId -> record.ack()) 
> .exceptionally(getPublishErrorHandler(record, true));
> }
> 
> To sum up, users may be confused when configuring Guarantees and autoAck, and 
> cannot judge their correct expected behavior.
> 
> I would like to discuss whether it is possible to cancel the autoAck 
> configuration and add a CUSTOM type for Guarantees.
> 
> switch (processingGuarantees) {
>   case Guarantees.ATMOST_ONCE: After the framework consumes the message, 
> it immediately acks
>   case Guarantees.ATLEAST_ONCE: After processing on the source side, 
> perform ack again
>   case Guarantees.EFFECTIVELY_ONCE: After processing on the source side, 
> perform ack again
>   case  Guarantees.CUSTOM: The function framework does not help users 
> with any ack operations and semantic guarantees
> }
> 
> If you have any ideas, welcome to discuss. If everyone agrees with this idea, 
> I will mention a PIP to promote implementation.
> 
> Thanks,
> Baodi Shi
> 
> 

Re: Build Pulsar Server on Java 17- too strict ?

[Matteo Merli]

We have already had several discussions on this subject, on the
by-weekly community call, on the PIP proposal and finally the PIP
vote.

It is not that the PR came out of the blue. Obviously every decision
can be re-visited if there are additional details, though it would be
better if we get the feedback at the time the proposal.

To reiterate the rationale for going directly to 17:

 1. Requiring Java 11 won't buy us anything new and will at the same
time require changes from the part of the users.
 2. 17 is a Java LTS release that will be out for 1 year from the
moment in which we release Pulsar 2.11
 3. It is a stable release with widely available packages for every
platform and from every Java vendor.
 4. We are setting up for 4 years of active support of Java 17,
compared to just 1 year of Java 11
 5. There are several source-level features introduced in 12+ that we
can take advantage of in our codebase


--
Matteo Merli


On Mon, May 9, 2022 at 9:33 AM Neng Lu  wrote:
>
> +1 for requiring JDK11 and prepare for JDK17
>
> On 2022/05/09 11:03:27 Enrico Olivelli wrote:
> > I am sorry,
> > I have missed this thread.
> >
> > I believe that requiring JDK17 to build and especially to RUN the
> > Pulsar broker is not a good idea currently.
> > Many enterprises, especially the bigger, or banks, insurance
> > companieshave strict requirements on some components and they are
> > very slow to accept bleeding edge tecnologie.
> >
> > I believe that it is good to run CI on JDK17 and also to build the
> > docker images on JDK17.
> > But I know a few companies who won't be able to switch to JDK17 very 
> > quickly.
> >
> > I think it is better to require JDK11 at this moment, and not JDK17,
> > otherwise users will be stuck with Pulsar 2.10 for a long time.
> >
> > Requiring JDK17 would be justified only if there is some required new
> > feature, but this is not the case.
> >
> > So I propose to change the required JDK version to build and run to
> > JDK11 for the server part and JDK8 for the client.
> >
> > Enrico
> >
> > Il giorno lun 9 mag 2022 alle ore 12:03 Lari Hotari
> >  ha scritto:
> > >
> > > PIP-156 PR https://github.com/apache/pulsar/pull/15264 has been merged to 
> > > master branch.
> > >
> > > Please notice that Java 17 is now required for building Pulsar master 
> > > branch.
> > >
> > > btw. https://sdkman.io/ is handy for managing multiple JDK versions in 
> > > local development environments.
> > >
> > > -Lari
> > >
> > >
> > > On 2022/04/20 16:37:21 Heesung Sohn wrote:
> > > > Dear Pulsar Community,
> > > >
> > > > Please review and vote on this PIP.
> > > >
> > > > PIP link : https://github.com/apache/pulsar/issues/15207
> > > >
> > > > Thank you,
> > > > --
> > > >
> > > > 
> > > >
> > > > Heesung Sohn
> > > >
> > > > Platform Engineer
> > > >
> > > > e: heesung.s...@streamnative.io
> > > >
> > > > streamnative.io
> > > >
> > > > 
> > > > 
> > > > 
> > > >
> >

Re: Discuss about creating partitioned topic in V1

[Michael Marshall]

Great find, Jiwei Guo. I agree that this should be classified as a bug
and that we should change the V1 implementation so that a topic can
only be created if its namespace already exists.

Thanks,
Michael

On Mon, May 9, 2022 at 7:13 AM guo jiwei  wrote:
>
> Hi, community:
>I found that in the admin API V1, it is possible to successfully create
> a partitioned topic without a namespace.
> However, in V1, when getting the partitioned topic list
> (getPartitionedTopicList) and when creating a non-partitioned topic, the
> existence of the namespace is checked.
> In V2, we all need to check whether the namespace exists.
> For consistency, I think creating a partitioned topic in V1 also needs to
> check for the existence of the namespace.
> So, I push out patch #14858 .
> If the patch is passed, the user's existing topics will not be affected,
> the newly created partitioned topic will fail, and a namespace needs to be
> created first.
>
> Above, I hope you can give some other ideas.
>
>
> Regards
> Jiwei Guo (Tboy)

Re: Discuss about creating partitioned topic in V1

[Enrico Olivelli]

+1
I agree totally. It is a bug

Enrico

Il Lun 9 Mag 2022, 20:59 Michael Marshall  ha scritto:

> Great find, Jiwei Guo. I agree that this should be classified as a bug
> and that we should change the V1 implementation so that a topic can
> only be created if its namespace already exists.
>
> Thanks,
> Michael
>
> On Mon, May 9, 2022 at 7:13 AM guo jiwei  wrote:
> >
> > Hi, community:
> >I found that in the admin API V1, it is possible to successfully
> create
> > a partitioned topic without a namespace.
> > However, in V1, when getting the partitioned topic list
> > (getPartitionedTopicList) and when creating a non-partitioned topic, the
> > existence of the namespace is checked.
> > In V2, we all need to check whether the namespace exists.
> > For consistency, I think creating a partitioned topic in V1 also needs to
> > check for the existence of the namespace.
> > So, I push out patch #14858  >.
> > If the patch is passed, the user's existing topics will not be affected,
> > the newly created partitioned topic will fail, and a namespace needs to
> be
> > created first.
> >
> > Above, I hope you can give some other ideas.
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
>

Re: Discuss about creating partitioned topic in V1

[Matteo Merli]

Why I agree that it "looks like a bug", "walks like a bug" and "talks
like a bug"...
this was actually a "feature" :)

not a good one, or with a good reason to exist, but something we've
been trying not to break. I'm not going to dig into the "wrong"
reasoning on why this was done in the first place, unless someone is
really curious. :D

Having said that, V1 admin API was in effect considered as "soft
deprecated" since Pulsar 2.0. All documentation has been focused on v2
APIs.

I believe a better approach now would be to disable v1 APIs using the
configuration, and actually disabling it by default in next Pulsar
versions.

Matteo

Instead
--
Matteo Merli


On Mon, May 9, 2022 at 12:06 PM Enrico Olivelli  wrote:
>
> +1
> I agree totally. It is a bug
>
> Enrico
>
> Il Lun 9 Mag 2022, 20:59 Michael Marshall  ha scritto:
>
> > Great find, Jiwei Guo. I agree that this should be classified as a bug
> > and that we should change the V1 implementation so that a topic can
> > only be created if its namespace already exists.
> >
> > Thanks,
> > Michael
> >
> > On Mon, May 9, 2022 at 7:13 AM guo jiwei  wrote:
> > >
> > > Hi, community:
> > >I found that in the admin API V1, it is possible to successfully
> > create
> > > a partitioned topic without a namespace.
> > > However, in V1, when getting the partitioned topic list
> > > (getPartitionedTopicList) and when creating a non-partitioned topic, the
> > > existence of the namespace is checked.
> > > In V2, we all need to check whether the namespace exists.
> > > For consistency, I think creating a partitioned topic in V1 also needs to
> > > check for the existence of the namespace.
> > > So, I push out patch #14858  > >.
> > > If the patch is passed, the user's existing topics will not be affected,
> > > the newly created partitioned topic will fail, and a namespace needs to
> > be
> > > created first.
> > >
> > > Above, I hope you can give some other ideas.
> > >
> > >
> > > Regards
> > > Jiwei Guo (Tboy)
> >

Re: [DISCUSS] Cancel the configuration of autoAck in Function framework

[Baozi]

Thanks reply,

> If AUTO_ACK is TRUE, then the JavaInstanceRunnable will be acking messages.
> If AUTO_ACK is  FALSE, then the acking will be done by Sink implementation.

A little confused, I want to know why AUTO_ACK is designed this way.

I'll give another example:

> If AUTO_ACK is TRUE, then the JavaInstanceRunnable will be acking messages.


And if Guarantees != ATMOST_ONCE，then the JavaInstanceRunnable not will ack 
message.

> JavaInstanceRunnable#run():Line273


Thanks,
Baodi Shi

> 2022年5月10日 01:0009，Neng Lu  写道：
> 
> Thanks for this detailed discussion about processing guarantee and ack.
> These two settings are together affecting the behavior of a running function.
> 
> One thing I want to clarify is: 
> AUTO_ACK setting means if the function runtime will ack messages or not. 
> ("function runtime" here specifically refers to the JavaInstanceRunnable. If 
> the ack happens inside a sink's implemented write method, it's not auto-ack). 
> 
> If AUTO_ACK is TRUE, then the JavaInstanceRunnable will be acking messages.
> If AUTO_ACK is  FALSE, then the acking will be done by Sink implementation.
> 
> Now with this context, let's review your two scenarios:
> 
>> 1.If the user set Guarantees == ATMOST_ONCE and autoAck == false.
> To be precise, the processing semantics is not ATLEAST_ONCE. It's actually 
> left to the implemented Sink to decide which semantics it is. It can be 
> ATMOST_ONCE, ATLEAST_ONCE and probably EFFECTIVELLY_ONCE.
> 
>> 2. If the user thinks that the framework doesn’t auto ack when autoAck == 
>> false
> This behavior is actually correct based on our previous context.
> 
> A real problematic scenario here is when USER sets 
> ATLEAST_ONCE/EFFECTIVELY_ONCE and AUTO_ACK=true. I don't think the 
> JavaInstanceRunnable can ack for use under these cases. So there should be 
> some check to ban user submit function with such configs.
> 
> 
> 
> On 2022/05/09 09:02:12 Baozi wrote:
>> Hi, guys:
>> 
>> I found out that autoAck configuration in function framework now affects 
>> Delivery semantics, and make it difficult for users to understand. Refer to 
>> the following two scenarios.
>> 
>> 1. If the user understands that the semantics of Guarantees shall prevail
>> 
>> If the user set Guarantees == ATMOST_ONCE and autoAck == false. Then the 
>> processing semantics of the actual Function will become ATLEAST_ONCE. Refer 
>> to the following code, this scenario will not immediately ack.
>> 
>> JavaInstanceRunnable#run():Line273  
>> 
>> if (instanceConfig.getFunctionDetails().getProcessingGuarantees() == 
>> org.apache.pulsar.functions
>>.proto.Function.ProcessingGuarantees.ATMOST_ONCE) {
>>if (instanceConfig.getFunctionDetails().getAutoAck()) { // just when 
>> autoAck == true to auto ack
>>currentRecord.ack();
>>}
>> }
>> 
>> 2. If the user thinks that the framework doesn’t auto ack when autoAck == 
>> false
>> 
>> According to the following code, the framework is still automatically acked.
>> 
>> PulsarSinkAtLeastOnceProcessor#sendOutputMessage():Line275 
>> 
>> PulsarSinkEffectivelyOnceProcessor#sendOutputMessage():Line325 
>> 
>> 
>> public void sendOutputMessage(TypedMessageBuilder msg, SinkRecord 
>> record) {
>>msg.sendAsync()
>>.thenAccept(messageId -> record.ack()) 
>>.exceptionally(getPublishErrorHandler(record, true));
>> }
>> 
>> To sum up, users may be confused when configuring Guarantees and autoAck, 
>> and cannot judge their correct expected behavior.
>> 
>> I would like to discuss whether it is possible to cancel the autoAck 
>> configuration and add a CUSTOM type for Guarantees.
>> 
>> switch (processingGuarantees) {
>>  case Guarantees.ATMOST_ONCE: After the framework consumes the message, 
>> it immediately acks
>>  case Guarantees.ATLEAST_ONCE: After processing on the source side, 
>> perform ack again
>>  case Guarantees.EFFECTIVELY_ONCE: After processing on the source side, 
>> perform ack again
>>  case  Guarantees.CUSTOM: The function framework does not help users 
>> with any ack operations and semantic guarantees
>> }
>> 
>> If you have any ideas, welcome to discuss. If everyone agrees with this 
>> idea, I will mention a PIP to promote implementation.
>> 
>> Thanks,
>> Baodi Shi
>> 
>> 

Re: [DISCUSS] Cancel the configuration of autoAck in Function framework

[Baozi]

> AUTO_ACK setting means if the function runtime will ack messages or not. 
> ("function runtime" here specifically refers to the JavaInstanceRunnable. If 
> the ack happens inside a sink's implemented write method, it's not auto-ack). 
The description of the official website document is：Whether or not the 
framework acknowledges messages automatically.
For users, sink is also part of the function framework.


Thanks,
Baodi Shi

> 2022年5月10日 09:1407，Baozi  写道：
> 
> Thanks for this detailed discussion about processing guarantee and ack.
> These two settings are together affecting the behavior of a running function.
> 
> One thing I want to clarify is: 
> AUTO_ACK setting means if the function runtime will ack messages or not. 
> ("function runtime" here specifically refers to the JavaInstanceRunnable. If 
> the ack happens inside a sink's implemented write method, it's not auto-ack). 
> 
> If AUTO_ACK is TRUE, then the JavaInstanceRunnable will be acking messages.
> If AUTO_ACK is FALSE, then the acking will be done by Sink implementation.
> 
> Now with this context, let's review your two scenarios:
> 
>> 1.If the user set Guarantees == ATMOST_ONCE and autoAck == false.
> To be precise, the processing semantics is not ATLEAST_ONCE. It's actually 
> left to the implemented Sink to decide which semantics it is. It can be 
> ATMOST_ONCE, ATLEAST_ONCE and probably EFFECTIVELLY_ONCE.
> 
>> 2. If the user thinks that the framework doesn’t auto ack when autoAck == 
>> false
> This behavior is actually correct based on our previous context.
> 
> A real problematic scenario here is when USER sets 
> ATLEAST_ONCE/EFFECTIVELY_ONCE and AUTO_ACK=true. I don't think the 
> JavaInstanceRunnable can ack for use under these cases. So there should be 
> some check to ban user submit function with such configs.
> 
> 
> 
> On 2022/05/09 09:02:12 Baozi wrote:
>> Hi, guys:
>> 
>> I found out that autoAck configuration in function framework now affects 
>> Delivery semantics, and make it difficult for users to understand. Refer to 
>> the following two scenarios.
>> 
>> 1. If the user understands that the semantics of Guarantees shall prevail
>> 
>> If the user set Guarantees == ATMOST_ONCE and autoAck == false. Then the 
>> processing semantics of the actual Function will become ATLEAST_ONCE. Refer 
>> to the following code, this scenario will not immediately ack.
>> 
>> JavaInstanceRunnable#run():Line273 
>> >  
>> >
>> if (instanceConfig.getFunctionDetails().getProcessingGuarantees() == 
>> org.apache.pulsar.functions
>> .proto.Function.ProcessingGuarantees.ATMOST_ONCE) {
>> if (instanceConfig.getFunctionDetails().getAutoAck()) { // just when autoAck 
>> == true to auto ack
>> currentRecord.ack();
>> }
>> }
>> 
>> 2. If the user thinks that the framework doesn’t auto ack when autoAck == 
>> false
>> 
>> According to the following code, the framework is still automatically acked.
>> 
>> PulsarSinkAtLeastOnceProcessor#sendOutputMessage():Line275 
>> >  
>> >
>> PulsarSinkEffectivelyOnceProcessor#sendOutputMessage():Line325 
>> >  
>> >
>> 
>> public void sendOutputMessage(TypedMessageBuilder msg, SinkRecord 
>> record) {
>> msg.sendAsync()
>> .thenAccept(messageId -> record.ack()) 
>> .exceptionally(getPublishErrorHandler(record, true));
>> }
>> 
>> To sum up, users may be confused when configuring Guarantees and autoAck, 
>> and cannot judge their correct expected behavior.
>> 
>> I would like to discuss whether it is possible to cancel the autoAck 
>> configuration and add a CUSTOM type for Guarantees.
>> 
>> switch (processingGuarantees) {
>>  case Guarantees.ATMOST_ONCE: After the framework consumes the message, 
>> it immediately acks
>>  case Guarantees.ATLEAST_ONCE: After processing on the source side, 
>> perform ack again
>>  case Guarantees.EFFECTIVELY_ONCE: After processing on the source side, 
>> perform ack again
>>  case Guar

Re: Call for projects and mentors for OSPP 2022

[Dianjin Wang]

Hi guys, thanks for your project ideas. I have helped submit your ideas to
the OSPP website. I believe you have received the invitation emails, PTAL.
Next, the interested students will email you to learn more about your
project, please be patient with their questions. :>

Reference:
[1] Mentor Guide: https://summer-ospp.ac.cn/help/en/mentor/
[2] Student Guide: https://summer-ospp.ac.cn/help/en/student/

Best,
Dianjin Wang


On Sat, May 7, 2022 at 9:25 PM Lin Lin  wrote:

> Thanks for Penghui and Dianjin,
>
> Project Name: Implement the MongoDB Connector
>
> Project Description:
>
> Pulsar IO connectors enable we to easily create, deploy, and manage
> connectors that interact with external systems, such as Apache Cassandra,
> Aerospike, and many others.
>
> Currently, the MongoDB Connector is still in a demo state, we need a
> MongoDB Connector that can run in a production environment.
> We need to support both sink and source, full data transfer, multi-table
> data transfer, etc..
>
> Difficulty Level:
> - [x] Basic
> - [ ] Advanced
>
> Project Validation Items:
> Item 1: Understand and try to use pulsar connector.
> Item 2: Prepare a design for this feature.
> Item 3: Start coding.
> Item 4: Add unit and integration tests.
> Item 5: Add doc for this feature.
>
> Project Mentor:
> Your Name: Lin Lin
> Your Email: lin...@apache.org
> Your Apache ID: linlin
>

New Pulsar Website is Coming Soon!

[Yu]

Hi everyone,

The New Pulsar Website (beta) has been running for more than 3 months.
We've optimized the website based on the feedback from the New Pulsar
Website Survey [1].

Now everything goes fine except few known issues:
- Search function may not work very well
- Some pages are not completely translated

However, they do not block the switchover, and we're continually improving
them all the time.

Consequently, we plan to launch the New Pulsar Website on May 11, Wednesday
(UTC+8).

Feel free to comment if you have any suggestions or concerns on the
switchover, thanks!

[1] https://lists.apache.org/thread/xjccxbozlw7jqlldm2dw0yc74cydknvd

Yu and Lili

[GitHub] [pulsar-manager] urfreespace merged pull request #460: fix: launch error with log lib conflict

[GitBox]

urfreespace merged PR #460:
URL: https://github.com/apache/pulsar-manager/pull/460


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Cancel the configuration of autoAck in Function framework

[Neng Lu]

> For users, sink is also part of the function framework.

^^ Is this written inside any Pulsar documentation? If you look the code 
closely, the source and sink are actually configurable in Java runtime. User 
can actually provide their own source/sink implementation.

On 2022/05/10 01:38:48 Baozi wrote:
> > AUTO_ACK setting means if the function runtime will ack messages or not. 
> > ("function runtime" here specifically refers to the JavaInstanceRunnable. 
> > If the ack happens inside a sink's implemented write method, it's not 
> > auto-ack). 
> The description of the official website document is：Whether or not the 
> framework acknowledges messages automatically.
> For users, sink is also part of the function framework.
> 
> 
> Thanks,
> Baodi Shi
> 
> > 2022年5月10日 09:1407，Baozi  写道：
> > 
> > Thanks for this detailed discussion about processing guarantee and ack.
> > These two settings are together affecting the behavior of a running 
> > function.
> > 
> > One thing I want to clarify is: 
> > AUTO_ACK setting means if the function runtime will ack messages or not. 
> > ("function runtime" here specifically refers to the JavaInstanceRunnable. 
> > If the ack happens inside a sink's implemented write method, it's not 
> > auto-ack). 
> > 
> > If AUTO_ACK is TRUE, then the JavaInstanceRunnable will be acking messages.
> > If AUTO_ACK is FALSE, then the acking will be done by Sink implementation.
> > 
> > Now with this context, let's review your two scenarios:
> > 
> >> 1.If the user set Guarantees == ATMOST_ONCE and autoAck == false.
> > To be precise, the processing semantics is not ATLEAST_ONCE. It's actually 
> > left to the implemented Sink to decide which semantics it is. It can be 
> > ATMOST_ONCE, ATLEAST_ONCE and probably EFFECTIVELLY_ONCE.
> > 
> >> 2. If the user thinks that the framework doesn’t auto ack when autoAck == 
> >> false
> > This behavior is actually correct based on our previous context.
> > 
> > A real problematic scenario here is when USER sets 
> > ATLEAST_ONCE/EFFECTIVELY_ONCE and AUTO_ACK=true. I don't think the 
> > JavaInstanceRunnable can ack for use under these cases. So there should be 
> > some check to ban user submit function with such configs.
> > 
> > 
> > 
> > On 2022/05/09 09:02:12 Baozi wrote:
> >> Hi, guys:
> >> 
> >> I found out that autoAck configuration in function framework now affects 
> >> Delivery semantics, and make it difficult for users to understand. Refer 
> >> to the following two scenarios.
> >> 
> >> 1. If the user understands that the semantics of Guarantees shall prevail
> >> 
> >> If the user set Guarantees == ATMOST_ONCE and autoAck == false. Then the 
> >> processing semantics of the actual Function will become ATLEAST_ONCE. 
> >> Refer to the following code, this scenario will not immediately ack.
> >> 
> >> JavaInstanceRunnable#run():Line273 
> >>  >>  
> >> >
> >> if (instanceConfig.getFunctionDetails().getProcessingGuarantees() == 
> >> org.apache.pulsar.functions
> >> .proto.Function.ProcessingGuarantees.ATMOST_ONCE) {
> >> if (instanceConfig.getFunctionDetails().getAutoAck()) { // just when 
> >> autoAck == true to auto ack
> >> currentRecord.ack();
> >> }
> >> }
> >> 
> >> 2. If the user thinks that the framework doesn’t auto ack when autoAck == 
> >> false
> >> 
> >> According to the following code, the framework is still automatically 
> >> acked.
> >> 
> >> PulsarSinkAtLeastOnceProcessor#sendOutputMessage():Line275 
> >>  >>  
> >> >
> >> PulsarSinkEffectivelyOnceProcessor#sendOutputMessage():Line325 
> >>  >>  
> >> >
> >> 
> >> public void sendOutputMessage(TypedMessageBuilder msg, SinkRecord 
> >> record) {
> >> msg.sendAsync()
> >> .thenAccept(messageId -> record.ack()) 
> >> .exceptionally(getPublishErrorHandler(record, true));
> >> }
> >> 
> >> To sum up, users may be confused when configuring Guarantees and autoAck, 
> >> and cannot judge their correc

Re: Discuss about creating partitioned topic in V1

[guo jiwei]

Thanks.
@Matteo I will track this.

Regards
Jiwei Guo (Tboy)


On Tue, May 10, 2022 at 7:37 AM Matteo Merli  wrote:

> Why I agree that it "looks like a bug", "walks like a bug" and "talks
> like a bug"...
> this was actually a "feature" :)
>
> not a good one, or with a good reason to exist, but something we've
> been trying not to break. I'm not going to dig into the "wrong"
> reasoning on why this was done in the first place, unless someone is
> really curious. :D
>
> Having said that, V1 admin API was in effect considered as "soft
> deprecated" since Pulsar 2.0. All documentation has been focused on v2
> APIs.
>
> I believe a better approach now would be to disable v1 APIs using the
> configuration, and actually disabling it by default in next Pulsar
> versions.
>
> Matteo
>
> Instead
> --
> Matteo Merli
> 
>
> On Mon, May 9, 2022 at 12:06 PM Enrico Olivelli 
> wrote:
> >
> > +1
> > I agree totally. It is a bug
> >
> > Enrico
> >
> > Il Lun 9 Mag 2022, 20:59 Michael Marshall  ha
> scritto:
> >
> > > Great find, Jiwei Guo. I agree that this should be classified as a bug
> > > and that we should change the V1 implementation so that a topic can
> > > only be created if its namespace already exists.
> > >
> > > Thanks,
> > > Michael
> > >
> > > On Mon, May 9, 2022 at 7:13 AM guo jiwei  wrote:
> > > >
> > > > Hi, community:
> > > >I found that in the admin API V1, it is possible to successfully
> > > create
> > > > a partitioned topic without a namespace.
> > > > However, in V1, when getting the partitioned topic list
> > > > (getPartitionedTopicList) and when creating a non-partitioned topic,
> the
> > > > existence of the namespace is checked.
> > > > In V2, we all need to check whether the namespace exists.
> > > > For consistency, I think creating a partitioned topic in V1 also
> needs to
> > > > check for the existence of the namespace.
> > > > So, I push out patch #14858 <
> https://github.com/apache/pulsar/pull/14858
> > > >.
> > > > If the patch is passed, the user's existing topics will not be
> affected,
> > > > the newly created partitioned topic will fail, and a namespace needs
> to
> > > be
> > > > created first.
> > > >
> > > > Above, I hope you can give some other ideas.
> > > >
> > > >
> > > > Regards
> > > > Jiwei Guo (Tboy)
> > >
>

[DISCUSS] [PIP-165] Auto release client useless connections

[Yubiao Feng]

Hi Pulsar community:



I open a pip to discuss Auto release client useless connections.



link: [PIP-165] Auto release client useless connections · Issue #15516 ·
apache/pulsar (github.com) 

Re: [DISCUSS] Cancel the configuration of autoAck in Function framework

[Neng Lu]

Regarding your question "why AUTO_ACK is designed this way"

I think at the time when it's firstly implemented, the AUTO_ACK is just a 
convenient way to help user ack the message.

We can discuss the gap between expected behavior and actual behavior and try to 
resolve or simplify it.

On 2022/05/10 01:14:07 Baozi wrote:
> Thanks reply,
> 
> > If AUTO_ACK is TRUE, then the JavaInstanceRunnable will be acking messages.
> > If AUTO_ACK is  FALSE, then the acking will be done by Sink implementation.
> 
> A little confused, I want to know why AUTO_ACK is designed this way.
> 
> I'll give another example:
> 
> > If AUTO_ACK is TRUE, then the JavaInstanceRunnable will be acking messages.
> 
> 
> And if Guarantees != ATMOST_ONCE，then the JavaInstanceRunnable not will ack 
> message.
> 
> > JavaInstanceRunnable#run():Line273
> 
> 
> Thanks,
> Baodi Shi
> 
> > 2022年5月10日 01:0009，Neng Lu  写道：
> > 
> > Thanks for this detailed discussion about processing guarantee and ack.
> > These two settings are together affecting the behavior of a running 
> > function.
> > 
> > One thing I want to clarify is: 
> > AUTO_ACK setting means if the function runtime will ack messages or not. 
> > ("function runtime" here specifically refers to the JavaInstanceRunnable. 
> > If the ack happens inside a sink's implemented write method, it's not 
> > auto-ack). 
> > 
> > If AUTO_ACK is TRUE, then the JavaInstanceRunnable will be acking messages.
> > If AUTO_ACK is  FALSE, then the acking will be done by Sink implementation.
> > 
> > Now with this context, let's review your two scenarios:
> > 
> >> 1.If the user set Guarantees == ATMOST_ONCE and autoAck == false.
> > To be precise, the processing semantics is not ATLEAST_ONCE. It's actually 
> > left to the implemented Sink to decide which semantics it is. It can be 
> > ATMOST_ONCE, ATLEAST_ONCE and probably EFFECTIVELLY_ONCE.
> > 
> >> 2. If the user thinks that the framework doesn’t auto ack when autoAck == 
> >> false
> > This behavior is actually correct based on our previous context.
> > 
> > A real problematic scenario here is when USER sets 
> > ATLEAST_ONCE/EFFECTIVELY_ONCE and AUTO_ACK=true. I don't think the 
> > JavaInstanceRunnable can ack for use under these cases. So there should be 
> > some check to ban user submit function with such configs.
> > 
> > 
> > 
> > On 2022/05/09 09:02:12 Baozi wrote:
> >> Hi, guys:
> >> 
> >> I found out that autoAck configuration in function framework now affects 
> >> Delivery semantics, and make it difficult for users to understand. Refer 
> >> to the following two scenarios.
> >> 
> >> 1. If the user understands that the semantics of Guarantees shall prevail
> >> 
> >> If the user set Guarantees == ATMOST_ONCE and autoAck == false. Then the 
> >> processing semantics of the actual Function will become ATLEAST_ONCE. 
> >> Refer to the following code, this scenario will not immediately ack.
> >> 
> >> JavaInstanceRunnable#run():Line273  
> >> 
> >> if (instanceConfig.getFunctionDetails().getProcessingGuarantees() == 
> >> org.apache.pulsar.functions
> >>.proto.Function.ProcessingGuarantees.ATMOST_ONCE) {
> >>if (instanceConfig.getFunctionDetails().getAutoAck()) { // just when 
> >> autoAck == true to auto ack
> >>currentRecord.ack();
> >>}
> >> }
> >> 
> >> 2. If the user thinks that the framework doesn’t auto ack when autoAck == 
> >> false
> >> 
> >> According to the following code, the framework is still automatically 
> >> acked.
> >> 
> >> PulsarSinkAtLeastOnceProcessor#sendOutputMessage():Line275 
> >> 
> >> PulsarSinkEffectivelyOnceProcessor#sendOutputMessage():Line325 
> >> 
> >> 
> >> public void sendOutputMessage(TypedMessageBuilder msg, SinkRecord 
> >> record) {
> >>msg.sendAsync()
> >>.thenAccept(messageId -> record.ack()) 
> >>.exceptionally(getPublishErrorHandler(record, true));
> >> }
> >> 
> >> To sum up, users may be confused when configuring Guarantees and autoAck, 
> >> and cannot judge their correct expected behavior.
> >> 
> >> I would like to discuss whether it is possible to cancel the autoAck 
> >> configuration and add a CUSTOM type for Guarantees.
> >> 
> >> switch (processingGuarantees) {
> >>case Guarantees.ATMOST_ONCE: After the framework consumes the message, 
> >> it immediately acks
> >>case Guarantees.ATLEAST_ONCE: After processing on the source side, 
> >> perform ack again
> >>case Guarantees.EFFECTIVELY_

Re: Discuss about creating partitioned topic in V1

[Rajan Dhabalia]

> Having said that, V1 admin API was in effect considered as "soft
deprecated" since Pulsar 2.0. All documentation has been focused on v2 APIs.

There are Orgs which have large scale Pulsar clusters and use V1 apis.
There is no way we can deprecate V1 api and it has to be maintained until
we find a way to migrate V1 topics to V2. Right now, topics with V1 and V2
api names are having a separate state and it's not easy to migrate existing
V1 topics to V2 topics. So, any efforts which will cause incompatibility is
not acceptable and we can not deprecate V1.

Thanks,
Rajan

On Mon, May 9, 2022 at 4:37 PM Matteo Merli  wrote:

> Why I agree that it "looks like a bug", "walks like a bug" and "talks
> like a bug"...
> this was actually a "feature" :)
>
> not a good one, or with a good reason to exist, but something we've
> been trying not to break. I'm not going to dig into the "wrong"
> reasoning on why this was done in the first place, unless someone is
> really curious. :D
>
> Having said that, V1 admin API was in effect considered as "soft
> deprecated" since Pulsar 2.0. All documentation has been focused on v2
> APIs.
>
> I believe a better approach now would be to disable v1 APIs using the
> configuration, and actually disabling it by default in next Pulsar
> versions.
>
> Matteo
>
> Instead
> --
> Matteo Merli
> 
>
> On Mon, May 9, 2022 at 12:06 PM Enrico Olivelli 
> wrote:
> >
> > +1
> > I agree totally. It is a bug
> >
> > Enrico
> >
> > Il Lun 9 Mag 2022, 20:59 Michael Marshall  ha
> scritto:
> >
> > > Great find, Jiwei Guo. I agree that this should be classified as a bug
> > > and that we should change the V1 implementation so that a topic can
> > > only be created if its namespace already exists.
> > >
> > > Thanks,
> > > Michael
> > >
> > > On Mon, May 9, 2022 at 7:13 AM guo jiwei  wrote:
> > > >
> > > > Hi, community:
> > > >I found that in the admin API V1, it is possible to successfully
> > > create
> > > > a partitioned topic without a namespace.
> > > > However, in V1, when getting the partitioned topic list
> > > > (getPartitionedTopicList) and when creating a non-partitioned topic,
> the
> > > > existence of the namespace is checked.
> > > > In V2, we all need to check whether the namespace exists.
> > > > For consistency, I think creating a partitioned topic in V1 also
> needs to
> > > > check for the existence of the namespace.
> > > > So, I push out patch #14858 <
> https://github.com/apache/pulsar/pull/14858
> > > >.
> > > > If the patch is passed, the user's existing topics will not be
> affected,
> > > > the newly created partitioned topic will fail, and a namespace needs
> to
> > > be
> > > > created first.
> > > >
> > > > Above, I hope you can give some other ideas.
> > > >
> > > >
> > > > Regards
> > > > Jiwei Guo (Tboy)
> > >
>

[VOTE] Pulsar Manager Release 0.3.0 Candidate 3

[Li Li]

Hi everyone,
Please review and vote on the release candidate #3 for the version 0.3.0, as 
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* Release notes [1]
* The official Apache source and binary distributions to be deployed to 
dist.apache.org  [2]
* Source code tag "v0.3.0-candidate-3" [4] with git sha 
951095a71f7471dca028da0a330bc1a5e0707333a61fa4a09c8ea0f0a144d5628b511487e2442ebe290b9642b6b8ca7dee486a18a8339c893c37253724ad5fd4
  apache-pulsar-manager-0.3.0-src.tar.gz

PulsarManager's KEYS file contains PGP keys we used to sign this release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS 


Please download these packages and review this release candidate:

- Review release notes
- Download the source package (verify shasum, and asc) and follow the
instructions to build and run the pulsar-manager front end and back end service.
- Download the binary package (verify shasum, and asc) and follow the
instructions to run run the pulsar-manager front end and back end service.

The vote will be open for at least 72 hours. It is adopted by majority 
approval, with at least 3 PMC affirmative votes.


Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-manager/apache-pulsar-manager-0.3.0/apache-pulsar-manager-0.3.0-bin.tar.gz
 

https://dist.apache.org/repos/dist/dev/pulsar/pulsar-manager/apache-pulsar-manager-0.3.0/apache-pulsar-manager-0.3.0-src.tar.gz
 


SHA-512 checksums:

6ffa5921765ee94a404792e98eb4b3cbda9e016c6661ef12e4e873e7e452301bc05650709955b012d08048e418133948a628ad55bc91ac65836022e1ea426d6f
  apache-pulsar-manager-0.3.0-bin.tar.gz
951095a71f7471dca028da0a330bc1a5e0707333a61fa4a09c8ea0f0a144d5628b511487e2442ebe290b9642b6b8ca7dee486a18a8339c893c37253724ad5fd4
  apache-pulsar-manager-0.3.0-src.tar.gz

[GitHub] [pulsar-dotpulsar] blankensteiner commented on pull request #97: Add crude first version of DLQ

[GitBox]

blankensteiner commented on PR #97:
URL: https://github.com/apache/pulsar-dotpulsar/pull/97#issuecomment-1121982695

   Hi @GeroL 
   Let's start with a description of the feature before diving into the current 
implementation.
   How should this work? How is DotPulsar made aware that something is "dead"?
   What is the difference between a "deadLetterTopic" and "retryLetterTopic"?
   How is it decided when a message should go onto one or the other?
   Br
   db


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] benlongo commented on pull request #205: Tiered Storage config

[GitBox]

benlongo commented on PR #205:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/205#issuecomment-1121983501

   If needed I can open a new PR with this rebased.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org