[DISCUSS] Apache Pulsar 2.9.3 release

[mattison chao]

Hello, Pulsar community:

I'd like to propose to release Apache Pulsar 2.9.3

Currently, we have 192 commits [0] and there are many transaction
fixes, security fixes.

And there are 22 open PRs [1], I will follow them to make sure that
the important fixes could be contained in 2.9.3

If you have any important fixes or any questions,
please reply to this email, we will evaluate whether to
include it in 2.9.3

[0]
https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.9.3+
[1]
https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.9.3+

Best Regards
Mattison

Re: Discussing negative acknowledgment in batch default behavior

[Enrico Olivelli]

Asaf,
I don't have numbers to say something about the runtime impact.

It is a important feature and I am +1 to enabling it by default

You can send a PR in the meantime.
I am not sure about the impact on the tests, maybe we will have to
take a look carefully in order to not introduce new flaky tests

Enrico

Il giorno dom 22 mag 2022 alle ore 18:11 Asaf Mesika
 ha scritto:
>
> Resurfacing this as it seems (to me) an important correction to be made.
>
> Would love your opinions on this.
>
> Thanks!
>
> Asaf
>
> On Wed, May 11, 2022 at 1:16 PM Asaf Mesika  wrote:
>
> > Hello all,
> >
> > I have a question regarding a feature introduced in 2.6.0 called Negative
> > Acknowledgement in Batch Level index, described in PIP-54
> > 
> > .
> >
> > In short: Before this feature, if you would write in batches (in producer)
> > to Pulsar (which makes sense most would for performance reasons), then a
> > consumer sending a negative acknowledgment for 1 message in the batch (say
> > we have 500 in a single batch), would cause the consumer to get the all
> > messages in the batch (500) redelivered as if they were not acknowledge.
> >
> > PIP-54 fixes that by keeping the ack per message in a batch using a bitmap
> > index. The caveats are more memory consumption since the broker keeps those
> > bitmaps in-memory for any inflight batch.
> >
> > With PIP-54 Pulsar IMO becomes "complete" in that it acts the way you
> > would expect it to be, in normal circumstances (as opposed to disconnects,
> > machine crashing, etc).
> >
> > This feature was introduced in 2.6.0, roughly 2 years ago, and is off by
> > default.
> >
> > I was wondering what the developers community thoughts on turning it on by
> > default? Has anyone experienced any performance degradation to it? Have you
> > turned it on in your clusters?
> >
> > Thanks a lot for your time.
> >
> > Asaf Mesika
> >

Re: [DISCUSS] Apache Pulsar 2.9.3 release

[PengHui Li]

+1

Thanks
Penghui

On Mon, May 23, 2022 at 3:31 PM mattison chao 
wrote:

> Hello, Pulsar community:
>
> I'd like to propose to release Apache Pulsar 2.9.3
>
> Currently, we have 192 commits [0] and there are many transaction
> fixes, security fixes.
>
> And there are 22 open PRs [1], I will follow them to make sure that
> the important fixes could be contained in 2.9.3
>
> If you have any important fixes or any questions,
> please reply to this email, we will evaluate whether to
> include it in 2.9.3
>
> [0]
>
> https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.9.3+
> [1]
>
> https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.9.3+
>
> Best Regards
> Mattison
>

Re: Discussing negative acknowledgment in batch default behavior

[PengHui Li]

I support enabling it in 2.11.0

Without the `Acknowledgement in Batch Level index`
Users will see a weird behavior `the acked messages dispatched to the
consumer again`
Many new users thought it was a bug.

For the default configuration, we should be more friendly to new users,
because they have too much pulsar experience. For pulsar veterans, they can
disable it to reduce memory consumption, but they know what the expected
behavior is
after diable it.

And regarding memory consumption, we can optimize in later versions.

Thanks,
Penghui


On Mon, May 23, 2022 at 5:54 PM Enrico Olivelli  wrote:

> Asaf,
> I don't have numbers to say something about the runtime impact.
>
> It is a important feature and I am +1 to enabling it by default
>
> You can send a PR in the meantime.
> I am not sure about the impact on the tests, maybe we will have to
> take a look carefully in order to not introduce new flaky tests
>
> Enrico
>
> Il giorno dom 22 mag 2022 alle ore 18:11 Asaf Mesika
>  ha scritto:
> >
> > Resurfacing this as it seems (to me) an important correction to be made.
> >
> > Would love your opinions on this.
> >
> > Thanks!
> >
> > Asaf
> >
> > On Wed, May 11, 2022 at 1:16 PM Asaf Mesika 
> wrote:
> >
> > > Hello all,
> > >
> > > I have a question regarding a feature introduced in 2.6.0 called
> Negative
> > > Acknowledgement in Batch Level index, described in PIP-54
> > > <
> https://github.com/apache/pulsar/wiki/PIP-54:-Support-acknowledgment-at-batch-index-level
> >
> > > .
> > >
> > > In short: Before this feature, if you would write in batches (in
> producer)
> > > to Pulsar (which makes sense most would for performance reasons), then
> a
> > > consumer sending a negative acknowledgment for 1 message in the batch
> (say
> > > we have 500 in a single batch), would cause the consumer to get the all
> > > messages in the batch (500) redelivered as if they were not
> acknowledge.
> > >
> > > PIP-54 fixes that by keeping the ack per message in a batch using a
> bitmap
> > > index. The caveats are more memory consumption since the broker keeps
> those
> > > bitmaps in-memory for any inflight batch.
> > >
> > > With PIP-54 Pulsar IMO becomes "complete" in that it acts the way you
> > > would expect it to be, in normal circumstances (as opposed to
> disconnects,
> > > machine crashing, etc).
> > >
> > > This feature was introduced in 2.6.0, roughly 2 years ago, and is off
> by
> > > default.
> > >
> > > I was wondering what the developers community thoughts on turning it
> on by
> > > default? Has anyone experienced any performance degradation to it?
> Have you
> > > turned it on in your clusters?
> > >
> > > Thanks a lot for your time.
> > >
> > > Asaf Mesika
> > >
>

Re: [VOTE] [PIP-153] Optimize metadataPositions in MLPendingAckStore

[Enrico Olivelli]

+1 (binding)
it looks like the patch has already been committed

https://github.com/apache/pulsar/commit/ebca19b522fd9f4496689ca7d32ede345d28511a

Enrico

Il giorno lun 16 mag 2022 alle ore 07:18 Hang Chen
 ha scritto:
>
> +1 (binding)
>
> Thanks,
> Hang
>
> mattison chao  于2022年5月12日周四 21:17写道：
> >
> > +1 (non-binding)
> >
> > Best,
> > Mattison
> >
> > On Thu, 12 May 2022 at 12:15, Ran Gao  wrote:
> >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Ran
> > >
> > > On 2022/04/25 06:54:58 一苇以航 wrote:
> > > > Hi Pulsar Community.
> > > >
> > > > This is the voting thread for PIP-153. It will stay open for at least 48
> > > hours.
> > > >
> > > > The proposal can be found:https://github.com/apache/pulsar/issues/15073
> > > >
> > > > Discuss thread:
> > > > https://lists.apache.org/thread/svmbp8ybn6l8o0o8dmvsysnb86qj77r3
> > > >
> > > > Thanks,
> > > > Xiangying
> > > >
> > > >  
> > >

Re: Discussing negative acknowledgment in batch default behavior

[Enrico Olivelli]

Asaf,
would you like to send a PR ?

Enrico

Il giorno lun 23 mag 2022 alle ore 13:38 PengHui Li
 ha scritto:
>
> I support enabling it in 2.11.0
>
> Without the `Acknowledgement in Batch Level index`
> Users will see a weird behavior `the acked messages dispatched to the
> consumer again`
> Many new users thought it was a bug.
>
> For the default configuration, we should be more friendly to new users,
> because they have too much pulsar experience. For pulsar veterans, they can
> disable it to reduce memory consumption, but they know what the expected
> behavior is
> after diable it.
>
> And regarding memory consumption, we can optimize in later versions.
>
> Thanks,
> Penghui
>
>
> On Mon, May 23, 2022 at 5:54 PM Enrico Olivelli  wrote:
>
> > Asaf,
> > I don't have numbers to say something about the runtime impact.
> >
> > It is a important feature and I am +1 to enabling it by default
> >
> > You can send a PR in the meantime.
> > I am not sure about the impact on the tests, maybe we will have to
> > take a look carefully in order to not introduce new flaky tests
> >
> > Enrico
> >
> > Il giorno dom 22 mag 2022 alle ore 18:11 Asaf Mesika
> >  ha scritto:
> > >
> > > Resurfacing this as it seems (to me) an important correction to be made.
> > >
> > > Would love your opinions on this.
> > >
> > > Thanks!
> > >
> > > Asaf
> > >
> > > On Wed, May 11, 2022 at 1:16 PM Asaf Mesika 
> > wrote:
> > >
> > > > Hello all,
> > > >
> > > > I have a question regarding a feature introduced in 2.6.0 called
> > Negative
> > > > Acknowledgement in Batch Level index, described in PIP-54
> > > > <
> > https://github.com/apache/pulsar/wiki/PIP-54:-Support-acknowledgment-at-batch-index-level
> > >
> > > > .
> > > >
> > > > In short: Before this feature, if you would write in batches (in
> > producer)
> > > > to Pulsar (which makes sense most would for performance reasons), then
> > a
> > > > consumer sending a negative acknowledgment for 1 message in the batch
> > (say
> > > > we have 500 in a single batch), would cause the consumer to get the all
> > > > messages in the batch (500) redelivered as if they were not
> > acknowledge.
> > > >
> > > > PIP-54 fixes that by keeping the ack per message in a batch using a
> > bitmap
> > > > index. The caveats are more memory consumption since the broker keeps
> > those
> > > > bitmaps in-memory for any inflight batch.
> > > >
> > > > With PIP-54 Pulsar IMO becomes "complete" in that it acts the way you
> > > > would expect it to be, in normal circumstances (as opposed to
> > disconnects,
> > > > machine crashing, etc).
> > > >
> > > > This feature was introduced in 2.6.0, roughly 2 years ago, and is off
> > by
> > > > default.
> > > >
> > > > I was wondering what the developers community thoughts on turning it
> > on by
> > > > default? Has anyone experienced any performance degradation to it?
> > Have you
> > > > turned it on in your clusters?
> > > >
> > > > Thanks a lot for your time.
> > > >
> > > > Asaf Mesika
> > > >
> >

[Close] [PIP-153] Optimize metadataPositions in MLPendingAckStore

[????????]

Hi Pulsar community,


The PIP has been voted, and has three bindings.
voting 
list: https://lists.apache.org/thread/8z2f9m742yd3oy4hrxxc9vr8ngo0kwrd
So close this PIP.


Thanks, Xiangying

 

[GitHub] [pulsar-dotpulsar] blankensteiner commented on a diff in pull request #104: add server timeout functionality plus docker test

[GitBox]

blankensteiner commented on code in PR #104:
URL: https://github.com/apache/pulsar-dotpulsar/pull/104#discussion_r879466029


##
tests/DotPulsar.Tests/ProducerTests.cs:
##
@@ -152,6 +152,7 @@ private IPulsarClient CreateClient()
 => PulsarClient
 .Builder()
 .Authentication(AuthenticationFactory.Token(ct => 
ValueTask.FromResult(_fixture.CreateToken(Timeout.InfiniteTimeSpan
+.KeepAliveInterval(TimeSpan.FromSeconds(5))

Review Comment:
   Why is this added?



##
tests/DotPulsar.Tests/xunit.runner.json:
##
@@ -1,4 +1,5 @@
 {
   "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json";,
-  "diagnosticMessages": true
+  "diagnosticMessages": true,
+  "parallelizeTestCollections": false

Review Comment:
   This can be removed again if you go ahead with just one cluster for all 
integration tests.



##
DotPulsar.sln:
##
@@ -26,6 +26,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution 
Items", "Solution
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Processing", 
"samples\Processing\Processing.csproj", "{CC1494FA-4EB5-4DB9-8BE9-0A6E8D0D963E}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "DotPulsar.Consumer", 
"tests\DotPulsar.Consumer\DotPulsar.Consumer.csproj", 
"{36E6E6EF-A471-4AE4-B696-1C9DAAFA2770}"

Review Comment:
   Let's get the new tests into DotPulsar.Tests instead of creating new test 
projects.



##
tests/DotPulsar.Tests/IntegrationCollection.cs:
##
@@ -18,3 +18,6 @@ namespace DotPulsar.Tests;
 
 [CollectionDefinition("Integration")]
 public class IntegrationCollection : ICollectionFixture { }
+
+[CollectionDefinition("KeepAlive")]
+public class KeepAliveCollection : ICollectionFixture { }

Review Comment:
   Is a new (standalone) cluster needed? 
   Seems we could solve this with just one cluster/integration fixture.



##
src/DotPulsar/Abstractions/IPulsarClientBuilder.cs:
##
@@ -53,6 +53,16 @@ public interface IPulsarClientBuilder
 /// 
 IPulsarClientBuilder KeepAliveInterval(TimeSpan interval);
 
+/// 
+/// The maximum amount of time to wait without receiving any message from 
the server at
+/// which point the connection is assumed to be dead or the server is not 
responding.
+/// As we are sending pings the server should respond to those at a 
minimum within this specified timeout period.
+/// Once this happens the connection will be torn down and all 
consumers/producers will enter
+/// the disconnected state and attempt to reconnect
+/// The default is 60 seconds.
+/// 
+IPulsarClientBuilder ServerResponseTimeout(TimeSpan interval);

Review Comment:
   Is this also a configurable setting for other clients? If not, we could just 
hardcode it.



##
src/DotPulsar/Internal/PingPongHandler.cs:
##
@@ -25,29 +25,37 @@ public sealed class PingPongHandler : IAsyncDisposable
 {
 private readonly IConnection _connection;
 private readonly TimeSpan _keepAliveInterval;
+private readonly TimeSpan _serverResponseTimeout;
 private readonly Timer _timer;
 private readonly CommandPing _ping;
 private readonly CommandPong _pong;
 private long _lastCommand;
+private readonly TaskCompletionSource _serverNotRespondingTcs;

Review Comment:
   The non-generic TaskCompletionSource is a better fit since the object is 
never used.



##
src/DotPulsar/Internal/PingPongHandler.cs:
##
@@ -61,13 +69,25 @@ private void Watch(object? state)
 var lastCommand = Interlocked.Read(ref _lastCommand);
 var now = Stopwatch.GetTimestamp();
 var elapsed = TimeSpan.FromSeconds((now - lastCommand) / 
Stopwatch.Frequency);
-if (elapsed >= _keepAliveInterval)
+
+if (elapsed > _serverResponseTimeout)
 {
-Task.Factory.StartNew(() => SendPing());
-_timer.Change(_keepAliveInterval, TimeSpan.Zero);
+DotPulsarMeter.ServerTimedout();
+_serverNotRespondingTcs.SetResult(new object());

Review Comment:
   You might as well just return here instead of wrapping the following code in 
an else.



##
src/DotPulsar/Internal/Connection.cs:
##
@@ -294,6 +298,11 @@ private async Task Send(BaseCommand command, 
CancellationToken cancellationToken
 }
 
 public async Task ProcessIncommingFrames(CancellationToken 
cancellationToken)
+{
+await Task.WhenAny(ProcessIncommingFramesImpl(cancellationToken), 
_pingPongHandler.ServerNotResponding);

Review Comment:
   This needs a ConfigureAwait(false)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apac

Re: CFP for ApacheCon Asia 2022 is Open Now

[Dianjin Wang]

Hi all,

One week is left to submit your Pulsar session for ApacheCon Asia 2022.
Don't hesitate!

Best,
Dianjin Wang


On Wed, Apr 20, 2022 at 5:34 PM Dianjin Wang  wrote:

> Hi all,
>
> ApacheCon Asia 2022 is coming! It will be one virtual event and
> pre-recorded, on July 29-31, 2022. Its CFP is also open now, by Tuesday,
> May 31st, 2022 8:00 AM (Beijing time - UTC +8).
>
> Welcome to submit your talks on Pulsar and its ecosystem. Also, English &
> Chinese are both accepted.
>
> [1] https://apachecon.com/acasia2022/index.html
> [2] https://apachecon.com/acasia2022/cfp.html
>
> Best,
> Dianjin Wang
>

[GitHub] [pulsar-site] urfreespace merged pull request #91: add script for watch files change from pulsar repo

[GitBox]

urfreespace merged PR #91:
URL: https://github.com/apache/pulsar-site/pull/91


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

[Matteo Merli]

On Fri, May 20, 2022 at 3:04 AM Enrico Olivelli  wrote:
>
> Andras,
> even if our rules allow to close a vote with a binding -1, we should
> have to reach consensus and understand better Lari's motivations
>
> we are a community and accepting something with a -1 must be handled carefully
>
> I suggest to try to continue the discussion and try to converge

The place for reaching consensus is the [DISCUSSION] thread, and the
discussion thread was out for many days before the vote was started,
without any negative feedback.
When questions were asked on this voting thread, the clarifications were posted.

No one person has a "veto" power, and I don't think that Lari was
asking for that either.

This is a concrete proposal that removes a current limitation in the
system, in a way that does not impact existing users and provides a
simple mechanism for bucketing that can later be expanded to address
more complex scenarios (like migrations, enable/disable,
splitting/merging).

I've read through the points raised by Lari on the other thread and I
will reply on the ideas for MetadataStore API there, but I don't think
the concerns are related to this PIP-157 proposal and I don't see in
there a different suggestion on how to solve the problem raised by
PIP-157.

This PIP-157 is not about limitations of MetadataStore API, rather is
addressing limitations that are in the backend implementations (eg:
ZooKeeper) where the number of z-nodes in a single directory has to be
limited by the frame-size, or the getChildren() operation will fail.

Yes, there are matters that could be improved in the MetadataStore API
or implementation but I don't see how these can be tied with the fact
that we need to split the children in a directory.

Because of that, I don't think we should turn upside-down the PIP
process, which here was followed to the letter and to the spirit, and
we should still consider this PIP-157 as approved.

As I mentioned in an earlier thread, it's important to stick by the
decisions taken (unless new elements arise that were not initially
considered), or otherwise this PIP process would be just a pointless
exercise.

Matteo

--
Matteo Merli

Re: [DISCUSS] Apache Pulsar 2.10.1 release

[Yunze Xu]

+1

Thanks,
Yunze




> 2022年5月23日 11:34，Hang Chen  写道：
> 
> +1
> 
> There are a lot of transaction fixes.
> 
> Thanks,
> Hang
> 
> PengHui Li  于2022年5月21日周六 13:06写道：
>> 
>> Hello, Pulsar community:
>> 
>> I'd like to propose to release Apache Pulsar 2.10.1
>> 
>> Currently, we have 190 commits [0] and there are many transaction
>> fixes, security fixes.
>> 
>> And there are 22 open PRs [1], I will follow them to make sure that
>> the important fixes could be contained in 2.10.1
>> 
>> If you have any important fixes or any questions,
>> please reply to this email, we will evaluate whether to
>> include it in 2.10.1
>> 
>> [0]
>> https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.10.1+
>> [1]
>> https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.10.1+
>> 
>> Best Regards
>> Penghui

[GitHub] [pulsar-site] urfreespace merged pull request #92: optimize watch script

[GitBox]

urfreespace merged PR #92:
URL: https://github.com/apache/pulsar-site/pull/92


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org