Date Change of Pulsar Summit Asia 2021

[Pulsar Summit Team]

Hi Apache Pulsar Community,

Greetings from the Pulsar Summit team! Thank you everyone who has been
supporting Pulsar Summit Asia 2021. However, because of an unexpected
pandemic, we are sorry that the date of Pulsar Summit Asia 2021 has to be
extended from *Nov 20-21, 2021* to *Jan 15-16, 2022*. To ensure a safe
conference environment for our community members is always the priority.
Pulsar Summit Asia 2021 will still be a hybrid event as we originally
designed.

In the meantime, more session submissions are welcomed! We believe that
more Pulsar successes are foreseen during the extension, submit your story
here before *December 15th* if you have reached further achievements in
running Pulsar!

Please stay tuned and stay healthy. And we will keep the updates once the
situation changes.

[1] *Submit your session here*:
https://sessionize.com/pulsar-summit-asia-2021/
[2] *Register for Pulsar Summit Asia 2021*:
https://sessionize.com/pulsar-summit-asia-2021/


Best Regards,
Pulsar Summit Team

[PIP] Broker extensions to provide operators of enterprise-wide clusters better control and flexibility

[Narayanan, Madhavan]

Hi All,

   I request your help to review, discuss and resolve the problem and solution 
approach outlined in the PIP entry https://github.com/apache/pulsar/issues/12858

Regards,
Madhavan

Re: [PIP] Broker extensions to provide operators of enterprise-wide clusters better control and flexibility

[Enrico Olivelli]

Madhavan,
Thanks for sharing your PIP.
It looks interesting, but I see a major problem with this approach.
Basically we would be adding a way to tweak everything in Pulsar, from
Connections to what we are reading and writing to storage.

This feature will become very hard to maintain for users, as Pulsar changes
and there are things that may be different in the future.

We recently had other PIPs that try to add more flexibility and add code
into Pulsar.

It is not clear to me the kind of operations that you want to cover,
perhaps we could provide dedicated extensibility points to fulfill your
needs with specific APIs, that we can maintain and for which we can
guarantee
the compatibility in the future


Enrico

Il giorno gio 18 nov 2021 alle ore 16:31 Narayanan, Madhavan
 ha scritto:

> Hi All,
>
>I request your help to review, discuss and resolve the problem and
> solution approach outlined in the PIP entry
> https://github.com/apache/pulsar/issues/12858
>
> Regards,
> Madhavan
>

Re: [PIP] Broker extensions to provide operators of enterprise-wide clusters better control and flexibility

[Joe F]

Agree with Enrico.

I am not clear on how this (allow interception of write and read operations
of a managed ledger and modify the payload)  would work with e2e
encryption.
That is literally a MITM proposal.

Joe

On Thu, Nov 18, 2021 at 9:04 AM Enrico Olivelli  wrote:

> Madhavan,
> Thanks for sharing your PIP.
> It looks interesting, but I see a major problem with this approach.
> Basically we would be adding a way to tweak everything in Pulsar, from
> Connections to what we are reading and writing to storage.
>
> This feature will become very hard to maintain for users, as Pulsar changes
> and there are things that may be different in the future.
>
> We recently had other PIPs that try to add more flexibility and add code
> into Pulsar.
>
> It is not clear to me the kind of operations that you want to cover,
> perhaps we could provide dedicated extensibility points to fulfill your
> needs with specific APIs, that we can maintain and for which we can
> guarantee
> the compatibility in the future
>
>
> Enrico
>
> Il giorno gio 18 nov 2021 alle ore 16:31 Narayanan, Madhavan
>  ha scritto:
>
> > Hi All,
> >
> >I request your help to review, discuss and resolve the problem and
> > solution approach outlined in the PIP entry
> > https://github.com/apache/pulsar/issues/12858
> >
> > Regards,
> > Madhavan
> >
>

Re: [DISCUSS] Pulsar Protocol For Client Timeouts and Creating Producers

[Neng Lu]

How about making the behavior when timeout configurable? And by default, it 
will send the `CloseProducer` cmd?

On 2021/11/17 05:52:21 Michael Marshall wrote:
> Hi All,
> 
> I noticed that the `ServerCnxTest#testCreateProducerTimeout` test
> indicates that a producer will send a `CloserProducer` command when
> producer creation times out for the client.
> 
> The Java client does not follow this protocol. When the producer
> creation times out, it just schedules another attempt to create the
> producer.
> 
> The C++ client does follow this protocol and is annotated with the
> following comment:
> 
> > // Creating the producer has timed out. We need to ensure the broker closes 
> > the producer
> > // in case it was indeed created, otherwise it might prevent new create 
> > producer operation,
> > // since we are not closing the connection
> 
> I don't see anything in our official protocol spec indicating the
> "right" behavior. Given the test coverage, it seems like the initial
> design was to expect a `CloseProducer` command. However, I also see that
> the broker's `ServerCnx` class will reply to a redundant `Producer`
> command with a `ProducerSuccess` command, as long as the producer
> is already created.
> 
> Should I submit a PR to update the Java client to send a
> `CloseProducer` command when a `Producer` command times out?
> 
> Thanks,
> Michael
> 

RE: [DISCUSS] The processing of residual scheam and the convergence of schema operation auth

[Neng Lu]

+1 for dropping the schema when a topic is deleted by default.

I previously met a strange error that after creating a topic with the same name 
as previous deleted topic, the "ghost" schema is associated with the new topic 
again and caused a lot of confusion for us.


On 2021/11/18 03:35:50 Ruguo Yu wrote:
> Hi All,
> 
>  
> After discussing with Penghui and HangChen about the consistency of topic and 
> schema deletion, our preliminary conclusion is to drop the `--deleteSchema` 
> parameter in `bin/pulsar-admin topics delete`, which can ensure the schema is 
> deleted when the topic is deleted, and the default value is `true` on the 
> broker side to be compatible with the lower version client’s deletion request.
> This change plan to be merged in the next major version release. What about 
> your thoughts on this?
>  
> 
> Thanks,
> 
> Ruguo Yu
> 
>  
> 
> On 2021/11/14 14:14:52 yuruguo wrote:
> 
> > Dear all,
> 
> > 
> 
> > Currently, topic and schema are managed separately, and there will be a 
> > situation, that is, the topic has been deleted but its schema still exists. 
> > Should we deal with these residual schemas? For this problem I created an 
> > issue[1].
> 
> > 
> 
> > In addition, the operation auth of the schema should also converge. To a 
> > certain extent, it is related to the operation auth of topic. For this 
> > problem I created an issue[2].
> 
> > 
> 
> > Regarding the two problems of the schema, please give guidance or a better 
> > solution 😊
> 
> > 
> 
> >  
> 
> > 
> 
> > [1] https://github.com/apache/pulsar/issues/12795
> 
> > 
> 
> > [2] https://github.com/apache/pulsar/issues/12419
> 
> > 
> 
> >  
> 
> > Thanks,
> 
> > Ruguo Yu
> 
> >  
> 
> > 
> 
> > 
> 
> 

Re: [ANNOUNCE] Apache Pulsar Go Client 0.7.0 released

[Neng Lu]

Thank you Chris, I also updated the function's go runtime to utilize the most 
recent release.

On 2021/11/16 05:41:11 Chris Kellogg wrote:
> The Apache Pulsar team is proud to announce Apache Pulsar Go Client version
> 0.7.0.
> 
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management
> for
> subscribers, and cross-datacenter replication.
> 
> For Pulsar release details and downloads, visit:
> https://github.com/apache/pulsar-client-go/releases/tag/v0.7.0
> 
> Release Notes are at:
> https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG.md
> 
> We would like to thank the contributors that made the release possible.
> 
> Regards,
> 
> The Pulsar Team
> 

Re: [DISCUSS] Pulsar Protocol For Client Timeouts and Creating Producers

[Michael Marshall]

I view this as an edge case of the Pulsar Protocol that requires
clarification. Once we clarify the spec, we can update the clients to
conform to the spec. I don't think we need to give end users control
over this small part of the protocol.

After reviewing the design a bit more, I think we should update the
Java client to send the `CloseProducer` command, as well as update the
spec to recommend this implementation. While the `ServerCnx` class
"works" both ways, the current Java client implementation leads to
unnecessary calls, unnecessary errors, and a longer time to recovery.

Specifically, if the client fails to send a `CloseProducer` command,
it ends up getting into a sequence of retries where each new
`Producer` command receives an immediate `ErrorResponse` because the
`ServerCnx` already has a pending producer. By sending a
`CloseProducer` command, the client gives the broker permission to
stop keeping track of the original create producer request. It also
means that if the topic eventually loads, the broker will respond to
the right request id with a `ProducerSuccessResponse` command.

I will follow up with an update to the client and the protocol spec,
unless there are any objections.

Thanks,
Michael

On Thu, Nov 18, 2021 at 12:25 PM Neng Lu  wrote:
>
> How about making the behavior when timeout configurable? And by default, it 
> will send the `CloseProducer` cmd?
>
> On 2021/11/17 05:52:21 Michael Marshall wrote:
> > Hi All,
> >
> > I noticed that the `ServerCnxTest#testCreateProducerTimeout` test
> > indicates that a producer will send a `CloserProducer` command when
> > producer creation times out for the client.
> >
> > The Java client does not follow this protocol. When the producer
> > creation times out, it just schedules another attempt to create the
> > producer.
> >
> > The C++ client does follow this protocol and is annotated with the
> > following comment:
> >
> > > // Creating the producer has timed out. We need to ensure the broker 
> > > closes the producer
> > > // in case it was indeed created, otherwise it might prevent new create 
> > > producer operation,
> > > // since we are not closing the connection
> >
> > I don't see anything in our official protocol spec indicating the
> > "right" behavior. Given the test coverage, it seems like the initial
> > design was to expect a `CloseProducer` command. However, I also see that
> > the broker's `ServerCnx` class will reply to a redundant `Producer`
> > command with a `ProducerSuccess` command, as long as the producer
> > is already created.
> >
> > Should I submit a PR to update the Java client to send a
> > `CloseProducer` command when a `Producer` command times out?
> >
> > Thanks,
> > Michael
> >

Re: [DISCUSS] New repository for website - pulsar-site

[Dave Fisher]

I’m making progress here, but I need help getting the pulsarbot GH secret into 
the pulsar-site repository.

If that secret can be shared directly to me then I can fully test before adding 
my PR.

Thanks,
Dave

> On Nov 17, 2021, at 3:57 PM, Dave Fisher  wrote:
> 
> I’m going to work through 
> https://github.com/apache/pulsar/blob/master/site2/README.md
> 
> I’ll make sure that any changes related to the asf-site branch don’t have 
> issue with that.
> 
> We may want to be able to publish alternative web designs to a staging sites.
> 
>> On Nov 17, 2021, at 3:02 PM, Dave Fisher  wrote:
>> 
>> I’ve updated my fork of apache/pulsar
>> 
>> I’m not seeing how to run the workflow "CI - Pulsar Website build”. Any 
>> ideas?
>> 
>> If not then I’m going to need to test locally and it will take some time to 
>> ready it.
>> 
>> 
>>> On Nov 17, 2021, at 1:15 PM, Matteo Merli  wrote:
>>> 
>>> Yes, that should work.
>>> 
>>> After that we can go ahead and remove `asf-site` from the main repo,
>>> although we need to make it "unprotected" to be able to do so.
>> 
>> Yes once we have moved over to the new then we can ask Infra to take care of 
>> the branch protection along with deleting it.
>> 
>> When I create the new repository I will copy all of the asf-site branch 
>> which will take care of transferring the parts of the site not actively 
>> being built.
> 
> I have created the new repository and populated the asf-site branch: 
> https://github.com/apache/pulsar-site/tree/asf-site
> 
> It publishes to a staging url which you can see here: 
> https://pulsar.staged.apache.org
> 
> Once we are ready we alter: 
> https://github.com/apache/pulsar-site/blob/asf-site/.asf.yaml
> 
> Per: 
> https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features
> 
>> 
>> Regards,
>> Dave
>> 
>>> 
>>> 
>>> --
>>> Matteo Merli
>>> 
>>> 
>>> On Wed, Nov 17, 2021 at 12:46 PM Dave Fisher  wrote:
 
 If we change ORIGIN_REPO[1] to point to a new pulsar-site repos.
 Then with the correct .asf.yaml file changes we can remove the asf-site 
 branch.
 I see that the publish is run from this workflow [2]
 Let me think about a PR to make the move.
 
 Regards,
 Dave
 
 [1] 
 https://github.com/apache/pulsar/blob/7a34cebca25e6e584e8b758e6bd58c1c4fe8a58e/site2/tools/publish-website.sh#L25
 [2] 
 https://github.com/apache/pulsar/blob/master/.github/workflows/ci-pulsar-website-build.yaml
 
 
> On Nov 17, 2021, at 12:31 PM, Matteo Merli  wrote:
> 
> https://github.com/apache/pulsar/blob/master/site2/tools/publish-website.sh
> 
> 
> --
> Matteo Merli
> 
> 
> On Wed, Nov 17, 2021 at 12:29 PM Dave Fisher  wrote:
>> 
>> Show me where the code is that commits to the asf-site branch.
>> 
>>> On Nov 17, 2021, at 12:25 PM, Matteo Merli  
>>> wrote:
>>> 
>>> I agree with that.
>>> 
>>> I understand that there are tradeoffs for each approach, though the
>>> original intention was to allow for doc changes to be committed in the
>>> same PR as the code change. That doesn't have to be the case always,
>>> especially for larger multi-PR changes, but it makes it easier to do
>>> quick corrections to the docs.
>>> 
>>> I think the bigger problem here is to get rid of the generated site
>>> HTML stuff from the main pulsar repo.
>>> 
>>> --
>>> Matteo Merli
>>> 
>>> 
>>> On Wed, Nov 17, 2021 at 12:16 PM Enrico Olivelli  
>>> wrote:
 
 Dave,
 Having a new repo will make it harder for developers to contribute
 documentation.
 
 Usually engineers do  it like and do not have time to write docs.
 
 If we ask them to create two PRs only to add, for instance, a new
 configuration option, then it would be somehow a pain.
 
 I am not saying that we shouldn't go this way, but it would be kind of 
 a
 pain for someone and we need to ear more voices.
 
 Enrico
 
 Il Mer 17 Nov 2021, 19:28 Sijie Guo  ha scritto:
 
> I think we should have a PIP for this. Because this impacts all the
> developers who are making documentation changes.
> 
> - Sijie
> 
> On Tue, Nov 16, 2021 at 8:46 AM Dave Fisher  wrote:
> 
>> Hi -
>> 
>> There are two efforts happening in the community around website 
>> refresh.
>> 
>> (1) Docusaurus upgrades.
>> (2) New web design.
>> 
>> There is an effort to eliminate all the extra commits in the asf-site
>> branch of the main repository. In that thread I proposed a new 
>> repository
>> for the website.
>> 
>> We can then discuss migration and development both on this mailing 
>> list
>> and as PRs and Issues in that repository.
>>>

[GitHub] [pulsar-dotpulsar] blankensteiner commented on issue #84: Recover after network disconnect

[GitBox]

blankensteiner commented on issue #84:
URL: https://github.com/apache/pulsar-dotpulsar/issues/84#issuecomment-973318137


   @dunkymole, yes, DotPulsar will send pings to the broker with the new 
version (2.0.1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[OUTREACH] November '21 Edition of Happenings is out now

[Aaron Williams]

Hello Apache Pulsar Neighbors,

For this issue
,
we have some postponements of events, a new website design winner, more
events, and some great Bookkeeper blogs. Plus our normal features of a
Stack Overflow question and some community stats.

If you have anything that you think your neighbors would find interesting,
we have created #blogs-articles and #event-decks channels on the Apache
Pulsar slack Workspace to capture them.

Thank you,
Aaron Williams
Resident of the Apache Pulsar Neighborhood

Re: [VOTE] Apache Pulsar 2.9.0 candidate 4

[Yuri Mizushima]

+1 (non-binding)

* Verified checksum/signature
* Ran release candidate validation on JDK8+macOS

Thank you for being release manager.

nits:
Do we apply the https://github.com/apache/pulsar/pull/12659 commit to v2.9.0?
In my environment, StringIndexOutOfBoundsException occurs when I create a 
tenant.
I think it is not critical at least in my case because of 
https://github.com/apache/pulsar/blob/v2.9.0-candidate-4/pulsar-broker/src/main/java/org/apache/pulsar/broker/resourcegroup/ResourceGroupNamespaceConfigListener.java#L139-L141
 .
What do you think?

```
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1931) ~[?:1.8.0_312]
at 
org.apache.pulsar.broker.resources.NamespaceResources.pathIsFromNamespace(NamespaceResources.java:121)
 ~[org.apache.pulsar-pulsar-broker-common-2.9.0.jar:2.9.0]
at 
org.apache.pulsar.broker.resourcegroup.ResourceGroupNamespaceConfigListener.accept(ResourceGroupNamespaceConfigListener.java:139)
 ~[org.apache.pulsar-pulsar-broker-2.9.0.jar:2.9.0]
at 
org.apache.pulsar.broker.resourcegroup.ResourceGroupNamespaceConfigListener.accept(ResourceGroupNamespaceConfigListener.java:42)
 ~[org.apache.pulsar-pulsar-broker-2.9.0.jar:2.9.0]
at 
org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$0(AbstractMetadataStore.java:167)
 ~[org.apache.pulsar-pulsar-metadata-2.9.0.jar:2.9.0]
at 
java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:895)
 ~[?:1.8.0_312]
at 
org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$1(AbstractMetadataStore.java:165)
 ~[org.apache.pulsar-pulsar-metadata-2.9.0.jar:2.9.0]
at 
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
 [?:1.8.0_312]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_312]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_312]
at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 [io.netty-netty-common-4.1.68.Final.jar:4.1.68.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
```

Regards,
-- 
Yuri Mizushima
yumiz...@yahoo-corp.jp
 

"Enrico Olivelli"  wrote:

I have pushed in my personal dockerhub the docker images for this RC:

eolivelli/pulsar-all:2.9.0rc4
eolivelli/pulsar:2.9.0rc4

Enrico

Il giorno mer 17 nov 2021 alle ore 14:04 Enrico Olivelli <
eolive...@gmail.com> ha scritto:

> This is the fourth release candidate for Apache Pulsar, version 2.9.0.
>
> It fixes the following issues:
> https://github.com/apache/pulsar/milestone/30?closed=1
>
> *** Please download, test and vote on this release. This vote will stay 
open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.0-candidate-4/
>
> SHA-512 checksums:
>
> 
659afa98e2475fc95f3126aab47d365e3e43a9d419c9cbe20c99520029f0b28aec2f471da5ccf47b92017a4c7943fc113a96232b075eb675e2b9df3601b2a7ad
  apache-pulsar-2.9.0-SNAPSHOT-bin.tar.gz
> 
ed93aa1b57411c5d153bcd1433db6679ddb9b4d23896b39c1a1ad8edc70aaa7e95842a5770d6d55b466ab90500e17f50549673601556e767f7c12135ae360e85
  apache-pulsar-2.9.0-SNAPSHOT-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1104/
>
> The tag to be voted upon:
> v2.9.0-candidate-4 (bdd57b21a66b81aab72c4ec39d516ffd2a769c35)
> https://github.com/apache/pulsar/releases/tag/v2.9.0-candidate-4
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
>
> Enrico Olivelli
>
>