date:20220105

Re: [DISCUSSION] PIP-132: Include message header size when check maxMessageSize of non-batch message on the client side.

2022-01-05 Thread Zike Yang

> as we have the same header size for each chunk message.

The message metadata of different chunks will not be the same. There
are serval fields with different values such as chunk id, publish
time. Because some type of that fields is varint, I think the size of
the header is not always the same.


On Tue, Jan 4, 2022 at 11:51 AM Haiting Jiang  wrote:
>
> Good point,  I think we should shrink chunk size to 
> "ClientCnx.getMaxMessageSize() - chunkMessageHeaderSize", as we have the same 
> header size for each chunk message.
>
> Thanks,
> Haiting Jiang
>
> On 2022/01/04 03:27:00 Zike Yang wrote:
> > But how do we handle chunked messages? The chunked message is split
> > based on the maxMessageSize(max payload size). This would seem to make
> > `op.getMessageHeaderAndPayloadSize() > ClientCnx.getMaxMessageSize()`
> > always true.
> >
> > Thanks,
> > Zike
> >
> > On Fri, Dec 31, 2021 at 8:11 PM Haiting Jiang  
> > wrote:
> > >
> > > Sorry, it should be PIP-132.
> > >
> > > Thanks,
> > > Haiting Jiang
> > >
> > > On 2021/12/31 12:05:54 Haiting Jiang wrote:
> > > > https://github.com/apache/pulsar/issues/13591
> > > >
> > > > Pasted below for quoting convenience.
> > > >
> > > > ——
> > > >
> > > > ## Motivation
> > > >
> > > > Currently, Pulsar client (Java) only checks payload size for max 
> > > > message size validation.
> > > >
> > > > Client throws TimeoutException if we produce a message with too many 
> > > > properties, see [1].
> > > > But the root cause is that is trigged TooLongFrameException in broker 
> > > > server.
> > > >
> > > > In this PIP, I propose to include message header size when check 
> > > > maxMessageSize of non-batch
> > > > messages, this brings the following benefits.
> > > > 1. Clients can throw InvalidMessageException immediately if properties 
> > > > takes too much storage space.
> > > > 2. This will make the behaviour consistent with topic level max message 
> > > > size check in broker.
> > > > 3. Strictly limit the entry size less than maxMessageSize, avoid 
> > > > sending message to bookkeeper failed.
> > > >
> > > > ## Goal
> > > >
> > > > Include message header size when check maxMessageSize for non-batch 
> > > > message on the client side.
> > > >
> > > > ## Implementation
> > > >
> > > > ```
> > > > // Add a size check in 
> > > > org.apache.pulsar.client.impl.ProducerImpl#processOpSendMsg
> > > > if (op.msg != null // for non-batch messages only
> > > > && op.getMessageHeaderAndPayloadSize() > 
> > > > ClientCnx.getMaxMessageSize()) {
> > > > // finish send op with InvalidMessageException
> > > > releaseSemaphoreForSendOp(op);
> > > > op.sendComplete(new PulsarClientException(new 
> > > > InvalidMessageException, op.sequenceId));
> > > > }
> > > >
> > > >
> > > > // 
> > > > org.apache.pulsar.client.impl.ProducerImpl.OpSendMsg#getMessageHeaderAndPayloadSize
> > > >
> > > > public int getMessageHeaderAndPayloadSize() {
> > > > ByteBuf cmdHeader = cmd.getFirst();
> > > > cmdHeader.markReaderIndex();
> > > > int totalSize = cmdHeader.readInt();
> > > > int cmdSize = cmdHeader.readInt();
> > > > int msgHeadersAndPayloadSize = totalSize - cmdSize - 4;
> > > > cmdHeader.resetReaderIndex();
> > > > return msgHeadersAndPayloadSize;
> > > > }
> > > > ```
> > > >
> > > > ## Reject Alternatives
> > > > Add a new property like "maxPropertiesSize" or "maxHeaderSize" in 
> > > > broker.conf and pass it to
> > > > client like maxMessageSize. But the implementation is much more 
> > > > complex, and don't have the
> > > > benefit 2 and 3 mentioned in Motivation.
> > > >
> > > > ## Compatibility Issue
> > > > As a matter of fact, this PIP narrows down the sendable range. 
> > > > Previously, when maxMessageSize
> > > > is 1KB, it's ok to send message with 1KB properties and 1KB payload. 
> > > > But with this PIP, the
> > > > sending will fail with InvalidMessageException.
> > > >
> > > > One conservative way is to add a boolean config 
> > > > "includeHeaderInSizeCheck" to enable this
> > > > feature. But I think it's OK to enable this directly as it's more 
> > > > reasonable, and I don't see good
> > > > migration plan if we add a config for this.
> > > >
> > > > The compatibility issue is worth discussing. And any suggestions are 
> > > > appreciated.
> > > >
> > > > [1] https://github.com/apache/pulsar/issues/13560
> > > >
> > > > Thanks,
> > > > Haiting Jiang
> > > >
> >
> >
> >
> > --
> > Zike Yang
> >



-- 
Zike Yang

Re: [VOTE] PIP-130: Apply redelivery backoff policy for ack timeout

2022-01-05 Thread Ran Gao

+1

Best,
Ran Gao

On 2022/01/04 03:34:33 PengHui Li wrote:
> This is the voting thread for PIP-130. It will stay open for at least 48
> hours.
> 
> https://github.com/apache/pulsar/issues/13528
> 
> Pasted below for quoting convenience.
> 
> -
> 
> PIP 130: Apply redelivery backoff policy for ack timeout
> 
> ## Motivation
> 
> PIP 106
> https://github.com/apache/pulsar/wiki/PIP-106%3A-Negative-acknowledgment-backoff
> introduced negative acknowledgment message redelivery backoff which allows
> users to achieve
> more flexible message redelivery delay time control. But the redelivery
> backoff policy only
> apply to the negative acknowledgment API, for users who use ack timeout to
> trigger the message
> redelivery, not the negative acknowledgment API, they can't use the new
> features introduced by
> PIP 106.
> 
> So the proposal is to apply the message redelivery policy for the ack
> timeout mechanism.
> Users can specify an ack timeout redelivery backoff, for example, apply an
> exponential backoff
> with 10 seconds ack timeout:
> 
> ```java
> client.newConsumer()
> .ackTimeout(10, TimeUnit.SECOND)
> .ackTimeoutRedeliveryBackoff(
> ExponentialRedeliveryBackoff.builder()
> .minDelayMs(1000)
> .maxDelayMs(6).build());
> .subscribe();
> ```
> 
> The message redelivery behavior should be:
> 
> |  Redelivery count   | Redelivery delay  |
> |    |   |
> | 1 | 10 + 1 seconds |
> | 2 | 10 + 2 seconds |
> | 3 | 10 + 4 seconds |
> | 4 | 10 + 8 seconds |
> | 5 | 10 + 16 seconds |
> | 6 | 10 + 32 seconds |
> | 7 | 10 + 60 seconds |
> | 8 | 10 + 60 seconds |
> 
> ## Goal
> 
> Add an API to the Java Client to provide the ability to specify the ack
> timeout message redelivery
> backoff and the message redelivery behavior should abide by the redelivery
> backoff policy.
> 
> 
> ## API Changes
> 
> 1. Change `NegativeAckRedeliveryBackoff` to `RedeliveryBackoff`, so that we
> can use the
> MessageRedeliveryBackoff for both negative acknowledgment API and ack
> timeout message redelivery.
> 
> 2. Change `NegativeAckRedeliveryExponentialBackoff` to
> `ExponentialRedeliveryBackoff`, and add `multiplier`
> for `RedeliveryExponentialBackoff` with default value 2.
> 
> ExponentialRedeliveryBackoff.builder()
> .minDelayMs(1000)
> .maxDelayMs(6)
> .multiplier(5)
> .build()
> 
> 3. Add `ackTimeoutRedeliveryBackoff` method for the `ConsumerBuilder`:
> 
> ```java
> client.newConsumer()
> .ackTimeout(10, TimeUnit.SECOND)
> .ackTimeoutRedeliveryBackoff(
> ExponentialRedeliveryBackoff.builder()
> .minDelayMs(1000)
> .maxDelayMs(6).build());
> .subscribe();
> ```
> 
> ## Compatibility and migration plan
> 
> Since the negative acknowledgment message, redelivery backoff has not been
> released yet,
> so we can modify the API directly.
> 
> ## Tests plan
> 
> - Verify the introduced `multiplier` of ExponentialRedeliveryBackoff
> - Verify the ack timeout message redelivery work as expected
> 
> Regards,
> Penghui
>

[DISCUSS] Apache Pulsar 2.9.2 release

2022-01-05 Thread Ran Gao

Hello, Pulsar community:

I'd like to propose that we release Apache Pulsar 2.9.2.

Currently, compared to 2.9.1, branch-2.9 already merged 171 commits(refer
to [0]), they contain the log4j security patch and many important fixes.

I am happy to volunteer to be the release manager.

I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
labeled release/2.9.2, I'll follow them to make sure important fix could be
merged in 2.9.2.


Best,
Ran Gao


[0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9

Re: [DISCUSS] Apache Pulsar 2.9.2 release

2022-01-05 Thread PengHui Li

+1

Thanks,
Penghui

On Wed, Jan 5, 2022 at 5:23 PM Ran Gao  wrote:

> Hello, Pulsar community:
>
> I'd like to propose that we release Apache Pulsar 2.9.2.
>
> Currently, compared to 2.9.1, branch-2.9 already merged 171 commits(refer
> to [0]), they contain the log4j security patch and many important fixes.
>
> I am happy to volunteer to be the release manager.
>
> I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
> branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
> labeled release/2.9.2, I'll follow them to make sure important fix could be
> merged in 2.9.2.
>
>
> Best,
> Ran Gao
>
>
> [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
>

[GitHub] [pulsar-helm-chart] lhotari commented on pull request #138: automate initialize

2022-01-05 Thread GitBox



lhotari commented on pull request #138:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/138#issuecomment-1005614211


   @valeriano-manassero Thank you for the contribution. Would it be also 
possible to update README.md in this PR and remove the instructions of using 
`--set initialize=true` when doing the installation?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari opened a new pull request #196: [CI] Remove --ignore-errors=true parameter which requires newer kubectl

2022-01-05 Thread GitBox



lhotari opened a new pull request #196:
URL: https://github.com/apache/pulsar-helm-chart/pull/196


   - #192 changes with upgraded kubectl version would be required unless
 the parameter is removed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari commented on pull request #196: [CI] Remove --ignore-errors=true parameter which requires newer kubectl

2022-01-05 Thread GitBox



lhotari commented on pull request #196:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/196#issuecomment-1005645719


   It turns out that the current version of kubectl doesn't support 
`--prefix=true` which adds the pod name and container name as a prefix to each 
log line. It's better to get #192 merged to fix the kubectl log dumping 
solution which was added by #195


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari closed pull request #196: [CI] Remove --ignore-errors=true parameter which requires newer kubectl

2022-01-05 Thread GitBox



lhotari closed pull request #196:
URL: https://github.com/apache/pulsar-helm-chart/pull/196


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Lifting Kubernetes minimum version requirement for Apache Pulsar Helm Charts from k8s 1.14 to 1.18

2022-01-05 Thread Lari Hotari

Reminder, there's a PR to lift the minimum requirement for Apache Pulsar
Helm Charts to 1.18 so that we don't fall too far behind.
https://github.com/apache/pulsar-helm-chart/pull/192
Please review

BR, Lari

On Tue, Jan 4, 2022 at 11:40 AM Lari Hotari  wrote:

> Hi all,
>
> Currently k8s 1.14 version is used in CI to verify the Helm chart changes.
>
> k8s 1.14 became end-of-life 2019-12-11 , over 2 years ago [1].
> The oldest maintained version for Kubernetes is 1.20 and it will become
> end-of-life on 2022-02-28, in less than 2 months from now [2].
>
> There's a PR to lift the minimum requirement for Apache Pulsar Helm Charts
> to 1.18 so that we don't fall too far behind.
> https://github.com/apache/pulsar-helm-chart/pull/192
>
> Please review. If the PR gets approved and merged, the Kubernetes minimum
> version requirement will be lifted to Kubernetes 1.18.
>
>
> Best regards,
>
> Lari
>
>
> [1]
> https://kubernetes.io/releases/patch-releases/#non-active-branch-history
> [2] https://kubernetes.io/releases/patch-releases/#1-20
>

Re: [DISCUSS] Apache Pulsar 2.9.2 release

2022-01-05 Thread Lari Hotari

+1

On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:

> Hello, Pulsar community:
>
> I'd like to propose that we release Apache Pulsar 2.9.2.
>
> Currently, compared to 2.9.1, branch-2.9 already merged 171 commits(refer
> to [0]), they contain the log4j security patch and many important fixes.
>
> I am happy to volunteer to be the release manager.
>
> I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
> branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
> labeled release/2.9.2, I'll follow them to make sure important fix could be
> merged in 2.9.2.
>
>
> Best,
> Ran Gao
>
>
> [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
>

Re: upcoming change: Apache Pulsar Helm Chart switching from Pulsar 2.7.4 version to 2.8.2; known issue with ZK when TLS is enabled

2022-01-05 Thread Lari Hotari

Reminder: we need to decide about the developement of Apache Pulsar Helm
chart.
Please reply to the email or review
https://github.com/apache/pulsar-helm-chart/pull/190 .
That PR is blocked since a decision must be made whether it's fine to make
the change, although there's a known issue in Zookeeper when TLS is
enabled.
The issue is ZOOKEEPER-3988 /  https://github.com/apache/pulsar/issues/11070 .
The bug currently only impacts TLS since the change
https://github.com/apache/pulsar-helm-chart/pull/180 switched
to use NIOServerCnxnFactory for Zookeeper. NIOServerCnxnFactory doesn't
support TLS and the impacted NettyServerCnxnFactory must be used for TLS.

How do we handle the decision? Can we proceed in merging
https://github.com/apache/pulsar-helm-chart/pull/190 regardless of the
known issue?

BR,
Lari

On Mon, Jan 3, 2022 at 4:39 PM Lari Hotari  wrote:

> Hi all,
>
> There's an upcoming change in the Apache Pulsar Helm chart to finally
> switch to Pulsar 2.8.x, more specifically to Apache Pulsar version 2.8.2 .
> The latest Apache Pulsar Helm Chart release uses the Apache Pulsar 2.7.4
> image.
>
> The pull request to switch to Apache Pulsar image version 2.8.2 is current
> in review:
> https://github.com/apache/pulsar-helm-chart/pull/190
>
> There's a known issue that Zookeeper TLS isn't stable because of
> https://issues.apache.org/jira/browse/ZOOKEEPER-3988 (also reported in
> apache/pulsar as https://github.com/apache/pulsar/issues/11070) .
> The fix https://github.com/apache/zookeeper/pull/1770 is planned for
> Zookeeper 3.7.1 version.
> There's a workaround in the Apache Pulsar Helm chart when TLS isn't
> enabled for Zookeeper. That was added by
> https://github.com/apache/pulsar-helm-chart/pull/180 .
> However, the workaround cannot be applied when TLS is enabled for
> Zookeeper.
>
> Should we postpone switching to Apache Pulsar 2.8.2 in the Helm chart
> until there's a fix for ZOOKEEPER-3988 /
> https://github.com/apache/pulsar/issues/11070 ?
>
> BR,
> Lari
>

[GitHub] [pulsar-helm-chart] haorenfsa commented on pull request #176: Fix cluster initialize blocked when fail: #175

2022-01-05 Thread GitBox



haorenfsa commented on pull request #176:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/176#issuecomment-1005682255


   updated


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Apache Pulsar 2.9.2 release

2022-01-05 Thread 陳智弘

+1

Lari Hotari  於 2022年1月5日 週三 20:56 寫道：

> +1
>
> On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:
>
> > Hello, Pulsar community:
> >
> > I'd like to propose that we release Apache Pulsar 2.9.2.
> >
> > Currently, compared to 2.9.1, branch-2.9 already merged 171 commits(refer
> > to [0]), they contain the log4j security patch and many important fixes.
> >
> > I am happy to volunteer to be the release manager.
> >
> > I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
> > branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
> > labeled release/2.9.2, I'll follow them to make sure important fix could
> be
> > merged in 2.9.2.
> >
> >
> > Best,
> > Ran Gao
> >
> >
> > [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
> >
>

[GitHub] [pulsar-helm-chart] valeriano-manassero commented on pull request #138: automate initialize

2022-01-05 Thread GitBox



valeriano-manassero commented on pull request #138:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/138#issuecomment-1005708870


   > @valeriano-manassero Thank you for the contribution. Would it be also 
possible to update README.md in this PR and remove the instructions of using 
`--set initialize=true` when doing the installation?
   
   done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: upcoming change: Apache Pulsar Helm Chart switching from Pulsar 2.7.4 version to 2.8.2; known issue with ZK when TLS is enabled

2022-01-05 Thread 陳智弘

Hi everyone,

  From my side, I think currently merging the request without regarding the
known issue and announcing this information on the website is a good
option.


不含病毒。www.avg.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Lari Hotari  於 2022年1月5日 週三 下午9:04寫道：

> Reminder: we need to decide about the developement of Apache Pulsar Helm
> chart.
> Please reply to the email or review
> https://github.com/apache/pulsar-helm-chart/pull/190 .
> That PR is blocked since a decision must be made whether it's fine to make
> the change, although there's a known issue in Zookeeper when TLS is
> enabled.
> The issue is ZOOKEEPER-3988 /
> https://github.com/apache/pulsar/issues/11070 .
> The bug currently only impacts TLS since the change
> https://github.com/apache/pulsar-helm-chart/pull/180 switched
> to use NIOServerCnxnFactory for Zookeeper. NIOServerCnxnFactory doesn't
> support TLS and the impacted NettyServerCnxnFactory must be used for TLS.
>
> How do we handle the decision? Can we proceed in merging
> https://github.com/apache/pulsar-helm-chart/pull/190 regardless of the
> known issue?
>
> BR,
> Lari
>
> On Mon, Jan 3, 2022 at 4:39 PM Lari Hotari  wrote:
>
> > Hi all,
> >
> > There's an upcoming change in the Apache Pulsar Helm chart to finally
> > switch to Pulsar 2.8.x, more specifically to Apache Pulsar version 2.8.2
> .
> > The latest Apache Pulsar Helm Chart release uses the Apache Pulsar 2.7.4
> > image.
> >
> > The pull request to switch to Apache Pulsar image version 2.8.2 is
> current
> > in review:
> > https://github.com/apache/pulsar-helm-chart/pull/190
> >
> > There's a known issue that Zookeeper TLS isn't stable because of
> > https://issues.apache.org/jira/browse/ZOOKEEPER-3988 (also reported in
> > apache/pulsar as https://github.com/apache/pulsar/issues/11070) .
> > The fix https://github.com/apache/zookeeper/pull/1770 is planned for
> > Zookeeper 3.7.1 version.
> > There's a workaround in the Apache Pulsar Helm chart when TLS isn't
> > enabled for Zookeeper. That was added by
> > https://github.com/apache/pulsar-helm-chart/pull/180 .
> > However, the workaround cannot be applied when TLS is enabled for
> > Zookeeper.
> >
> > Should we postpone switching to Apache Pulsar 2.8.2 in the Helm chart
> > until there's a fix for ZOOKEEPER-3988 /
> > https://github.com/apache/pulsar/issues/11070 ?
> >
> > BR,
> > Lari
> >
>

[GitHub] [pulsar-helm-chart] lhotari closed issue #114: Use hooks instead of initialize=true

2022-01-05 Thread GitBox



lhotari closed issue #114:
URL: https://github.com/apache/pulsar-helm-chart/issues/114


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari merged pull request #138: automate initialize

2022-01-05 Thread GitBox



lhotari merged pull request #138:
URL: https://github.com/apache/pulsar-helm-chart/pull/138


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari merged pull request #176: Fix cluster initialize blocked when fail: #175

2022-01-05 Thread GitBox



lhotari merged pull request #176:
URL: https://github.com/apache/pulsar-helm-chart/pull/176


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari closed issue #175: Create may block if the initialize cluster job fails

2022-01-05 Thread GitBox



lhotari closed issue #175:
URL: https://github.com/apache/pulsar-helm-chart/issues/175


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[ANNOUNCE] Apache Pulsar 2.8.2 released

2022-01-05 Thread linlin

The Apache Pulsar team is proud to announce Apache Pulsar version 2.8.2.

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, automatic cursor management
for
subscribers, and cross-datacenter replication.

For Pulsar release details and downloads, visit:

https://pulsar.apache.org/download

Release Notes are at:
http://pulsar.apache.org/release-notes

We would like to thank the contributors that made the release possible.

Regards,

The Pulsar Team

[GitHub] [pulsar-helm-chart] lhotari merged pull request #181: Add ability to run extra commands in the initialization jobs e.g. to quit istio sidecars

2022-01-05 Thread GitBox



lhotari merged pull request #181:
URL: https://github.com/apache/pulsar-helm-chart/pull/181


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] frankjkelly commented on pull request #176: Fix cluster initialize blocked when fail: #175

2022-01-05 Thread GitBox



frankjkelly commented on pull request #176:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/176#issuecomment-1005731034


   @lhotari  without a change in Chart version will a new release be created? 
CC: @sijie 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari commented on pull request #176: Fix cluster initialize blocked when fail: #175

2022-01-05 Thread GitBox



lhotari commented on pull request #176:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/176#issuecomment-1005734036


   > @lhotari without a change in Chart version will a new release be created? 
CC: @sijie
   
   correct. Here's the logs: 
https://github.com/apache/pulsar-helm-chart/runs/4715446366?check_suite_focus=true
 .
   I think it's useful to have multiple PRs batched into a release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[ANNOUNCE] New Committer: Zhangjian He

2022-01-05 Thread Hang Chen

The Apache Pulsar Project Management Committee (PMC) has invited Zhangjian He
https://github.com/Shoothzj to become a committer and we are pleased to
announce that he has accepted.

Zhangjian He has joined the community for more than 1 year now and he is
active in the Pulsar community for more than 6 months

Welcome and Congratulations, Zhangjian!

Please join us in congratulating and welcoming Zhangjian onboard!

Best Regards,
Hang Chen on behalf of the Pulsar PMC

Re: [DISCUSS] Apache Pulsar 2.9.2 release

2022-01-05 Thread Hang Chen

+1

Best,
Hang

陳智弘  于2022年1月5日周三 21:30写道：
>
> +1
>
> Lari Hotari  於 2022年1月5日 週三 20:56 寫道：
>
> > +1
> >
> > On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:
> >
> > > Hello, Pulsar community:
> > >
> > > I'd like to propose that we release Apache Pulsar 2.9.2.
> > >
> > > Currently, compared to 2.9.1, branch-2.9 already merged 171 commits(refer
> > > to [0]), they contain the log4j security patch and many important fixes.
> > >
> > > I am happy to volunteer to be the release manager.
> > >
> > > I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
> > > branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
> > > labeled release/2.9.2, I'll follow them to make sure important fix could
> > be
> > > merged in 2.9.2.
> > >
> > >
> > > Best,
> > > Ran Gao
> > >
> > >
> > > [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
> > >
> >

Re: [DISCUSS] Apache Pulsar 2.9.2 release

2022-01-05 Thread Lan Liang

+1,Thanks for your work.






Best Regards,
Lan Liang
On 1/5/2022 23:01，Hang Chen wrote：
+1

Best,
Hang

陳智弘  于2022年1月5日周三 21:30写道：

+1

Lari Hotari  於 2022年1月5日 週三 20:56 寫道：

+1

On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:

Hello, Pulsar community:

I'd like to propose that we release Apache Pulsar 2.9.2.

Currently, compared to 2.9.1, branch-2.9 already merged 171 commits(refer
to [0]), they contain the log4j security patch and many important fixes.

I am happy to volunteer to be the release manager.

I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
labeled release/2.9.2, I'll follow them to make sure important fix could
be
merged in 2.9.2.


Best,
Ran Gao


[0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9

[GitHub] [pulsar-helm-chart] frankjkelly commented on pull request #176: Fix cluster initialize blocked when fail: #175

2022-01-05 Thread GitBox



frankjkelly commented on pull request #176:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/176#issuecomment-1005771570


   @lhotari Sounds good, thanks for the clarification.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] PIP-117: Change Pulsar standalone defaults

2022-01-05 Thread Lan Liang

+1 






Best Regards,
Lan Liang
On 12/23/2021 19:21，Haiting Jiang wrote：
+1

Thanks,
Haiting

On 2021/12/23 05:35:03 Michael Marshall wrote:
+1

- Michael

On Wed, Dec 22, 2021 at 6:18 PM Sijie Guo  wrote:

+1

On Tue, Dec 21, 2021 at 3:49 PM Matteo Merli  wrote:

This is the voting thread for PIP-117. It will stay open for at least 48h.

https://github.com/apache/pulsar/issues/13302



## Motivation

Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster,
where
all the components are started within the context of a single JVM process.

Users are using the standalone as a way to get quickly started with Pulsar
or
in all the cases where it makes sense to have a single node deployment.

Right now, the standalone is starting by default with many components,
several of
which are quite complex, since they are designed to be deployed in a
distributed
fashion.

## Goal

Simplify the components of Pulsar standalone to achieve:

1. Reduce complexity
2. Reduce startup time
3. Reduce memory and CPU footprint of running standalone

## Proposed changes

The proposal here is to change some of the default implementations that are
used for the Pulsar standalone.

1. **Metadata Store implementation** -->
Change from ZooKeeper to RocksDB

2. **Pulsar functions package backend** -->
Change from using DistributedLog to using local filesystem, storing
the
jars directly in the data folder instead of uploading them into BK.

3. **Pulsar functions state store implementation** -->
Change the state store to be backed by a MetadataStore based backed,
with the RocksDB implementation.

4. **Table Service** -->
Do not start BK table service by default

## Compatibility considerations

In order to avoid compatibility issues where users have existing Pulsar
standalone services that they want to upgrade without conflicts, we will
follow the principle of keeping the old defaults where there is existing
data on the disk.

We will add a file, serving the purpose as a flag, in the `data/standalone`
directory, for example `new-2.10-defaults`.

If the file is present, or if the data directory is completely missing, we
will adopt the new set of default configuration settings.

If the file is not there, we will continue to use existing defaults and we
will
not break the upgrade operation.


--
Matteo Merli

[DISCUSSION] PIP-132: Support Pulsar system event.

2022-01-05 Thread mattison chao

Original PIP : https://github.com/apache/pulsar/issues/13628

Pasted bellow for quoting convenience.

—— 

## Motivation

In some case or for better observability. We can support Pulsar event 
notification mechanism, that can let user subscribe this system topic to 
receive some **"we need notice to user"** event.

## Goal

Enable users to receive Pulsar system event.

## API Changes

1. Add config ``pulsarSystemEventEnabled``

```java
pulsarSystemEventEnabled=false
```
2. Add ``SystemEventMessageBuilder``
```java


public interface SystemEventMessageBuilder {

  SystemEventMessageBuilder eventType(SystemEvent event);

  SystemEventMessageBuilder properties(Map properties);

  SystemEventMessageBuilder addProperty(String key, Object value);

  SystemEventMessage build();

}

```
3. Add ``SystemEventService``

```java

public interface SystemEventService {

  void init();

  CompletableFuture trigger(SystemEventMessage message);

  void close();
}

```

## Implementation

This PIP just is a simple version, it's purpose is to discuss what event we 
need to trigger? how to trigger? and what is the event message structure.

I think we need to trigger some event as bellow:

- [ ] Client connected
- [ ] Client disconnected
- [ ] Consumer subscribe
- [ ] Consumer unsubscribe
- [ ] Producer publish
- [ ] Message delivered
- [ ] Message acked

These are far from enough, and we need to continue to make supplements here.
Please fell free to give me more advice for this PIP.

## Reject Alternatives

none.

Thanks,
Mattisonchao

Re: [DISCUSS] Apache Pulsar 2.9.2 release

2022-01-05 Thread mattison chao

+1

regards,
Mattisonchao

On Wed, 5 Jan 2022 at 23:04, Lan Liang  wrote:

> +1,Thanks for your work.
>
>
>
>
>
>
> Best Regards,
> Lan Liang
> On 1/5/2022 23:01，Hang Chen wrote：
> +1
>
> Best,
> Hang
>
> 陳智弘  于2022年1月5日周三 21:30写道：
>
> +1
>
> Lari Hotari  於 2022年1月5日 週三 20:56 寫道：
>
> +1
>
> On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:
>
> Hello, Pulsar community:
>
> I'd like to propose that we release Apache Pulsar 2.9.2.
>
> Currently, compared to 2.9.1, branch-2.9 already merged 171 commits(refer
> to [0]), they contain the log4j security patch and many important fixes.
>
> I am happy to volunteer to be the release manager.
>
> I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
> branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
> labeled release/2.9.2, I'll follow them to make sure important fix could
> be
> merged in 2.9.2.
>
>
> Best,
> Ran Gao
>
>
> [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
>
>
>

[GitHub] [pulsar-helm-chart] michaeljmarshall commented on pull request #176: Fix cluster initialize blocked when fail: #175

2022-01-05 Thread GitBox



michaeljmarshall commented on pull request #176:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/176#issuecomment-1005874420


   >  I think it's useful to have multiple PRs batched into a release.
   
   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] michaeljmarshall commented on pull request #150: added additionalCommand parameter

2022-01-05 Thread GitBox



michaeljmarshall commented on pull request #150:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/150#issuecomment-1005876996


   This PR adds value, even though the original goal of this feature might 
break when we move to a non-root docker image, so I am going to merge it now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] michaeljmarshall merged pull request #150: added additionalCommand parameter

2022-01-05 Thread GitBox



michaeljmarshall merged pull request #150:
URL: https://github.com/apache/pulsar-helm-chart/pull/150


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] michaeljmarshall closed issue #148: Add the ability to specify a custom command to the start up process of statefulset pods

2022-01-05 Thread GitBox



michaeljmarshall closed issue #148:
URL: https://github.com/apache/pulsar-helm-chart/issues/148


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Apache Pulsar 2.9.2 release

2022-01-05 Thread Enrico Olivelli

171 commits?

Why?

This is too much in my opinion for a point release.

I am pretty sure that we don't need all that changes.

Every change we cherry pick has a good chance to break the stability

Enrico

Il Mer 5 Gen 2022, 16:30 mattison chao  ha scritto:

> +1
>
> regards,
> Mattisonchao
>
> On Wed, 5 Jan 2022 at 23:04, Lan Liang  wrote:
>
> > +1,Thanks for your work.
> >
> >
> >
> >
> >
> >
> > Best Regards,
> > Lan Liang
> > On 1/5/2022 23:01，Hang Chen wrote：
> > +1
> >
> > Best,
> > Hang
> >
> > 陳智弘  于2022年1月5日周三 21:30写道：
> >
> > +1
> >
> > Lari Hotari  於 2022年1月5日 週三 20:56 寫道：
> >
> > +1
> >
> > On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:
> >
> > Hello, Pulsar community:
> >
> > I'd like to propose that we release Apache Pulsar 2.9.2.
> >
> > Currently, compared to 2.9.1, branch-2.9 already merged 171 commits(refer
> > to [0]), they contain the log4j security patch and many important fixes.
> >
> > I am happy to volunteer to be the release manager.
> >
> > I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
> > branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
> > labeled release/2.9.2, I'll follow them to make sure important fix could
> > be
> > merged in 2.9.2.
> >
> >
> > Best,
> > Ran Gao
> >
> >
> > [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
> >
> >
> >
>

[GitHub] [pulsar-helm-chart] michaeljmarshall commented on issue #185: Github Actions should enforce an increase in helm chart semantic version on each PR

2022-01-05 Thread GitBox



michaeljmarshall commented on issue #185:
URL: 
https://github.com/apache/pulsar-helm-chart/issues/185#issuecomment-1005897271


   I think a release for every change would lead to too many releases. However, 
there is an underlying problem that this issue points to: contributors to the 
helm chart feel that it can take too long to get a simple fix included in a 
release. As you mentioned, we use to only release the helm chart when Apache 
Pulsar was released. I'm not sure why we had the initial process, but I don't 
think we _need_ to synchronize releases of Pulsar and its official helm chart.
   
   We could add something to the README or to a CONTRIBUTORS file that 
indicates how we version the chart and explicitly lets contributors know how to 
request a release. Note that anyone wanting to initiate a release can open a PR 
to bump the version of the helm chart.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] michaeljmarshall commented on issue #156: chage the pulsar-manager port

2022-01-05 Thread GitBox



michaeljmarshall commented on issue #156:
URL: 
https://github.com/apache/pulsar-helm-chart/issues/156#issuecomment-1005903379


   I provided an answer to this issue here: 
https://github.com/apache/pulsar/issues/11909#issuecomment-917219256. It is not 
possible to configure the node port to 8100, as node ports must be in the range 
from 3 to 32767.
   
   @chlyzzo - are you still looking to configure the node port? If so, you 
could do this by modifying the service templates. Would you like to contribute 
this feature?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] michaeljmarshall closed issue #184: Helm Chart installs Pulsar Manager V1.0

2022-01-05 Thread GitBox



michaeljmarshall closed issue #184:
URL: https://github.com/apache/pulsar-helm-chart/issues/184


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] michaeljmarshall commented on issue #184: Helm Chart installs Pulsar Manager V1.0

2022-01-05 Thread GitBox



michaeljmarshall commented on issue #184:
URL: 
https://github.com/apache/pulsar-helm-chart/issues/184#issuecomment-1005907480


   @SapnaGirdhani1 - you can configure the version of the pulsar manager by 
adding the following to your helm values file: 
   
   ```yaml
   pulsar_manager:
 tag: v0.2.0
   ```
   
   Here are the default values for the current chart:
   
   
https://github.com/apache/pulsar-helm-chart/blob/cee3b5c5e6dd9edd1e7c584a941f076e5edf5a53/charts/pulsar/values.yaml#L190-L194
   
   Please re-open your issue if my solution does not work. Thanks!
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] michaeljmarshall commented on issue #167: Pulsar minikube installation pods fail

2022-01-05 Thread GitBox



michaeljmarshall commented on issue #167:
URL: 
https://github.com/apache/pulsar-helm-chart/issues/167#issuecomment-1005909843


   @pancudaniel7 - can you provide a bit more context? When you describe the 
pods that are in a "freeze state", do they have any kubernetes events? Does it 
work eventually? Note that the docker images are large, so it can take a while 
to download them to your minikube docker daemon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Apache Pulsar 2.9.2 release

2022-01-05 Thread Enrico Olivelli

BTW +1

I hope we will review carefully what we are including in the release

And that we will let branch-2.9 stabilise by not adding any other commits
that are not strictly required:
- security related issues
- data loss/data corruption cases


Enrico


Thank you Ran Gao for starting this thread.



Enrico

Il Mer 5 Gen 2022, 17:29 Enrico Olivelli  ha scritto:

> 171 commits?
>
> Why?
>
> This is too much in my opinion for a point release.
>
> I am pretty sure that we don't need all that changes.
>
> Every change we cherry pick has a good chance to break the stability
>
> Enrico
>
> Il Mer 5 Gen 2022, 16:30 mattison chao  ha
> scritto:
>
>> +1
>>
>> regards,
>> Mattisonchao
>>
>> On Wed, 5 Jan 2022 at 23:04, Lan Liang  wrote:
>>
>> > +1,Thanks for your work.
>> >
>> >
>> >
>> >
>> >
>> >
>> > Best Regards,
>> > Lan Liang
>> > On 1/5/2022 23:01，Hang Chen wrote：
>> > +1
>> >
>> > Best,
>> > Hang
>> >
>> > 陳智弘  于2022年1月5日周三 21:30写道：
>> >
>> > +1
>> >
>> > Lari Hotari  於 2022年1月5日 週三 20:56 寫道：
>> >
>> > +1
>> >
>> > On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:
>> >
>> > Hello, Pulsar community:
>> >
>> > I'd like to propose that we release Apache Pulsar 2.9.2.
>> >
>> > Currently, compared to 2.9.1, branch-2.9 already merged 171
>> commits(refer
>> > to [0]), they contain the log4j security patch and many important fixes.
>> >
>> > I am happy to volunteer to be the release manager.
>> >
>> > I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
>> > branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
>> > labeled release/2.9.2, I'll follow them to make sure important fix could
>> > be
>> > merged in 2.9.2.
>> >
>> >
>> > Best,
>> > Ran Gao
>> >
>> >
>> > [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
>> >
>> >
>> >
>>
>

Re: [DISCUSSION] PIP-121: Pulsar cluster level auto failover

2022-01-05 Thread Enrico Olivelli

I am commeting on the GH issue

Thanks

Enrico

Il Mer 5 Gen 2022, 04:56 PengHui Li  ha scritto:

> +1
>
> Penghui
>
> On Tue, Jan 4, 2022 at 4:51 PM Hang Chen  wrote:
>
> > https://github.com/apache/pulsar/issues/13315
> >
> > Pasted below for quoting convenience.
> >
> > 
> > ### Motivation
> > We have geo-replication to support Pulsar cluster level failover. We
> > can setup Pulsar cluster A as a primary cluster in data center A, and
> > setup Pulsar cluster B as backup cluster in data center B. Then we
> > configure geo-replication between cluster A and cluster B. All the
> > clients are connected to the Pulsar cluster by DNS. If cluster A is
> > down, we should switch the DNS to point the target Pulsar cluster from
> > cluster A to cluster B. After the clients are resolved to cluster B,
> > they can produce and consume messages normally. After cluster A
> > recovers, the administrator should switch the DNS back to cluster A.
> >
> > However, the current method has two shortcomings.
> > 1. The administrator should monitor the status of all Pulsar clusters,
> > and switch the DNS as soon as possible when cluster A is down. The
> > switch and recovery is not automatic and recovery time is controlled
> > by the administrator, which will put the administrator under heavy
> > load.
> > 2. The Pulsar client and DNS system have a cache. When the
> > administrator switches the DNS from cluster A to Cluster B, it will
> > take some time for cache trigger timeout, which will delay client
> > recovery time and lead to the product/consumer message failing.
> >
> > ### Goal
> > It's better to provide an automatic cluster level failure recovery
> > mechanism to make pulsar cluster failover more effective. We should
> > support pulsar clients auto switching from cluster A to cluster B when
> > it detects cluster A has been down according to the configured
> > detecting policy and switch back to cluster A when it has recovered.
> > The reason why we should switch back to cluster A is that most
> > applications may be deployed in data center A and they have low
> > network cost for communicating with pulsar cluster A. If they keep
> > visiting pulsar cluster B, they have high network cost, and cause high
> > produce/consume latency.
> >
> > In order to improve the DNS cache problem, we should provide an
> > administrator controlled switch provider for administrators to update
> > service URLs.
> >
> > In the end, we should provide an auto service URL switch provider and
> > administrator controlled switch provider.
> >
> > ### Design
> > We have already provided the `ServiceUrlProvider` interface to support
> > different service URLs. In order to support automatic cluster level
> > failure auto recovery, we can provide different ServiceUrlProvider
> > implementations. For current requirements, we can provide
> > `AutoClusterFailover` and `ControlledClusterFailover`.
> >
> >  AutoClusterFailover
> > In order to support auto switching from the primary cluster to the
> > secondary, we can provide a probe task, which will probe the activity
> > of the primary cluster and the secondary one. When it finds the
> > primary cluster failed more than `failoverDelayMs`, it will switch to
> > the secondary cluster by calling `updateServiceUrl`. After switching
> > to the secondary cluster, the `AutoClusterFailover` will continue to
> > probe the primary cluster. If the primary cluster comes back and
> > remains active for `switchBackDelayMs`, it will switch back to the
> > primary cluster.
> > The APIs are listed as follows.
> >
> > In order to support multiple secondary clusters, use List to store
> > secondary cluster urls. When the primary cluster probe fails for
> > failoverDelayMs, it will start to probe the secondary cluster list one
> > by one, once it finds the active cluster, it will switch to the target
> > cluster. Notice: If you configured multiple clusters, you should turn
> > on cluster level geo-replication to ensure the topic data sync between
> > all primary and secondary clusters. Otherwise, it may distribute the
> > topic data into different clusters. And the consumers won’t get the
> > whole data of the topic.
> >
> > In order to support different authentication configurations between
> > clusters, we provide the authentication relation configurations
> > updated with the target cluster.
> >
> > ```Java
> > public class AutoClusterFailover implements ServiceUrlProvider {
> >
> >private AutoClusterFailover(String primary, List secondary,
> > long failoverDelayNs, long switchBackDelayNs,
> > long intervalMs, Authentication
> > primaryAuthentication,
> > List
> > secondaryAuthentications, String primaryTlsTrustCertsFilePath,
> > List
> > secondaryTlsTrustCertsFilePaths, String primaryTlsTrustStorePath,
> > List
> > secondaryTlsTrustStorePaths, String primaryTlsTrustStorePasswor

Re: [DISCUSS] Apache Pulsar 2.9.2 release

2022-01-05 Thread Michael Marshall

+1 - thanks for starting this thread.

>171 commits?
> Why?
> This is too much in my opinion for a point release.

I agree that this is a lot of commits, and especially since we only
just released 2.9.1. Note though that this is not a new phenomenon:
2.8.2 had 251 commits on top of 2.8.1. [0]

> Every change we cherry pick has a good chance to break the stability

This is one of the challenges to our current cherry-pick based git
workflow. It's hard to decide which commits to cherry-pick, and it
requires a high level of effort to later verify the correctness of
what was cherry-picked. This is one of the reasons I am interested in
exploring a merge based git workflow.

Our project has grown a ton and will continue to grow. Our git process
should prioritize the stability of our existing release branches.

> And that we will let branch-2.9 stabilise by not adding any other commits
> that are not strictly required:
> - security related issues
> - data loss/data corruption cases

If we don't change our git workflow, we should at least add a
cherry-picking guide to the wiki. We have added many new committers in
the past year, but we don't provide them with much documented guidance
on how to exercise their new rights.

Thanks,
Michael

[0] https://github.com/apache/pulsar/compare/v2.8.1...v2.8.2


On Wed, Jan 5, 2022 at 11:44 AM Enrico Olivelli  wrote:
>
> BTW +1
>
> I hope we will review carefully what we are including in the release
>
> And that we will let branch-2.9 stabilise by not adding any other commits
> that are not strictly required:
> - security related issues
> - data loss/data corruption cases
>
>
> Enrico
>
>
> Thank you Ran Gao for starting this thread.
>
>
>
> Enrico
>
> Il Mer 5 Gen 2022, 17:29 Enrico Olivelli  ha scritto:
>
> > 171 commits?
> >
> > Why?
> >
> > This is too much in my opinion for a point release.
> >
> > I am pretty sure that we don't need all that changes.
> >
> > Every change we cherry pick has a good chance to break the stability
> >
> > Enrico
> >
> > Il Mer 5 Gen 2022, 16:30 mattison chao  ha
> > scritto:
> >
> >> +1
> >>
> >> regards,
> >> Mattisonchao
> >>
> >> On Wed, 5 Jan 2022 at 23:04, Lan Liang  wrote:
> >>
> >> > +1,Thanks for your work.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Best Regards,
> >> > Lan Liang
> >> > On 1/5/2022 23:01，Hang Chen wrote：
> >> > +1
> >> >
> >> > Best,
> >> > Hang
> >> >
> >> > 陳智弘  于2022年1月5日周三 21:30写道：
> >> >
> >> > +1
> >> >
> >> > Lari Hotari  於 2022年1月5日 週三 20:56 寫道：
> >> >
> >> > +1
> >> >
> >> > On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:
> >> >
> >> > Hello, Pulsar community:
> >> >
> >> > I'd like to propose that we release Apache Pulsar 2.9.2.
> >> >
> >> > Currently, compared to 2.9.1, branch-2.9 already merged 171
> >> commits(refer
> >> > to [0]), they contain the log4j security patch and many important fixes.
> >> >
> >> > I am happy to volunteer to be the release manager.
> >> >
> >> > I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
> >> > branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
> >> > labeled release/2.9.2, I'll follow them to make sure important fix could
> >> > be
> >> > merged in 2.9.2.
> >> >
> >> >
> >> > Best,
> >> > Ran Gao
> >> >
> >> >
> >> > [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
> >> >
> >> >
> >> >
> >>
> >

Re: [PR] Pulsar non root docker image

2022-01-05 Thread Michael Marshall

PR 13376 is ready for review, PTAL.

What approach should we take regarding docker image size?
I propose providing a minimal image along with documentation
on how to add your own debugging tools. Is that sufficient?

I'd like to include this feature in 2.10.0.

Note that you can test the new docker image here:
michaelmarshall/pulsar:2.10.0-SNAPSHOT-1dec8b9

Thanks,
Michael



On Wed, Dec 22, 2021 at 1:51 PM Michael Marshall  wrote:
>
> Thanks for raising this important topic, Enrico.
>
> > Basically if you are running as non root, you cannot add tools on demand,
> > so we need to add basic tools, like netstat/vim/unzip otherwise when
> > you have a problem you are trapped.
>
> I agree that running as a non root user presents challenges for
> debugging. However, I don't think we should optimize our production
> image for debugging. We should instead optimize for a minimal docker
> image with as few dependencies as possible to decrease the image's
> attack surface.
>
> Also, I think it would be valuable to audit the current
> dependencies we already ship in our docker image. For example, this
> single `RUN` command [0] adds 1 GB of dependencies to our
> docker image.
>
> If there is a need for an image with debug tools, we could publish a
> "debug" image that extends our production image with extra tooling.
> Users could swap out the prod image with the debug image, as needed.
> However, it is pretty easy and quick to extend a public docker image
> to add your own tooling. By documenting how to extend our docker
> image, we could avoid decisions about which tools should be in our
> image.
>
> > there are ways to inject tools, but they are very hard to execute for
> > people who is not very experienced
>
> We can solve this through additional documentation. Freeznet asked a
> similar question about debugging on the PR. I provided some methods for
> debugging here [1].
>
> Thanks,
> Michael
>
> [0] 
> https://github.com/apache/pulsar/blob/5dd60dbd748e446f8da396b448a5bb16a2ae6902/docker/pulsar/Dockerfile#L46-L55
> [1] https://github.com/apache/pulsar/pull/13376#discussion_r773580954
>
>
>
>
> On Wed, Dec 22, 2021 at 1:29 AM Enrico Olivelli  wrote:
> >
> > Michael,
> >
> > I would like to add this item to the list
> > https://github.com/apache/pulsar/pull/10815
> >
> > Basically if you are running as non root, you cannot add tools on demand,
> > so we need to add basic tools, like netstat/vim/unzip otherwise when
> > you have a problem you are trapped.
> >
> > there are ways to inject tools, but they are very hard to execute for
> > people who is not very experienced
> >
> > Enrico
> >
> >
> > Il giorno mer 22 dic 2021 alle ore 04:40 Haiting Jiang <
> > jianghait...@apache.org> ha scritto:
> >
> > > > 1. Pulsar configuration is read in only from configuration files in
> > > > `/pulsar/conf`. A non root user must be able to update these files to
> > > > have run with custom configuration.
> > >
> > > About the configurations, I also see similar require like this lately [0].
> > > IMHO, update any configs without redeploy service is a useful feature.
> > > I would like post a PIP for this later.
> > > Basic idea is like make all configs dynamic by default except
> > > `metadataStoreUrl` and all configs are stored under path
> > > "/admin/configuration" in metadata store.
> > >
> > > [0] https://github.com/apache/pulsar/pull/13074
> > >
> > > On 2021/12/21 20:16:44 Michael Marshall wrote:
> > > > All tests are now passing for this PR [0]. I built the docker image
> > > > and pushed it to my personal repository to simplify testing [1] for
> > > > anyone interested in verifying the changes.
> > > >
> > > > I would like our docker image to be as close to immutable as possible.
> > > > As far as I can tell, here are the only reasons the image cannot be
> > > > immutable:
> > > >
> > > > 1. Pulsar configuration is read in only from configuration files in
> > > > `/pulsar/conf`. A non root user must be able to update these files to
> > > > have run with custom configuration.
> > > > 2. The Pulsar function worker unpacks nar files to
> > > > `/pulsar/download/pulsar_functions` by default.
> > > > 3. Pulsar tiered storage writes to `/pulsar` by default when using
> > > > filesystem storage.
> > > > 4. The Presto SQL worker writes to `/pulsar/lib/presto/` by default.
> > > > 5. Pulsar-admin and functions write to `/pulsar/log` by default
> > > > (possibly other components too).
> > > >
> > > > Note that even though bookkeepers and zookeepers write to
> > > > `/pulsar/data`, they should be writing to docker volumes, in which
> > > > case, the host's file system permissions are all that matter.
> > > >
> > > > If we can remove any of the above reasons, we can decrease the
> > > > privilege in the docker image.
> > > >
> > > > The PR has more detail. Please take a look.
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > > > [0] https://github.com/apache/pulsar/pull/13376
> > > > [1] michaelmarshall/pulsar:2.10.0-SNAPS

Re: [DISCUSS] Remove Grafana and Prometheus Dockerfiles; Remove DCOS example

2022-01-05 Thread Michael Marshall

Here is the PR to remove the DC/OS example:
https://github.com/apache/pulsar/pull/13632

In the PR, I remove the `site2/docs/deploy-dcos.md` file, as it won't
have much meaning without the underlying example. Further, it contains
5 broken links to Streamlio's GitHub repo (the repo no longer exists).

An alternative approach would be to remove the prometheus and grafana
examples from the DC/OS example, fix all of the broken links, and
update the content to ensure correctness.

- Michael


On Thu, Dec 23, 2021 at 4:22 PM Enrico Olivelli  wrote:
>
> +1
>
> Enrico
>
> Il Gio 23 Dic 2021, 19:01 Matteo Merli  ha scritto:
>
> > +1
> >
> > On Thu, Dec 23, 2021 at 9:53 AM Michael Marshall 
> > wrote:
> >
> > > Update: the PR [0] to remove the grafana docker image is now merged. I
> > > still need reviews for the PR to remove the Prometheus docker image
> > > [1]. I will follow up with a PR to remove the DCOS example.
> > >
> > > Thanks,
> > > Michael
> > >
> > > [0] https://github.com/apache/pulsar/pull/13389
> > > [1] https://github.com/apache/pulsar/pull/13387
> > >
> > > On Mon, Dec 20, 2021 at 1:29 PM Michael Marshall 
> > > wrote:
> > > >
> > > > Hi Pulsar Community,
> > > >
> > > > I would like to clean up our main project's `docker/` directory. We
> > > > have a couple of old Dockerfiles that either need to be updated or
> > > > removed. My vote is to remove them.
> > > >
> > > > 1. Grafana: with every release, we build and upload a custom grafana
> > > > docker image just to package dashboards. Grafana provides other
> > > > mechanisms to load dashboards. I propose we remove the grafana image
> > > > from our release process and then move our dashboard json files to a
> > > > new top level directory named `grafana/`. [0]
> > > >
> > > > Our image is based on grafana/grafana:4.3.1. The current grafana
> > > > release is grafana/grafana:8.3.3.
> > > >
> > > > 2. Prometheus: we have a prometheus Dockerfile that was initially
> > > > merged as part of a DCOS example 4 years ago. We have never published
> > > > this docker image as part of a release [1]. There is nothing pulsar
> > > > specific in these files. [2]
> > > >
> > > > Our image is based on prom/prometheus:v1.8.2. The latest prometheus
> > > > image prom/prometheus:v2.32.1.
> > > >
> > > > 3. Can we remove our DCOS examples in `deployment/dcos`? We haven't
> > > > updated our DCOS example deployment spec in 3.5 years. The current
> > > > spec relies on our custom grafana docker image and on a community
> > > > member's custom build [3] of our prometheus-dcos docker image. I don't
> > > > think our example deployment should rely on a community member's
> > > > custom docker image. (I don't have a PR to remove this example yet.)
> > > >
> > > > Alternatively, we could update the dockerfiles and the DCOS example.
> > > > However, the lack of activity in keeping these three project
> > > > dependencies up to date indicates to me that we should remove them.
> > > >
> > > > Let me know what you think.
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > > > [0] https://github.com/apache/pulsar/pull/13389
> > > > [1] https://hub.docker.com/u/apachepulsar/
> > > > [2] https://github.com/apache/pulsar/pull/13387
> > > > [3]
> > >
> > https://github.com/apache/pulsar/blob/c62ca808fc8c5aed3bb96b99bf6ef0c8ed382e3f/deployment/dcos/PulsarGroups.json#L251
> > >
> > --
> > --
> > Matteo Merli
> > 
> >

[GitHub] [pulsar-helm-chart] michaeljmarshall commented on a change in pull request #130: Bump pulsar 2.8.0

2022-01-05 Thread GitBox



michaeljmarshall commented on a change in pull request #130:
URL: https://github.com/apache/pulsar-helm-chart/pull/130#discussion_r779156795



##
File path: charts/pulsar/values.yaml
##
@@ -278,7 +278,7 @@ zookeeper:
   replicaCount: 3
   updateStrategy:
 type: RollingUpdate
-  podManagementPolicy: OrderedReady
+  podManagementPolicy: Parallel

Review comment:
   I think `Parallel` is the right choice for us. It'll lead to faster 
cluster initialization. 
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#parallel-pod-management




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[DISCUSSION] PIP-133 Pulsar Functions Add API For Accessing Other Function States

2022-01-05 Thread Ethan Merrill

Original PIP: https://github.com/apache/pulsar/issues/13633

Pasted below for quoting convenience.

-

## Motivation

Certain uses of Pulsar functions could benefit from the ability to access the 
states of other functions. Currently functions can only access their own 
states, and so sharing information between functions requires other solutions 
such as writing to a separate database.

## Goal

The goal is to enable the ability for a function to access another function's 
state. Given another function's tenant, namespace, and name, any function 
should be able to access the other function's state for read and write 
purposes. This PIP is not concerned with expanding the capabilities of function 
states, It only deals with expanding access to function states.

## API Changes

The Pulsar function API would be modified to have the function context 
implement the following interface for accessing function states using a tenant, 
namespace, and name.

```
public interface SharedContext {
/**
 * Update the state value for the key.
 *
 * @param key   name of the key
 * @param value state value of the key
 * @param tenant the state tenant name
 * @param ns the state namespace name
 * @param name the state store name
 */
void putState(String key, ByteBuffer value, String tenant, String ns, 
String name);

/**
 * Update the state value for the key, but don't wait for the operation to 
be completed
 *
 * @param key   name of the key
 * @param value state value of the key
 * @param tenant the state tenant name
 * @param ns the state namespace name
 * @param name the state store name
 */
CompletableFuture putStateAsync(String key, ByteBuffer value, String 
tenant, String ns, String name);

/**
 * Retrieve the state value for the key.
 *
 * @param key name of the key
 * @param tenant the state tenant name
 * @param ns the state namespace name
 * @param name the state store name
 * @return the state value for the key.
 */
ByteBuffer getState(String key, String tenant, String ns, String name);

/**
 * Retrieve the state value for the key, but don't wait for the operation 
to be completed
 *
 * @param key name of the key
 * @param tenant the state tenant name
 * @param ns the state namespace name
 * @param name the state store name
 * @return the state value for the key.
 */
CompletableFuture getStateAsync(String key, String tenant, 
String ns, String name);

/**
 * Delete the state value for the key.
 *
 * @param key   name of the key
 * @param tenant the state tenant name
 * @param ns the state namespace name
 * @param name the state store name
 */
void deleteState(String key, String tenant, String ns, String name);

/**
 * Delete the state value for the key, but don't wait for the operation to 
be completed
 *
 * @param key   name of the key
 * @param tenant the state tenant name
 * @param ns the state namespace name
 * @param name the state store name
 */
CompletableFuture deleteStateAsync(String key, String tenant, String 
ns, String name);

/**
 * Increment the builtin distributed counter referred by key.
 *
 * @param keyThe name of the key
 * @param amount The amount to be incremented
 * @param tenant the state tenant name
 * @param ns the state namespace name
 * @param name the state store name
 */
void incrCounter(String key, long amount, String tenant, String ns, String 
name);

/**
 * Increment the builtin distributed counter referred by key
 * but dont wait for the completion of the increment operation
 *
 * @param keyThe name of the key
 * @param amount The amount to be incremented
 * @param tenant the state tenant name
 * @param ns the state namespace name
 * @param name the state store name
 */
CompletableFuture incrCounterAsync(String key, long amount, String 
tenant, String ns, String name);

/**
 * Retrieve the counter value for the key.
 *
 * @param key name of the key
 * @param tenant the state tenant name
 * @param ns the state namespace name
 * @param name the state store name
 * @return the amount of the counter value for this key
 */
long getCounter(String key, String tenant, String ns, String name);

/**
 * Retrieve the counter value for the key, but don't wait
 * for the operation to be completed
 *
 * @param key name of the key
 * @param tenant the state tenant name
 * @param ns the state namespace name
 * @param name the state store name
 * @return the amount of the counter value for this key
 */
CompletableFuture getCounterAsync(String key, String tenant, String 
ns, String name);
}
```

And the python context would have the following added:
```
class Context(object):
  @abstractmethod

[GitHub] [pulsar-client-node] Hoskins355 commented on issue #188: When installing the Pulsar Node.js client, the operation steps need to be optimized

2022-01-05 Thread GitBox



Hoskins355 commented on issue #188:
URL: 
https://github.com/apache/pulsar-client-node/issues/188#issuecomment-1006180927


   I am facing the same. @zhaoyajun2009 did you get it working?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Apache Pulsar 2.9.2 release

2022-01-05 Thread guo jiwei

+1

Regards
Jiwei Guo (Tboy)


On Thu, Jan 6, 2022 at 2:32 AM Michael Marshall 
wrote:

> +1 - thanks for starting this thread.
>
> >171 commits?
> > Why?
> > This is too much in my opinion for a point release.
>
> I agree that this is a lot of commits, and especially since we only
> just released 2.9.1. Note though that this is not a new phenomenon:
> 2.8.2 had 251 commits on top of 2.8.1. [0]
>
> > Every change we cherry pick has a good chance to break the stability
>
> This is one of the challenges to our current cherry-pick based git
> workflow. It's hard to decide which commits to cherry-pick, and it
> requires a high level of effort to later verify the correctness of
> what was cherry-picked. This is one of the reasons I am interested in
> exploring a merge based git workflow.
>
> Our project has grown a ton and will continue to grow. Our git process
> should prioritize the stability of our existing release branches.
>
> > And that we will let branch-2.9 stabilise by not adding any other commits
> > that are not strictly required:
> > - security related issues
> > - data loss/data corruption cases
>
> If we don't change our git workflow, we should at least add a
> cherry-picking guide to the wiki. We have added many new committers in
> the past year, but we don't provide them with much documented guidance
> on how to exercise their new rights.
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar/compare/v2.8.1...v2.8.2
>
>
> On Wed, Jan 5, 2022 at 11:44 AM Enrico Olivelli 
> wrote:
> >
> > BTW +1
> >
> > I hope we will review carefully what we are including in the release
> >
> > And that we will let branch-2.9 stabilise by not adding any other commits
> > that are not strictly required:
> > - security related issues
> > - data loss/data corruption cases
> >
> >
> > Enrico
> >
> >
> > Thank you Ran Gao for starting this thread.
> >
> >
> >
> > Enrico
> >
> > Il Mer 5 Gen 2022, 17:29 Enrico Olivelli  ha
> scritto:
> >
> > > 171 commits?
> > >
> > > Why?
> > >
> > > This is too much in my opinion for a point release.
> > >
> > > I am pretty sure that we don't need all that changes.
> > >
> > > Every change we cherry pick has a good chance to break the stability
> > >
> > > Enrico
> > >
> > > Il Mer 5 Gen 2022, 16:30 mattison chao  ha
> > > scritto:
> > >
> > >> +1
> > >>
> > >> regards,
> > >> Mattisonchao
> > >>
> > >> On Wed, 5 Jan 2022 at 23:04, Lan Liang 
> wrote:
> > >>
> > >> > +1,Thanks for your work.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Best Regards,
> > >> > Lan Liang
> > >> > On 1/5/2022 23:01，Hang Chen wrote：
> > >> > +1
> > >> >
> > >> > Best,
> > >> > Hang
> > >> >
> > >> > 陳智弘  于2022年1月5日周三 21:30写道：
> > >> >
> > >> > +1
> > >> >
> > >> > Lari Hotari  於 2022年1月5日 週三 20:56 寫道：
> > >> >
> > >> > +1
> > >> >
> > >> > On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:
> > >> >
> > >> > Hello, Pulsar community:
> > >> >
> > >> > I'd like to propose that we release Apache Pulsar 2.9.2.
> > >> >
> > >> > Currently, compared to 2.9.1, branch-2.9 already merged 171
> > >> commits(refer
> > >> > to [0]), they contain the log4j security patch and many important
> fixes.
> > >> >
> > >> > I am happy to volunteer to be the release manager.
> > >> >
> > >> > I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
> > >> > branch-2.9, I'll cherry-pick them, and there are also 20 open PRs
> are
> > >> > labeled release/2.9.2, I'll follow them to make sure important fix
> could
> > >> > be
> > >> > merged in 2.9.2.
> > >> >
> > >> >
> > >> > Best,
> > >> > Ran Gao
> > >> >
> > >> >
> > >> > [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
> > >> >
> > >> >
> > >> >
> > >>
> > >
>

Re: [ANNOUNCE] New Committer: Zhangjian He

2022-01-05 Thread guo jiwei

Congrats to Zhangjian!


Regards
Jiwei Guo (Tboy)


On Wed, Jan 5, 2022 at 10:59 PM Hang Chen  wrote:

> The Apache Pulsar Project Management Committee (PMC) has invited Zhangjian
> He
> https://github.com/Shoothzj to become a committer and we are pleased to
> announce that he has accepted.
>
> Zhangjian He has joined the community for more than 1 year now and he is
> active in the Pulsar community for more than 6 months
>
> Welcome and Congratulations, Zhangjian!
>
> Please join us in congratulating and welcoming Zhangjian onboard!
>
> Best Regards,
> Hang Chen on behalf of the Pulsar PMC
>

Re: [ANNOUNCE] New Committer: Zhangjian He

2022-01-05 Thread Jia Zhai

Congratulations to Zhangjian!
Thanks for your contributions to Pulsar.



On Thu, Jan 6, 2022 at 9:25 AM guo jiwei  wrote:

> Congrats to Zhangjian!
>
>
> Regards
> Jiwei Guo (Tboy)
>
>
> On Wed, Jan 5, 2022 at 10:59 PM Hang Chen  wrote:
>
> > The Apache Pulsar Project Management Committee (PMC) has invited
> Zhangjian
> > He
> > https://github.com/Shoothzj to become a committer and we are pleased to
> > announce that he has accepted.
> >
> > Zhangjian He has joined the community for more than 1 year now and he is
> > active in the Pulsar community for more than 6 months
> >
> > Welcome and Congratulations, Zhangjian!
> >
> > Please join us in congratulating and welcoming Zhangjian onboard!
> >
> > Best Regards,
> > Hang Chen on behalf of the Pulsar PMC
> >
>

Re: [ANNOUNCE] New Committer: Zhangjian He

2022-01-05 Thread Haiting Jiang

Congrats, Zhangjian!

Thanks,
Haiting Jiang

On 2022/01/06 02:05:53 Jia Zhai wrote:
> Congratulations to Zhangjian!
> Thanks for your contributions to Pulsar.
> 
> 
> 
> On Thu, Jan 6, 2022 at 9:25 AM guo jiwei  wrote:
> 
> > Congrats to Zhangjian!
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
> >
> > On Wed, Jan 5, 2022 at 10:59 PM Hang Chen  wrote:
> >
> > > The Apache Pulsar Project Management Committee (PMC) has invited
> > Zhangjian
> > > He
> > > https://github.com/Shoothzj to become a committer and we are pleased to
> > > announce that he has accepted.
> > >
> > > Zhangjian He has joined the community for more than 1 year now and he is
> > > active in the Pulsar community for more than 6 months
> > >
> > > Welcome and Congratulations, Zhangjian!
> > >
> > > Please join us in congratulating and welcoming Zhangjian onboard!
> > >
> > > Best Regards,
> > > Hang Chen on behalf of the Pulsar PMC
> > >
> >
>

Re: [ANNOUNCE] New Committer: Zhangjian He

2022-01-05 Thread Yu

Kudos to Zhangjian! And thanks for your documentation contribution, welcome
to submit more commits!

On Thu, Jan 6, 2022 at 10:20 AM Haiting Jiang 
wrote:

> Congrats, Zhangjian!
>
> Thanks,
> Haiting Jiang
>
> On 2022/01/06 02:05:53 Jia Zhai wrote:
> > Congratulations to Zhangjian!
> > Thanks for your contributions to Pulsar.
> >
> >
> >
> > On Thu, Jan 6, 2022 at 9:25 AM guo jiwei  wrote:
> >
> > > Congrats to Zhangjian!
> > >
> > >
> > > Regards
> > > Jiwei Guo (Tboy)
> > >
> > >
> > > On Wed, Jan 5, 2022 at 10:59 PM Hang Chen  wrote:
> > >
> > > > The Apache Pulsar Project Management Committee (PMC) has invited
> > > Zhangjian
> > > > He
> > > > https://github.com/Shoothzj to become a committer and we are
> pleased to
> > > > announce that he has accepted.
> > > >
> > > > Zhangjian He has joined the community for more than 1 year now and
> he is
> > > > active in the Pulsar community for more than 6 months
> > > >
> > > > Welcome and Congratulations, Zhangjian!
> > > >
> > > > Please join us in congratulating and welcoming Zhangjian onboard!
> > > >
> > > > Best Regards,
> > > > Hang Chen on behalf of the Pulsar PMC
> > > >
> > >
> >
>

Re: [DISCUSSION] PIP-132: Support Pulsar system event.

2022-01-05 Thread Haiting Jiang



>   CompletableFuture trigger(SystemEventMessage message);

Maybe `record` make more sense? 

>   SystemEventMessageBuilder properties(Map properties);
> 
>   SystemEventMessageBuilder addProperty(String key, Object value);

How to handle  the serialization and deserialization if the value type is 
Object?  

> - [ ] Producer publish
> - [ ] Message delivered
> - [ ] Message acked

I think we don't need to record normal message producing and consuming event? 
That would make the system topic store too much data.


On 2022/01/05 15:27:33 mattison chao wrote:
> Original PIP : https://github.com/apache/pulsar/issues/13628
> 
> Pasted bellow for quoting convenience.
> 
> —— 
> 
> ## Motivation
> 
> In some case or for better observability. We can support Pulsar event 
> notification mechanism, that can let user subscribe this system topic to 
> receive some **"we need notice to user"** event.
> 
> ## Goal
> 
> Enable users to receive Pulsar system event.
> 
> ## API Changes
> 
> 1. Add config ``pulsarSystemEventEnabled``
> 
> ```java
> pulsarSystemEventEnabled=false
> ```
> 2. Add ``SystemEventMessageBuilder``
> ```java
> 
> 
> public interface SystemEventMessageBuilder {
> 
>   SystemEventMessageBuilder eventType(SystemEvent event);
> 
>   SystemEventMessageBuilder properties(Map properties);
> 
>   SystemEventMessageBuilder addProperty(String key, Object value);
> 
>   SystemEventMessage build();
> 
> }
> 
> ```
> 3. Add ``SystemEventService``
> 
> ```java
> 
> public interface SystemEventService {
> 
>   void init();
> 
>   CompletableFuture trigger(SystemEventMessage message);
> 
>   void close();
> }
> 
> ```
> 
> ## Implementation
> 
> This PIP just is a simple version, it's purpose is to discuss what event we 
> need to trigger? how to trigger? and what is the event message structure.
> 
> I think we need to trigger some event as bellow:
> 
> - [ ] Client connected
> - [ ] Client disconnected
> - [ ] Consumer subscribe
> - [ ] Consumer unsubscribe
> - [ ] Producer publish
> - [ ] Message delivered
> - [ ] Message acked
> 
> These are far from enough, and we need to continue to make supplements here.
> Please fell free to give me more advice for this PIP.
> 
> ## Reject Alternatives
> 
> none.
> 
> Thanks,
> Mattisonchao

Thanks,
Haiting Jiang

Re: [DISCUSSION] PIP-132: Support Pulsar system event.

2022-01-05 Thread Dave Fisher




Sent from my iPhone

> On Jan 5, 2022, at 8:10 PM, Haiting Jiang  wrote:
> 
> 
>>  CompletableFuture trigger(SystemEventMessage message);
> 
> Maybe `record` make more sense? 
> 
>>  SystemEventMessageBuilder properties(Map properties);
>> 
>>  SystemEventMessageBuilder addProperty(String key, Object value);
> 
> How to handle  the serialization and deserialization if the value type is 
> Object?  
> 
>> - [ ] Producer publish
>> - [ ] Message delivered
>> - [ ] Message acked
> 
> I think we don't need to record normal message producing and consuming event? 
> That would make the system topic store too much data.

These bother me too. Producer and consumers can accomplish the same use cases 
in their application logic.

Any notification mechanism must be very lightweight. What is the implantation 
plan?

Also if this pip is for user use cases then there needs to be isolation at the 
tenant level.

Thanks for the proposal.

Regards,
Dave

> 
> 
>> On 2022/01/05 15:27:33 mattison chao wrote:
>> Original PIP : https://github.com/apache/pulsar/issues/13628
>> 
>> Pasted bellow for quoting convenience.
>> 
>> —— 
>> 
>> ## Motivation
>> 
>> In some case or for better observability. We can support Pulsar event 
>> notification mechanism, that can let user subscribe this system topic to 
>> receive some **"we need notice to user"** event.
>> 
>> ## Goal
>> 
>> Enable users to receive Pulsar system event.
>> 
>> ## API Changes
>> 
>> 1. Add config ``pulsarSystemEventEnabled``
>> 
>> ```java
>> pulsarSystemEventEnabled=false
>> ```
>> 2. Add ``SystemEventMessageBuilder``
>> ```java
>> 
>> 
>> public interface SystemEventMessageBuilder {
>> 
>>  SystemEventMessageBuilder eventType(SystemEvent event);
>> 
>>  SystemEventMessageBuilder properties(Map properties);
>> 
>>  SystemEventMessageBuilder addProperty(String key, Object value);
>> 
>>  SystemEventMessage build();
>> 
>> }
>> 
>> ```
>> 3. Add ``SystemEventService``
>> 
>> ```java
>> 
>> public interface SystemEventService {
>> 
>>  void init();
>> 
>>  CompletableFuture trigger(SystemEventMessage message);
>> 
>>  void close();
>> }
>> 
>> ```
>> 
>> ## Implementation
>> 
>> This PIP just is a simple version, it's purpose is to discuss what event we 
>> need to trigger? how to trigger? and what is the event message structure.
>> 
>> I think we need to trigger some event as bellow:
>> 
>> - [ ] Client connected
>> - [ ] Client disconnected
>> - [ ] Consumer subscribe
>> - [ ] Consumer unsubscribe
>> - [ ] Producer publish
>> - [ ] Message delivered
>> - [ ] Message acked
>> 
>> These are far from enough, and we need to continue to make supplements here.
>> Please fell free to give me more advice for this PIP.
>> 
>> ## Reject Alternatives
>> 
>> none.
>> 
>> Thanks,
>> Mattisonchao
> 
> Thanks,
> Haiting Jiang

Re: [DISCUSSION] PIP-132: Support Pulsar system event.

2022-01-05 Thread mattison chao

Hi, Haiting Jiang


>>  CompletableFuture trigger(SystemEventMessage message);
> 
> Maybe `record` make more sense? 

+1

>>  SystemEventMessageBuilder properties(Map properties);
>> 
>>  SystemEventMessageBuilder addProperty(String key, Object value);
> 
> How to handle  the serialization and deserialization if the value type is 
> Object?  

For this part, I think we need to discuss. Do we create a specific event class 
or use properties in json format?


> I think we don't need to record normal message producing and consuming event? 
> That would make the system topic store too much data.


Agree with you.

Maybe the better way is to record some system internal event like node changed, 
ledger deleted etc.






> On Jan 6, 2022, at 12:10 PM, Haiting Jiang  > wrote:
> 
> 
>>  CompletableFuture trigger(SystemEventMessage message);
> 
> Maybe `record` make more sense? 
> 
>>  SystemEventMessageBuilder properties(Map properties);
>> 
>>  SystemEventMessageBuilder addProperty(String key, Object value);
> 
> How to handle  the serialization and deserialization if the value type is 
> Object?  
> 
>> - [ ] Producer publish
>> - [ ] Message delivered
>> - [ ] Message acked
> 
> I think we don't need to record normal message producing and consuming event? 
> That would make the system topic store too much data.
> 
> 
> On 2022/01/05 15:27:33 mattison chao wrote:
>> Original PIP : https://github.com/apache/pulsar/issues/13628 
>> 
>> 
>> Pasted bellow for quoting convenience.
>> 
>> —— 
>> 
>> ## Motivation
>> 
>> In some case or for better observability. We can support Pulsar event 
>> notification mechanism, that can let user subscribe this system topic to 
>> receive some **"we need notice to user"** event.
>> 
>> ## Goal
>> 
>> Enable users to receive Pulsar system event.
>> 
>> ## API Changes
>> 
>> 1. Add config ``pulsarSystemEventEnabled``
>> 
>> ```java
>> pulsarSystemEventEnabled=false
>> ```
>> 2. Add ``SystemEventMessageBuilder``
>> ```java
>> 
>> 
>> public interface SystemEventMessageBuilder {
>> 
>>  SystemEventMessageBuilder eventType(SystemEvent event);
>> 
>>  SystemEventMessageBuilder properties(Map properties);
>> 
>>  SystemEventMessageBuilder addProperty(String key, Object value);
>> 
>>  SystemEventMessage build();
>> 
>> }
>> 
>> ```
>> 3. Add ``SystemEventService``
>> 
>> ```java
>> 
>> public interface SystemEventService {
>> 
>>  void init();
>> 
>>  CompletableFuture trigger(SystemEventMessage message);
>> 
>>  void close();
>> }
>> 
>> ```
>> 
>> ## Implementation
>> 
>> This PIP just is a simple version, it's purpose is to discuss what event we 
>> need to trigger? how to trigger? and what is the event message structure.
>> 
>> I think we need to trigger some event as bellow:
>> 
>> - [ ] Client connected
>> - [ ] Client disconnected
>> - [ ] Consumer subscribe
>> - [ ] Consumer unsubscribe
>> - [ ] Producer publish
>> - [ ] Message delivered
>> - [ ] Message acked
>> 
>> These are far from enough, and we need to continue to make supplements here.
>> Please fell free to give me more advice for this PIP.
>> 
>> ## Reject Alternatives
>> 
>> none.
>> 
>> Thanks,
>> Mattisonchao
> 
> Thanks,
> Haiting Jiang

Re: [DISCUSSION] PIP-132: Support Pulsar system event.

2022-01-05 Thread mattison chao

Hi, Dave Fisher

> These bother me too. Producer and consumers can accomplish the same use cases 
> in their application logic.
> 
> Any notification mechanism must be very lightweight. What is the implantation 
> plan?

Agree with you.

I think we just need to record some system internal event that need human 
process.

> Also if this pip is for user use cases then there needs to be isolation at 
> the tenant level.

I think this pip just for middleware team to handle the pulsar internal event 
that need human process. 

E.g. Due to the network issue or distributed issue we can't to delete ledger.

Therefore, whether the global system topic is enough?

Thanks,
Mattisonchao


> On Jan 6, 2022, at 12:35 PM, Dave Fisher  wrote:
> 
> 
> 
> Sent from my iPhone
> 
>> On Jan 5, 2022, at 8:10 PM, Haiting Jiang  wrote:
>> 
>> 
>>> CompletableFuture trigger(SystemEventMessage message);
>> 
>> Maybe `record` make more sense? 
>> 
>>> SystemEventMessageBuilder properties(Map properties);
>>> 
>>> SystemEventMessageBuilder addProperty(String key, Object value);
>> 
>> How to handle  the serialization and deserialization if the value type is 
>> Object?  
>> 
>>> - [ ] Producer publish
>>> - [ ] Message delivered
>>> - [ ] Message acked
>> 
>> I think we don't need to record normal message producing and consuming 
>> event? 
>> That would make the system topic store too much data.
> 
> These bother me too. Producer and consumers can accomplish the same use cases 
> in their application logic.
> 
> Any notification mechanism must be very lightweight. What is the implantation 
> plan?
> 
> Also if this pip is for user use cases then there needs to be isolation at 
> the tenant level.
> 
> Thanks for the proposal.
> 
> Regards,
> Dave
> 
>> 
>> 
>>> On 2022/01/05 15:27:33 mattison chao wrote:
>>> Original PIP : https://github.com/apache/pulsar/issues/13628
>>> 
>>> Pasted bellow for quoting convenience.
>>> 
>>> —— 
>>> 
>>> ## Motivation
>>> 
>>> In some case or for better observability. We can support Pulsar event 
>>> notification mechanism, that can let user subscribe this system topic to 
>>> receive some **"we need notice to user"** event.
>>> 
>>> ## Goal
>>> 
>>> Enable users to receive Pulsar system event.
>>> 
>>> ## API Changes
>>> 
>>> 1. Add config ``pulsarSystemEventEnabled``
>>> 
>>> ```java
>>> pulsarSystemEventEnabled=false
>>> ```
>>> 2. Add ``SystemEventMessageBuilder``
>>> ```java
>>> 
>>> 
>>> public interface SystemEventMessageBuilder {
>>> 
>>> SystemEventMessageBuilder eventType(SystemEvent event);
>>> 
>>> SystemEventMessageBuilder properties(Map properties);
>>> 
>>> SystemEventMessageBuilder addProperty(String key, Object value);
>>> 
>>> SystemEventMessage build();
>>> 
>>> }
>>> 
>>> ```
>>> 3. Add ``SystemEventService``
>>> 
>>> ```java
>>> 
>>> public interface SystemEventService {
>>> 
>>> void init();
>>> 
>>> CompletableFuture trigger(SystemEventMessage message);
>>> 
>>> void close();
>>> }
>>> 
>>> ```
>>> 
>>> ## Implementation
>>> 
>>> This PIP just is a simple version, it's purpose is to discuss what event we 
>>> need to trigger? how to trigger? and what is the event message structure.
>>> 
>>> I think we need to trigger some event as bellow:
>>> 
>>> - [ ] Client connected
>>> - [ ] Client disconnected
>>> - [ ] Consumer subscribe
>>> - [ ] Consumer unsubscribe
>>> - [ ] Producer publish
>>> - [ ] Message delivered
>>> - [ ] Message acked
>>> 
>>> These are far from enough, and we need to continue to make supplements here.
>>> Please fell free to give me more advice for this PIP.
>>> 
>>> ## Reject Alternatives
>>> 
>>> none.
>>> 
>>> Thanks,
>>> Mattisonchao
>> 
>> Thanks,
>> Haiting Jiang

57 matches

Mail list logo