[DISCUSS] The default value of maxPendingChunkedMessage

[Zike Yang]

Hi, Pulsar community,

We found that there are inconsistencies between the code and the
documentation regarding the default value of maxPendingChunkedMessage.

In the java client code, we use 10 as the default value. [1] But in
the java doc, we use 100 as the default value. [2]
We need to fix this inconsistency. But what should we take as the
default value? From the code or the doc? I would like to hear your
discussions.

[1] 
https://github.com/apache/pulsar/blob/d11147616aa6cc7888420f6325bb71cd7f7ab065/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConsumerConfigurationData.java#L112-L113
[2] 
https://github.com/apache/pulsar/blob/1e2ff8a3941b7cc6d583f528ceedc393b7e607fb/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java#L690

Thanks,
Zike Yang

Re: [DISCUSS] The default value of maxPendingChunkedMessage

[Yunze Xu]

After thinking for a while, I’d prefer 10 as the default value and I changed
the default value to 10 in C++ client, see
https://github.com/apache/pulsar/pull/14070.

A chunked buffer to contain all chunks could use much memory, for example, if a
message was split into N chunks, since each chunk is 5MB by default, then 100
buffers will use N*500 MB. It could reach 1GB if N > 2.

In addition, normally, only if at least 100 producers sent messages to a
partition would it be meaningful to configure maxPendingChunkedMessages to 100.
IMO, it's hard to see so many producers on a partition in production.

Thanks,
Yunze Xu

> 2022年1月30日 下午6:32，Zike Yang  写道：
> 
> Hi, Pulsar community,
> 
> We found that there are inconsistencies between the code and the
> documentation regarding the default value of maxPendingChunkedMessage.
> 
> In the java client code, we use 10 as the default value. [1] But in
> the java doc, we use 100 as the default value. [2]
> We need to fix this inconsistency. But what should we take as the
> default value? From the code or the doc? I would like to hear your
> discussions.
> 
> [1] 
> https://github.com/apache/pulsar/blob/d11147616aa6cc7888420f6325bb71cd7f7ab065/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConsumerConfigurationData.java#L112-L113
> [2] 
> https://github.com/apache/pulsar/blob/1e2ff8a3941b7cc6d583f528ceedc393b7e607fb/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java#L690
> 
> Thanks,
> Zike Yang

Re: [DISCUSS] The default value of maxPendingChunkedMessage

[Zike Yang]

Hi, Yunze,

Thanks for your opinion.

> A chunked buffer to contain all chunks could use much memory, for example, if 
> a
> message was split into N chunks, since each chunk is 5MB by default, then 100
> buffers will use N*500 MB. It could reach 1GB if N > 2.
>
> In addition, normally, only if at least 100 producers sent messages to a
> partition would it be meaningful to configure maxPendingChunkedMessages to 
> 100.
> IMO, it's hard to see so many producers on a partition in production.

+1. I agree with you. And keeping the current default value in the
code (10) will not change the default behavior of the current client.
If there are no other objections, I would like to fix this
inconsistency in the java client.

Thanks,
Zike


On Sun, Jan 30, 2022 at 7:26 PM Yunze Xu  wrote:
>
> After thinking for a while, I’d prefer 10 as the default value and I changed
> the default value to 10 in C++ client, see
> https://github.com/apache/pulsar/pull/14070.
>
> A chunked buffer to contain all chunks could use much memory, for example, if 
> a
> message was split into N chunks, since each chunk is 5MB by default, then 100
> buffers will use N*500 MB. It could reach 1GB if N > 2.
>
> In addition, normally, only if at least 100 producers sent messages to a
> partition would it be meaningful to configure maxPendingChunkedMessages to 
> 100.
> IMO, it's hard to see so many producers on a partition in production.
>
> Thanks,
> Yunze Xu
>
> > 2022年1月30日 下午6:32，Zike Yang  写道：
> >
> > Hi, Pulsar community,
> >
> > We found that there are inconsistencies between the code and the
> > documentation regarding the default value of maxPendingChunkedMessage.
> >
> > In the java client code, we use 10 as the default value. [1] But in
> > the java doc, we use 100 as the default value. [2]
> > We need to fix this inconsistency. But what should we take as the
> > default value? From the code or the doc? I would like to hear your
> > discussions.
> >
> > [1] 
> > https://github.com/apache/pulsar/blob/d11147616aa6cc7888420f6325bb71cd7f7ab065/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConsumerConfigurationData.java#L112-L113
> > [2] 
> > https://github.com/apache/pulsar/blob/1e2ff8a3941b7cc6d583f528ceedc393b7e607fb/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java#L690
> >
> > Thanks,
> > Zike Yang
>

[DISCUSS] PIP-136: Sync Pulsar policies across multiple clouds

[Rajan Dhabalia]

Hi,

I would like to start a discussion about PIP-136: Sync Pulsar policies
across multiple clouds.

PIP documentation: https://github.com/apache/pulsar/issues/13728

*Motivation*
Apache Pulsar is a cloud-native, distributed messaging framework which
natively provides geo-replication. Many organizations deploy pulsar
instances on-prem and on multiple different cloud providers and at the same
time they would like to enable replication between multiple clusters
deployed in different cloud providers. Pulsar already provides various
proxy options (Pulsar proxy/ enterprise proxy solutions on SNI) to fulfill
security requirements when brokers are deployed on different security zones
connected with each other. However, sometimes it's not possible to share
metadata-store (global zookeeper) between pulsar clusters deployed on
separate cloud provider platforms, and synchronizing configuration metadata
(policies) can be a critical path to share tenant/namespace/topic policies
between clusters and administrate pulsar policies uniformly across all
clusters. Therefore, we need a mechanism to sync configuration metadata
between clusters deployed on the different cloud platforms.

*Sync Pulsar policies across multiple clouds*
https://github.com/apache/pulsar/issues/13728
Prototype git-hub-link

Thanks,
Rajan

[GitHub] [pulsar-helm-chart] lhotari opened a new pull request #220: Increase Zookeeper probe timeouts

[GitBox]

lhotari opened a new pull request #220:
URL: https://github.com/apache/pulsar-helm-chart/pull/220


   ### Motivation
   
   - 5 seconds seems to be a too short probe timeout on a system with low 
resources such as in CI
   
   ### Modifications
   
   - change default timeout to 30 seconds (which matches the probe's 
periodSeconds parameter)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org