[DISCUSS] Strategy for pushing the latest tag to Pulsar docker images
Hi, everyone, There seems to be a gap in our current release process concerning the pushing of the latest tag to our Docker images. Specifically, we need to decide which version of Pulsar the latest tag should point to. I have come up with two options: 1. The latest tag could point to the most recent feature release or the subsequent patch of that feature release. For instance, it could currently point to 3.1.1, and in the future, it could point to 3.1.2 or 3.2.0. 2. The latest tag could point to the most recent LTS release or the subsequent patch of that LTS release. As an example, it could currently point to 3.0.2, and in the future, it could point to 3.0.3 or 4.0.0. What do you think? I would prefer the option 1. I have pushed a PR to update the release docker image process: https://github.com/apache/pulsar-site/pull/745. PTAL and leave your comments and feedback here. Thanks, Zike Yang
Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters
Hi Joe, You're correct. The initial design of the replication policy leaves room for improvement. To address this, we aim to refine the cluster settings at the namespace level in a way that won't impact the existing system. The replication clusters should solely be used to establish full mesh replication for that specific namespace, without having any other definitions or functionalities. BR, Xiangying On Mon, Dec 4, 2023 at 1:52 PM Joe F wrote: > >if users want to change the replication policy for > topic-n and do not change the replication policy of other topics, they need > to change all the topic policy under this namespace. > > This PIP unfortunately flows from attempting to solve bad application > design > > A namespace is supposed to represent an application, and the namespace > policy is an umbrella for a similar set of policies that applies to all > topics. The exceptions would be if a topic had a need for a deficit, The > case of one topic in the namespace sticking out of the namespace policy > umbrella is bad application design in my opinion > > -Joe. > > > > On Sun, Dec 3, 2023 at 6:00 PM Xiangying Meng > wrote: > > > Hi Rajan and Girish, > > Thanks for your reply. About the question you mentioned, there is some > > information I want to share with you. > > >If anyone wants to setup different replication clusters then either > > >those topics can be created under different namespaces or defined at > topic > > >level policy. > > > > >And users can anyway go and update the namespace's cluster list to add > the > > >missing cluster. > > Because the replication clusters also mean the clusters where the topic > can > > be created or loaded, the topic-level replication clusters can only be > the > > subset of namespace-level replication clusters. > > Just as Girish mentioned, the users can update the namespace's cluster > list > > to add the missing cluster. > > But there is a problem because the replication clusters as the namespace > > level will create a full mesh replication for that namespace across the > > clusters defined in > > replication-clusters if users want to change the replication policy for > > topic-n and do not change the replication policy of other topics, they > need > > to change all the topic policy under this namespace. > > > > > Pulsar is being used by many legacy systems and changing its > > >semantics for specific usecases without considering consequences are > > >creating a lot of pain and incompatibility problems for other existing > > >systems and let's avoid doing it as we are struggling with such changes > > and > > >breaking compatibility or changing semantics are just not acceptable. > > > > This proposal will not introduce an incompatibility problem, because the > > default value of the namespace policy of allowed-clusters and > > topic-policy-synchronized-clusters are the replication-clusters. > > > > >Allowed clusters defined at tenant level > > >will restrict tenants to create namespaces in regions/clusters where > they > > >are not allowed. > > >As Rajan also mentioned, allowed-clusters field has a different > > meaning/purpose. > > > > Allowed clusters defined at the tenant level will restrict tenants from > > creating namespaces in regions/clusters where they are not allowed. > > Similarly, the allowed clusters defined at the namespace level will > > restrict the namespace from creating topics in regions/clusters where > they > > are not allowed. > > What's wrong with this? > > > > Regards, > > Xiangying > > > > On Fri, Dec 1, 2023 at 2:35 PM Girish Sharma > > wrote: > > > > > Hi Xiangying, > > > > > > Shouldn't the solution to the issue mentioned in #21564 [0] mostly be > > > around validating that topic level replication clusters are subset of > > > namespace level replication clusters? > > > It would be a completely compatible change as even today the case > where a > > > topic has a cluster not defined in namespaces's replication-clusters > > > doesn't really work. > > > And users can anyway go and update the namespace's cluster list to add > > the > > > missing cluster. > > > > > > As Rajan also mentioned, allowed-clusters field has a different > > > meaning/purpose. > > > Regards > > > > > > On Thu, Nov 30, 2023 at 10:56 AM Xiangying Meng > > > wrote: > > > > > > > Hi, Pulsar Community > > > > > > > > I drafted a proposal to make the configuration of clusters at the > > > namespace > > > > level clearer. This helps solve the problem of geo-replication not > > > working > > > > correctly at the topic level. > > > > > > > > https://github.com/apache/pulsar/pull/21648 > > > > > > > > I'm looking forward to hearing from you. > > > > > > > > BR > > > > Xiangying > > > > > > > > > > > > > -- > > > Girish Sharma > > > > > >
Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters
Hello Xiangying, I believe what Joe here is referring to as "application design" is not the design of pulsar or namespace level replication but the design of your application and the dependency that you've put on topic level replication. In general, I am aligned with Joe from an application design standpoint. A namespace is supposed to represent a single application use case, topic level override of replication clusters helps in cases where there are a few exceptional topics which do not need replication in all of the namespace clusters. This helps in saving network bandwidth, storage, CPU, RAM etc But the reason why you've raised this PIP is to bring down the actual replication semantics at a topic level. Yes, namespace level still exists as per your PIP as well, but is basically left only to be a "default in case topic level is missing". This brings me to a very basic question - What's the use case that you are trying to solve that needs these changes? Because, then what's stopping us from bringing every construct that's at a namespace level (bundling, hardware affinity, etc) down to a topic level? Regards On Tue, Dec 5, 2023 at 2:52 PM Xiangying Meng wrote: > Hi Joe, > > You're correct. The initial design of the replication policy leaves room > for improvement. To address this, we aim to refine the cluster settings at > the namespace level in a way that won't impact the existing system. The > replication clusters should solely be used to establish full mesh > replication for that specific namespace, without having any other > definitions or functionalities. > > BR, > Xiangying > > > On Mon, Dec 4, 2023 at 1:52 PM Joe F wrote: > > > >if users want to change the replication policy for > > topic-n and do not change the replication policy of other topics, they > need > > to change all the topic policy under this namespace. > > > > This PIP unfortunately flows from attempting to solve bad application > > design > > > > A namespace is supposed to represent an application, and the namespace > > policy is an umbrella for a similar set of policies that applies to all > > topics. The exceptions would be if a topic had a need for a deficit, The > > case of one topic in the namespace sticking out of the namespace policy > > umbrella is bad application design in my opinion > > > > -Joe. > > > > > > > > On Sun, Dec 3, 2023 at 6:00 PM Xiangying Meng > > wrote: > > > > > Hi Rajan and Girish, > > > Thanks for your reply. About the question you mentioned, there is some > > > information I want to share with you. > > > >If anyone wants to setup different replication clusters then either > > > >those topics can be created under different namespaces or defined at > > topic > > > >level policy. > > > > > > >And users can anyway go and update the namespace's cluster list to add > > the > > > >missing cluster. > > > Because the replication clusters also mean the clusters where the topic > > can > > > be created or loaded, the topic-level replication clusters can only be > > the > > > subset of namespace-level replication clusters. > > > Just as Girish mentioned, the users can update the namespace's cluster > > list > > > to add the missing cluster. > > > But there is a problem because the replication clusters as the > namespace > > > level will create a full mesh replication for that namespace across the > > > clusters defined in > > > replication-clusters if users want to change the replication policy for > > > topic-n and do not change the replication policy of other topics, they > > need > > > to change all the topic policy under this namespace. > > > > > > > Pulsar is being used by many legacy systems and changing its > > > >semantics for specific usecases without considering consequences are > > > >creating a lot of pain and incompatibility problems for other existing > > > >systems and let's avoid doing it as we are struggling with such > changes > > > and > > > >breaking compatibility or changing semantics are just not acceptable. > > > > > > This proposal will not introduce an incompatibility problem, because > the > > > default value of the namespace policy of allowed-clusters and > > > topic-policy-synchronized-clusters are the replication-clusters. > > > > > > >Allowed clusters defined at tenant level > > > >will restrict tenants to create namespaces in regions/clusters where > > they > > > >are not allowed. > > > >As Rajan also mentioned, allowed-clusters field has a different > > > meaning/purpose. > > > > > > Allowed clusters defined at the tenant level will restrict tenants from > > > creating namespaces in regions/clusters where they are not allowed. > > > Similarly, the allowed clusters defined at the namespace level will > > > restrict the namespace from creating topics in regions/clusters where > > they > > > are not allowed. > > > What's wrong with this? > > > > > > Regards, > > > Xiangying > > > > > > On Fri, Dec 1, 2023 at 2:35 PM Girish Sharma > > > wrote: > > > > > > > Hi Xiangying, > > >
Re: [ANNOUNCE] Apache Pulsar Client C++ 3.2.0 released
I focused on other projects in the last few months so I might not have the plan recently. If someone else is interested in implementing the transaction feature for the C++ client, welcome your contributions. Thanks, Yunze On Thu, Nov 30, 2023 at 7:55 PM Karthikeyan A wrote: > > Hi Team, > > Any updates on including transaction support on C++ client ? > > Thanks & Regards > Karthikeyan A > > -Original Message- > From: Yunze Xu > Sent: 17 May 2023 08:33 > To: dev@pulsar.apache.org > Subject: Re: [ANNOUNCE] Apache Pulsar Client C++ 3.2.0 released > > I have a plan to support the transaction on C++ client next month (June). It > might be included in the 3.3.0 release. > > Thanks, > Yunze > > On Tue, May 16, 2023 at 10:09 PM Karthikeyan A > wrote: > > > > Hi, > > > > We have a requirement where the message has been consumed from one topic, > > processed it and written to another topic in single transaction. > > Looks like transaction support related client APIs in C++ is still not > > available and that is in roadmap of pulsar delivery. > > > > Please let us know the details on when the C++ client library for > > Transaction Support will be available for use. > > > > Thanks & Regards > > Karthikeyan A > > > > > > -Original Message- > > From: Yunze Xu > > Sent: 16 May 2023 18:17 > > To: dev@pulsar.apache.org; us...@pulsar.apache.org; > > annou...@apache.org > > Subject: [ANNOUNCE] Apache Pulsar Client C++ 3.2.0 released > > > > The Apache Pulsar team is proud to announce Apache Pulsar Client C++ > > version 3.2.0. > > > > Pulsar is a highly scalable, low latency messaging platform running on > > commodity hardware. It provides simple pub-sub semantics over topics, > > guaranteed at-least-once delivery of messages, automatic cursor management > > for subscribers, and cross-datacenter replication. > > > > For Pulsar release details and downloads, visit: > > > > https://pulsar.apache.org/download > > > > Release Notes are at: > > https://pulsar.apache.org/release-notes/client-cpp/ > > > > We would like to thank the contributors that made the release possible. > > > > Regards, > > > > The Pulsar Team
Re: [DISCUSS] Strategy for pushing the latest tag to Pulsar docker images
I prefer the 1st way as well. Thanks, Yunze On Tue, Dec 5, 2023 at 5:11 PM Zike Yang wrote: > > Hi, everyone, > > There seems to be a gap in our current release process concerning the > pushing of the latest tag to our Docker images. Specifically, we need > to decide which version of Pulsar the latest tag should point to. > > I have come up with two options: > 1. The latest tag could point to the most recent feature release or > the subsequent patch of that feature release. For instance, it could > currently point to 3.1.1, and in the future, it could point to 3.1.2 > or 3.2.0. > 2. The latest tag could point to the most recent LTS release or the > subsequent patch of that LTS release. As an example, it could > currently point to 3.0.2, and in the future, it could point to 3.0.3 > or 4.0.0. > > What do you think? I would prefer the option 1. I have pushed a PR to > update the release docker image process: > https://github.com/apache/pulsar-site/pull/745. > PTAL and leave your comments and feedback here. > > Thanks, > Zike Yang
[Discuss] Release C++ Client 3.4.2
Hi all, There is a regression related to OAuth2 authentication for C++ Client 3.4.0 and 3.4.1: https://github.com/apache/pulsar-client-cpp/issues/363 Generally, for C++ client users, you won't be affected if you - don't use OAuth2 authentication - or build the library from source in your platform - or use the RPM/DEB/APK pre-built libraries If you're using pre-built libraries for macOS or you built the Linux libraries in a specific platform (e.g. RedHat) but provided the libraries in various Linux platforms (e.g. Ubuntu), there would be no way to use OAuth2 authentication. Python and Node.js clients would also be affected because they rely on the customized `tlsTrustCertsFilePath` config that specifies the CA cert path for OAuth2 authentication. Therefore, I pushed a fix (https://github.com/apache/pulsar-client-cpp/pull/364) and will start the 3.4.2 release. Thanks, Yunze
Re: [DISCUSS] Release Pulsar Python clieng 3.4.0
Hi Zike, I found a regression [1] that would affect the Python or Node.js client if any of them depends on the C++ Client 3.4.1. So I will prepare a 3.4.2 release soon. Please wait until the C++ client 3.4.2 is released. [1] https://lists.apache.org/thread/rjolgrlp4x1lmfj678k3hjco80kcb73c Thanks, Yunze On Tue, Dec 5, 2023 at 1:38 PM Jia Zhai wrote: > > +1 > > On Mon, Dec 4, 2023 at 6:14 PM Enrico Olivelli wrote: > > > +1 > > > > Enrico > > > > Il giorno lun 4 dic 2023 alle ore 08:31 Zike Yang ha > > scritto: > > > > > > Hi all, > > > > > > I would like to propose releasing the Pulsar Python client 3.4.0. > > > > > > It's over three months since the release of 3.3.0, and there are 46 new > > commits: > > > https://github.com/apache/pulsar-client-python/compare/v3.0.0...main > > > > > > The Pulsar C++ client 3.4.1 has been released. I have upgraded the C++ > > > client for the Python client to 3.4.1: > > > https://github.com/apache/pulsar-client-python/pull/167 > > > > > > We need to cut a new release. Please let me know if you have any > > > important fixes that need to be included in Pulsar Python client 3.4.0. > > > > > > Thanks, > > > Zike Yang > >
[Discussion] (Extensible Load Balancer) Expose loadBalancerInFlightServiceUnitStateWaitingTimeInMillis and loadBalancerServiceUnitStateMonitorIntervalInSeconds to ServiceConfiguration
Hello Pulsar dev, I propose that ExtensibleLoadBalancer expose the following two configurations in the ServiceConfiguration. loadBalancerInFlightServiceUnitStateWaitingTimeInMillis=3 loadBalancerServiceUnitStateMonitorIntervalInSeconds=60 PR: https://github.com/apache/pulsar/pull/21668/files#diff-cc761e782083f37db72cd91684fee07b931c188dd9397c62b0a4c45a657eR2664-R2684 Currently, these are hardcoded in the ServiceUnitStateChannel class, but I think they can be exposed to the ServiceConfiguration for users to fine-tune the behavior. I updated the PIP-192 to list the ExtensibleLoadBalancer's Service Configurations. PIP-192: https://github.com/apache/pulsar/issues/16691 Regards, Heesung
Re: [Discussion] (Extensible Load Balancer) Expose loadBalancerInFlightServiceUnitStateWaitingTimeInMillis and loadBalancerServiceUnitStateMonitorIntervalInSeconds to ServiceConfiguration
Looks good to me. Thank you, Dragos > On Dec 5, 2023, at 9:36 AM, Heesung Sohn wrote: > > Hello Pulsar dev, > > I propose that ExtensibleLoadBalancer expose the following two > configurations in the ServiceConfiguration. > > loadBalancerInFlightServiceUnitStateWaitingTimeInMillis=3 > loadBalancerServiceUnitStateMonitorIntervalInSeconds=60 > > PR: > https://github.com/apache/pulsar/pull/21668/files#diff-cc761e782083f37db72cd91684fee07b931c188dd9397c62b0a4c45a657eR2664-R2684 > > Currently, these are hardcoded in the ServiceUnitStateChannel class, but I > think they can be exposed to the ServiceConfiguration for users to > fine-tune the behavior. I updated the PIP-192 to list the > ExtensibleLoadBalancer's Service Configurations. > > PIP-192: https://github.com/apache/pulsar/issues/16691 > > Regards, > Heesung
[ANNOUNCE] Apache Pulsar 3.0.2 released
The Apache Pulsar team is proud to announce Apache Pulsar version 3.0.2. Pulsar is a highly scalable, low-latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication. For Pulsar release details and downloads, visit: https://pulsar.apache.org/download Release Notes are at: https://pulsar.apache.org/release-notes We would like to thank the contributors that made the release possible. Regards, The Pulsar Team
Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters
Girish, Thank you for making my point much better than I did .. -Joe On Tue, Dec 5, 2023 at 1:45 AM Girish Sharma wrote: > Hello Xiangying, > > I believe what Joe here is referring to as "application design" is not the > design of pulsar or namespace level replication but the design of your > application and the dependency that you've put on topic level replication. > > In general, I am aligned with Joe from an application design standpoint. A > namespace is supposed to represent a single application use case, topic > level override of replication clusters helps in cases where there are a few > exceptional topics which do not need replication in all of the namespace > clusters. This helps in saving network bandwidth, storage, CPU, RAM etc > > But the reason why you've raised this PIP is to bring down the actual > replication semantics at a topic level. Yes, namespace level still exists > as per your PIP as well, but is basically left only to be a "default in > case topic level is missing". > This brings me to a very basic question - What's the use case that you are > trying to solve that needs these changes? Because, then what's stopping us > from bringing every construct that's at a namespace level (bundling, > hardware affinity, etc) down to a topic level? > > Regards > > On Tue, Dec 5, 2023 at 2:52 PM Xiangying Meng > wrote: > > > Hi Joe, > > > > You're correct. The initial design of the replication policy leaves room > > for improvement. To address this, we aim to refine the cluster settings > at > > the namespace level in a way that won't impact the existing system. The > > replication clusters should solely be used to establish full mesh > > replication for that specific namespace, without having any other > > definitions or functionalities. > > > > BR, > > Xiangying > > > > > > On Mon, Dec 4, 2023 at 1:52 PM Joe F wrote: > > > > > >if users want to change the replication policy for > > > topic-n and do not change the replication policy of other topics, they > > need > > > to change all the topic policy under this namespace. > > > > > > This PIP unfortunately flows from attempting to solve bad application > > > design > > > > > > A namespace is supposed to represent an application, and the namespace > > > policy is an umbrella for a similar set of policies that applies to > all > > > topics. The exceptions would be if a topic had a need for a deficit, > The > > > case of one topic in the namespace sticking out of the namespace policy > > > umbrella is bad application design in my opinion > > > > > > -Joe. > > > > > > > > > > > > On Sun, Dec 3, 2023 at 6:00 PM Xiangying Meng > > > wrote: > > > > > > > Hi Rajan and Girish, > > > > Thanks for your reply. About the question you mentioned, there is > some > > > > information I want to share with you. > > > > >If anyone wants to setup different replication clusters then either > > > > >those topics can be created under different namespaces or defined at > > > topic > > > > >level policy. > > > > > > > > >And users can anyway go and update the namespace's cluster list to > add > > > the > > > > >missing cluster. > > > > Because the replication clusters also mean the clusters where the > topic > > > can > > > > be created or loaded, the topic-level replication clusters can only > be > > > the > > > > subset of namespace-level replication clusters. > > > > Just as Girish mentioned, the users can update the namespace's > cluster > > > list > > > > to add the missing cluster. > > > > But there is a problem because the replication clusters as the > > namespace > > > > level will create a full mesh replication for that namespace across > the > > > > clusters defined in > > > > replication-clusters if users want to change the replication policy > for > > > > topic-n and do not change the replication policy of other topics, > they > > > need > > > > to change all the topic policy under this namespace. > > > > > > > > > Pulsar is being used by many legacy systems and changing its > > > > >semantics for specific usecases without considering consequences are > > > > >creating a lot of pain and incompatibility problems for other > existing > > > > >systems and let's avoid doing it as we are struggling with such > > changes > > > > and > > > > >breaking compatibility or changing semantics are just not > acceptable. > > > > > > > > This proposal will not introduce an incompatibility problem, because > > the > > > > default value of the namespace policy of allowed-clusters and > > > > topic-policy-synchronized-clusters are the replication-clusters. > > > > > > > > >Allowed clusters defined at tenant level > > > > >will restrict tenants to create namespaces in regions/clusters where > > > they > > > > >are not allowed. > > > > >As Rajan also mentioned, allowed-clusters field has a different > > > > meaning/purpose. > > > > > > > > Allowed clusters defined at the tenant level will restrict tenants > from > > > > creating namespaces in regions/clusters where the
