Re: [VOTE] Pulsar Client C++ Release 3.1.0 Candidate 3

[PengHui Li]

+1 binding

- Verified the signatures
- Verified the Linux platform follows the documentation [0]
 - RPM
 - DEB
 - APK

[0] https://github.com/BewareMyPower/pulsar-client-cpp-demo

Thanks,
Penghui

On Fri, Dec 2, 2022 at 2:06 PM Matteo Merli  wrote:

> +1 binding.
>
> Checked:
>  * Source compiling and tested with perf producer
>  * Signatures
>  * RPM & Deb packages installing
>
>
> --
> Matteo Merli
> 
>
> On Thu, Dec 1, 2022 at 5:57 PM Zike Yang  wrote:
> >
> > This is the third release candidate for Apache Pulsar Client C++,
> > version 3.1.0.
> >
> > It fixes the following issues:
> > https://github.com/apache/pulsar-client-cpp/milestone/2?closed=1
> >
> > *** Please download, test and vote on this release. This vote will stay
> open
> > for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> >
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-cpp/pulsar-client-cpp-3.1.0-candidate-3/
> >
> > SHA-512 checksums:
> >
> a09cbf9c3e1208545ed9150a4ff6cba20cb29d01a3c8177aa9f17aa2285a080e19a4cd146a73e5bdbfd165562c4e63beb1aef7a3c655dd7fe01a9a92139dd0ac
> > ./apache-pulsar-client-cpp-3.1.0.tar.gz
> >
> > The tag to be voted upon:
> > v3.1.0-candidate-3 (f9cca17b5bb90819be78eabf921d60c99cf02a13)
> >
> https://github.com/apache/pulsar-client-cpp/releases/tag/v3.1.0-candidate-3
> >
> > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Please download the source package, and follow the README to compile and
> test.
> >
> > Thanks,
> > Zike Yang
>

Re: Support for Java LTS 17 (or at least 15+)

[Nicolò Boschi]

Frank,
Thanks for reporting, there may be several users hitting this common issue
without even being aware of it

My guess is that the bug you're facing is related to
https://bugs.openjdk.org/browse/JDK-8274349 and it has been fixed in
openjdk 11.0.17.

Pulsar 2.10.2 should be compatible with JDK 17 at runtime even though there
are no tests and it's recommended to test it in your dev environment.

Pulsar 2.11.0 and onwards will REQUIRE jdk 17 on the server side.
Pulsar 2.11.0 rc2 includes openjdk 17.0.5 that contains the above fix.

Cheers,
Nicolò Boschi


Il giorno sab 3 dic 2022 alle ore 10:03 Enrico Olivelli 
ha scritto:

> I think that Pulsar 2.10 should work on jdk17 but we have no automated
> tests on CI
>
> Enrico
>
> Il Ven 2 Dic 2022, 23:23 Frank Kelly  ha
> scritto:
>
> > Thanks - I see they switched to JRE 17 in 2.11
> >
> > https://github.com/apache/pulsar/pull/14355
> >
> > -Frank
> >
> >
> > On Fri, Dec 2, 2022 at 1:17 PM Dave Fisher  wrote:
> >
> > >
> > >
> > > > On Dec 2, 2022, at 8:22 AM, Frank Kelly
>  > >
> > > wrote:
> > > >
> > > > Hello Pulsar Devs,
> > > >
> > > >  We just hit an odd issue in the Proxy where when we set our K8S
> > > > `requests.cpu` exactly to "1" (or "1000m") the proxy thought it had
> > > access
> > > > to all 32 cores on the underlying VM and would not startup without a
> > > config
> > > > change.
> > > >
> > > > This seems to be an issue in Open JDK 11?
> > > > See
> > > > https://github.com/kubernetes/kubernetes/issues/84876
> > > >
> > >
> >
> https://stackoverflow.com/questions/58719842/kubernetes-pods-getting-different-cpu-limits
> > > >
> > > > As you can imagine this JDK bug might cause strange behavior also for
> > > > brokers/bookies etc. whose thread pools are bigger than expected. So
> > this
> > > > is partly just an FYI.
> > > >
> > > > Anyway an easy work-around (we set the requests.cpu to "999m") but
> just
> > > > wondering if there are plans to upgrade the JRE to 15+ (perhaps the
> > next
> > > > LTS 17?).
> > >
> > > You could try JDK17.
> > >
> > > Best,
> > > Dave
> > >
> > > >
> > > > Thanks!
> > > >
> > > > -Frank
> > > >
> > > > --
> > > >
> > > > Frank Kelly | Principal Engineer, Platform Team
> > > >
> > > > 100 High Street, 7th Floor, Boston, MA 02110 USA
> > > >
> > > > www.cogitocorp.com | Cogito on LinkedIn
> > > > 
> > > >
> > > > Confidentiality Notice : This email is the property of Cogito
> > > Corporation.
> > > > This message, including any attachments, is for the sole use of the
> > > > intended recipient(s) and may contain confidential and/or privileged
> > > > information. Any unauthorized review, use, disclosure or distribution
> > is
> > > > prohibited. If you are not the intended recipient, please contact the
> > > > sender by reply email and destroy all copies of the original message.
> > >
> > >
> >
>

[DISCUSS] Transactions isolation design

[Nicolò Boschi]

Hi folks,

I recently opened an issue about transactions [0]. The specific issue is
the client requires to be able to lookup the system topic
pulsar/system/transaction_coordinator_assign to get all the transaction
coordinators to dial with.

Since multi-tenancy is a core feature in Pulsar, this requirement may lead
to authorization issues in multi-tenant clusters breaking the tenant
isolation principle.

In this thread I'd like to discuss, more in general, the approach that has
been taken while designing transactions.
The main concern I have is that TCs are global.
A TC is backed by two system topics: transaction_coordinator_assign and
__transaction_log_x.
Both are used in the transaction's hot path with different workloads.
This leads to potential critical issues:

1. *if somehow one of these topics is unloaded, ALL the tenants using
transactions will suffer micro-outages*. (haven't looked at the error
handling but I suppose the error would be thrown in the client's face). In
general, availability and performance are not granted anymore
per-tenant/namespace.

I believe the TC should be per-tenant (maybe per-namespace?).
*Is there any strong reason why this shouldn't be possible by design?* (and
I mean, regardless of the current implementation and client-server
compatibility, we can handle them somehow but it's a detail atm)


One thing that I believe should be possible at the moment (but I'm not
sure) are cross-tenant transactions. This wouldn't be possible anymore with
per-tenant TC-

2. *the clients need lookup permission to get all the TCs*.
(transaction_coordinator_assign partitions). This can be solved in
different ways, even keeping using the TC as a system entity.

At the moment the java client, when starting, needs to get all the
available TCs to spread transactions over them. The call it does
is getPartitionedTopicMetadata to the system topic.
To fix this there are multiple ways:
a. Suggest to users to extend their own PulsarAuthorizationProvider to
always allow lookup to that particular topic. (quick, works with all the
existing clients and it only requires broker/proxy restarts without token
invalidations) However it's not builtin so this is not optimal. More
details here: [1]
b. Add a new auth action LOOKUP in order to allow cluster admins to give
this permission to their clients without affecting the produce or consume
ability. This would require only broker restarts plus operational costs for
the admin.
c. Creates a new specific endpoint (in the binary protocol) to give all the
required info to the TC client to properly initialize. This would be the
preferred solution because the permission would be granular to this
protocol call and it wouldn't require any permission changes for the
current applications. However, only new clients (and brokers) may use this
solution.

I believe the c. option would be great for the mid-term.
Anyway, if the per-tenant TC is designable, then this issue would be
resolved as well.


[0] https://github.com/apache/pulsar/issues/18716
[1] https://github.com/apache/pulsar/pull/18718


BR,
Nicolò Boschi

[DISCUSS][PIP-225] Pulsar Functions fetch parameters from local config file

[Yufei Zhang]

Hi all,

I opened a PIP to discuss: https://github.com/apache/pulsar/issues/18744

It allows Pulsar Function to read configs from a local config file, while
currently it only supports passing in configs from the command line and
cannot be changed dynamically. With reading configs from files it allows
easier configuration of Pulsar functions in cloud environment, In
particular, when Pulsar Functions are running in Kubernetes, passing in
specific data by updating the command line arguments will trigger a
reschedule (restart for pod) of the Pulsar Instance.

We proposed an implementation plan to support this feature, PTAL.

Thanks,
Yufei.

Re: [DISCUSS] Transactions isolation design

[PengHui Li]

Thanks for starting the discussion.

One transaction is allowed to work with multiple topics from multiple
tenants/namespaces.
And there are some real cases of injecting data from multiple tenants and
publishing
the calculated result to one or many topics under exactly-once semantics.

But I agree we should find a solution to replace the centralized
transaction coordinator.
One rough idea is to have a tenant/namespace for a transaction. It looks
like

Txn txn = new Txn("public/default")

which means the client will use the TC served by `public/default` namespace.
Then we can have permission to access the TC of the namespace.
The transaction can still work with multiple topics from multiple
namespaces/tenants.
Only the TC service or transaction log is isolated.

For the short-term solution. I think we can skip the lookup permission
check for the
transaction_assignment topic. We will not write any data on this topic.
Just using the
topic lookup to find where the TC is, get the service URL, and send the
transaction commands
to TC directly. But can we skip it in the broker? not the Authorization
Provider?
Otherwise, the user-defined Authorization Provider is also required to
apply the changes.

For option c, the broker and client must upgrade to the new version.
And in essence, they are(new API vs. lookup) almost the same, just
different APIs.

Thanks,
Penghui

On Mon, Dec 5, 2022 at 7:41 PM Nicolò Boschi  wrote:

> Hi folks,
>
> I recently opened an issue about transactions [0]. The specific issue is
> the client requires to be able to lookup the system topic
> pulsar/system/transaction_coordinator_assign to get all the transaction
> coordinators to dial with.
>
> Since multi-tenancy is a core feature in Pulsar, this requirement may lead
> to authorization issues in multi-tenant clusters breaking the tenant
> isolation principle.
>
> In this thread I'd like to discuss, more in general, the approach that has
> been taken while designing transactions.
> The main concern I have is that TCs are global.
> A TC is backed by two system topics: transaction_coordinator_assign and
> __transaction_log_x.
> Both are used in the transaction's hot path with different workloads.
> This leads to potential critical issues:
>
> 1. *if somehow one of these topics is unloaded, ALL the tenants using
> transactions will suffer micro-outages*. (haven't looked at the error
> handling but I suppose the error would be thrown in the client's face). In
> general, availability and performance are not granted anymore
> per-tenant/namespace.
>
> I believe the TC should be per-tenant (maybe per-namespace?).
> *Is there any strong reason why this shouldn't be possible by design?* (and
> I mean, regardless of the current implementation and client-server
> compatibility, we can handle them somehow but it's a detail atm)
>
>
> One thing that I believe should be possible at the moment (but I'm not
> sure) are cross-tenant transactions. This wouldn't be possible anymore with
> per-tenant TC-
>
> 2. *the clients need lookup permission to get all the TCs*.
> (transaction_coordinator_assign partitions). This can be solved in
> different ways, even keeping using the TC as a system entity.
>
> At the moment the java client, when starting, needs to get all the
> available TCs to spread transactions over them. The call it does
> is getPartitionedTopicMetadata to the system topic.
> To fix this there are multiple ways:
> a. Suggest to users to extend their own PulsarAuthorizationProvider to
> always allow lookup to that particular topic. (quick, works with all the
> existing clients and it only requires broker/proxy restarts without token
> invalidations) However it's not builtin so this is not optimal. More
> details here: [1]
> b. Add a new auth action LOOKUP in order to allow cluster admins to give
> this permission to their clients without affecting the produce or consume
> ability. This would require only broker restarts plus operational costs for
> the admin.
> c. Creates a new specific endpoint (in the binary protocol) to give all the
> required info to the TC client to properly initialize. This would be the
> preferred solution because the permission would be granular to this
> protocol call and it wouldn't require any permission changes for the
> current applications. However, only new clients (and brokers) may use this
> solution.
>
> I believe the c. option would be great for the mid-term.
> Anyway, if the per-tenant TC is designable, then this issue would be
> resolved as well.
>
>
> [0] https://github.com/apache/pulsar/issues/18716
> [1] https://github.com/apache/pulsar/pull/18718
>
>
> BR,
> Nicolò Boschi
>

Re: [VOTE] Pulsar Release 2.11.0 Candidate-2

[Enrico Olivelli]

FYI I am investigating some errors I see while running the Jakarta JMS
TCK against a Pulsar 2.11.0rc2 server with the provided docker images.

I still can't understand the problem, so I will keep you posted.

Please hold on

Enrico

Il giorno dom 4 dic 2022 alle ore 19:13 Christophe Bornet
 ha scritto:
>
> +1 (non-binding)
>
> - Start docker image standalone
> - Test HTTP Sink
> - Test Transformation Function
>
> Best regards
>
> Christophe
>
> Le lun. 28 nov. 2022 à 13:57, guo jiwei  a écrit :
>
> > This is the second release candidate for Apache Pulsar, version 2.11.0.
> >
> > This release contains 1574 commits by 64 contributors.
> > https://github.com/apache/pulsar/compare/v2.10.2...v2.11.0-candidate-2
> >
> > CI for this release candidate
> > https://github.com/Technoboy-/pulsar/pull/14
> >
> > *** Please download, test and vote on this release. This vote will stay
> > open
> > for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.0-candidate-2
> >
> > SHA-512 checksums:
> >
> >
> > 199eb731bad0635fbc0388adf090e8c60ce189634e6b1692a12cfdf43f15347371d79c42d040b91dd7b2d0302061ad48bff850638e3bb48540ab24908792dd5e
> >
> >  ./apache-pulsar-2.11.0-bin.tar.gz
> >
> >
> > 035d5a656cdd4c2430ae4f7c4fef952ad05e432219a64474fcfaa097fcc7ad6cd76fd82b77e91d667e0ec8495c35fc08a5bb25a54058ccdfaf4072a22b94f6d7
> >
> >  ./apache-pulsar-2.11.0-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1192/
> > 
> >
> > The tag to be voted upon:
> > v2.11.0-candidate-2 (c0a823a242a834dac35b9a6fcd6a2064a0e4bfb5)
> > https://github.com/apache/pulsar/releases/tag/v2.11.0-candidate-2
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Docker images:
> >
> > https://hub.docker.com/layers/mattison/pulsar-all/2.11.0-rc2/images/sha256-1b6b04673d62ccefe6fd6eed27a001ef8a9607668fcc5fb79459fed73f07e9e4
> >
> > https://hub.docker.com/layers/mattison/pulsar/2.11.0-rc2/images/sha256-21a4afa5a83ed5780c2d4986a3e13bc9d43142614f610434d5d304c1d61a65eb
> >
> > Please download the source package, and follow the
> > release-candidate-validation doc to build
> > and run the Pulsar standalone service.
> > https://pulsar.apache.org/contribute/validate-release-candidate
> >
> > Since the metadata store is changed from ZK to RocksDB, the verification of
> > the `stateful functions` needs to set the parameter "export
> > PULSAR_STANDALONE_USE_ZOOKEEPER=1"
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >

Re: Support for Java LTS 17 (or at least 15+)

[Nicolò Boschi]

Sorry I meant this JDK issue: https://bugs.openjdk.org/browse/JDK-8281181
Nicolò Boschi


Il giorno lun 5 dic 2022 alle ore 11:15 Nicolò Boschi 
ha scritto:

> Frank,
> Thanks for reporting, there may be several users hitting this common issue
> without even being aware of it
>
> My guess is that the bug you're facing is related to
> https://bugs.openjdk.org/browse/JDK-8274349 and it has been fixed in
> openjdk 11.0.17.
>
> Pulsar 2.10.2 should be compatible with JDK 17 at runtime even though
> there are no tests and it's recommended to test it in your dev environment.
>
> Pulsar 2.11.0 and onwards will REQUIRE jdk 17 on the server side.
> Pulsar 2.11.0 rc2 includes openjdk 17.0.5 that contains the above fix.
>
> Cheers,
> Nicolò Boschi
>
>
> Il giorno sab 3 dic 2022 alle ore 10:03 Enrico Olivelli <
> eolive...@gmail.com> ha scritto:
>
>> I think that Pulsar 2.10 should work on jdk17 but we have no automated
>> tests on CI
>>
>> Enrico
>>
>> Il Ven 2 Dic 2022, 23:23 Frank Kelly  ha
>> scritto:
>>
>> > Thanks - I see they switched to JRE 17 in 2.11
>> >
>> > https://github.com/apache/pulsar/pull/14355
>> >
>> > -Frank
>> >
>> >
>> > On Fri, Dec 2, 2022 at 1:17 PM Dave Fisher  wrote:
>> >
>> > >
>> > >
>> > > > On Dec 2, 2022, at 8:22 AM, Frank Kelly
>> > > >
>> > > wrote:
>> > > >
>> > > > Hello Pulsar Devs,
>> > > >
>> > > >  We just hit an odd issue in the Proxy where when we set our K8S
>> > > > `requests.cpu` exactly to "1" (or "1000m") the proxy thought it had
>> > > access
>> > > > to all 32 cores on the underlying VM and would not startup without a
>> > > config
>> > > > change.
>> > > >
>> > > > This seems to be an issue in Open JDK 11?
>> > > > See
>> > > > https://github.com/kubernetes/kubernetes/issues/84876
>> > > >
>> > >
>> >
>> https://stackoverflow.com/questions/58719842/kubernetes-pods-getting-different-cpu-limits
>> > > >
>> > > > As you can imagine this JDK bug might cause strange behavior also
>> for
>> > > > brokers/bookies etc. whose thread pools are bigger than expected. So
>> > this
>> > > > is partly just an FYI.
>> > > >
>> > > > Anyway an easy work-around (we set the requests.cpu to "999m") but
>> just
>> > > > wondering if there are plans to upgrade the JRE to 15+ (perhaps the
>> > next
>> > > > LTS 17?).
>> > >
>> > > You could try JDK17.
>> > >
>> > > Best,
>> > > Dave
>> > >
>> > > >
>> > > > Thanks!
>> > > >
>> > > > -Frank
>> > > >
>> > > > --
>> > > >
>> > > > Frank Kelly | Principal Engineer, Platform Team
>> > > >
>> > > > 100 High Street, 7th Floor, Boston, MA 02110 USA
>> > > >
>> > > > www.cogitocorp.com | Cogito on LinkedIn
>> > > > 
>> > > >
>> > > > Confidentiality Notice : This email is the property of Cogito
>> > > Corporation.
>> > > > This message, including any attachments, is for the sole use of the
>> > > > intended recipient(s) and may contain confidential and/or privileged
>> > > > information. Any unauthorized review, use, disclosure or
>> distribution
>> > is
>> > > > prohibited. If you are not the intended recipient, please contact
>> the
>> > > > sender by reply email and destroy all copies of the original
>> message.
>> > >
>> > >
>> >
>>
>

Re: [Vote] PIP-223: Add metrics for all Rest Endpoints

[Jiuming Tao]

bump

Jiuming Tao  于2022年11月28日周一 16:47写道：

> Dear Pulsar Community,
>
> Please review and vote on this PIP.
>
> PIP link: https://github.com/apache/pulsar/issues/18560
>
> Discuss thread:
> https://lists.apache.org/thread/z74vcn0yolzzrcc4ftonm9j3nbk4pzxm
>
> Thanks,
> Tao Jiuming
>

Re: [Vote] PIP-223: Add metrics for all Rest Endpoints

[mattisonchao]

+1

Best,
Mattison
On Nov 28, 2022, 16:47 +0800, Jiuming Tao , 
wrote:
> Dear Pulsar Community,
>
> Please review and vote on this PIP.
>
> PIP link: https://github.com/apache/pulsar/issues/18560
>
> Discuss thread:
> https://lists.apache.org/thread/z74vcn0yolzzrcc4ftonm9j3nbk4pzxm
>
> Thanks,
> Tao Jiuming

Re: [Vote] PIP-223: Add metrics for all Rest Endpoints

[PengHui Li]

+1 binding

Penghui

> On Dec 6, 2022, at 09:34, mattisonc...@gmail.com wrote:
> 
> +1
> 
> Best,
> Mattison
> On Nov 28, 2022, 16:47 +0800, Jiuming Tao , 
> wrote:
>> Dear Pulsar Community,
>> 
>> Please review and vote on this PIP.
>> 
>> PIP link: https://github.com/apache/pulsar/issues/18560
>> 
>> Discuss thread:
>> https://lists.apache.org/thread/z74vcn0yolzzrcc4ftonm9j3nbk4pzxm
>> 
>> Thanks,
>> Tao Jiuming