[GitHub] [pulsar-client-node] massakam commented on a change in pull request #138: Support end to end encryption for node client

[GitBox]

massakam commented on a change in pull request #138:
URL: https://github.com/apache/pulsar-client-node/pull/138#discussion_r559977594



##
File path: tests/end_to_end.test.js
##
@@ -665,5 +669,87 @@ const Pulsar = require('../index.js');
   await consumer.close();
   await client.close();
 });
+test('Basic produce and consume encryption', async () => {
+  const client = new Pulsar.Client({
+serviceUrl: 'pulsar://localhost:6650',
+operationTimeoutSeconds: 30,
+  });
+
+  const topic = 'persistent://public/default/encryption-produce-consume';
+  const producer = await client.createProducer({
+topic,
+sendTimeoutMs: 3,
+batchingEnabled: true,
+publicKeyPath: './certificate/public-rsa.pem',
+privateKeyPath: './certificate/private-rsa.pem',
+encryptionKey: 'encryption-key',
+  });
+
+  const consumer = await client.subscribe({
+topic,
+subscription: 'sub1',
+subscriptionType: 'Shared',
+ackTimeoutMs: 1,
+publicKeyPath: './public-rsa.pem',
+privateKeyPath: './private-rsa.pem',

Review comment:
   Are these paths correct?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-node] massakam commented on a change in pull request #138: Support end to end encryption for node client

[GitBox]

massakam commented on a change in pull request #138:
URL: https://github.com/apache/pulsar-client-node/pull/138#discussion_r559980339



##
File path: tests/end_to_end.test.js
##
@@ -665,5 +669,87 @@ const Pulsar = require('../index.js');
   await consumer.close();
   await client.close();
 });
+test('Basic produce and consume encryption', async () => {
+  const client = new Pulsar.Client({
+serviceUrl: 'pulsar://localhost:6650',
+operationTimeoutSeconds: 30,
+  });
+
+  const topic = 'persistent://public/default/encryption-produce-consume';
+  const producer = await client.createProducer({
+topic,
+sendTimeoutMs: 3,
+batchingEnabled: true,
+publicKeyPath: './certificate/public-rsa.pem',
+privateKeyPath: './certificate/private-rsa.pem',
+encryptionKey: 'encryption-key',
+  });
+
+  const consumer = await client.subscribe({
+topic,
+subscription: 'sub1',
+subscriptionType: 'Shared',
+ackTimeoutMs: 1,
+publicKeyPath: './public-rsa.pem',
+privateKeyPath: './private-rsa.pem',
+  });
+
+  const messages = [];
+  for (let i = 0; i < 10; i += 1) {
+const msg = `my-message-${i}`;
+producer.send({
+  data: Buffer.from(msg),
+});
+messages.push(msg);
+  }
+  await producer.flush();
+
+  const results = [];
+  for (let i = 0; i < 10; i += 1) {
+const msg = await consumer.receive();
+consumer.acknowledge(msg);
+results.push(msg.getData().toString());
+  }
+  expect(lodash.difference(messages, results)).toEqual([]);
+  await producer.close();
+  await consumer.close();
+  await client.close();
+});
+
+test('Failed produce and consume encryption', async () => {

Review comment:
   Is this test case necessary? It doesn't seem to be a test related to 
encryption.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] Pulsar Release 2.6.3 Candidate 2

[Jia Zhai]

+1 binding


OS: MacOS 10.15.6

- verified packages checksum and signatures ( asc and sha all good)
- the source package rat check(mvn apache-rat:check), licence check,  build
and unit test all run successfully.
- bin package build from source, 'bin/pulsar standalone' and client
produce/consume runs well.
- bin package, 'bin/pulsar standalone' and client produce/consume runs well.

Best Regards.


Jia Zhai

Beijing, China

Mobile: +86 15810491983




On Mon, Jan 18, 2021 at 6:30 PM Enrico Olivelli  wrote:

> +1 (non binding)
> - checked checksums/signatures
> - built from sources with JDK8 on Mac (a few flaky tests)
> - run produce/consume smoke tests with pulsar-per from the binaries (on
> JDK8 + Mac + Pulsar standalone)
>
> My concerns about the differences between the source tarball has been
> addressed in a PR, but we are going to
> commit that change only on master branch, this is a bugfix release, it is
> not worth to add risks to the release.
>
> Enrico
>
>
> Il giorno lun 18 gen 2021 alle ore 10:23 Yuto Furuta <
> yfur...@yahoo-corp.jp>
> ha scritto:
>
> > +1 (non binding)
> >
> >  - Check checksums
> >  - Validate Produce/Consume
> >  - Validate Pulsar IO
> >  -- pulsar-io-cassandra-2.6.3.nar
> >
> > Regards,
> > Yuto
> > --
> > Yuto Furuta
> > yfur...@yahoo-corp.jp
> > 
> > 差出人: Hiroyuki Sakai 
> > 送信日時: 2021年1月18日 16:03
> > 宛先: Dev 
> > 件名: Re: [VOTE] Pulsar Release 2.6.3 Candidate 2
> >
> > +1 (binding)
> >
> >  - check signatures/checksums
> >  - Validate Pub/Sub and Java Functions
> >  - Validate Stateful Functions
> >
> > Regards,
> > Hiroyuki
> >
> >
> >
> > 
> > From: PengHui Li 
> > Sent: Wednesday, January 13, 2021 20:42
> > To: Dev 
> > Subject: Re: [VOTE] Pulsar Release 2.6.3 Candidate 2
> >
> > Sorry, forget to update the link address, you can copy the address
> directly
> > for validation.
> >
> > Thanks,
> > Penghui
> >
> > PengHui Li  于2021年1月13日周三 下午7:35写道：
> >
> > > This is the second release candidate for Apache Pulsar, version 2.6.3.
> > >
> > > It fixes the following issues:
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.6.3+is%3Aclosed
> > >
> > > *** Please download, test and vote on this release. This vote will stay
> > open
> > > for at least 72 hours ***
> > >
> > > Note that we are voting upon the source (tag), binaries are provided
> for
> > > convenience.
> > >
> > > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.6.3-candidate-2/
> <
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.6.3-candidate-1/>
> > >
> > > SHA-1 checksums:
> > >
> > > e620672cb560b5947069e4e22be9b0e1bd08c327
> apache-pulsar-2.6.3-src.tar.gz
> > > d1f3f9907ce54c002a7ec0a73dbe84be933cb4d1
> apache-pulsar-2.6.3-bin.tar.gz
> > >
> > > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1078/
> <
> > https://repository.apache.org/content/repositories/orgapachepulsar-1077/
> >
> > >
> > > The tag to be voted upon:
> > > v2.6.3-candidate-2 (801ba85350f9a1c5dff9a7288c1f8fc9c22d8868)
> > https://github.com/apache/pulsar/releases/tag/v2.6.3-candidate-2
> > >
> > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/release/pulsar/KEYS
> > >
> > > Please download the the source package, and follow the README to build
> > > and run the Pulsar standalone service.
> > >
> > >
> >
>

[GitHub] [pulsar-helm-chart] yufan022 opened a new issue #94: Broker ClusterRole resources bug

[GitBox]

yufan022 opened a new issue #94:
URL: https://github.com/apache/pulsar-helm-chart/issues/94


   **Describe the bug**
   Get some error log on k8s.
   ```
   10:33:59.441 [Timer-0] ERROR 
org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory - Error 
while trying to fetch configmap pulsar-mini-functions-worker-config at 
namespace default
   io.kubernetes.client.openapi.ApiException: Forbidden
at 
io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:971) 
~[io.kubernetes-client-java-api-9.0.2.jar:?]
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:883) 
~[io.kubernetes-client-java-api-9.0.2.jar:?]
at 
io.kubernetes.client.openapi.apis.CoreV1Api.readNamespacedConfigMapWithHttpInfo(CoreV1Api.java:44821)
 ~[io.kubernetes-client-java-api-9.0.2.jar:?]
at 
io.kubernetes.client.openapi.apis.CoreV1Api.readNamespacedConfigMap(CoreV1Api.java:44791)
 ~[io.kubernetes-client-java-api-9.0.2.jar:?]
at 
org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory.fetchConfigMap(KubernetesRuntimeFactory.java:369)
 [org.apache.pulsar-pulsar-functions-runtime-2.7.0.jar:2.7.0]
at 
org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory$1.run(KubernetesRuntimeFactory.java:358)
 [org.apache.pulsar-pulsar-functions-runtime-2.7.0.jar:2.7.0]
at java.util.TimerThread.mainLoop(Timer.java:555) [?:1.8.0_275]
at java.util.TimerThread.run(Timer.java:505) [?:1.8.0_275]
   ```
   
   **To Reproduce**
   Steps to reproduce the behavior:
   1. deploy pulsar on k8s
   
   **Expected behavior**
   no forbidden
   
   **Desktop (please complete the following information):**
- OS: k8s
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] yufan022 opened a new pull request #95: fix broker configmap forbidden

[GitBox]

yufan022 opened a new pull request #95:
URL: https://github.com/apache/pulsar-helm-chart/pull/95


   Fixes #94
   
   ### Motivation
   
   fix `io.kubernetes.client.openapi.ApiException: Forbidden`
   
   ### Modifications
   
   fix typo
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PIP-78] Split the individual acknowledgments into multiple entries

[Joe F]

I have a simpler question. Just storing the message-ids raw will fit ~300K
entries in one ledger entry. With the bitmap  changes, we can store a
couple of million  within one 5MB ledger entry.  So can you tell us what
numbers of unacked messages are  creating a problem?  What exactly are the
issues you face, and at what numbers of unacked messages/memory use etc?

I have my own concerns about this proposal, but I would like to understand
the problem first

Joe

On Sun, Jan 17, 2021 at 10:16 PM Sijie Guo  wrote:

> Hi Lin,
>
> Thanks you and Penghui for drafting this! We have seen a lot of pain points
> of `managedLedgerMaxUnackedRangesToPersist` when enabling delayed messages.
> Glad that you and Penghui are spending time on resolving this!
>
> Overall the proposal looks good. But I have a couple of questions about the
> proposal.
>
> 1. What happens if the broker fails to write the entry marker? For example,
> at t0, the broker flushes dirty pages and successfully writes an entry
> marker. At t1, the broker tries to flushes dirty pages but failed to write
> the new entry marker. How can you recover the entry marker?
>
> 2.  When a broker crashes and recovers the managed ledger, the cursor
> ledger is not writable anymore. Are you going to create a new cursor ledger
> and copy all the entries from the old cursor ledger to the new one?
>
> It would be good if you can clarify these two questions.
>
> - Sijie
>
> On Sun, Jan 17, 2021 at 9:48 PM linlin  wrote:
>
> > Hi, community:
> > Recently we encountered some problems when using individual
> > acknowledgments, such as:
> > when the amount of acknowledgment is large, entry writing fails; a large
> > amount of cache causes OOM, etc.
> > So I drafted a PIP in `
> >
> >
> https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBcGYZbqK8XsVJ4qkU/edit?usp=sharing`
> 
> > <
> https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBcGYZbqK8XsVJ4qkU/edit?usp=sharing
> >
> > ,
> > any voice is welcomed.
> >
>