[GitHub] [pulsar-client-node] hrsakai closed issue #135: send delay message, but deliver immediately

[GitBox]

hrsakai closed issue #135:
URL: https://github.com/apache/pulsar-client-node/issues/135


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-node] hrsakai commented on issue #135: send delay message, but deliver immediately

[GitBox]

hrsakai commented on issue #135:
URL: 
https://github.com/apache/pulsar-client-node/issues/135#issuecomment-842938122


   @ilaipi 
We released v1.3.0, so closed this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[Java][Client] Important behaviour change on empty key from 2.7.x to 2.8.x

[Enrico Olivelli]

Hello,
I have found this behaviour in the Java Client while switching from 2.7 to 2.8.

Short version of the story:
- on 2.7.2 a null key is received as an empty key
- on 2.8.0 a null key is received as a null key

The behaviour of 2.8 is better, because it is what you expect.

But if you migrate an application from 2.7.2 to 2.8 you can start to
see Nulls instead of empty strings and this will lead to unpredictable
behaviour and possibly NullPointerExceptions.

We can accept the new behaviour but I would like to check if the
community is aware of this and this is acceptable.

This is the issue, with a reproducer
https://github.com/apache/pulsar/issues/10625

Enrico

[GitHub] [pulsar-client-node] k2la opened a new pull request #154: [Do Not Merge]This is a test

[GitBox]

k2la opened a new pull request #154:
URL: https://github.com/apache/pulsar-client-node/pull/154


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] klaus-pd opened a new issue #120: pulsar deploy in k8s zookeeper error

[GitBox]

klaus-pd opened a new issue #120:
URL: https://github.com/apache/pulsar-helm-chart/issues/120


   **Describe the bug**
   The Pulsar cluster was deployed on K8S using the officially provided Helm 
cluster, but the ZooKeeper cluster deployment could not complete the 
initialization of the cluster
   
   https://user-images.githubusercontent.com/9957221/118625702-d1892600-b7fc-11eb-9880-97511025c7ff.png";>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Pulsar WebSite Builder is failing

[Enrico Olivelli]

Hello,
we are still stuck.

the website builder does not work.
This time is because the build image picks up Python 3.5.2 and so the
job is not able to build the Python client that apparently cannot be
compiled with that version.

I tried to tweak the build image in order to force Python 3.9 but
without success.

I cannot announce 2.7.2 release, and also all the website/docs
improvements are stuck

Any help is appreciated
:-)

Enrico

Il giorno ven 14 mag 2021 alle ore 14:33 Enrico Olivelli
 ha scritto:
>
> Il giorno ven 14 mag 2021 alle ore 13:43 Enrico Olivelli
>  ha scritto:
> >
> > Hello,
> > the website builder is failing on CI, below you can find the error
> >
> > Is there anyone who knows how it works and how to fix it ?
> > Is there a way to update it manually ?
> >
> > I cannot announce 2.7.2 release until the website is updated.
> >
> >
> > this is the link:
> > https://github.com/apache/pulsar/runs/2581534691?check_suite_focus=true
> >
> > this is the error:
> > Please, upgrade your dependencies to the actual version of core-js.
> > 38596warning jest > jest-cli > jest-config > babel-core >
> > babel-register > core-js@2.6.12: core-js@<3.3 is no longer maintained
> > and not recommended for usage due to the number of issues. Because of
> > the V8 engine whims, feature detection in old core-js versions could
> > cause a slowdown up to 100x even if nothing is polyfilled. Please,
> > upgrade your dependencies to the actual version of core-js.
> > 38597warning jest > jest-cli > jest-environment-jsdom > jsdom >
> > left-pad@1.3.0: use String.prototype.padStart()
> > 38598warning jest > jest-cli > jest-environment-jsdom > jsdom >
> > request-promise-native@1.0.9: request-promise-native has been
> > deprecated because it extends the now deprecated request package, see
> > https://github.com/request/request/issues/3142
> > 38599warning jest > jest-cli > jest-haste-map > sane >
> > fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using
> > insecure binaries. Upgrade to fsevents 2.
> > 38600warning highlight.js@9.18.5: Support has ended for 9.x series.
> > Upgrade to @latest
> > 38601[2/4] Fetching packages...
> > 38602info fsevents@1.2.13: The platform "linux" is incompatible with
> > this module.
> > 38603info "fsevents@1.2.13" is an optional dependency and failed
> > compatibility check. Excluding it from installation.
> > 38604error @redocly/openapi-core@1.0.0-beta.45: The engine "node" is
> > incompatible with this module. Expected version ">=12.0.0". Got
> > "10.23.3"
> > 38605error Found incompatible module.
> > 38606info Visit https://yarnpkg.com/en/docs/cli/install for
> > documentation about this command.
> > 38607Error: Process completed with exit code 1.
>
> it looks like the problem is in the "pulsar build  image" that still
> installs Node 10 instead of Node 12
>
> I am trying to prepare a quick fix, but I will need help from someone
> who can push the image to dockerhub
>
>
> Enrico
>
> >
> > Enrico

[GitHub] [pulsar-manager] andikercuku commented on issue #297: Question:how to change the default login account and password

[GitBox]

andikercuku commented on issue #297:
URL: https://github.com/apache/pulsar-manager/issues/297#issuecomment-843249637


   For this issue you can deploy a pulsar-manager-batch-job.yaml file which 
will fix or change the login access in backend-service in this 
[link](https://github.com/andikercuku/pulsar-manager-login-fix) 
   
   After this batch job, you can access your pulsar-manager via dashboard.
   Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[Pulsar Community Weekly Update] 2021-05-10 ~ 2021-05-16

[Huanli Meng]

Dear Pulsar enthusiast,

This is the Pulsar community weekly update for 2021-05-10 ~ 2021-05-16,
with updates on Pulsar client, broker, and so on.

This Pulsar community weekly update is also available at
https://streamnative.io/weekly/2021/2021-05/2021-05-17-pulsar-weekly.

*All Pulsar community weekly updates are available at
**https://streamnative.io/weekly/
.*

===
*Pulsar Highlight*
- [Transaction] Handle transaction pending ack persistence.

  https://github.com/apache/pulsar/pull/8881 ([@congbobo184](
https://github.com/congbobo184))

- [Broker] Migrate the schema storage metadata to `MetadataStore`.

  https://github.com/apache/pulsar/pull/10545 ([@merlimat](
https://github.com/merlimat))

- [Docker] Add a new docker compose that includes all the Pulsar components.

  https://github.com/apache/pulsar/pull/10409 ([@odmarkj](
https://github.com/odmarkj))
===
*Development*
- [PIP-85] Add the schema information to messages in the Java Client API.

  https://github.com/apache/pulsar/pull/10476 ([@eolivelli](
https://github.com/eolivelli))
===
*Notable Feature*
- [Python Client] Replace `Exceptions` with `PulsarExceptions`.

  https://github.com/apache/pulsar/pull/7600 ([@lbenc135](
https://github.com/lbenc135))

- [Enhancement] The reader supports seeking messages from the separate
message ID or time.

  https://github.com/apache/pulsar/pull/10348 ([@315157973](
https://github.com/315157973))

- [CLI] Support setting the time-based limit on the backlog quota through
CLI.

  https://github.com/apache/pulsar/pull/10401 ([@MarvinCai](
https://github.com/MarvinCai))

- [Pulsar IO] Use `Message.getReaderSchema()` in Pulsar sink connectors
when possible.

  https://github.com/apache/pulsar/pull/10557 ([@eolivelli](
https://github.com/eolivelli))

- [Broker] Expose the topic-level `averageMsgSize` to metrics.

  https://github.com/apache/pulsar/pull/10553 ([@315157973](
https://github.com/315157973))

- [Enhancement] Support truncating all data of a topic without
disconnecting the producers and consumers.

  https://github.com/apache/pulsar/pull/10326 ([@jangwind](
https://github.com/jangwind))

- [Enhancement] Support creating `MetadataCache` with the custom SerDe.

  https://github.com/apache/pulsar/pull/10543 ([@merlimat](
https://github.com/merlimat))

- [Broker] Get durable subscription without handling
`startMessageRollbackDurationSec`.

  https://github.com/apache/pulsar/pull/10520 ([@linlinnn](
https://github.com/linlinnn))

- [Auth] Enable Conscrypt for Jetty in the Pulsar broker and in the Pulsar
proxy.

  https://github.com/apache/pulsar/pull/10541 ([@lhotari](
https://github.com/lhotari))

- [Broker] Dispatch messages to consumers with permits.

  https://github.com/apache/pulsar/pull/10417 ([@rdhabalia](
https://github.com/rdhabalia))

- [Enhancement] Support disabling the maximum queue size for the producers.

  https://github.com/apache/pulsar/pull/9650 ([@merlimat](
https://github.com/merlimat))

- [Auth] Support the optional authentication method name header in HTTP
authentication.

  https://github.com/apache/pulsar/pull/6799 ([@KannarFr](
https://github.com/KannarFr))
===
*Notable Bug Fix*
- [Broker] Fix the issue that the `AdvertisedAddress` in `PusarService` and
in `conf.` is inconsistent.

  https://github.com/apache/pulsar/pull/10312 ([@315157973](
https://github.com/315157973))

- [Client] Fix the NPE that is thrown when an ACK grouping tracker checks
duplicated message IDs.

  https://github.com/apache/pulsar/pull/10586 ([@BewareMyPower](
https://github.com/BewareMyPower))

- [Test] Fix the `GracefulExecutorServicesShutdownTest` flaky test.

  https://github.com/apache/pulsar/pull/10599 ([@lhotari](
https://github.com/lhotari))

- [Function] Fix the sink or source exception stats.

  https://github.com/apache/pulsar/pull/10549 ([@linlinnn](
https://github.com/linlinnn))

- [Client] Fix the default retry letter and the dead letter topic name.

  https://github.com/apache/pulsar/pull/10129 ([@wangjialing218](
https://github.com/wangjialing218))

- [Broker] Fix the bug that occurs when checking whether a partitioned
topic is a system topic.

  https://github.com/apache/pulsar/pull/10529 ([@hangc0276](
https://github.com/hangc0276))

- [Broker] Make `OpAddEntry.toString()` more robust to nulls to prevent
NPEs.

  https://github.com/apache/pulsar/pull/10548 ([@devinbost](
https://github.com/devinbost))

- [Transaction] Fix the transaction buffer delete marker issue.

  https://github.com/apache/pulsar/pull/10525 ([@congbobo184](
https://github.com/congbobo184))

- [Test] Refactor the function integration tests for easier maintainability.

  https://github.com/apache/pulsar/pull/10140 ([@david-streamlio](
https://github.com/david-streamlio))
===
*Event / News*

- Pulsar Virtual Summ

Re: [Java][Client] Important behaviour change on empty key from 2.7.x to 2.8.x

[Matteo Merli]

I believe this is a by-product of the protobuf changes. Google
Protobuf returns an empty string if you're trying to access an
optional field with no default value associated with it. This can lead
to subtle bugs..

Users of `Message` API should always check `hasKey()` before trying to
do `getKey()`. (A better API would have been to return an
`Optional`).

In any case, I'd rather not change the behavior of this right now, but
leave it in the same form as 2.7 and earlier versions.


--
Matteo Merli


On Tue, May 18, 2021 at 12:50 AM Enrico Olivelli  wrote:
>
> Hello,
> I have found this behaviour in the Java Client while switching from 2.7 to 
> 2.8.
>
> Short version of the story:
> - on 2.7.2 a null key is received as an empty key
> - on 2.8.0 a null key is received as a null key
>
> The behaviour of 2.8 is better, because it is what you expect.
>
> But if you migrate an application from 2.7.2 to 2.8 you can start to
> see Nulls instead of empty strings and this will lead to unpredictable
> behaviour and possibly NullPointerExceptions.
>
> We can accept the new behaviour but I would like to check if the
> community is aware of this and this is acceptable.
>
> This is the issue, with a reproducer
> https://github.com/apache/pulsar/issues/10625
>
> Enrico

[GitHub] [pulsar-client-node] Romstar opened a new pull request #155: added functionality for orderingKey in Message.cc

[GitBox]

Romstar opened a new pull request #155:
URL: https://github.com/apache/pulsar-client-node/pull/155


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Major performance issue uncovered

[Lari Hotari]

Thanks Devin, very thoughtful observations.

Do you think that it would be possible to create a repro of the case with
some "dummy" code in the functions? Perhaps messages could be produced to
the input topic partitions with pulsar-perf?

There's a lot of interesting and useful details in your observation. I
haven't had the chance to go through all the details. I have a few comments
or questions on some topics.

> 3. When a subscription is "frozen," the broker that owns the "frozen"
topic is doing nothing but processing pendingAddEntries in the do loop of
*ManagedLedgerImpl.updateLedgersIdsComplete(..)*, here:
https://github.com/apache/pulsar/blob/30d48cbbec8b522e8ab595f70fb1edf8b31bb51b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L1507

How did you detect that nothing else is executed? Based on log output?

> 5. Functions get blocked from producing due to waiting for ack's from the
broker on pending messages, which blocks the function semaphore and results
in large backlog accumulation.

For this case, it would be useful to do a jstack thread dump for both
broker and client side when the problem is ongoing. Multiple subsequent
thread dump taken a few seconds apart are useful in detecting
blocked/dead-locked threads. Heap dumps would also be useful, but those
contain confidential details about the messages and environment
credentials, so they shouldn't be shared publicly or by using in-secure
channels.

-Lari

On Tue, May 18, 2021 at 4:26 AM Devin Bost  wrote:

> While researching https://github.com/apache/pulsar/issues/6054, I
> discovered some key things that revealed a major performance issue that
> manifests in high-velocity workloads:
>
> 1. The problem can be reproduced when batching is disabled on all Pulsar
> functions
> 2. When a subscription is "frozen," it's actually still processing
> messages but extremely slowly
> 3. When a subscription is "frozen," the broker that owns the "frozen"
> topic is doing nothing but processing pendingAddEntries in the do loop of
> *ManagedLedgerImpl.updateLedgersIdsComplete(..)*, here:
> https://github.com/apache/pulsar/blob/30d48cbbec8b522e8ab595f70fb1edf8b31bb51b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L1507
> 4. Restarting the broker that owns the topic will unblock the "freeze" for
> a non-deterministic amount of time (sometimes minutes, sometimes hours.)
> 5. Functions get blocked from producing due to waiting for ack's from the
> broker on pending messages, which blocks the function semaphore and results
> in large backlog accumulation.
> 6. If input topics stop being written to, the backlogs will eventually
> clear (after some amount of hours), even on topics that are "frozen."
> 7. The issue is easiest to reproduce in high-velocity/high-volume
> workloads, especially if partitioned topics and parallel function instances
> are used.
> 8. Permit behavior is not abnormal.
>
> Here are some thoughts:
> 1. I noticed that *ManagedLedgerImpl.updateLedgersIdsComplete(..)* is
> synchronized. I'm concerned that the mutex may be locking up too much of
> ManagedLedgerImpl, but I suspect this is not the root cause of the issue,
> and I'm not sure if it's possible to add parallelization in this area.
> (Someone with a deeper understanding of ManagedLedger could chime in here.)
> 2. Load isn't sufficiently balanced across the brokers. Checking debug
> logs for executions of *ManagedLedgerImpl.updateLedgersIdsComplete(..)*
> indicated:
>broker1: 0 executions
>broker2: 2228 executions
>broker3: 8221 executions. (This is the broker that owns the "frozen"
> topic.)
> In the flow that reproduced the issue, there were 3 topics separated by
> Pulsar Functions, the first topic (the input topic) has 4 partitions, both
> Pulsar Functions are filtering messages, and the first function (consuming
> from the first topic) has 4 parallel instances. Topics are all persistent
> without retention or tiered storage, and the subscriptions are all shared.
> Here is a graphic that shows the flow:
> [image: image.png]
> (The green numbers are backlog counts.)
> In the image, the topic that is "frozen" is the "siege-filtered" topic on
> broker3.
>
> Looking at the sizes of debug logs that accumulated on the brokers (with
> the most frequently executed debug log line occurring inside the do loop of
> *ManagedLedgerImpl.updateLedgersIdsComplete(..)*, the load imbalance is
> even clearer:
>broker1: 7.1MB
>broker2: 242 MB
>broker3: 6.6 GB
>
> CPU isn't fully utilized:
>   broker1: 35.30%
>   broker2: 53.24%
>   broker3: 36.49%
>
> The bookies' CPU also isn't fully utilized:
>   bookie1 (on broker1): 14.27%
>   bookie2 (on broker2): 16.72%
>   bookie3 (on broker3): 9.40%
>
> The thing that's not clear to me yet is why unloading the "frozen" topic
> would clear the blockage. I'd think it would just move the performance
> issue to another broker. Any ideas on why unloading the "frozen" top

[GitHub] [pulsar-helm-chart] TC-robV opened a new pull request #121: add enableAdminApi for prometheus

[GitBox]

TC-robV opened a new pull request #121:
URL: https://github.com/apache/pulsar-helm-chart/pull/121


   Fixes #
   
   ### Motivation
   
   would be nice to have this option here so people can run admin commands 
against the prometheus. 
   
   ### Modifications
   
   added a new value and modified the deployment, taken from the official prom 
helm.
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Major performance issue uncovered

[Devin Bost]

Lari,

Thanks for sharing your thoughts. I'll work on making this easier to
reproduce. That's a good suggestion.

> How did you detect that nothing else is executed? Based on log output?

I suppose it may be misleading for me to say that "nothing" else is
executed since I'm judging based on debug log output, and not all modules
are outputting to the log. However, I'm logging the modules mentioned in
https://github.com/apache/pulsar/issues/6054#issuecomment-842626569, which
are:

  - name: org.apache.pulsar.broker.service
level: trace
ref: "${sys:pulsar.log.appender}"
  - name: org.apache.pulsar.common
level: trace
ref: "${sys:pulsar.log.appender}"
  - name: org.apache.pulsar.client
level: trace
ref: "${sys:pulsar.log.appender}"
  - name: org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl
level: trace
ref: "${sys:pulsar.log.appender}"

Despite that, without adding custom log lines throughout the code, the
debug lines go quiet on the broker that owns the topic that stops
performing, as shown in
https://github.com/apache/pulsar/issues/6054#issuecomment-834127635

Also, as a quick update, I did some performance testing in
*ManagedLedgerImpl.updateLedgersIdsComplete(..)* and didn't find any
obvious issues, so that may have been misleading as well. I'm continuing my
performance testing by testing other methods that are called prior to the
broker sending ack's to the client (since that's where the slowdown appears
to be.)

Devin G. Bost


On Tue, May 18, 2021 at 1:15 PM Lari Hotari  wrote:

> Thanks Devin, very thoughtful observations.
>
> Do you think that it would be possible to create a repro of the case with
> some "dummy" code in the functions? Perhaps messages could be produced to
> the input topic partitions with pulsar-perf?
>
> There's a lot of interesting and useful details in your observation. I
> haven't had the chance to go through all the details. I have a few comments
> or questions on some topics.
>
> > 3. When a subscription is "frozen," the broker that owns the "frozen"
> topic is doing nothing but processing pendingAddEntries in the do loop of
> *ManagedLedgerImpl.updateLedgersIdsComplete(..)*, here:
>
> https://github.com/apache/pulsar/blob/30d48cbbec8b522e8ab595f70fb1edf8b31bb51b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L1507
>
> How did you detect that nothing else is executed? Based on log output?
>
> > 5. Functions get blocked from producing due to waiting for ack's from the
> broker on pending messages, which blocks the function semaphore and results
> in large backlog accumulation.
>
> For this case, it would be useful to do a jstack thread dump for both
> broker and client side when the problem is ongoing. Multiple subsequent
> thread dump taken a few seconds apart are useful in detecting
> blocked/dead-locked threads. Heap dumps would also be useful, but those
> contain confidential details about the messages and environment
> credentials, so they shouldn't be shared publicly or by using in-secure
> channels.
>
> -Lari
>
> On Tue, May 18, 2021 at 4:26 AM Devin Bost  wrote:
>
> > While researching https://github.com/apache/pulsar/issues/6054, I
> > discovered some key things that revealed a major performance issue that
> > manifests in high-velocity workloads:
> >
> > 1. The problem can be reproduced when batching is disabled on all Pulsar
> > functions
> > 2. When a subscription is "frozen," it's actually still processing
> > messages but extremely slowly
> > 3. When a subscription is "frozen," the broker that owns the "frozen"
> > topic is doing nothing but processing pendingAddEntries in the do loop of
> > *ManagedLedgerImpl.updateLedgersIdsComplete(..)*, here:
> >
> https://github.com/apache/pulsar/blob/30d48cbbec8b522e8ab595f70fb1edf8b31bb51b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L1507
> > 4. Restarting the broker that owns the topic will unblock the "freeze"
> for
> > a non-deterministic amount of time (sometimes minutes, sometimes hours.)
> > 5. Functions get blocked from producing due to waiting for ack's from the
> > broker on pending messages, which blocks the function semaphore and
> results
> > in large backlog accumulation.
> > 6. If input topics stop being written to, the backlogs will eventually
> > clear (after some amount of hours), even on topics that are "frozen."
> > 7. The issue is easiest to reproduce in high-velocity/high-volume
> > workloads, especially if partitioned topics and parallel function
> instances
> > are used.
> > 8. Permit behavior is not abnormal.
> >
> > Here are some thoughts:
> > 1. I noticed that *ManagedLedgerImpl.updateLedgersIdsComplete(..)* is
> > synchronized. I'm concerned that the mutex may be locking up too much of
> > ManagedLedgerImpl, but I suspect this is not the root cause of the issue,
> > and I'm not sure if it's possible to add parallelization in this area

[GitHub] [pulsar-client-node] hrsakai opened a new pull request #156: [Do Not Merge] This is a test

[GitBox]

hrsakai opened a new pull request #156:
URL: https://github.com/apache/pulsar-client-node/pull/156


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-node] k2la closed pull request #154: [Do Not Merge]This is a test

[GitBox]

k2la closed pull request #154:
URL: https://github.com/apache/pulsar-client-node/pull/154


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [Java][Client] Important behaviour change on empty key from 2.7.x to 2.8.x

[Enrico Olivelli]

Il Mar 18 Mag 2021, 18:15 Matteo Merli  ha scritto:

> I believe this is a by-product of the protobuf changes. Google
> Protobuf returns an empty string if you're trying to access an
> optional field with no default value associated with it. This can lead
> to subtle bugs..
>
> Users of `Message` API should always check `hasKey()` before trying to
> do `getKey()`. (A better API would have been to return an
> `Optional`).
>
> In any case, I'd rather not change the behavior of this right now, but
> leave it in the same form as 2.7 and earlier versions.
>

I can send a fix, at least for the Key, leaving the new behaviour for other
properties.

Let's wait for other comments or proposals


Enrico




>
> --
> Matteo Merli
> 
>
> On Tue, May 18, 2021 at 12:50 AM Enrico Olivelli 
> wrote:
> >
> > Hello,
> > I have found this behaviour in the Java Client while switching from 2.7
> to 2.8.
> >
> > Short version of the story:
> > - on 2.7.2 a null key is received as an empty key
> > - on 2.8.0 a null key is received as a null key
> >
> > The behaviour of 2.8 is better, because it is what you expect.
> >
> > But if you migrate an application from 2.7.2 to 2.8 you can start to
> > see Nulls instead of empty strings and this will lead to unpredictable
> > behaviour and possibly NullPointerExceptions.
> >
> > We can accept the new behaviour but I would like to check if the
> > community is aware of this and this is acceptable.
> >
> > This is the issue, with a reproducer
> > https://github.com/apache/pulsar/issues/10625
> >
> > Enrico
>