Re: [BUGS] Pulsar 3.0 GKE Cluster Won't Start

2023-04-14 Thread Hang Chen
Hi guys, Thanks for Micheal's great job, and he already pushed one PR [1] to fix this bug on the BookKeeper side. This comment [2] shows how this bug happens. I will cut BookKeeper 4.16.1 release soon. Thanks, Hang [1] https://github.com/apache/bookkeeper/pull/3919 [2] https://github.com/ap

Re: [BUGS] Pulsar 3.0 GKE Cluster Won't Start

2023-04-13 Thread Michael Marshall
I ended up digging a little longer. I have some partial results to share. It seems to me the problem was likely introduced by this Bookkeeper PR https://github.com/apache/bookkeeper/pull/3783. These are likely the problematic lines: https://github.com/apache/bookkeeper/blob/234b817cdb4e054887ffd5

Re: [BUGS] Pulsar 3.0 GKE Cluster Won't Start

2023-04-13 Thread Michael Marshall
By my testing using the above steps to reproduce the issue, I do not see the bug on commit [0] but I do see it on the subsequent commit [1]. That indicates [1], which is the bookkeeper client upgrade to 4.16.0 commit, introduced the problem. I need to sign off soon. I appreciate any help you can p

Re: [BUGS] Pulsar 3.0 GKE Cluster Won't Start

2023-04-13 Thread Michael Marshall
Here's an update on my progress. I reproduced Dave's observations with the following steps: 1. Download and unpack 3.0.0 RC 1. 2. Add these two settings to the conf/standalone.conf: exposingBrokerEntryMetadataToClientEnabled=true brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.

Re: [BUGS] Pulsar 3.0 GKE Cluster Won't Start

2023-04-12 Thread Enrico Olivelli
Dave, IIUC (from offline discussion) those tests started to fail only recently, as we run those tests against the master branch of apache/pulsar repo. Does the beginning of failures match the upgrade to BK 4.16 ? The problem may be related to the handling of ByteBufs, IIRC there are a few changes

Re: [BUGS] Pulsar 3.0 GKE Cluster Won't Start

2023-04-12 Thread Hang Chen
Hi Dave, Thanks for verifying this release candidate. I have setup the Pulsar cluster with 3.0 on my laptop and test the produce and consume, it works fine. We also deploy the it on k8s, the produce, consume and health check also works fine. Would you please share your broker configurations? T