Re: [VOTE] Pulsar Manager Release 0.3.0 Candidate 3

[Enrico Olivelli]

+1 (binding)

Checks:
- Build from Sources, all tests are passing
- Tested with Pulsar Standalone, it works
- BKVM boots as well

Some problems:
- I see one problem in the Source tarball, when I unpack it I see the
sources are created in a directory named
'pulsar-manager-0.3.0-candidate-3'
- The build works only on JDK8

I am voting +1 but we should fix the two problems for the next release

Enrico

Il giorno gio 19 mag 2022 alle ore 18:32 PengHui Li
 ha scritto:
>
> +1 (binding)
>
> - Validate checksum
> - Deploy pulsar-manager
>
> Thanks,
> Penghui
>
> On Wed, May 18, 2022 at 11:03 PM Max Xu  wrote:
>
> > +1 (non-binding)
> >
> > - Validate checksum
> > - Start pulsar-manager
> > - Create an environment (add a pulsar instance)
> > - Create and delete tenants/namespaces/topics. But unable to create token
> >
> > Thanks,
> > Max Xu
> >
> >
> >
> > On Wed, May 18, 2022 at 6:58 PM Hang Chen  wrote:
> >
> > > +1(binding)
> > > - Validate checksum
> > > - Deploy pulsar-manager service and add pulsar cluster
> > > - Create tenants, namespace and topics, delete topics.
> > >
> > > Thanks,
> > > Hang
> > >
> > > Guangning E  于2022年5月12日周四 20:39写道：
> > > >
> > > > +1(non-binding)
> > > > Validate checksum
> > > > Start pulsar-manager service
> > > > Create tenant and topic
> > > >
> > > > Thanks,
> > > > Guangning
> > > >
> > > > Li Li  于2022年5月10日周二 14:14写道：
> > > >
> > > > > Hi everyone,
> > > > > Please review and vote on the release candidate #3 for the version
> > > 0.3.0,
> > > > > as follows:
> > > > > [ ] +1, Approve the release
> > > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > > >
> > > > > The complete staging area is available for your review, which
> > includes:
> > > > > * Release notes [1]
> > > > > * The official Apache source and binary distributions to be deployed
> > to
> > > > > dist.apache.org  [2]
> > > > > * Source code tag "v0.3.0-candidate-3" [4] with git sha
> > > > >
> > >
> > 951095a71f7471dca028da0a330bc1a5e0707333a61fa4a09c8ea0f0a144d5628b511487e2442ebe290b9642b6b8ca7dee486a18a8339c893c37253724ad5fd4
> > > > > apache-pulsar-manager-0.3.0-src.tar.gz
> > > > >
> > > > > PulsarManager's KEYS file contains PGP keys we used to sign this
> > > release:
> > > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS <
> > > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS>
> > > > >
> > > > > Please download these packages and review this release candidate:
> > > > >
> > > > > - Review release notes
> > > > > - Download the source package (verify shasum, and asc) and follow the
> > > > > instructions to build and run the pulsar-manager front end and back
> > end
> > > > > service.
> > > > > - Download the binary package (verify shasum, and asc) and follow the
> > > > > instructions to run run the pulsar-manager front end and back end
> > > service.
> > > > >
> > > > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > > approval, with at least 3 PMC affirmative votes.
> > > > >
> > > > >
> > > > > Source and binary files:
> > > > >
> > > > >
> > >
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-manager/apache-pulsar-manager-0.3.0/apache-pulsar-manager-0.3.0-bin.tar.gz
> > > > > <
> > > > >
> > >
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-manager/apache-pulsar-manager-0.3.0/apache-pulsar-manager-0.3.0-bin.tar.gz
> > > > > >
> > > > >
> > > > >
> > >
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-manager/apache-pulsar-manager-0.3.0/apache-pulsar-manager-0.3.0-src.tar.gz
> > > > > <
> > > > >
> > >
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-manager/apache-pulsar-manager-0.3.0/apache-pulsar-manager-0.3.0-src.tar.gz
> > > > > >
> > > > >
> > > > > SHA-512 checksums:
> > > > >
> > > > >
> > >
> > 6ffa5921765ee94a404792e98eb4b3cbda9e016c6661ef12e4e873e7e452301bc05650709955b012d08048e418133948a628ad55bc91ac65836022e1ea426d6f
> > > > > apache-pulsar-manager-0.3.0-bin.tar.gz
> > > > >
> > >
> > 951095a71f7471dca028da0a330bc1a5e0707333a61fa4a09c8ea0f0a144d5628b511487e2442ebe290b9642b6b8ca7dee486a18a8339c893c37253724ad5fd4
> > > > > apache-pulsar-manager-0.3.0-src.tar.gz
> > > > >
> > > > >
> > > > >
> > > > >
> > >
> >

[DISCUSS] Pulsar 3.0 brainstorming: Going beyond PIP-45 Pluggable metadata interface

[Lari Hotari]

Hi all,

I started writing this email as feedback to "PIP-157: Bucketing topic
metadata to allow more topics per namespace" [3].
This email expanded to cover some analysis of "PIP-45: Pluggable metadata
interface" [4] design. (A good introduction to PIP-45 is the StreamNative
blog post "Moving Toward a ZooKeeper-Less Apache Pulsar" [5]).

The intention is to start discussions for Pulsar 3.0 and beyond. Bouncing
ideas and challenging the existing design with good intentions and the
benefit of all.

I'll share some thoughts that have come up in discussions together with my
colleague Michael Marshall.  We have been bouncing some ideas together and
that has been very helpful in being able to start building some
understanding of the existing challenges and possible direction for solving
these challenges. I hope that we could have broader conversations in the
Pulsar Community for improving Pulsar's metadata management and load
balancing designs in the long term.

There are few areas where there are challenges with the current Metadata
Store / PIP-45 solution:

1) Metadata consistency from user's point of view
  - Summarized well in this great analysis and comment [1] by Zac Bentley
   "Ideally, the resolution of all of these issues would be the same: a
management API operation--any operation--should not return successfully
until all observable side effects of that operation across a Pulsar cluster
(including brokers, proxies, bookies, and ZK) were completed." (see [1] for
the full analysis and comment)

2) Metadata consistency issues within Pulsar
  - There are issues where the state in a single broker gets left in a bad
state as a result of consistency and concurrency issues with metadata
handling and caching.
Possible example https://github.com/apache/pulsar/issues/13946

3) Scalability issue: all metadata changes are broadcasted to all brokers -
model doesn't scale out
   - This is due to the change made in
https://github.com/apache/pulsar/pull/11198 , "Use ZK persistent watches".
   - The global broadcasting design of metadata changes doesn't follow
typical scalable design principles such as the "Scale cube". This will pose
limits on Pulsar clusters with large number of brokers. The current
metadata change notification solution doesn't support scaling out when it's
based on a design that broadcast all notifications to every participant.

When doing some initial analysis and brainstorming on the above areas,
there have been thoughts that PIP-45 Metadata Store API [2] abstractions
are somewhat not optimal.

A lot of the functionality that is provided in the PIP-45 Metadata Store
API interface [2] could be solved more efficiently in a way where Pulsar
itself would be a key part of the metadata storage solution.

For example, listing topics in a namespace could be a "scatter-gather"
query to all "metadata shards" that hold namespace topics. There's not
necessarily a need to have a centralized external Metadata Store API
interface [2] that replies to all queries. Pulsar metadata handling could
be moving towards a distributed database type of design where consistent
hashing plays a key role. Since the Metadata handling is an internal
concern, the interface doesn't need to provide services directly to
external users of Pulsar. The Pulsar Admin API should also be improved to
scale for queries and listing of namespaces with millions of topics, and
should have pagination to limit results. This implementation can internally
handle possible "scatter-gather" queries when the metadata handling backend
is not centralized. The point is that Metadata Store API [2] abstraction
doesn't necessarily need to provide service for this, since it could be a
different concern.

Most of the complexity in the current PIP-45 MetaData Store comes from data
consistency challenges. The solution is heavily based on caches and having
ways to handle cache expirations and keeping data consistent. There are
gaps in the caching solution since there are metadata consistency problems,
as described in 1) and 2) above. A lot of the problems go away in a model
where most processing and data access is local. Similar to how the broker
handles the topics. The topic is owned on a single broker at a time. The
approach could be extended to cover metadata changes and queries.

What is interesting here regarding PIP-157 is that brainstorming led to a
sharding (aka "bucketing") solution, where there are metadata shards in the
system.

metadata shard
   |
namespace bundle  (existing)
   |
namespace  (existing)

Instead of having a specific solution in mind for dealing with the storage
of the metadata, the main idea is that each metadata shard is independent
and would be able to perform operations without coordination with other
metadata shards. This does impact the storage of metadata so that
operations to the storage system can be isolated (for example, it is
necessary to be able to list the topics for a bundle without listing
everything. P

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

[Lari Hotari]

-1

Hi Andras & Matteo, I'm sorry for the delay in my feedback. 

I voted -1 since I think that it would be more valuable to first analyze the 
existing problems before adding more intermediate solutions such as PIP-157. 
I'm fine if you want to go ahead and implement PIP-157 so I don't see the 
proposal a problem itself.  

I have written a long write-up in a new thread "Pulsar 3.0 brainstorming: Going 
beyond PIP-45 Pluggable metadata interface", 
https://lists.apache.org/thread/tvco1orf0hsyt59pjtfbwoq0vf6hfrcj . The last 
part of the long email handles PIP-157 and hopefully clarifies why I voted -1. 
We can continue discussion in the context of that thread.

-Lari

On 2022/05/02 16:40:54 Matteo Merli wrote:
> Lari & Enrico, the discussion thread was out for 11 days and there
> were 2 positive feedbacks.
> I don't think this qualifies as "too early for a vote" and it would
> have been better if the discussion happened then.
> 
> As for the comments in the other thread, I think there are only a
> couple of misconceptions on the proposal itself, as they are not
> actual problems (eg: managed ledger is not affected at all by this
> proposal, as the naming happens on top of it).
> 
> Some parts can be clarified (as it is always the case), though I don't
> think it's a good idea to stop a vote at this point.
> 
> 
> Matteo
> 
> --
> Matteo Merli
> 
> 
> 
> On Mon, May 2, 2022 at 3:31 AM Lari Hotari  wrote:
> >
> > -1. It's too early to start a vote. Let's first have discussions.
> >
> > -Lari
> >
> > ma 2. toukok. 2022 klo 9.50 Andras Beni 
> > 
> > kirjoitti:
> >
> > > Hi Pulsar Community,
> > >
> > > I would like to start a VOTE on "Bucketing topic metadata to allow more
> > > topics per namespace" (PIP-157).
> > >
> > > The proposal can be read at https://github.com/apache/pulsar/issues/15254
> > > and the discussion thead is available at
> > > https://lists.apache.org/thread/zx6s7hyrl2vy7nhdl79wh6gn88kxpd6k.
> > >
> > > Voting will stay open for at least 48h.
> > >
> > > Thanks,
> > > Andras
> > >
> 

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

[Andras Beni]

Thanks for your participation.
I'm closing the vote.
The proposal has been accepted with 4 binding +1, 1 binding -1 and 2
non-binding +1 votes.

Best regards,
Andras

On Fri, May 20, 2022 at 11:11 AM Lari Hotari  wrote:

> -1
>
> Hi Andras & Matteo, I'm sorry for the delay in my feedback.
>
> I voted -1 since I think that it would be more valuable to first analyze
> the existing problems before adding more intermediate solutions such as
> PIP-157. I'm fine if you want to go ahead and implement PIP-157 so I don't
> see the proposal a problem itself.
>
> I have written a long write-up in a new thread "Pulsar 3.0 brainstorming:
> Going beyond PIP-45 Pluggable metadata interface",
> https://lists.apache.org/thread/tvco1orf0hsyt59pjtfbwoq0vf6hfrcj . The
> last part of the long email handles PIP-157 and hopefully clarifies why I
> voted -1. We can continue discussion in the context of that thread.
>
> -Lari
>
> On 2022/05/02 16:40:54 Matteo Merli wrote:
> > Lari & Enrico, the discussion thread was out for 11 days and there
> > were 2 positive feedbacks.
> > I don't think this qualifies as "too early for a vote" and it would
> > have been better if the discussion happened then.
> >
> > As for the comments in the other thread, I think there are only a
> > couple of misconceptions on the proposal itself, as they are not
> > actual problems (eg: managed ledger is not affected at all by this
> > proposal, as the naming happens on top of it).
> >
> > Some parts can be clarified (as it is always the case), though I don't
> > think it's a good idea to stop a vote at this point.
> >
> >
> > Matteo
> >
> > --
> > Matteo Merli
> > 
> >
> >
> > On Mon, May 2, 2022 at 3:31 AM Lari Hotari  wrote:
> > >
> > > -1. It's too early to start a vote. Let's first have discussions.
> > >
> > > -Lari
> > >
> > > ma 2. toukok. 2022 klo 9.50 Andras Beni  .invalid>
> > > kirjoitti:
> > >
> > > > Hi Pulsar Community,
> > > >
> > > > I would like to start a VOTE on "Bucketing topic metadata to allow
> more
> > > > topics per namespace" (PIP-157).
> > > >
> > > > The proposal can be read at
> https://github.com/apache/pulsar/issues/15254
> > > > and the discussion thead is available at
> > > > https://lists.apache.org/thread/zx6s7hyrl2vy7nhdl79wh6gn88kxpd6k.
> > > >
> > > > Voting will stay open for at least 48h.
> > > >
> > > > Thanks,
> > > > Andras
> > > >
> >
>

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

[Enrico Olivelli]

Andras,
even if our rules allow to close a vote with a binding -1, we should
have to reach consensus and understand better Lari's motivations

we are a community and accepting something with a -1 must be handled carefully

I suggest to try to continue the discussion and try to converge

Enrico

Il giorno ven 20 mag 2022 alle ore 11:49 Andras Beni
 ha scritto:
>
> Thanks for your participation.
> I'm closing the vote.
> The proposal has been accepted with 4 binding +1, 1 binding -1 and 2
> non-binding +1 votes.



>
> Best regards,
> Andras
>
> On Fri, May 20, 2022 at 11:11 AM Lari Hotari  wrote:
>
> > -1
> >
> > Hi Andras & Matteo, I'm sorry for the delay in my feedback.
> >
> > I voted -1 since I think that it would be more valuable to first analyze
> > the existing problems before adding more intermediate solutions such as
> > PIP-157. I'm fine if you want to go ahead and implement PIP-157 so I don't
> > see the proposal a problem itself.
> >
> > I have written a long write-up in a new thread "Pulsar 3.0 brainstorming:
> > Going beyond PIP-45 Pluggable metadata interface",
> > https://lists.apache.org/thread/tvco1orf0hsyt59pjtfbwoq0vf6hfrcj . The
> > last part of the long email handles PIP-157 and hopefully clarifies why I
> > voted -1. We can continue discussion in the context of that thread.
> >
> > -Lari
> >
> > On 2022/05/02 16:40:54 Matteo Merli wrote:
> > > Lari & Enrico, the discussion thread was out for 11 days and there
> > > were 2 positive feedbacks.
> > > I don't think this qualifies as "too early for a vote" and it would
> > > have been better if the discussion happened then.
> > >
> > > As for the comments in the other thread, I think there are only a
> > > couple of misconceptions on the proposal itself, as they are not
> > > actual problems (eg: managed ledger is not affected at all by this
> > > proposal, as the naming happens on top of it).
> > >
> > > Some parts can be clarified (as it is always the case), though I don't
> > > think it's a good idea to stop a vote at this point.
> > >
> > >
> > > Matteo
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> > >
> > > On Mon, May 2, 2022 at 3:31 AM Lari Hotari  wrote:
> > > >
> > > > -1. It's too early to start a vote. Let's first have discussions.
> > > >
> > > > -Lari
> > > >
> > > > ma 2. toukok. 2022 klo 9.50 Andras Beni  > .invalid>
> > > > kirjoitti:
> > > >
> > > > > Hi Pulsar Community,
> > > > >
> > > > > I would like to start a VOTE on "Bucketing topic metadata to allow
> > more
> > > > > topics per namespace" (PIP-157).
> > > > >
> > > > > The proposal can be read at
> > https://github.com/apache/pulsar/issues/15254
> > > > > and the discussion thead is available at
> > > > > https://lists.apache.org/thread/zx6s7hyrl2vy7nhdl79wh6gn88kxpd6k.
> > > > >
> > > > > Voting will stay open for at least 48h.
> > > > >
> > > > > Thanks,
> > > > > Andras
> > > > >
> > >
> >

[GitHub] [pulsar-helm-chart] michaeljmarshall commented on pull request #266: Add bk, zk securityContext to support upgrade to non-root docker image

[GitBox]

michaeljmarshall commented on PR #266:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/266#issuecomment-1133142563

   @maxsxu - thank you for testing. Do you know if OpenShift works when you set 
`securityContext: {}` for zookeeper and bookkeeper? I had assumed (perhaps 
incorrectly) the PR's current security context would work because the OpenShift 
documentation for how to [create a docker 
image](https://docs.openshift.com/container-platform/4.10/openshift_images/create-images.html)
 to run on OpenShift explicitly says:
   
   > Because the container user is always a member of the root group, the 
container user can read and write these files.
   
   By setting `securityContext: {}`, the user should be a member of the root 
group, but I'm not sure that OpenShift will recursively make the persistent 
volumes root group writable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[DISCUSS] Apache Pulsar 2.10.1 release

[PengHui Li]

Hello, Pulsar community:

I'd like to propose to release Apache Pulsar 2.10.1

Currently, we have 190 commits [0] and there are many transaction
fixes, security fixes.

And there are 22 open PRs [1], I will follow them to make sure that
the important fixes could be contained in 2.10.1

If you have any important fixes or any questions,
please reply to this email, we will evaluate whether to
include it in 2.10.1

[0]
https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.10.1+
[1]
https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.10.1+

Best Regards
Penghui