Re: Revote: Pulsar website concepts

[Yu]

Thanks, Mellisa!

We are making final updates to the new website, including updating the
Ecosystem page, adjusting font size, fixing URL issues, etc.

Feel free to comment if you have more suggestions, thank you!

On Sat, Apr 23, 2022 at 11:31 PM Aaron Williams  wrote:

> Thank you Melissa, et al! The site looks great and looking forward to it
> being live.
>
> Aaron Williams
>
> On Fri, Apr 22, 2022, 11:13 AM Melissa Logan 
> wrote:
>
> > Update on the Pulsar website launch: we're getting very close.
> >
> > The group in the #website channel is Slack is taking a final look at the
> > ecosystem and docs pages. We will soon have a launch date to share.
> >
> > Thanks to all who provided input via the survey Yu shared a few weeks
> ago.
> > (1) Note: There was a Docusaurus update that caused breaking changes
> during
> > that time, so some of you noticed odd behaviors on the site which have
> > since been resolved. You can preview the new site here --
> > https://pulsar-next.staged.apache.org/ -- but keep in mind there are PRs
> > that haven't been committed (2) and syncing issues that will resolve.
> >
> > If you see any red flags NOT already noted in the PR or weekly status doc
> > (3), please let us know here or on #website channel by April 29. Much
> > appreciated.
> >
> > (1) https://lists.apache.org/thread/ndrkkcm5s8xojxsgbn8wg4j1kxj37wqj
> > (2) https://github.com/apache/pulsar-site/pull/53
> > (3)
> >
> >
> https://docs.google.com/document/d/1QtkqXMxhF8pqfJcdkep4Wd7mm1HlQquUEkwQwHC67t8/edit#
> >
> > On Fri, Dec 10, 2021 at 9:58 AM Melissa Logan 
> > wrote:
> >
> > > For record-keeping, we have added a new PIP that captures details for
> the
> > > website design/content project:
> > > https://github.com/apache/pulsar/issues/13235
> > >
> > > Anonymitaet, it references we'll collaborate with you and others on the
> > > work being done for PIP 87. I plan to start a website channel in Slack
> to
> > > aid in our collaboration.
> > >
> > > Any questions, please let me know. Thanks!
> > >
> > > Melissa
> > >
> > >
> > > On Wed, Nov 3, 2021 at 7:57 PM Anonymitaet _  >
> > > wrote:
> > >
> > >> Hi Melissa, thanks for your proposal!
> > >>
> > >> Hi community, as discussed before1, we're performing these tasks:
> > >>
> > >>- Upgrading the Pulsar website framework2. We're making progress
> > >>every week. You can check the details and progress here3.
> > >>- Designing the information architecture4. Since it is a large
> > >>project, we need more time to design and will send out the proposal
> > later.
> > >>
> > >> Feel free to comment, thanks.
> > >>
> > >> 1: https://lists.apache.org/thread/39ct84dlxbo6zf0kktm15pyc7fd1n8cc
> > >> 2:
> > >>
> >
> https://docs.google.com/document/d/1IV35SI_F8G8cL-Vuzknc6RTGLK9_edRMpZpnrHvAWNs/edit#
> > >> 3: https://github.com/apache/pulsar/issues/11766
> > >> 4:
> > >>
> >
> https://docs.google.com/document/d/1O19v1znrNkoahFpsr8puVQ_20jDQfUjC2eRxdFOL91U/edit#
> > >>
> > >>
> > >> <
> >
> https://docs.google.com/document/d/1O19v1znrNkoahFpsr8puVQ_20jDQfUjC2eRxdFOL91U/edit#
> > >
> > >> Redesign Information Architecture
> > >> <
> >
> https://docs.google.com/document/d/1O19v1znrNkoahFpsr8puVQ_20jDQfUjC2eRxdFOL91U/edit#
> > >
> > >> This page was last updated on 2021/11/4 by Anonymitaet.
> > >>  Landing page
> > Changes
> > >> Content blocks On the landing page, categorize tons of info to several
> > >> blocks, which allows users to locate desired info quickly. Pulsar Good
> > >> example Docker doc we...
> > >> docs.google.com
> > >>
> > >> 
> > >> [website] website upgrade · Issue #11766 · apache/pulsar · GitHub
> > >> 
> > >> Preview make preview supported on netlify #11770 New syntax rules
> Click
> > >> here to see details. Action Base framework init website-next base
> > project
> > >> #11770 Compatibility process implement custom link remark parser match
> > >> {@Inject: prefix:nam...
> > >> github.com
> > >>
> > >> 4:
> > >>
> >
> https://docs.google.com/document/d/1O19v1znrNkoahFpsr8puVQ_20jDQfUjC2eRxdFOL91U/edit#
> > >>
> > >> <
> >
> https://docs.google.com/document/d/1O19v1znrNkoahFpsr8puVQ_20jDQfUjC2eRxdFOL91U/edit#
> > >
> > >> Redesign Information Architecture
> > >> <
> >
> https://docs.google.com/document/d/1O19v1znrNkoahFpsr8puVQ_20jDQfUjC2eRxdFOL91U/edit#
> > >
> > >> This page was last updated on 2021/11/4 by Anonymitaet.
> > >>  Landing page
> > Changes
> > >> Content blocks On the landing page, categorize tons of info to several
> > >> blocks, which allows users to locate desired info quickly. Pulsar Good
> > >> example Docker doc we...
> > >> docs.google.com
> > >>
> > >>
> > >> <
> >
> https://docs.google.com/document/d/1IV35SI_F8G8cL-Vuzknc6RTGLK9_edRMpZpnrHvAWNs/edit#
> > >
> > >> Upgrade Docusaurus
> > >> <
> >
> https://docs.google.com/docum

Re: Enable GitHub Discussions?

[Yu]

Hi Dave,

Thanks for your suggestions!

1. Using GitHub Discussions does not mean abandoning Google Docs, we can
use both and leverage both advantages.

2. One question on this:

> Another important part of The Apache Way is to make decisions by reaching
consensus (and avoiding votes).

But the Apache Website says "one of the fundamental aspects of
accomplishing things within the Apache framework is doing so by consensus,
we need a way to tell whether we have reached consensus. We do this by
voting."  [1] — which means we make decisions by voting, correct?

[1] https://www.apache.org/foundation/voting.html



On Fri, Apr 22, 2022 at 11:00 PM Dave Fisher  wrote:

> Hi Yu,
>
> > On Apr 21, 2022, at 11:46 PM, Yu  wrote:
> >
> > Thanks all!
> >
> > I understand that we need to follow the Apache way (If it didn’t happen
> on
> > the mailing list, it didn’t happen) to *make final decisions*, but we can
> > find a better place to *discuss issues and reuse information effectively*
> > [1].
>
> Another important part of The Apache Way is to make decisions by reaching
> consensus (and avoiding votes). [3]
>
> This discussion is not finished and this should not be a call to a vote.
> When a VOTE is called it must be clear and not buried at the bottom of a
> long email.
>
> The project has been using Google Docs for collaboration. Is the real
> motivation to use GitHub Discussions instead of Google Docs?
>
> >
> > 
> >
> > # Current issues
> >
> > We have a lot of work in various places, but the information is not
> > organized and reused effectively.
> >
> > ## Slack
> > - Information is not publicly accessible. You need to upgrade to paid
> plans
> > or else you can see just the 10,000 most recent messages.
> > - Information is scattered and not categorized.
>
> The Pulsar slack has existed since the beginning of the project. We have
> 6688 subscribers to the #general and that is great.
>
> >
> > # StackOveflow
> > - Inactive. Only 24 results after searching "apachepulsar".
>
> When I search “pulsar” I get over 500 results. This is an active channel.
> It’s a place where developers go.
>
> >
> > # Mailing List
> > - Information is scattered and not categorized.
> > - User experience is terrible. Plain text format decreases email
> > readability. Search function is hard to use.
>
> The search function at lists.apache.org is hard to use, but it is
> powerful. In other projects I’ve searched for and found email from 15 years
> ago. The new monthly calendar is very powerful.
>
> If we use GitHub Discussions then the discussion of the decision won’t be
> on the mailing list. People will write emails with links to the GitHub
> Discussions. These links will eventually get stale.
>
> If you want to improve lists.apache.org then you can influence that here:
> d...@ponymail.apache.org. Pony Mail is an Apache Incubator podling.
>
>
> >
> > 
> >
> > # Why try GitHub Discussions?
>
> >
> > Compared with the platforms above and GitHub Issue, GitHub Discussions
> make
> > collaboration more efficient by:
> >
> > # Making it easier to categorize, archive, and search information
> > - It has categories (Announcements, General, Ideas, Polls, Q&A, Show and
> > Tell, etc) and labels to categorize and filter information quickly.
> > - Information is permanently saved.
> > - Search is easy to use.
> >
> > # Providing a centralized community space for conversations and questions
> > - As instructed by GitHub, we can use GitHub Discussions to *discuss
> > things*, and use GitHub Issues to *catalog the work* we need to do after
> > we’ve reached a decision about how to move forward.
> >
> > # Boosting community engagement and encouraging people to make
> contributions
> > - It highlights project contributors and shows Most Helpful Ranking of
> > Contributors.
> > - It creates a place for newcomers and non-code collaborators to
> > participate in a larger community.
> >
> > # Calling users' attention to important issues or events
> > -  Important discussions can be pinned to home pages for easy reference.
> >
> > 
> >
> > Feel free to share your thoughts. This vote will be open for 72 hrs.
>
> We are not ready to vote. While it might be good to do I cannot agree with
> all of your reasoning. If forced to vote then I am -1.
>
> I might agree to GitHub Discussions as a replacement for GoogleDocs.
>
> We have no power to move users away from StackOverflow. A huge number
> follow Pulsar on Slack. If it’s a matter of paying for a plan then perhaps
> there is a way. The PMC should discuss that.
>
> All The Best,
> Dave
>
> [3] http://theapacheway.com/consensus/
>
> >
> > Then we can ask Infra to do the needed GitHub Admin tasks if we reach a
> > consensus on enabling it, thank you!
> >
> > [1] http://theapacheway.com/on-list/
> >
> > Anonymitaet
> >
> > On Fri, Apr 22, 2022 at 2:34 AM Dave Fisher  wrote:
> >
> >> I asked Infra. GitHub Discussions are not synchronized into ASF
> >> Infrastructure's GitBox. That means that th

Re: Enable GitHub Discussions?

[Niclas Hedhman]

On 2022-04-25 12:34, Yu wrote:
Another important part of The Apache Way is to make decisions by 
reaching

consensus (and avoiding votes).

But the Apache Website says "one of the fundamental aspects of
accomplishing things within the Apache framework is doing so by 
consensus,

we need a way to tell whether we have reached consensus. We do this by
voting."  [1] — which means we make decisions by voting, correct?

[1] https://www.apache.org/foundation/voting.html


Not necessarily... Many communities use "Lazy Consensus", meaning "I 
will do this..." announcement (or even "I have done this...") and go 
ahead with it UNLESS someone raises a concern. This typically works 
well, if the community is reasonably well-aligned with the vision and 
path forward.


Niclas

Re: Call for projects and mentors for OSPP 2022

[Yu]

Thanks Penghui and Dianjin.

Lili and I will submit applications to improve our user experience on the
Pulsar website and documentation. We aim to fix some of these issues [1].
Feel free to comment if you have more suggestions on that, thanks!

[1]
https://github.com/apache/pulsar/issues?q=is%3Aopen+is%3Aissue+label%3Awebsite



On Sun, Apr 24, 2022 at 9:23 PM PengHui Li  wrote:

> Hi all,
>
> Open Source Promotion Plan, OSPP[1] is open for project registration,
> like Google Summer of Code. This Plan aims to encourage college students
> to participate in open-source projects and make contributions.
>
> The project idea is open from 4/21, 2022 to 5/9, 2022(UTC+8).
> Apache Pulsar ended up with 6 projects seats. If you have great ideas,
> please copy the following template and fill in the fields.
> Once you have completed your project ideas, you can reply to this email.
> We will help you register for the program. Note that One mentor can
> mentor one project at a time and both mentors and students can
> receive financial awards for completed projects. Only Pulsar committers
> can be the mentors as asked by the OSPP.
>
> The project can include feature catchup, ecological tool construction,
> new features of the pulsar and so on.
>
> Welcome to join the Pulsar
> slack channel: #ospp-summer for asking any questions about OSPP.
>
> [1] https://summer-ospp.ac.cn
>
> Thanks,
> Penghui
>
> 
> Template
>
> Project Name:
> Project Description: (at most 1000 words)
> Difficulty Level:
> - [ ] Basic
> - [ ] Advanced
> Project Validation Items:
> Item 1:__
> Item 2:__
> Item 3:__
> …
> Project Mentor:
> Your Name:
> Your Email:
> Your Apache ID:
>

RE: [VOTE] [PIP-154] Max active transaction limitation for transaction coordinator

[Dezhi Liu]

+1 

Thanks,
Dezhi

{CHERRY-PICK concerns] Re: [pulsar] branch master updated: [improve][broker] Tidy up the system topic. (#15252)

[Dave Fisher]

I am not saying we need to cherry pick this change. I want to note that this 
seemingly small change does touch a large number of files with 44 changed. 
Changing that many files may impact future changes that will need to be cherry 
picked. Those cherry picks of future PRs to 2.8.4, 2.9.3, and 2.10.1 will be 
more difficult.

I think we need to make sure that PRs that are made to the master branch that 
will be cherry picked are quickly cherry picked.

Let’s keep our branches in release order!

All The Best,
Dave

> On Apr 24, 2022, at 6:39 PM, peng...@apache.org wrote:
> 
> This is an automated email from the ASF dual-hosted git repository.
> 
> penghui pushed a commit to branch master
> in repository https://gitbox.apache.org/repos/asf/pulsar.git
> 
> 
> The following commit(s) were added to refs/heads/master by this push:
> new aa4df1b3af1 [improve][broker] Tidy up the system topic. (#15252)
> aa4df1b3af1 is described below
> 
> commit aa4df1b3af1298b087661a28b1d5e0f601ac6925
> Author: Jiwei Guo 
> AuthorDate: Mon Apr 25 09:39:42 2022 +0800
> 
>[improve][broker] Tidy up the system topic. (#15252)
> 
>### Motivation
> 
>For #13520, #14643, #14949, we fix some issues related to system topic but 
> result in checking the system topic in different method. So it's better to 
> tidy up the system topic.
> 
>So put these system topic names into a new class called SystemTopicNames.
> ---
> .../apache/pulsar/PulsarClusterMetadataSetup.java  |  3 +-
> .../java/org/apache/pulsar/PulsarStandalone.java   | 17 ++---
> .../PulsarTransactionCoordinatorMetadataSetup.java |  5 +--
> .../org/apache/pulsar/broker/PulsarService.java| 23 ++---
> .../broker/TransactionMetadataStoreService.java|  7 ++--
> .../broker/admin/impl/PersistentTopicsBase.java|  6 ++--
> .../pulsar/broker/admin/impl/TransactionsBase.java | 13 +++
> .../pulsar/broker/namespace/NamespaceService.java  | 12 +--
> .../ResourceUsageTopicTransportManager.java|  8 ++---
> .../pulsar/broker/service/BrokerService.java   | 30 
> .../SystemTopicBasedTopicPoliciesService.java  |  3 +-
> .../service/nonpersistent/NonPersistentTopic.java  |  8 ++---
> .../service/persistent/PersistentSubscription.java |  4 +--
> .../broker/service/persistent/PersistentTopic.java | 16 -
> .../broker/service/persistent/SystemTopic.java | 12 +++
> .../stats/prometheus/TransactionAggregator.java|  4 +--
> .../NamespaceEventsSystemTopicFactory.java | 10 +++---
> .../pendingack/impl/MLPendingAckStore.java |  9 ++---
> .../pulsar/broker/web/PulsarWebResource.java   |  6 +---
> .../pulsar/broker/admin/TopicPoliciesTest.java |  4 +--
> .../v3/AdminApiTransactionMultiBrokerTest.java | 18 +++---
> .../broker/admin/v3/AdminApiTransactionTest.java   | 17 +
> .../ResourceUsageTransportManagerTest.java |  3 +-
> .../BrokerServiceAutoTopicCreationTest.java|  6 ++--
> .../pulsar/broker/service/BrokerServiceTest.java   |  4 +--
> .../pulsar/broker/service/BrokerTestBase.java  | 15 
> .../pulsar/broker/service/ReplicatorTest.java  | 25 --
> .../TransactionMetadataStoreServiceTest.java   |  9 +++--
> .../broker/stats/ManagedLedgerMetricsTest.java |  4 +--
> .../broker/stats/TransactionMetricsTest.java   | 14 
> .../NamespaceEventsSystemTopicServiceTest.java |  4 +--
> .../systopic/PartitionedSystemTopicTest.java   |  4 +--
> .../TopicTransactionBufferRecoverTest.java |  5 ++-
> .../broker/transaction/TransactionConsumeTest.java |  4 +--
> .../pulsar/broker/transaction/TransactionTest.java |  9 +++--
> .../broker/transaction/TransactionTestBase.java| 14 ++--
> .../buffer/TransactionBufferCloseTest.java |  5 ++-
> .../buffer/TransactionLowWaterMarkTest.java|  8 ++---
> .../TransactionMetaStoreAssignmentTest.java|  4 +--
> .../TransactionCoordinatorClientImpl.java  |  8 +++--
> .../SystemTopicNames.java} | 40 ++
> .../org/apache/pulsar/common/naming/TopicName.java |  6 
> .../Oauth2PerformanceTransactionTest.java  |  9 +++--
> .../testclient/PerformanceTransactionTest.java |  8 +++--
> 44 files changed, 234 insertions(+), 209 deletions(-)
> 
> diff --git 
> a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
>  
> b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
> index 5146b513fb5..1dbb480f29d 100644
> --- 
> a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
> +++ 
> b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java
> @@ -37,6 +37,7 @@ import org.apache.pulsar.broker.resources.PulsarResources;
> import org.apache.pulsar.broker.resources.TenantResources;
> import org.apache.pulsar.common.conf.InternalConfigurationData;
> import org.apache.pulsar.common.naming.NamespaceN

Re: Call for projects and mentors for OSPP 2022

[Hang Chen]

Thanks for Penghui and Dianjin.

Project Name: Compaction with tiered storage

Project Description: (at most 1000 words)

Use the tiered storage to store the compacted data to save costs and
to avoid the impaction of the BookKeeper cluster.

The topic compaction reads all the compacted data along with the
additional data after the last compaction. If many topics enabled
topic compaction or with a large size compacted data, the compaction
task will impact the BookKeeper cluster. Leverage the tiered storage,
the compacted data will read from the tiered storage directly so that
the Bookkeeper cluster can provide more reliable and stable service
quality.


Difficulty Level:
- [ ] Basic
- [x] Advanced

Project Validation Items:
Item 1: Write a proposal to describe your design
Item 2: Write a MVP for this proposal
Item 3: Implement this feature and Push the PR out
Item 4: Add unit tests


Project Mentor: Hang Chen
Your Name: Hang Chen
Your Email: chenh...@apache.org
Your Apache ID: chenhang

Yu  于2022年4月25日周一 18:47写道：
>
> Thanks Penghui and Dianjin.
>
> Lili and I will submit applications to improve our user experience on the
> Pulsar website and documentation. We aim to fix some of these issues [1].
> Feel free to comment if you have more suggestions on that, thanks!
>
> [1]
> https://github.com/apache/pulsar/issues?q=is%3Aopen+is%3Aissue+label%3Awebsite
>
>
>
> On Sun, Apr 24, 2022 at 9:23 PM PengHui Li  wrote:
>
> > Hi all,
> >
> > Open Source Promotion Plan, OSPP[1] is open for project registration,
> > like Google Summer of Code. This Plan aims to encourage college students
> > to participate in open-source projects and make contributions.
> >
> > The project idea is open from 4/21, 2022 to 5/9, 2022(UTC+8).
> > Apache Pulsar ended up with 6 projects seats. If you have great ideas,
> > please copy the following template and fill in the fields.
> > Once you have completed your project ideas, you can reply to this email.
> > We will help you register for the program. Note that One mentor can
> > mentor one project at a time and both mentors and students can
> > receive financial awards for completed projects. Only Pulsar committers
> > can be the mentors as asked by the OSPP.
> >
> > The project can include feature catchup, ecological tool construction,
> > new features of the pulsar and so on.
> >
> > Welcome to join the Pulsar
> > slack channel: #ospp-summer for asking any questions about OSPP.
> >
> > [1] https://summer-ospp.ac.cn
> >
> > Thanks,
> > Penghui
> >
> > 
> > Template
> >
> > Project Name:
> > Project Description: (at most 1000 words)
> > Difficulty Level:
> > - [ ] Basic
> > - [ ] Advanced
> > Project Validation Items:
> > Item 1:__
> > Item 2:__
> > Item 3:__
> > …
> > Project Mentor:
> > Your Name:
> > Your Email:
> > Your Apache ID:
> >