date:20240305

Hi,

For the upcoming 3.1.3 release, we encountered a problem where the staging of 
release artifacts failed because the size of apache-pulsar-3.1.3-bin.tar.gz is 
now 455MB. The current limit for SVN file uploads is 429MB. The size of 
apache-pulsar-3.1.2-bin.tar.gz was 428MB. 

The ASF infra ticket can be found at 
https://issues.apache.org/jira/browse/INFRA-25579 (visible to Apache 
committers). I have requested an increase in the limit by about 100MB, from 
45000 bytes (429MB) to 55000 bytes (524MB). 

Meanwhile, we will investigate why the size increased from 428MB to 455MB 
between apache-pulsar-3.1.2-bin.tar.gz and apache-pulsar-3.1.3-bin.tar.gz.

-Lari

Re: [DISCUSS] Broken builds and CI Failures in Maintenance Branches; improving maintenance strategy to address root causes

To enhance our maintenance processes, I've created a guide for
configuring "git mergetool" to resolve merge conflicts:

https://pulsar.apache.org/contribute/setup-mergetool/

For Apache Pulsar core developers, managing git merge conflict
resolution is a necessary task. To streamline this process, it's crucial
to set up tools that aid in visualizing and resolving these conflicts.

I encourage you to follow the guide to set up a git mergetool. Your
feedback is valuable, and you're welcome to contribute improvements
directly to the website. You can do this by creating a PR by editing
https://github.com/apache/pulsar-site/edit/main/contribute/setup-mergetool.md
directly in your browser.

-Lari

On 2024/03/01 14:01:55 Lari Hotari wrote:
> Dear Pulsar Community,
> 
> As we prepare for new releases in our maintenance branches, we have once
> again encountered issues with our cherry-picking process. Some of our
> maintenance branches are currently broken or were recently broken,
> containing compilation errors or failing tests. Many have encountered
> these issues, as we have seen new PRs come in to address the
> problems. The compilation problems are already being addressed by
> Heesung (release manager for 3.0.3) and myself. We aim to resolve these
> issues as soon as possible. Please join #dev channel on Apache Pulsar
> Slack to collaborate in real time to help with this and get updates.
> 
> The cherry-picking process has always been problematic and lacks clear
> documentation in Apache Pulsar. This often leads to our maintenance
> branches breaking, especially as we approach release dates and begin
> cherry-picking fixes. This recurring issue has been the subject of
> multiple discussions over the years. The "feature freeze" in the release
> process does not mitigate the key problem with the cherry-picking
> approach.
> 
> Furthermore, the cherry-picking process is mostly based on tribal
> knowledge and lacks clear documentation. I have previously expressed my
> concerns about this on the mailing list in this thread:
> https://lists.apache.org/thread/69mwjso51kzkrv5xgdmw04d9wngbg8br
> 
> Many problems with cherry-picking arise because cherry-picks occur in
> the wrong order, or dependent changes are not picked. Some dependent
> changes shouldn't be picked since when we have made bug fixes in the
> master branch, it can already contain changes for new features that
> shouldn't be applied to maintenance branches. In those cases
> a backport of the fix is needed and the original developer of the 
> PR might not be available to do this and there could be a significant
> delay for the release if delivering the backport takes time.
> 
> When cherry-picking and backporting is delegated to other developers, 
> in addition to delays, it can lead to coordination problems and commits
> being picked and applied in an order that results in even more merge 
> conflicts. Thankfully, this isn't usually too painful, but it does
> happen once in a while.
> 
> A few days ago, I began working on improving the documentation of the
> current process. I have added a section where I share some thoughts and
> a tool to prevent future problems. You can find the document here:
> https://pulsar.apache.org/contribute/release-process/#cherry-picking-changes-scheduled-for-the-release.
> However, this does not fully describe the current process and will only
> help to some extent.
> 
> The added section should help prevent cherry-picking in the wrong order,
> but it still has many gaps. Many developers do not have proper merge
> conflict resolution tools configured. Without proper 3-way diff
> visualization and merge tools, it's very difficult to resolve many of
> the merge conflicts without making mistakes. This also requires a deep
> understanding of the module where the conflicts occur.
> 
> After we have made the next set of maintenance releases, I plan to
> propose an alternative to the cherry-picking process that will address
> the main issues that the Apache Pulsar project has been struggling with
> every time we do releases.
> 
> The alternative would be to designate the LTS branch as the default
> branch, make bug fixes primarily in the LTS branch, merge fixes to newer
> branches, and cherry-pick to possible older branches. This common
> approach in many projects leverages what Git does well: handling
> development across multiple branches. This solution ensures that our LTS
> branch is always immediately in a releasable state and the branch will
> also become the most stable version of Pulsar since bug fixes are
> continuously evaluated and integrated into the LTS branch with our CI
> where bug fix PRs are targeted to the LTS branch.
> Stability was the original goal of PIP-175 where the LTS concept was
> introduced to Pulsar.
> 
> I hope that our community would be open to making changes to the
> maintenance strategy to help resolve the pain that we have to deal with
> each time we make releases. Sometimes, this "cherry-picking v

Re: Staging release artifacts fail when file size exceeds 429MB, ASF Infra ticket created to increase limit by about 100MB

The size had increased from 428MiB to 434MiB. I had mixed the units 
MiB/1048576B and MB/100B. We found an explanation to the increase. 
byte-buddy-1.14.12.jar adds 4MiB to the release and the remaining 2MiB comes 
from upgraded jars.

-Lari

On 2024/03/05 12:31:28 Lari Hotari wrote:
> Hi,
> 
> For the upcoming 3.1.3 release, we encountered a problem where the staging of 
> release artifacts failed because the size of apache-pulsar-3.1.3-bin.tar.gz 
> is now 455MB. The current limit for SVN file uploads is 429MB. The size of 
> apache-pulsar-3.1.2-bin.tar.gz was 428MB. 
> 
> The ASF infra ticket can be found at 
> https://issues.apache.org/jira/browse/INFRA-25579 (visible to Apache 
> committers). I have requested an increase in the limit by about 100MB, from 
> 45000 bytes (429MB) to 55000 bytes (524MB). 
> 
> Meanwhile, we will investigate why the size increased from 428MB to 455MB 
> between apache-pulsar-3.1.2-bin.tar.gz and apache-pulsar-3.1.3-bin.tar.gz.
> 
> -Lari
>

[VOTE] PIP-332: peek messages from topic subscription with offset value

2024-03-05 Thread Balaji Lakshmanan

Hi,

This is the vote for the PIP-332 to peek messages from a topic subscription
with an offset value.

Can you please reply with your vote?

Here is the link to the PIP pull request
.

Thanks,
Balaji

Re: [ANNOUNCE] New Committer: Kiryl Valkovich

2024-03-05 Thread Asaf Mesika

Congrats! You are definitely a great addition to Pulsar committers!

On Thu, Feb 29, 2024 at 4:58 AM Zixuan Liu  wrote:

> Congrats!
>
> Lari Hotari  于2024年2月27日周二 14:14写道：
>
> > Congrats, Kiryl!
> >
> > -Lari
> >
> > On Tue, 27 Feb 2024 at 06:53, tison  wrote:
> > >
> > > The Apache Pulsar Project Management Committee (PMC) has invited
> > > Kiryl Valkovich https://github.com/visortelle to become a committer,
> > and we
> > > are pleased to announce that he has accepted.
> > >
> > > Welcome and Congratulations, Kiryl Valkovich!
> > >
> > > Please join us in congratulating and welcoming Kiryl onboard!
> > >
> > > Best Regards,
> > >
> > > tison
> > > on behalf of the Pulsar PMC
> >
>

[VOTE] Pulsar Release 3.1.3 Candidate 1

2024-03-05 Thread Ran Gao

This is the first release candidate for Apache Pulsar version 3.1.3.

It fixes the following issues:
https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Amerged+label%3Arelease%2F3.1.3+label%3Acherry-picked%2Fbranch-3.1+

*** Please download, test and verify on this release. This vote will stay
open for at least 72 hours ***

Note that we are verifying upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.1.3-candidate-1/

SHA-512 checksums:

66e2cdbc28af2c1bc5aafa8f0a9c726c2d3bb33128f81632dadbb49d51397c9295d82c797094988eda80494ad217053e609e135fca3506e135f198f7d3ce18ed
 apache-pulsar-3.1.3-bin.tar.gz
ca4ffeeb13b77b1d36f2e8e121a7698f2b989f73f88d1f708745cf45875daf033a8395bb2941f43b5e0f014ee482dfdc01ad805938456511ddfa9777a07c
 apache-pulsar-3.1.3-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1268

The tag to verify:
v3.1.3-candidate-1 (7e28e8404f50d065ed37a590c188b8934cae4a75)
https://github.com/apache/pulsar/commits/v3.1.3-candidate-1

Pulsar's KEYS file containing PGP keys you use to sign the release:
https://downloads.apache.org/pulsar/KEYS

Docker images:

https://hub.docker.com/layers/gaoran10/pulsar/3.1.3-7e28e84/images/sha256-fe637cca3c98137f70237568f12d589b62d00483b5563ac8c63ac0a6a5d33b5c?context=repo

https://hub.docker.com/layers/gaoran10/pulsar-all/3.1.3-7e28e84/images/sha256-219cf93ea289254de127bb52e0c0f1d324a6891fda583ec2ade02c0b0b5a508f?context=repo


Please download the source package, and follow the README to build
and run the Pulsar standalone service.


Regards
Ran Gao (gaoran10)

Re: [VOTE] Pulsar Release 3.2.1 Candidate 1

2024-03-05 Thread Ran Gao

+1(non-binding)

1. verify the GPG signature for the below files
- apache-pulsar-3.2.1-bin.tar.gz.asc
- apache-pulsar-3.2.1-src.tar.gz.asc
2. build the source code
3. test following the release verifying doc.

Regards,
Ran Gao

On 2024/03/05 07:05:05 guo jiwei wrote:
> This is the first release candidate for Apache Pulsar version 3.2.1.
> 
> It fixes the following issues:
> https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Amerged+label%3Arelease%2F3.2.1+label%3Acherry-picked%2Fbranch-3.2+
> 
> *** Please download, test and verify on this release. This vote will stay
> open for at least 72 hours ***
> 
> Note that we are verifying upon the source (tag), binaries are provided for
> convenience.
> 
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.2.1-candidate-1/
> 
> SHA-512 checksums:
> 
> d500ba21305d56b0f7b4355e1270e50ae5f60e5632b6632d66fec6d1178bb9fdf3f24caa709b6b36ea8273e5a2c094868cf30e389154bc8bb6397e7de4f1bf1d
> 
> apache-pulsar-3.2.1-bin.tar.gz
> 
> 7aee1623db6dc95058cd0e7a4f108f8a3f43163d798a8eeeaecf5f335225c04f5a0518ce2dfb2d4fa29b7b5e54d27ba24af82fbb3542eb3ffc9e3602cb577878
> 
> apache-pulsar-3.2.1-src.tar.gz
> 
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1267/
> 
> The tag to verify:
> v3.2.1-candidate-1 (158d5eb670c9fd7b123c204533ac6cf8cb439ccd)
> https://github.com/apache/pulsar/commits/v3.2.1-candidate-1/
> 
> Pulsar's KEYS file containing PGP keys you use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> 
> Docker images:
> 
> pulsar images:
> https://hub.docker.com/layers/technoboy8/pulsar/3.2.1-158d5eb/images/sha256-23f0cbd54b1fb504dcab16dfcce562ed735ab94db1c5f6fabbe9145f3a2d0fa8?context=repo
> 
> pulsar-all images:
> https://hub.docker.com/layers/technoboy8/pulsar-all/3.2.1-158d5eb/images/sha256-80ab1d748eff18655a9c247beba74aa107624b9d3e7cc3a2f22e1246f0d3de83?context=repo
> 
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
> 
> 
> 
> Regards
> Jiwei Guo (Tboy)
>

Re: [ANNOUNCE] New Committer: Kiryl Valkovich

Congratulations!

Enrico

Il Mar 5 Mar 2024, 17:03 Asaf Mesika  ha scritto:

> Congrats! You are definitely a great addition to Pulsar committers!
>
> On Thu, Feb 29, 2024 at 4:58 AM Zixuan Liu  wrote:
>
> > Congrats!
> >
> > Lari Hotari  于2024年2月27日周二 14:14写道：
> >
> > > Congrats, Kiryl!
> > >
> > > -Lari
> > >
> > > On Tue, 27 Feb 2024 at 06:53, tison  wrote:
> > > >
> > > > The Apache Pulsar Project Management Committee (PMC) has invited
> > > > Kiryl Valkovich https://github.com/visortelle to become a committer,
> > > and we
> > > > are pleased to announce that he has accepted.
> > > >
> > > > Welcome and Congratulations, Kiryl Valkovich!
> > > >
> > > > Please join us in congratulating and welcoming Kiryl onboard!
> > > >
> > > > Best Regards,
> > > >
> > > > tison
> > > > on behalf of the Pulsar PMC
> > >
> >
>

Re: Staging release artifacts fail when file size exceeds 429MB, ASF Infra ticket created to increase limit by about 100MB

Il Mar 5 Mar 2024, 14:33 Lari Hotari  ha scritto:

> The size had increased from 428MiB to 434MiB. I had mixed the units
> MiB/1048576B and MB/100B. We found an explanation to the increase.
> byte-buddy-1.14.12.jar adds 4MiB to the release and the remaining 2MiB
> comes from upgraded jars.
>

Thanks

Enrico

>
> -Lari
>
> On 2024/03/05 12:31:28 Lari Hotari wrote:
> > Hi,
> >
> > For the upcoming 3.1.3 release, we encountered a problem where the
> staging of release artifacts failed because the size of
> apache-pulsar-3.1.3-bin.tar.gz is now 455MB. The current limit for SVN file
> uploads is 429MB. The size of apache-pulsar-3.1.2-bin.tar.gz was 428MB.
> >
> > The ASF infra ticket can be found at
> https://issues.apache.org/jira/browse/INFRA-25579 (visible to Apache
> committers). I have requested an increase in the limit by about 100MB, from
> 45000 bytes (429MB) to 55000 bytes (524MB).
> >
> > Meanwhile, we will investigate why the size increased from 428MB to
> 455MB between apache-pulsar-3.1.2-bin.tar.gz and
> apache-pulsar-3.1.3-bin.tar.gz.
> >
> > -Lari
> >
>

Re: [DISCUSS] PIP-324: Alpine Docker images

Since the discussion has been open for a while now, I will proceed to a
vote.

Thanks everyone for bringing out all the perspectives here.

Matteo


--
Matteo Merli



On Fri, Mar 1, 2024 at 10:17 AM Alexander Hall
 wrote:

> Matteo,
>
> Yes, I'm seeing a similar number of vulnerabilities with Stackrox. It's
> important to point out that most of the vulnerabilities are of a low
> severity, with 22 of them being moderate.  Neither Trivy nor Stackrox found
> any vulnerabilities of an important or critical severity. That's a good
> thing.
>
> (TOTAL-COMPONENTS: 41, TOTAL-VULNERABILITIES: 165, LOW: 143, MODERATE: 22,
> IMPORTANT: 0, CRITICAL: 0)
> https://gist.github.com/compuguy/984697f779ec2be2e71f9b3c89db39ae
>
> Thanks,
>
> Alex Hall
> 
>
> -Original Message-
> From: Matteo Merli 
> Sent: Friday, March 1, 2024 12:31 PM
> To: dev@pulsar.apache.org
> Subject: Re: [DISCUSS] PIP-324: Alpine Docker images
>
> [You don't often get email from *REDACTED*. Learn why this is important at
> https://aka.ms/LearnAboutSenderIdentification ]
>
> Hi Alexander,
>
> it doesn't look to be only 4 issues in that image:
>
>
> https://catalog.redhat.com/software/containers/ubi9/ubi/615bcf606feffc5384e8452e
> points to this Docker image
>
>
> registry.access.redhat.com/ubi9/ubi@sha256:1fafb0905264413501df60d90a92ca32df8a2011cbfb4876ddff5ceb20c8f165
> (redhat 9.3)
> Total: 172 (UNKNOWN: 0, LOW: 144, MEDIUM: 28, HIGH: 0, CRITICAL: 0)
>
> Full list:
> https://gist.github.com/merlimat/a4fcea7336cab83ad64dcf01d9b15a1c
> --
> Matteo Merli
> 
>
>
> On Fri, Mar 1, 2024 at 9:19 AM Alexander Hall
> 
> wrote:
>
> > According to Red Hat their latest tagged release for UBI9.3, 9.3-1552,
> has
> > four moderate CVE's (
> >
> https://catalog.redhat.com/software/containers/ubi9/ubi/615bcf606feffc5384e8452e
> ).
> > There is also the option of basing the Pulsar image on the UBI9-minimal
> > image (
> >
> https://catalog.redhat.com/software/containers/ubi9/ubi-minimal/615bd9b4075b022acc111bf5
> ).
> > That may have a better security footprint.
> >
> > Thank You,
> >
> > Alex Hall
> > 
> >
> > -Original Message-
> > From: Matteo Merli 
> > Sent: Thursday, February 15, 2024 12:55 PM
> > To: dev@pulsar.apache.org
> > Subject: ''Re: Re: [DISCUSS] PIP-324: Alpine Docker images
> >
> > [You don't often get email from *REDACTED*. Learn why this is
> > important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > Hi Alex,
> >
> > the situation for UBI9 doesn't look much different from Ubuntu:
> >
> > registry.access.redhat.com/ubi9/ubi (redhat 9.3)
> > Total: 166 (UNKNOWN: 0, LOW: 138, MEDIUM: 28, HIGH: 0, CRITICAL: 0)
> >
> > Full list:
> > https://gist.github.com/merlimat/ba96b91ea49709bb218ddc3906bb9e95>
> >
> > --
> > Matteo Merli
> > 
> >
> >
> > On Thu, Feb 15, 2024 at 9:10 AM Alexander Hall
> >  wrote:
> >
> > > Reviving a previous tangent from this discussion. Using UBI9 as a base
> > > is also a great option. Some end-users use that as a base and copy the
> > > files from the pulsar and pulsar-all containers as an upstream source.
> > >
> > > -Alex H
> > >
> > > -Original Message-
> > > From: Matteo Merli 
> > > Sent: Wednesday, February 14, 2024 2:01 PM
> > > To: david.chris...@discordapp.com.invalid
> > > Cc: dev@pulsar.apache.org
> > > Subject: ''Re: Re: [DISCUSS] PIP-324: Alpine Docker images
> > >
> > > [You don't often get email from *REDACTED*. Learn why this is
> > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > >
> > > Reviving the discussion thread.
> > >
> > >
> > > > For Netty, I think netty-transport-native-epoll is only built
> > > > against glibc (
> > >
> > > https://nett/
> > > y.io%2Fwiki%2Fnative-transports.html%23using-the-linux-native-transpor
> > > t&data=05%7C02%7Cahall%40teknoluxion.com%7C079cd39e87b240332d3108dc2e4
> > > f53d5%7Cfcceb892218c4d6f9e27223a522b9791%7C0%7C0%7C638436165473628082%
> > > 7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik
> > > 1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=kJQqQ5o6ZnuIlqO6Chq0P0Z6axc6Ji
> > > WSP%2F5Qd7bN7xw%3D&reserved=0
> > > ).
> > > > Is there a workaround ?
> > >
> > > Yes, there is a workaround for Netty. It works perfectly fine by
> > > including the GLibc compatibility library. Same for Kinesis producer
> > (side note:
> > > Kinesis SDK is the worst train wreck I've seen in many many years:
> > > it's a
> > > C++ binary that it spawned from Java and communicates through a pipe...
> > > anyway it works fine with the GLibc compatibility lib).
> > >
> > > > Other than that, there is the DNS caching issue Lari mentioned.
> > >
> > > I think the DNS issue was already solved a few releases ago. In any
> > > case, it wouldn't affect Pulsar/BK since we use the Netty DNS client.
> > > In the same way, I believe that JDK also doesn't use the glibc provided
> > DNS client:
> > > that's why we configure the DNS cache directly in the JVM
> configuration.
> > >
> > > >> - Using a smaller base image like Alpine can save spac

[VOTE] PIP-324: Alpine Docker images

Opening the vote thread for PIP-324: Alpine Docker images

https://github.com/apache/pulsar/pull/21716

WIP PR: https://github.com/apache/pulsar/pull/22054

Thank you,
Matteo


--
Matteo Merli

[DISCUSS] Retire pulsar-all Docker image and spin-off Python Functions runtime

The docker image `pulsar-all` is a convenience image that is created on top
of the base `pulsar` image, including all the Pulsar IO connectors as well
as the tiered storage offloaders.

The Dockerfile for `pulsar-all` can be found here:
https://github.com/apache/pulsar/blob/master/docker/pulsar-all/Dockerfile

The resulting image is very big:

```
apachepulsar/pulsar-all 3.1.2
3d1aa250bf6c 2 months ago3.68GB
```

This poses a challenge in many ways:
1. Our CI pipeline needs to build these images and cache them across
different stages of the pipeline
2. It takes a lot of time for release managers to build and push these
images to Docker Hub
3. Users using this image in production see very long download times,
something that can affect the availability of the system (eg: more chances
of a 2nd broker to crash if a restart takes a very long time).
4. It's very unlikely that one user will require all the connectors, most
likely, it would use just 2-3 of them.

The problem is that `pulsar-all` was introduced at a time when there were
~3 Pulsar IO connectors. Right now we do have 35 connectors, with a 1.9 GB
total size.

The proposal here is to drop this image altogether. Users will be able to
construct their own targeted images in a very simple way:

```
FROM apachepulsar/pulsar:latest
RUN mkdir -p connectors && \
cd connectors && \
wget
https://downloads.apache.org/pulsar/pulsar-3.2.0/connectors/pulsar-io-elastic-search-3.2.0.nar
```

### Pulsar Functions Python Runtime

In order to support Python functions runtime, we have been including the
Pulsar base image with quite a bit of dependencies, from `pulsar-client`
Python SDK, to gRPC which is quite a heavy package with many transitive
dependencies.

Given that the vast majority would be using the `pulsar` base image to run
brokers and not python functions, it would make sense to split the Python
support into a different image, like `pulsar-functions-python`, which
extends from the base image and adds all the needed Python dependencies.

This way it will be very easy for users to select the appropriate image and
we wouldn't be carrying a big amount of useless Python dependencies to
users who don't need them.

What are people's opinions with respect to this?

Matteo

--
Matteo Merli

Re: [ANNOUNCE] New Committer: Kiryl Valkovich

2024-03-05 Thread ZhangJian He

Congratulations!

Thanks
ZhangJian He


On Wed, Mar 6, 2024 at 5:17 AM Enrico Olivelli  wrote:

> Congratulations!
>
> Enrico
>
> Il Mar 5 Mar 2024, 17:03 Asaf Mesika  ha scritto:
>
> > Congrats! You are definitely a great addition to Pulsar committers!
> >
> > On Thu, Feb 29, 2024 at 4:58 AM Zixuan Liu  wrote:
> >
> > > Congrats!
> > >
> > > Lari Hotari  于2024年2月27日周二 14:14写道：
> > >
> > > > Congrats, Kiryl!
> > > >
> > > > -Lari
> > > >
> > > > On Tue, 27 Feb 2024 at 06:53, tison  wrote:
> > > > >
> > > > > The Apache Pulsar Project Management Committee (PMC) has invited
> > > > > Kiryl Valkovich https://github.com/visortelle to become a
> committer,
> > > > and we
> > > > > are pleased to announce that he has accepted.
> > > > >
> > > > > Welcome and Congratulations, Kiryl Valkovich!
> > > > >
> > > > > Please join us in congratulating and welcoming Kiryl onboard!
> > > > >
> > > > > Best Regards,
> > > > >
> > > > > tison
> > > > > on behalf of the Pulsar PMC
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Committer: Asaf Mesika

2024-03-05 Thread Kalwit S

Congratulations.!


On Tue, Feb 20, 2024 at 8:50 AM Lari Hotari  wrote:

> The Apache Pulsar Project Management Committee (PMC) has invited
> Asaf Mesika https://github.com/asafm to become a committer and we
> are pleased to announce that he has accepted.
>
> Welcome and Congratulations, Asaf Mesika!
>
> Please join us in congratulating and welcoming Asaf onboard!
>
> Best Regards,
>
> Lari Hotari
> on behalf of the Pulsar PMC
>

Re: [ANNOUNCE] New Committer: Kiryl Valkovich

2024-03-05 Thread Kalwit S

Congratulations!

On Tue, Mar 5, 2024 at 3:13 PM ZhangJian He  wrote:

> Congratulations!
>
> Thanks
> ZhangJian He
>
>
> On Wed, Mar 6, 2024 at 5:17 AM Enrico Olivelli 
> wrote:
>
> > Congratulations!
> >
> > Enrico
> >
> > Il Mar 5 Mar 2024, 17:03 Asaf Mesika  ha scritto:
> >
> > > Congrats! You are definitely a great addition to Pulsar committers!
> > >
> > > On Thu, Feb 29, 2024 at 4:58 AM Zixuan Liu  wrote:
> > >
> > > > Congrats!
> > > >
> > > > Lari Hotari  于2024年2月27日周二 14:14写道：
> > > >
> > > > > Congrats, Kiryl!
> > > > >
> > > > > -Lari
> > > > >
> > > > > On Tue, 27 Feb 2024 at 06:53, tison  wrote:
> > > > > >
> > > > > > The Apache Pulsar Project Management Committee (PMC) has invited
> > > > > > Kiryl Valkovich https://github.com/visortelle to become a
> > committer,
> > > > > and we
> > > > > > are pleased to announce that he has accepted.
> > > > > >
> > > > > > Welcome and Congratulations, Kiryl Valkovich!
> > > > > >
> > > > > > Please join us in congratulating and welcoming Kiryl onboard!
> > > > > >
> > > > > > Best Regards,
> > > > > >
> > > > > > tison
> > > > > > on behalf of the Pulsar PMC
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Retire pulsar-all Docker image and spin-off Python Functions runtime

These are very welcome changes! Let's go ahead asap.

-Lari

On Wed, 6 Mar 2024 at 01:04, Matteo Merli  wrote:
>
> The docker image `pulsar-all` is a convenience image that is created on top
> of the base `pulsar` image, including all the Pulsar IO connectors as well
> as the tiered storage offloaders.
>
> The Dockerfile for `pulsar-all` can be found here:
> https://github.com/apache/pulsar/blob/master/docker/pulsar-all/Dockerfile
>
> The resulting image is very big:
>
> ```
> apachepulsar/pulsar-all   3.1.2
>  3d1aa250bf6c   2 months ago3.68GB
> ```
>
> This poses a challenge in many ways:
>  1. Our CI pipeline needs to build these images and cache them across
> different stages of the pipeline
>  2. It takes a lot of time for release managers to build and push these
> images to Docker Hub
>  3. Users using this image in production see very long download times,
> something that can affect the availability of the system (eg: more chances
> of a 2nd broker to crash if a restart takes a very long time).
>  4. It's very unlikely that one user will require all the connectors, most
> likely, it would use just 2-3 of them.
>
> The problem is that `pulsar-all` was introduced at a time when there were
> ~3 Pulsar IO connectors. Right now we do have 35 connectors, with a 1.9 GB
> total size.
>
> The proposal here is to drop this image altogether. Users will be able to
> construct their own targeted images in a very simple way:
>
> ```
> FROM apachepulsar/pulsar:latest
> RUN mkdir -p connectors && \
> cd connectors && \
> wget
> https://downloads.apache.org/pulsar/pulsar-3.2.0/connectors/pulsar-io-elastic-search-3.2.0.nar
> ```
>
>
>
> ### Pulsar Functions Python Runtime
>
> In order to support Python functions runtime, we have been including the
> Pulsar base image with quite a bit of dependencies, from `pulsar-client`
> Python SDK, to gRPC which is quite a heavy package with many transitive
> dependencies.
>
> Given that the vast majority would be using the `pulsar` base image to run
> brokers and not python functions, it would make sense to split the Python
> support into a different image, like `pulsar-functions-python`, which
> extends from the base image and adds all the needed Python dependencies.
>
> This way it will be very easy for users to select the appropriate image and
> we wouldn't be carrying a big amount of useless Python dependencies to
> users who don't need them.
>
>
> What are people's opinions with respect to this?
>
> Matteo
>
> --
> Matteo Merli
>

Re: [VOTE] PIP-324: Alpine Docker images

+1 (binding)


-Lari

On Tue, 5 Mar 2024 at 23:48, Matteo Merli  wrote:
>
> Opening the vote thread for PIP-324: Alpine Docker images
>
> https://github.com/apache/pulsar/pull/21716
>
> WIP PR: https://github.com/apache/pulsar/pull/22054
>
> Thank you,
> Matteo
>
>
> --
> Matteo Merli
>

Re: [VOTE] PIP-324: Alpine Docker images

2024-03-05 Thread PengHui Li

+1 (binding)

Regards,
Penghui

On Wed, Mar 6, 2024 at 7:54 AM Lari Hotari  wrote:

> +1 (binding)
>
>
> -Lari
>
> On Tue, 5 Mar 2024 at 23:48, Matteo Merli  wrote:
> >
> > Opening the vote thread for PIP-324: Alpine Docker images
> >
> > https://github.com/apache/pulsar/pull/21716
> >
> > WIP PR: https://github.com/apache/pulsar/pull/22054
> >
> > Thank you,
> > Matteo
> >
> >
> > --
> > Matteo Merli
> > 
>

Re: [ANNOUNCE] New Committer: Kiryl Valkovich

2024-03-05 Thread PengHui Li

Congratulations!

On Wed, Mar 6, 2024 at 7:35 AM Kalwit S  wrote:

> Congratulations!
>
> On Tue, Mar 5, 2024 at 3:13 PM ZhangJian He  wrote:
>
> > Congratulations!
> >
> > Thanks
> > ZhangJian He
> >
> >
> > On Wed, Mar 6, 2024 at 5:17 AM Enrico Olivelli 
> > wrote:
> >
> > > Congratulations!
> > >
> > > Enrico
> > >
> > > Il Mar 5 Mar 2024, 17:03 Asaf Mesika  ha
> scritto:
> > >
> > > > Congrats! You are definitely a great addition to Pulsar committers!
> > > >
> > > > On Thu, Feb 29, 2024 at 4:58 AM Zixuan Liu 
> wrote:
> > > >
> > > > > Congrats!
> > > > >
> > > > > Lari Hotari  于2024年2月27日周二 14:14写道：
> > > > >
> > > > > > Congrats, Kiryl!
> > > > > >
> > > > > > -Lari
> > > > > >
> > > > > > On Tue, 27 Feb 2024 at 06:53, tison 
> wrote:
> > > > > > >
> > > > > > > The Apache Pulsar Project Management Committee (PMC) has
> invited
> > > > > > > Kiryl Valkovich https://github.com/visortelle to become a
> > > committer,
> > > > > > and we
> > > > > > > are pleased to announce that he has accepted.
> > > > > > >
> > > > > > > Welcome and Congratulations, Kiryl Valkovich!
> > > > > > >
> > > > > > > Please join us in congratulating and welcoming Kiryl onboard!
> > > > > > >
> > > > > > > Best Regards,
> > > > > > >
> > > > > > > tison
> > > > > > > on behalf of the Pulsar PMC
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Retire pulsar-all Docker image and spin-off Python Functions runtime

2024-03-05 Thread Dave Fisher

What would be the name of the image that contains the functions runtime?

Best,
Dave

> On Mar 5, 2024, at 6:37 PM, Lari Hotari  wrote:
> 
> These are very welcome changes! Let's go ahead asap.
> 
> -Lari
> 
> On Wed, 6 Mar 2024 at 01:04, Matteo Merli  wrote:
>> 
>> The docker image `pulsar-all` is a convenience image that is created on top
>> of the base `pulsar` image, including all the Pulsar IO connectors as well
>> as the tiered storage offloaders.
>> 
>> The Dockerfile for `pulsar-all` can be found here:
>> https://github.com/apache/pulsar/blob/master/docker/pulsar-all/Dockerfile
>> 
>> The resulting image is very big:
>> 
>> ```
>> apachepulsar/pulsar-all   3.1.2
>> 3d1aa250bf6c   2 months ago3.68GB
>> ```
>> 
>> This poses a challenge in many ways:
>> 1. Our CI pipeline needs to build these images and cache them across
>> different stages of the pipeline
>> 2. It takes a lot of time for release managers to build and push these
>> images to Docker Hub
>> 3. Users using this image in production see very long download times,
>> something that can affect the availability of the system (eg: more chances
>> of a 2nd broker to crash if a restart takes a very long time).
>> 4. It's very unlikely that one user will require all the connectors, most
>> likely, it would use just 2-3 of them.
>> 
>> The problem is that `pulsar-all` was introduced at a time when there were
>> ~3 Pulsar IO connectors. Right now we do have 35 connectors, with a 1.9 GB
>> total size.
>> 
>> The proposal here is to drop this image altogether. Users will be able to
>> construct their own targeted images in a very simple way:
>> 
>> ```
>> FROM apachepulsar/pulsar:latest
>> RUN mkdir -p connectors && \
>>cd connectors && \
>>wget
>> https://downloads.apache.org/pulsar/pulsar-3.2.0/connectors/pulsar-io-elastic-search-3.2.0.nar
>> ```
>> 
>> 
>> 
>> ### Pulsar Functions Python Runtime
>> 
>> In order to support Python functions runtime, we have been including the
>> Pulsar base image with quite a bit of dependencies, from `pulsar-client`
>> Python SDK, to gRPC which is quite a heavy package with many transitive
>> dependencies.
>> 
>> Given that the vast majority would be using the `pulsar` base image to run
>> brokers and not python functions, it would make sense to split the Python
>> support into a different image, like `pulsar-functions-python`, which
>> extends from the base image and adds all the needed Python dependencies.
>> 
>> This way it will be very easy for users to select the appropriate image and
>> we wouldn't be carrying a big amount of useless Python dependencies to
>> users who don't need them.
>> 
>> 
>> What are people's opinions with respect to this?
>> 
>> Matteo
>> 
>> --
>> Matteo Merli
>>

[VOTE] Pulsar Release 2.10.6 Candidate 1

2024-03-05 Thread Xiangying Meng

This is the first release candidate for Apache Pulsar, version 2.10.6.

It fixes the following issues:
https://github.com/apache/pulsar/pulls?q=is:pr+label:cherry-picked/branch-2.10+label:release/2.10.6+is:closed

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.6-candidate-1/

SHA-512 checksums:
09f29265f8173331d4c05b470c4e77a31146172b27ef333f45d8c8a19074ef25061cb1e80872fc45c323c9ce8e2e17989c6df5d991ef84c4d245197303d9e6d7
 apache-pulsar-2.10.6-bin.tar.gz
49c8836882818c6f38748dae26b51c598f163606c16993a3287ab1ce9f853a4aaa43c6729c1b6f6957738b4dead3818cd12026da68b328eb2d4ac0d0214957bb
 apache-pulsar-2.10.6-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1270

The tag to be voted upon:
v2.10.6-candidate-1 (9c29b76ff2be865429ad44df8683aec80deacfba)
https://github.com/apache/pulsar/releases/tag/v2.10.6-candidate-1

Pulsar's KEYS file containing PGP keys you use to sign the release:
https://downloads.apache.org/pulsar/KEYS

Docker images:


https://hub.docker.com/layers/xiangyingmeng/pulsar/2.10.6/images/sha256-bf8f36e49ff44ef810ab2c76742121205e51d3a04c79afdb5d288c7d8a06443f?context=repo


https://hub.docker.com/layers/xiangyingmeng/pulsar-all/2.10.6/images/sha256-1b3a10db12f6d5a0acd2d4ed73eb11864b6b598294bb905b6ede34aef1157f23?context=repo

Please download the source package, and follow the README to build
and run the Pulsar standalone service.

(Apache committer criteria) [ANNOUNCE] New Committer: Asaf Mesika

2024-03-05 Thread Kalwit S

Congratulations Asaf.

Btw, does the Apache project have any promotion criteria for committers? I
looked at Asaf's commits at
https://github.com/apache/pulsar/commits?author=asafm  and found that 99%
of the commits are simple documentation changes and 1% are related to PIP
monitoring. Most of the PIP monitoring involves adding plugins to existing
metrics APIs. He has also contributed to the PIP reviews, but his
contribution is more philosophical rather than technical. Most of his
comments are comparing Pulsar to other projects, rather than focusing on
the internal insights that Pulsar brings to the table. Our team has been
running production traffic using Apache Pulsar for over a year now. We have
tried several different versions of Pulsar (which we have to constantly
upgrade due to unknown issues in live production traffic) and have never
seen a stable version of Pulsar. Our team has also tried to submit multiple
enhancements and also PIP, but most of them are bogged down by reviewers
who are very new to Pulsar, might not understand messaging correctly, or
don’t find such enhancements useful for their usecases.
I would say that most of these reviewers are brand new to Pulsar, and
almost all of them are from the same company that is also the provider of
Pulsar. The same company controls Pulsar, prevents others from
contributing, and avoids having non-pulsar committers. This is why we
wanted to replace our existing Kafka cluster with Pulsar but we see no
difference in Pulsar provider and Confluent because Pulsar is also largely
controlled by one provider and this company's reviewers are not well-versed
in such systems.
In addition, we can see that almost all the reviewers are from the same
company, and PIP approval requires 3+ votes, which means only specific
reviewers belonging to one company participate and because of that, no one
can promote their improvements without the approval of the provider
company. The Pulsar community needs to break away from the monopolies of
the provider companies, start focusing on stable releases, and let other
companies make their enhancements to meet their requirements, and
experienced contributors or Pulsar creators should be active to prevent
unfairness in the community.

On Tue, Mar 5, 2024 at 3:31 PM Kalwit S  wrote:

> Congratulations.!
>
>
> On Tue, Feb 20, 2024 at 8:50 AM Lari Hotari  wrote:
>
>> The Apache Pulsar Project Management Committee (PMC) has invited
>> Asaf Mesika https://github.com/asafm to become a committer and we
>> are pleased to announce that he has accepted.
>>
>> Welcome and Congratulations, Asaf Mesika!
>>
>> Please join us in congratulating and welcoming Asaf onboard!
>>
>> Best Regards,
>>
>> Lari Hotari
>> on behalf of the Pulsar PMC
>>
>

Re: [DISCUSS] Retire pulsar-all Docker image and spin-off Python Functions runtime

I was proposing `pulsar-functions-python`, though I'm open to any other
name
--
Matteo Merli



On Tue, Mar 5, 2024 at 6:43 PM Dave Fisher  wrote:

> What would be the name of the image that contains the functions runtime?
>
> Best,
> Dave
>
> > On Mar 5, 2024, at 6:37 PM, Lari Hotari  wrote:
> >
> > These are very welcome changes! Let's go ahead asap.
> >
> > -Lari
> >
> > On Wed, 6 Mar 2024 at 01:04, Matteo Merli 
> wrote:
> >>
> >> The docker image `pulsar-all` is a convenience image that is created on
> top
> >> of the base `pulsar` image, including all the Pulsar IO connectors as
> well
> >> as the tiered storage offloaders.
> >>
> >> The Dockerfile for `pulsar-all` can be found here:
> >>
> https://github.com/apache/pulsar/blob/master/docker/pulsar-all/Dockerfile
> >>
> >> The resulting image is very big:
> >>
> >> ```
> >> apachepulsar/pulsar-all   3.1.2
> >> 3d1aa250bf6c   2 months ago3.68GB
> >> ```
> >>
> >> This poses a challenge in many ways:
> >> 1. Our CI pipeline needs to build these images and cache them across
> >> different stages of the pipeline
> >> 2. It takes a lot of time for release managers to build and push these
> >> images to Docker Hub
> >> 3. Users using this image in production see very long download times,
> >> something that can affect the availability of the system (eg: more
> chances
> >> of a 2nd broker to crash if a restart takes a very long time).
> >> 4. It's very unlikely that one user will require all the connectors,
> most
> >> likely, it would use just 2-3 of them.
> >>
> >> The problem is that `pulsar-all` was introduced at a time when there
> were
> >> ~3 Pulsar IO connectors. Right now we do have 35 connectors, with a 1.9
> GB
> >> total size.
> >>
> >> The proposal here is to drop this image altogether. Users will be able
> to
> >> construct their own targeted images in a very simple way:
> >>
> >> ```
> >> FROM apachepulsar/pulsar:latest
> >> RUN mkdir -p connectors && \
> >>cd connectors && \
> >>wget
> >>
> https://downloads.apache.org/pulsar/pulsar-3.2.0/connectors/pulsar-io-elastic-search-3.2.0.nar
> >> ```
> >>
> >>
> >>
> >> ### Pulsar Functions Python Runtime
> >>
> >> In order to support Python functions runtime, we have been including the
> >> Pulsar base image with quite a bit of dependencies, from `pulsar-client`
> >> Python SDK, to gRPC which is quite a heavy package with many transitive
> >> dependencies.
> >>
> >> Given that the vast majority would be using the `pulsar` base image to
> run
> >> brokers and not python functions, it would make sense to split the
> Python
> >> support into a different image, like `pulsar-functions-python`, which
> >> extends from the base image and adds all the needed Python dependencies.
> >>
> >> This way it will be very easy for users to select the appropriate image
> and
> >> we wouldn't be carrying a big amount of useless Python dependencies to
> >> users who don't need them.
> >>
> >>
> >> What are people's opinions with respect to this?
> >>
> >> Matteo
> >>
> >> --
> >> Matteo Merli
> >> 
>
>

Re: (Apache committer criteria) [ANNOUNCE] New Committer: Asaf Mesika

I will not enter into debating the long list of grievances here, though I
thought I needed to clarify at least 2 points:

1. You can ask any questions and direct any feedback to the PMC (and if
you're not happy with the response you can take it all the way up to the
ASF Board), but personal attacks are not OK here

2. I don't think it's good looking when you're reacting to people
disagreeing with you by claiming they either are incompetent or have some
hidden agenda. Perhaps trying to understand why they disagreed with you
would be more helpful.


Matteo

--
Matteo Merli



On Tue, Mar 5, 2024 at 7:22 PM Kalwit S  wrote:

> Congratulations Asaf.
>
> Btw, does the Apache project have any promotion criteria for committers? I
> looked at Asaf's commits at
> https://github.com/apache/pulsar/commits?author=asafm  and found that 99%
> of the commits are simple documentation changes and 1% are related to PIP
> monitoring. Most of the PIP monitoring involves adding plugins to existing
> metrics APIs. He has also contributed to the PIP reviews, but his
> contribution is more philosophical rather than technical. Most of his
> comments are comparing Pulsar to other projects, rather than focusing on
> the internal insights that Pulsar brings to the table. Our team has been
> running production traffic using Apache Pulsar for over a year now. We have
> tried several different versions of Pulsar (which we have to constantly
> upgrade due to unknown issues in live production traffic) and have never
> seen a stable version of Pulsar. Our team has also tried to submit multiple
> enhancements and also PIP, but most of them are bogged down by reviewers
> who are very new to Pulsar, might not understand messaging correctly, or
> don’t find such enhancements useful for their usecases.
> I would say that most of these reviewers are brand new to Pulsar, and
> almost all of them are from the same company that is also the provider of
> Pulsar. The same company controls Pulsar, prevents others from
> contributing, and avoids having non-pulsar committers. This is why we
> wanted to replace our existing Kafka cluster with Pulsar but we see no
> difference in Pulsar provider and Confluent because Pulsar is also largely
> controlled by one provider and this company's reviewers are not well-versed
> in such systems.
> In addition, we can see that almost all the reviewers are from the same
> company, and PIP approval requires 3+ votes, which means only specific
> reviewers belonging to one company participate and because of that, no one
> can promote their improvements without the approval of the provider
> company. The Pulsar community needs to break away from the monopolies of
> the provider companies, start focusing on stable releases, and let other
> companies make their enhancements to meet their requirements, and
> experienced contributors or Pulsar creators should be active to prevent
> unfairness in the community.
>
> On Tue, Mar 5, 2024 at 3:31 PM Kalwit S  wrote:
>
> > Congratulations.!
> >
> >
> > On Tue, Feb 20, 2024 at 8:50 AM Lari Hotari  wrote:
> >
> >> The Apache Pulsar Project Management Committee (PMC) has invited
> >> Asaf Mesika https://github.com/asafm to become a committer and we
> >> are pleased to announce that he has accepted.
> >>
> >> Welcome and Congratulations, Asaf Mesika!
> >>
> >> Please join us in congratulating and welcoming Asaf onboard!
> >>
> >> Best Regards,
> >>
> >> Lari Hotari
> >> on behalf of the Pulsar PMC
> >>
> >
>

Re: [DISCUSS] Retire pulsar-all Docker image and spin-off Python Functions runtime

2024-03-05 Thread Nicolò Boschi

+1, great ideas

Let's make sure there's a dedicated section in the docs on how to "migrate"
from pulsar-all:3.2.0 to "build your own -all image"

Nicolò Boschi


Il giorno mer 6 mar 2024 alle ore 04:22 Matteo Merli 
ha scritto:

> I was proposing `pulsar-functions-python`, though I'm open to any other
> name
> --
> Matteo Merli
> 
>
>
> On Tue, Mar 5, 2024 at 6:43 PM Dave Fisher  wrote:
>
> > What would be the name of the image that contains the functions runtime?
> >
> > Best,
> > Dave
> >
> > > On Mar 5, 2024, at 6:37 PM, Lari Hotari  wrote:
> > >
> > > These are very welcome changes! Let's go ahead asap.
> > >
> > > -Lari
> > >
> > > On Wed, 6 Mar 2024 at 01:04, Matteo Merli 
> > wrote:
> > >>
> > >> The docker image `pulsar-all` is a convenience image that is created
> on
> > top
> > >> of the base `pulsar` image, including all the Pulsar IO connectors as
> > well
> > >> as the tiered storage offloaders.
> > >>
> > >> The Dockerfile for `pulsar-all` can be found here:
> > >>
> >
> https://github.com/apache/pulsar/blob/master/docker/pulsar-all/Dockerfile
> > >>
> > >> The resulting image is very big:
> > >>
> > >> ```
> > >> apachepulsar/pulsar-all   3.1.2
> > >> 3d1aa250bf6c   2 months ago3.68GB
> > >> ```
> > >>
> > >> This poses a challenge in many ways:
> > >> 1. Our CI pipeline needs to build these images and cache them across
> > >> different stages of the pipeline
> > >> 2. It takes a lot of time for release managers to build and push these
> > >> images to Docker Hub
> > >> 3. Users using this image in production see very long download times,
> > >> something that can affect the availability of the system (eg: more
> > chances
> > >> of a 2nd broker to crash if a restart takes a very long time).
> > >> 4. It's very unlikely that one user will require all the connectors,
> > most
> > >> likely, it would use just 2-3 of them.
> > >>
> > >> The problem is that `pulsar-all` was introduced at a time when there
> > were
> > >> ~3 Pulsar IO connectors. Right now we do have 35 connectors, with a
> 1.9
> > GB
> > >> total size.
> > >>
> > >> The proposal here is to drop this image altogether. Users will be able
> > to
> > >> construct their own targeted images in a very simple way:
> > >>
> > >> ```
> > >> FROM apachepulsar/pulsar:latest
> > >> RUN mkdir -p connectors && \
> > >>cd connectors && \
> > >>wget
> > >>
> >
> https://downloads.apache.org/pulsar/pulsar-3.2.0/connectors/pulsar-io-elastic-search-3.2.0.nar
> > >> ```
> > >>
> > >>
> > >>
> > >> ### Pulsar Functions Python Runtime
> > >>
> > >> In order to support Python functions runtime, we have been including
> the
> > >> Pulsar base image with quite a bit of dependencies, from
> `pulsar-client`
> > >> Python SDK, to gRPC which is quite a heavy package with many
> transitive
> > >> dependencies.
> > >>
> > >> Given that the vast majority would be using the `pulsar` base image to
> > run
> > >> brokers and not python functions, it would make sense to split the
> > Python
> > >> support into a different image, like `pulsar-functions-python`, which
> > >> extends from the base image and adds all the needed Python
> dependencies.
> > >>
> > >> This way it will be very easy for users to select the appropriate
> image
> > and
> > >> we wouldn't be carrying a big amount of useless Python dependencies to
> > >> users who don't need them.
> > >>
> > >>
> > >> What are people's opinions with respect to this?
> > >>
> > >> Matteo
> > >>
> > >> --
> > >> Matteo Merli
> > >> 
> >
> >
>

Re: [VOTE] Pulsar Release 3.1.3 Candidate 1

2024-03-05 Thread guo jiwei

+1 (binding)

- Built from source
- Checked the signatures
- Run standalone
- Checked producer and consumer
- Verified the Cassandra connector
- Verified the Stateful function

Regards
Jiwei Guo (Tboy)


On Wed, Mar 6, 2024 at 2:25 AM Ran Gao  wrote:

> This is the first release candidate for Apache Pulsar version 3.1.3.
>
> It fixes the following issues:
>
> https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Amerged+label%3Arelease%2F3.1.3+label%3Acherry-picked%2Fbranch-3.1+
>
> *** Please download, test and verify on this release. This vote will stay
> open for at least 72 hours ***
>
> Note that we are verifying upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.1.3-candidate-1/
>
> SHA-512 checksums:
>
>
> 66e2cdbc28af2c1bc5aafa8f0a9c726c2d3bb33128f81632dadbb49d51397c9295d82c797094988eda80494ad217053e609e135fca3506e135f198f7d3ce18ed
>  apache-pulsar-3.1.3-bin.tar.gz
>
> ca4ffeeb13b77b1d36f2e8e121a7698f2b989f73f88d1f708745cf45875daf033a8395bb2941f43b5e0f014ee482dfdc01ad805938456511ddfa9777a07c
>  apache-pulsar-3.1.3-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1268
>
> The tag to verify:
> v3.1.3-candidate-1 (7e28e8404f50d065ed37a590c188b8934cae4a75)
> https://github.com/apache/pulsar/commits/v3.1.3-candidate-1
>
> Pulsar's KEYS file containing PGP keys you use to sign the release:
> https://downloads.apache.org/pulsar/KEYS
>
> Docker images:
>
>
> https://hub.docker.com/layers/gaoran10/pulsar/3.1.3-7e28e84/images/sha256-fe637cca3c98137f70237568f12d589b62d00483b5563ac8c63ac0a6a5d33b5c?context=repo
>
>
> https://hub.docker.com/layers/gaoran10/pulsar-all/3.1.3-7e28e84/images/sha256-219cf93ea289254de127bb52e0c0f1d324a6891fda583ec2ade02c0b0b5a508f?context=repo
>
>
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
>
>
> Regards
> Ran Gao (gaoran10)
>

Re: [DISCUSS] Retire pulsar-all Docker image and spin-off Python Functions runtime

2024-03-05 Thread Zixuan Liu

+1

This is a good idea, and then we must provide a document on building the
own connector image and python functions runtime image.

Thanks,
Zixuan

Matteo Merli  于2024年3月6日周三 07:04写道：

> The docker image `pulsar-all` is a convenience image that is created on top
> of the base `pulsar` image, including all the Pulsar IO connectors as well
> as the tiered storage offloaders.
>
> The Dockerfile for `pulsar-all` can be found here:
> https://github.com/apache/pulsar/blob/master/docker/pulsar-all/Dockerfile
>
> The resulting image is very big:
>
> ```
> apachepulsar/pulsar-all   3.1.2
>  3d1aa250bf6c   2 months ago3.68GB
> ```
>
> This poses a challenge in many ways:
>  1. Our CI pipeline needs to build these images and cache them across
> different stages of the pipeline
>  2. It takes a lot of time for release managers to build and push these
> images to Docker Hub
>  3. Users using this image in production see very long download times,
> something that can affect the availability of the system (eg: more chances
> of a 2nd broker to crash if a restart takes a very long time).
>  4. It's very unlikely that one user will require all the connectors, most
> likely, it would use just 2-3 of them.
>
> The problem is that `pulsar-all` was introduced at a time when there were
> ~3 Pulsar IO connectors. Right now we do have 35 connectors, with a 1.9 GB
> total size.
>
> The proposal here is to drop this image altogether. Users will be able to
> construct their own targeted images in a very simple way:
>
> ```
> FROM apachepulsar/pulsar:latest
> RUN mkdir -p connectors && \
> cd connectors && \
> wget
>
> https://downloads.apache.org/pulsar/pulsar-3.2.0/connectors/pulsar-io-elastic-search-3.2.0.nar
> ```
>
>
>
> ### Pulsar Functions Python Runtime
>
> In order to support Python functions runtime, we have been including the
> Pulsar base image with quite a bit of dependencies, from `pulsar-client`
> Python SDK, to gRPC which is quite a heavy package with many transitive
> dependencies.
>
> Given that the vast majority would be using the `pulsar` base image to run
> brokers and not python functions, it would make sense to split the Python
> support into a different image, like `pulsar-functions-python`, which
> extends from the base image and adds all the needed Python dependencies.
>
> This way it will be very easy for users to select the appropriate image and
> we wouldn't be carrying a big amount of useless Python dependencies to
> users who don't need them.
>
>
> What are people's opinions with respect to this?
>
> Matteo
>
> --
> Matteo Merli
> 
>

Re: [DISCUSS] Retire pulsar-all Docker image and spin-off Python Functions runtime

+1

Great idea

Enrico

Il Mer 6 Mar 2024, 08:23 Zixuan Liu  ha scritto:

> +1
>
> This is a good idea, and then we must provide a document on building the
> own connector image and python functions runtime image.
>
> Thanks,
> Zixuan
>
> Matteo Merli  于2024年3月6日周三 07:04写道：
>
> > The docker image `pulsar-all` is a convenience image that is created on
> top
> > of the base `pulsar` image, including all the Pulsar IO connectors as
> well
> > as the tiered storage offloaders.
> >
> > The Dockerfile for `pulsar-all` can be found here:
> >
> https://github.com/apache/pulsar/blob/master/docker/pulsar-all/Dockerfile
> >
> > The resulting image is very big:
> >
> > ```
> > apachepulsar/pulsar-all   3.1.2
> >  3d1aa250bf6c   2 months ago3.68GB
> > ```
> >
> > This poses a challenge in many ways:
> >  1. Our CI pipeline needs to build these images and cache them across
> > different stages of the pipeline
> >  2. It takes a lot of time for release managers to build and push these
> > images to Docker Hub
> >  3. Users using this image in production see very long download times,
> > something that can affect the availability of the system (eg: more
> chances
> > of a 2nd broker to crash if a restart takes a very long time).
> >  4. It's very unlikely that one user will require all the connectors,
> most
> > likely, it would use just 2-3 of them.
> >
> > The problem is that `pulsar-all` was introduced at a time when there were
> > ~3 Pulsar IO connectors. Right now we do have 35 connectors, with a 1.9
> GB
> > total size.
> >
> > The proposal here is to drop this image altogether. Users will be able to
> > construct their own targeted images in a very simple way:
> >
> > ```
> > FROM apachepulsar/pulsar:latest
> > RUN mkdir -p connectors && \
> > cd connectors && \
> > wget
> >
> >
> https://downloads.apache.org/pulsar/pulsar-3.2.0/connectors/pulsar-io-elastic-search-3.2.0.nar
> > ```
> >
> >
> >
> > ### Pulsar Functions Python Runtime
> >
> > In order to support Python functions runtime, we have been including the
> > Pulsar base image with quite a bit of dependencies, from `pulsar-client`
> > Python SDK, to gRPC which is quite a heavy package with many transitive
> > dependencies.
> >
> > Given that the vast majority would be using the `pulsar` base image to
> run
> > brokers and not python functions, it would make sense to split the Python
> > support into a different image, like `pulsar-functions-python`, which
> > extends from the base image and adds all the needed Python dependencies.
> >
> > This way it will be very easy for users to select the appropriate image
> and
> > we wouldn't be carrying a big amount of useless Python dependencies to
> > users who don't need them.
> >
> >
> > What are people's opinions with respect to this?
> >
> > Matteo
> >
> > --
> > Matteo Merli
> > 
> >
>

Re: (Apache committer criteria) [ANNOUNCE] New Committer: Asaf Mesika

Kalwit,
All the committers are invited by the PMC, you can reach out to
priv...@pulsar.apache.org if you have any problems.

Being a committer is not only about doing code contributions, but also
talking care of the project and the community.

I am sorry to hear that you feel that your contributions are blocked,
please start a new thread with this problem.

This is a community,  it is not a product managed by one single company.

Best regards
Enrico Olivelli

Il Mer 6 Mar 2024, 06:27 Matteo Merli  ha scritto:

> I will not enter into debating the long list of grievances here, though I
> thought I needed to clarify at least 2 points:
>
> 1. You can ask any questions and direct any feedback to the PMC (and if
> you're not happy with the response you can take it all the way up to the
> ASF Board), but personal attacks are not OK here
>
> 2. I don't think it's good looking when you're reacting to people
> disagreeing with you by claiming they either are incompetent or have some
> hidden agenda. Perhaps trying to understand why they disagreed with you
> would be more helpful.
>
>
> Matteo
>
> --
> Matteo Merli
> 
>
>
> On Tue, Mar 5, 2024 at 7:22 PM Kalwit S  wrote:
>
> > Congratulations Asaf.
> >
> > Btw, does the Apache project have any promotion criteria for committers?
> I
> > looked at Asaf's commits at
> > https://github.com/apache/pulsar/commits?author=asafm  and found that
> 99%
> > of the commits are simple documentation changes and 1% are related to PIP
> > monitoring. Most of the PIP monitoring involves adding plugins to
> existing
> > metrics APIs. He has also contributed to the PIP reviews, but his
> > contribution is more philosophical rather than technical. Most of his
> > comments are comparing Pulsar to other projects, rather than focusing on
> > the internal insights that Pulsar brings to the table. Our team has been
> > running production traffic using Apache Pulsar for over a year now. We
> have
> > tried several different versions of Pulsar (which we have to constantly
> > upgrade due to unknown issues in live production traffic) and have never
> > seen a stable version of Pulsar. Our team has also tried to submit
> multiple
> > enhancements and also PIP, but most of them are bogged down by reviewers
> > who are very new to Pulsar, might not understand messaging correctly, or
> > don’t find such enhancements useful for their usecases.
> > I would say that most of these reviewers are brand new to Pulsar, and
> > almost all of them are from the same company that is also the provider of
> > Pulsar. The same company controls Pulsar, prevents others from
> > contributing, and avoids having non-pulsar committers. This is why we
> > wanted to replace our existing Kafka cluster with Pulsar but we see no
> > difference in Pulsar provider and Confluent because Pulsar is also
> largely
> > controlled by one provider and this company's reviewers are not
> well-versed
> > in such systems.
> > In addition, we can see that almost all the reviewers are from the same
> > company, and PIP approval requires 3+ votes, which means only specific
> > reviewers belonging to one company participate and because of that, no
> one
> > can promote their improvements without the approval of the provider
> > company. The Pulsar community needs to break away from the monopolies of
> > the provider companies, start focusing on stable releases, and let other
> > companies make their enhancements to meet their requirements, and
> > experienced contributors or Pulsar creators should be active to prevent
> > unfairness in the community.
> >
> > On Tue, Mar 5, 2024 at 3:31 PM Kalwit S  wrote:
> >
> > > Congratulations.!
> > >
> > >
> > > On Tue, Feb 20, 2024 at 8:50 AM Lari Hotari 
> wrote:
> > >
> > >> The Apache Pulsar Project Management Committee (PMC) has invited
> > >> Asaf Mesika https://github.com/asafm to become a committer and we
> > >> are pleased to announce that he has accepted.
> > >>
> > >> Welcome and Congratulations, Asaf Mesika!
> > >>
> > >> Please join us in congratulating and welcoming Asaf onboard!
> > >>
> > >> Best Regards,
> > >>
> > >> Lari Hotari
> > >> on behalf of the Pulsar PMC
> > >>
> > >
> >
>

Re: [VOTE] Pulsar Release 2.10.6 Candidate 1

2024-03-05 Thread guo jiwei

+1 (binding)

- Built from source
- Checked the signatures
- Run standalone
- Checked producer and consumer
- Verified the Cassandra connector
- Verified the Stateful function


Regards
Jiwei Guo (Tboy)


On Wed, Mar 6, 2024 at 11:15 AM Xiangying Meng  wrote:

> This is the first release candidate for Apache Pulsar, version 2.10.6.
>
> It fixes the following issues:
>
> https://github.com/apache/pulsar/pulls?q=is:pr+label:cherry-picked/branch-2.10+label:release/2.10.6+is:closed
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.6-candidate-1/
>
> SHA-512 checksums:
>
> 09f29265f8173331d4c05b470c4e77a31146172b27ef333f45d8c8a19074ef25061cb1e80872fc45c323c9ce8e2e17989c6df5d991ef84c4d245197303d9e6d7
>  apache-pulsar-2.10.6-bin.tar.gz
>
> 49c8836882818c6f38748dae26b51c598f163606c16993a3287ab1ce9f853a4aaa43c6729c1b6f6957738b4dead3818cd12026da68b328eb2d4ac0d0214957bb
>  apache-pulsar-2.10.6-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1270
>
> The tag to be voted upon:
> v2.10.6-candidate-1 (9c29b76ff2be865429ad44df8683aec80deacfba)
> https://github.com/apache/pulsar/releases/tag/v2.10.6-candidate-1
>
> Pulsar's KEYS file containing PGP keys you use to sign the release:
> https://downloads.apache.org/pulsar/KEYS
>
> Docker images:
>
> 
>
> https://hub.docker.com/layers/xiangyingmeng/pulsar/2.10.6/images/sha256-bf8f36e49ff44ef810ab2c76742121205e51d3a04c79afdb5d288c7d8a06443f?context=repo
>
> 
>
> https://hub.docker.com/layers/xiangyingmeng/pulsar-all/2.10.6/images/sha256-1b3a10db12f6d5a0acd2d4ed73eb11864b6b598294bb905b6ede34aef1157f23?context=repo
>
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
>

Re: [VOTE] Pulsar Release 3.2.1 Candidate 1

2024-03-05 Thread guo jiwei

+1 (binding)

- Built from source
- Checked the signatures
- Run standalone
- Checked producer and consumer
- Verified the Cassandra connector
- Verified the Stateful function


Regards
Jiwei Guo (Tboy)


On Wed, Mar 6, 2024 at 2:33 AM Ran Gao  wrote:

> +1(non-binding)
>
> 1. verify the GPG signature for the below files
> - apache-pulsar-3.2.1-bin.tar.gz.asc
> - apache-pulsar-3.2.1-src.tar.gz.asc
> 2. build the source code
> 3. test following the release verifying doc.
>
> Regards,
> Ran Gao
>
> On 2024/03/05 07:05:05 guo jiwei wrote:
> > This is the first release candidate for Apache Pulsar version 3.2.1.
> >
> > It fixes the following issues:
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Amerged+label%3Arelease%2F3.2.1+label%3Acherry-picked%2Fbranch-3.2+
> >
> > *** Please download, test and verify on this release. This vote will stay
> > open for at least 72 hours ***
> >
> > Note that we are verifying upon the source (tag), binaries are provided
> for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.2.1-candidate-1/
> >
> > SHA-512 checksums:
> >
> >
> d500ba21305d56b0f7b4355e1270e50ae5f60e5632b6632d66fec6d1178bb9fdf3f24caa709b6b36ea8273e5a2c094868cf30e389154bc8bb6397e7de4f1bf1d
> >
> > apache-pulsar-3.2.1-bin.tar.gz
> >
> >
> 7aee1623db6dc95058cd0e7a4f108f8a3f43163d798a8eeeaecf5f335225c04f5a0518ce2dfb2d4fa29b7b5e54d27ba24af82fbb3542eb3ffc9e3602cb577878
> >
> > apache-pulsar-3.2.1-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1267/
> >
> > The tag to verify:
> > v3.2.1-candidate-1 (158d5eb670c9fd7b123c204533ac6cf8cb439ccd)
> > https://github.com/apache/pulsar/commits/v3.2.1-candidate-1/
> >
> > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Docker images:
> >
> > pulsar images:
> >
> https://hub.docker.com/layers/technoboy8/pulsar/3.2.1-158d5eb/images/sha256-23f0cbd54b1fb504dcab16dfcce562ed735ab94db1c5f6fabbe9145f3a2d0fa8?context=repo
> > <
> https://hub.docker.com/layers/mattison/pulsar/3.1.0-candidate-1/images/sha256-0efbaad7d893cc5041a46a2d4d56432bda855ae4068a38349777d1be6e98d27d?context=explore
> >
> > pulsar-all images:
> >
> https://hub.docker.com/layers/technoboy8/pulsar-all/3.2.1-158d5eb/images/sha256-80ab1d748eff18655a9c247beba74aa107624b9d3e7cc3a2f22e1246f0d3de83?context=repo
> >
> > Please download the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
>

Re: [VOTE] PIP-324: Alpine Docker images