Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Lari, I would start a new thread with an 'Heads up' People may not follow this long thread and miss this piece of information Great work! Thank you very much Enrico Il Mer 30 Mar 2022, 18:32 Lari Hotari ha scritto: > The refactored Pulsar CI workflow PR > https://github.com/apache/pulsar/pull

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

The refactored Pulsar CI workflow PR https://github.com/apache/pulsar/pull/14819 has been merged. It unblocks CI and makes it work again. Pulsar SQL integration tests are disabled temporarily until https://github.com/apache/pulsar/issues/14951 has been addressed. Please rebase your PR or close/

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Merging the PR is blocked by https://github.com/apache/pulsar/issues/14951 . Pulsar SQL doesn't work with Java 11.0.14.1 version. It fails with this error message: Exception in thread "main" java.lang.IllegalArgumentException: Cannot parse version 11.0.14.1 at io.prestosql.server.JavaV

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Thank you for the reviews and feedback. I have started making the switch to the new refactored Pulsar CI. Merging new PRs are blocked until the switch is ready. The reason for this is that I have merged https://github.com/apache/pulsar/pull/14939 preparing for merging https://github.com/apache/

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Great work, Lari! It's great news that GitHub's new feature helps this valuable work move forward. I look forward to seeing your PR merged, and I am happy to help resolve any issues that might pop up. Thanks, Michael On Tue, Mar 29, 2022 at 7:55 AM Lari Hotari wrote: > > The PR has sufficient re

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

The PR has sufficient reviews, and I'll proceed with merging it today or tomorrow. Please provide feedback now if you want to do that before the PR is merged. Thanks! -Lari On 2022/03/28 20:05:14 Lari Hotari wrote: > The PR https://github.com/apache/pulsar/pull/14819 is now ready for review. >

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

The PR https://github.com/apache/pulsar/pull/14819 is now ready for review. The changes in the PR now use GitHub Actions Artifacts for sharing binary files (such as docker images) between the build steps. This saves a lot of GitHub Actions VM resources since the docker images are built once and

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

I have created a separate PR to make pulsarbot support the new GitHub Actions feature of rerunning failed jobs (instead of all jobs in a workflow). Please review https://github.com/apache/pulsar-test-infra/pull/27 too. -Lari On 2022/03/23 10:46:10 Lari Hotari wrote: > I have submitted the PR fo

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

I have submitted the PR for refactoring the apache/pulsar GitHub Actions based CI. Please review https://github.com/apache/pulsar/pull/14819 . BR, -Lari On 2022/03/22 13:38:36 Enrico Olivelli wrote: > Lari, > > Il Mar 22 Mar 2022, 14:32 Lari Hotari ha scritto: > > > I have resumed work to imp

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Lari, Il Mar 22 Mar 2022, 14:32 Lari Hotari ha scritto: > I have resumed work to improve our GitHub Actions based Pulsar CI. > > Last year, I worked on a proof-of-concept which significantly reduced the > resource consumption and improved the usability of the build by combining > multiple workfl

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

I have resumed work to improve our GitHub Actions based Pulsar CI. Last year, I worked on a proof-of-concept which significantly reduced the resource consumption and improved the usability of the build by combining multiple workflows into a single larger workflow. The showstopper a year ago wa

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

On Tue, Mar 16, 2021 at 3:11 AM Sijie Guo wrote: > > The prototype has demonstrated about 60% reduction in > resource consumption. > > It is hard to quantify. Merging them into one large workflow can result in > more failures. Re-running those failures can consume resources as well. > Yes, you a

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

> The prototype has demonstrated about 60% reduction in resource consumption. It is hard to quantify. Merging them into one large workflow can result in more failures. Re-running those failures can consume resources as well. > Isn't it urgent to resolve it? I think we are in a stage that gives u

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Thanks for the feedback Sijie. > We are using a lazy consensus approach. Typically if there is no objection, > please go ahead and not need to wait for approval. > If people raise concerns, please address the concerns. You and Ali have raised concerns about changing the existing GitHub Actions wo

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

> *Sijie, how far are we from getting the draft PIP "Changes to GitHub Actions based Pulsar CI" into an actual PIP that gets put on the wiki page https://github.com/apache/pulsar/wiki ?* I see what you referred to before now. This can be easily done. I (or a

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Thanks for the feedback, Sijie. > The "Fail fast" approach is great. That would be super helpful if there are > multiple workflows and each workflow is retryable. > However, I am not sure how much it will help if you run all workflows in > one giant workflow. Or is it making things worse? We can

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

This is good progress. However, my main concern is still merging all workflows into one giant workflow. The "Fail fast" approach is great. That would be super helpful if there are multiple workflows and each workflow is retryable. However, I am not sure how much it will help if you run all workflo

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Thanks for the feedback Michael. > I left a question on the doc about how concurrent runs affect the > repository's 5 GB cache limit. This is a good question. There isn't a clear answer in the GitHub Actions Cache documentation. The documentation is https://docs.github.com/en/actions/guides/cach

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Lari, Thank you for this work. I have been following it very closely and I am happy that we are close to be able to leverage it very soon. What is the next step in order to apply these changes to the Pulsar repo ? IIUC all of the changes that blocked you on building up this work have been merged (

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

This will be a great improvement. I read through the PIP, and overall, it looks good to me. I left a question on the doc about how concurrent runs affect the repository's 5 GB cache limit. I also think it could be helpful to explicitly document, or reference github documentation, on how failure w

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

This is great news. Thanks Hari , Mateo and pulsar community On Fri, Mar 12, 2021, 2:04 AM Lari Hotari wrote: > Dear Pulsar community members, > > The work on "Changes to GitHub Actions based Pulsar CI" has gone forward > based on your feedback. Here are some updates about the work. > > The draf

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Dear Pulsar community members, The work on "Changes to GitHub Actions based Pulsar CI" has gone forward based on your feedback. Here are some updates about the work. The draft PIP proposal document is here: https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#headi

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Thank you, Ali. This sounds good! BR, Lari On Mon, Feb 1, 2021 at 11:47 PM Ali Ahmed wrote: > We will need some simple connectors without dependencies, to replace the > existing ones for basic integration testing. I can write those. > > > On Mon, Feb 1, 2021 at 1:17 PM Sijie Guo wrote: > > >

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Hi Sijie, > Yes. I was doing the work there and hopefully will get there soon. Is the plan to do the 2.8.0 release with the new repositories and repository structure in place? I'd be interested in understanding how releasing is intended to be carried out with multiple repositories without runnin

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

We will need some simple connectors without dependencies, to replace the existing ones for basic integration testing. I can write those. On Mon, Feb 1, 2021 at 1:17 PM Sijie Guo wrote: > Yes. I was doing the work there and hopefully will get there soon. > > - Sijie > > On Mon, Feb 1, 2021 at 1

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Yes. I was doing the work there and hopefully will get there soon. - Sijie On Mon, Feb 1, 2021 at 12:40 PM Ali Ahmed wrote: > I recommend we move the connectors away for the pulsar repo to reduce the > load on the main ci pipeline. The new repo seems ready. > https://github.com/apache/pulsar-co

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

I recommend we move the connectors away for the pulsar repo to reduce the load on the main ci pipeline. The new repo seems ready. https://github.com/apache/pulsar-connectors. -Ali On Fri, Jan 29, 2021 at 9:22 AM Sijie Guo wrote: > Currently, Github Actions are shared across one large `apache` >

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Currently, Github Actions are shared across one large `apache` organization. It is the main problem for GA-based CI besides flaky tests. If we use Azure Pipeline, we can have a dedicated project for the pulsar. So we will have more resources to run. It will solve the problem that this proposal tri

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Hi Sijie, Let's keep this work going since resolving the problems with Pulsar CI are urgent. I took a quick glance on the Azure Pipelines solution in Flink. By Googling I found https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines . In the repository I found https://github.com/apache/

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Lari, Yes, we can keep this proposal open for discussion. That's for sure. I just don't have any good solution at this moment with a multiple-workflow approach using Github Actions. An alternative is to look into Azure Pipeline, which the Flink community is using. We are still learning there. Wi

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Thanks for the feedback, Sijie. > If this proposal is blocked by the other proposal, we should focus on getting the changes for the other proposal before talking about merging them. Yes, the current proposal depends on the draft PIP for "Changes to flaky test handling". I'll follow up on fixing t

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Lari, Thank you for bringing this proposal up! This is a great initiative. However, I agreed with Yong. We have spent tons of effort splitting one large workflow into multiple smaller workflows. If this proposal is blocked by the other proposal, we should focus on getting the changes for the oth

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Thank you for the comments Penghui. Exactly what you said, we should make the tests stable. The proposals in the other draft PIP "Changes to flaky test handling" deals with that. It's currently a draft and needs more eyes. Would you be able to take a closer look at that too? BR, Lari On Fri, Jan

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Thanks for the comments, Yong. Very good point. Re-running a single job isn't supported. Currently, re-rerunning is required because it is the way we deal with flaky tests. There is a separate draft PIP for "Changes to flaky test handling" in discussion, https://lists.apache.org/thread.html/r4c23

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

Currently, especially for the integration tests, a lot time to build pulsar distributions and docker images. I think before merge tests we should to make the test stable, otherwise rerun the test will become more expensive. Thanks, Penghui On Jan 29, 2021, 11:55 AM +0800, Yong Zhang , wrote: > I

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

I am not sure that merge all the workflows into one workflow is a good idea. As I know, Github Actions doesn't allow to rerun a single job in a workflow. That means if there has any failure in the workflow, we need to rerun all steps/stage. There has a worst-case is we failed in the different test