I have submitted the PR for refactoring the apache/pulsar GitHub Actions based CI. Please review https://github.com/apache/pulsar/pull/14819 .
BR, -Lari On 2022/03/22 13:38:36 Enrico Olivelli wrote: > Lari, > > Il Mar 22 Mar 2022, 14:32 Lari Hotari <lhot...@apache.org> ha scritto: > > > I have resumed work to improve our GitHub Actions based Pulsar CI. > > > > Last year, I worked on a proof-of-concept which significantly reduced the > > resource consumption and improved the usability of the build by combining > > multiple workflows into a single larger workflow. > > > > The showstopper a year ago was the lack of being able to re-run a single > > failed job in a larger workflow. > > GitHub has since then delivered this feature and no showstoppers are > > present. > > > > I have been posting updates to > > https://github.com/apache/pulsar/issues/14401 "Speed up CI workflows" > > about the progress. > > I have rebased the changes from last year's PoC, and I'm finalizing and > > testing the changes in my fork under > > https://github.com/lhotari/pulsar/pull/59 . I'll send a PR to > > apache/pulsar, when the refactoring is ready. > > > > This is great news ! > > Looking forward to your patch > > Enrico > > > > > -Lari > > > > On 2021/03/16 01:10:52 Sijie Guo wrote: > > > > The prototype has demonstrated about 60% reduction in > > > resource consumption. > > > > > > It is hard to quantify. Merging them into one large workflow can result > > in > > > more failures. Re-running those failures can consume resources as well. > > > > > > > Isn't it urgent to resolve it? > > > > > > I think we are in a stage that gives us breathing room to fix flaky tests > > > and solve other problems, no? > > > I don't mean we stop the effort here. I mean we have other enhancements > > > that we can do to improve the situation. > > > Once we get into a position where the flakiness is reduced, we can merge > > > them into one workflow. > > > > > > - Sijie > > > > > > > > > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <l...@hotari.net> wrote: > > > > > > > Thanks for the feedback Sijie. > > > > > > > > > We are using a lazy consensus approach. Typically if there is no > > > > objection, > > > > > please go ahead and not need to wait for approval. > > > > > If people raise concerns, please address the concerns. > > > > > > > > You and Ali have raised concerns about changing the existing GitHub > > Actions > > > > workflows in a way where multiple workflows would be combined to a > > single > > > > workflow. Before proceeding, there is a need to address the concerns. > > We > > > > might end up with a completely different type of solution of what has > > been > > > > proposed initially. :) > > > > > > > > > Yes. So I am in favor of addressing flaky tests than merging all > > > > workflows > > > > > into one giant workflow. > > > > > > > > I agree that addressing flaky tests is favorable. The main reason for > > PIP > > > > "Changes to GitHub Actions based Pulsar CI" is to > > > > 1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds > > > > 2) Reduce lead times for Pull Request feedback > > > > We cannot ignore these problems. If we don't change anything, the > > problems > > > > won't get fixed. The prototype has demonstrated about 60% reduction in > > > > resource consumption. Measuring the lead times hasn't been done in the > > > > prototype, but since the reason for long lead times has been long build > > > > queues due to excessive resource consumption, it's likely that the lead > > > > times would be reduced. > > > > > > > > I know that switching to a single workflow isn't the only solution to > > the > > > > above problems. I had a discussion with Ali. He recommended reducing > > the > > > > modules in Pulsar repository (PIP-62), reducing the docker container > > size > > > > and improving the Pulsar Broker unit test harness so that tests using > > it > > > > would be less flaky and that it would be easier to fix the issues in > > > > failing test when there would be better information about what was the > > > > state problem that caused the test to fail. > > > > > > > > As mentioned in the earlier email about the optimizations in the > > Pulsar CI > > > > refactoring prototype, the main benefits come from reusing binary > > artifacts > > > > from previous build stages so that each job doesn't have to build > > > > everything from scratch. This becomes irrelevant when the build is very > > > > fast and there isn't a benefit of reusing artifacts. > > > > This means that it's possible to resolve the resource consumption > > problem > > > > of Pulsar PR builds in the way that Ali is recommending, without > > switching > > > > from multiple workflows to a single workflow that can reuse binary > > > > artifacts in the build. > > > > > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing > > > > more > > > > > on solving flaky tests. > > > > > Consider merging them into one workflow when the tests are in a > > better > > > > > situation. > > > > > > > > Makes sense for minimizing the risk of change, but we cannot just wait > > for > > > > things to fix themselves. > > > > How long will other Apache projects tolerate the resource consumption > > > > issues Pulsar is causing in the shared GitHub Actions Runner VM quota? > > For > > > > example, > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396 > > > > . > > > > Isn't it urgent to resolve it? > > > > > > > > I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar > > CI" > > > > based on the community feedback in the upcoming days. That might mean > > that > > > > the current solution is pivoted. The goal is to solve the problems of > > high > > > > resource consumption and long lead time for PR build in Pulsar CI. > > Please > > > > continue to provide feedback so that we get a revisited plan together > > that > > > > addresses these problems.Thank you! > > > > > > > > BR, > > > > -Lari > > > > > > > > On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <guosi...@gmail.com> wrote: > > > > > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to > > GitHub > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki > > > > > page https://github.com/apache/pulsar/wiki > > > > > <https://github.com/apache/pulsar/wiki> ?* > > > > > > > > > > I see what you referred to before now. This can be easily done. I > > (or any > > > > > other committer) can do it for you. > > > > > > > > > > There is no real blocker for you to continue work even there are > > concerns > > > > > or it is not listed in the PIP. > > > > > We are using a lazy consensus approach. Typically if there is no > > > > objection, > > > > > please go ahead and not need to wait for approval. > > > > > If people raise concerns, please address the concerns. > > > > > > > > > > > The reason why > > > > > re-runs happen currently is because of the high flakiness of tests. > > > > > > > > > > Yes. So I am in favor of addressing flaky tests than merging all > > > > workflows > > > > > into one giant workflow. > > > > > It is not about "No pain, no gain". The community has suffered a lot > > with > > > > > giant workflow before. > > > > > There were a lot of committers and contributors working hard to > > split one > > > > > giant workflow into multiple > > > > > current workflows. Unless there is really strong evidence that > > merging > > > > them > > > > > back to one will improve > > > > > the entire CI experience, I will still have concerns about one giant > > > > > workflow approach. > > > > > > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing > > > > more > > > > > on solving flaky tests. > > > > > Consider merging them into one workflow when the tests are in a > > better > > > > > situation. > > > > > > > > > > > This solution would also require disabling > > > > > required status checks > > > > > > > > > > I don't think it is a good idea to disable status checks. We can > > consider > > > > > running "dark mode" but it will just overload the action quota. > > > > > > > > > > Another alternative is to mirror the pull requests into another > > Github > > > > > account to test that and get more concrete statistics on the > > flakiness > > > > rate > > > > > of one workflow approach. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <l...@hotari.net> wrote: > > > > > > > > > > > Thanks for the feedback, Sijie. > > > > > > > > > > > > > The "Fail fast" approach is great. That would be super helpful if > > > > there > > > > > > are > > > > > > > multiple workflows and each workflow is retryable. > > > > > > > However, I am not sure how much it will help if you run all > > workflows > > > > > in > > > > > > > one giant workflow. Or is it making things worse? > > > > > > > > > > > > We can reduce the need for re-running workflow runs. The reason why > > > > > > re-runs happen currently is because of high flakiness of tests. > > > > > > Addressing flakiness continues to be top-priority. Now that the > > Pulsar > > > > CI > > > > > > workflow prototype is finished, I'll be focusing more in the other > > > > draft > > > > > > PIP, "Changes to flaky test handling". > > > > > > We as a community should address the critical problem that the > > current > > > > > > retry solution has: it can mask bugs in production code and make > > the > > > > > build > > > > > > pass and allow changes to be merged that cause regressions. > > > > > > It's a false sense of security what the green builds after all the > > > > > retries > > > > > > bring us. Bringing Pulsar to the next level in stability requires > > > > > > addressing this. > > > > > > > > > > > > If something doesn't work, it can be adapted and improved. Changes > > can > > > > be > > > > > > rolled back and revisited when things go worse. We also need a > > leap of > > > > > > faith. > > > > > > "No pain, no gain", like any change, it will be painful at first, > > but > > > > we > > > > > > will get over the bump. > > > > > > > > > > > > > Secondly, your test has been done in your folk where there are > > not a > > > > > lot > > > > > > of > > > > > > > concurrent pushes and pull requests. I am not sure how your > > approach > > > > > will > > > > > > > behave once it is merged into master. Can you simulate multiple > > > > > > concurrent > > > > > > > pull requests in your account to prove your approach doesn't > > bring > > > > side > > > > > > > effects? > > > > > > > > > > > > One possibility to address this is to introduce the new workflow > > in a > > > > > mode > > > > > > where you need to opt-in to the new workflow in some way. > > > > > > This was an idea brought up by my colleagues Enrico and Andrey. > > > > > > It might be possible to configure the existing workflow and this > > new > > > > > > workflow in a way where some condition (for example whitelisted > > github > > > > > user > > > > > > name or a certain keyword in the PR title/description) chooses > > either > > > > one > > > > > > for the Pull request. This solution would also require disabling > > > > > > required status checks ("Require status checks to pass before > > merging" > > > > > > feature in GitHub branch protection rules) since the names of the > > > > checks > > > > > > would be different. > > > > > > > > > > > > > Lastly, can we apply those optimizations to current workflows > > without > > > > > > > merging them into one giant workflow? > > > > > > > > > > > > This is what I have been doing. All individual optimizations have > > > > already > > > > > > been sent as PRs in the last months. I guess there's 20-30 PRs that > > > > have > > > > > > already been merged. > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged > > > > > > There's also 2 build related PRs from yesterday which haven't been > > > > merged > > > > > > yet: > > > > > > Fix Maven download issues (ported from the prototype to our > > existing > > > > > Pulsar > > > > > > CI): https://github.com/apache/pulsar/pull/9883 > > > > > > improve Maven module build order (required for more efficient > > builds > > > > that > > > > > > selectively build required artifacts): > > > > > > https://github.com/apache/pulsar/pull/9882 > > > > > > > > > > > > There aren't many optimizations left that could be ported from the > > > > > > prototype to the existing workflow. There are a few, but the > > impact is > > > > > > minor. > > > > > > The reason for this is that the optimization with the greatest > > impact > > > > are > > > > > > the ones that build a binary artifacts (maven libs, docker images) > > once > > > > > and > > > > > > share it with the downstream jobs in the pipeline. > > > > > > Applying this type of solution has certain challenges when there > > are > > > > > > multiple separate workflow. Sharing binary artifacts to other > > workflows > > > > > > would require that the workflow to reuse the artifacts gets > > triggered > > > > by > > > > > > the workflow that produced the artifacts. This wouldn't be secure > > or > > > > > > practical for handling pull requests. Triggering a workflow > > explicitly > > > > > > would require a token from the main repository and using that for > > pull > > > > > > request builds would be a serious security vulnerability. (more > > details > > > > > > about the GitHub Actions security model in > > > > > > > > > > > > > > > > > > > > > > > https://securitylab.github.com/research/github-actions-preventing-pwn-requests > > > > > > ) > > > > > > > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to > > GitHub > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the > > wiki > > > > > > page https://github.com/apache/pulsar/wiki > > > > > > <https://github.com/apache/pulsar/wiki> ?* > > > > > > > > > > > > -Lari > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <guosi...@gmail.com> > > wrote: > > > > > > > > > > > > > This is good progress. However, my main concern is still merging > > all > > > > > > > workflows into one giant workflow. > > > > > > > > > > > > > > The "Fail fast" approach is great. That would be super helpful if > > > > there > > > > > > are > > > > > > > multiple workflows and each workflow is retryable. > > > > > > > However, I am not sure how much it will help if you run all > > workflows > > > > > in > > > > > > > one giant workflow. Or is it making things worse? > > > > > > > > > > > > > > Secondly, your test has been done in your folk where there are > > not a > > > > > lot > > > > > > of > > > > > > > concurrent pushes and pull requests. I am not sure how your > > approach > > > > > will > > > > > > > behave once it is merged into master. Can you simulate multiple > > > > > > concurrent > > > > > > > pull requests in your account to prove your approach doesn't > > bring > > > > side > > > > > > > effects? > > > > > > > > > > > > > > Lastly, can we apply those optimizations to current workflows > > without > > > > > > > merging them into one giant workflow? > > > > > > > > > > > > > > - Sijie > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <l...@hotari.net> > > > > wrote: > > > > > > > > > > > > > > > Thanks for the feedback Michael. > > > > > > > > > > > > > > > > > I left a question on the doc about how concurrent runs > > affect the > > > > > > > > > repository's 5 GB cache limit. > > > > > > > > > > > > > > > > This is a good question. There isn't a clear answer in the > > GitHub > > > > > > Actions > > > > > > > > Cache documentation. > > > > > > > > > > > > > > > > The documentation is > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows > > > > > > > > . > > > > > > > > Based on this document and some testing, I have made these > > > > > conclusions: > > > > > > > > For GitHub Actions Cache, pull requests get executed in the > > context > > > > > of > > > > > > > the > > > > > > > > forked repository. > > > > > > > > The workflow triggered by a pull request event can only update > > it's > > > > > own > > > > > > > > cache. It has read-only access to upstream caches. > > > > > > > > If there's a cache miss, the entry will get written to the > > cache of > > > > > the > > > > > > > > forked repository. If the PR could to write to the upstream > > cache, > > > > it > > > > > > > would > > > > > > > > be a security issue since this would be vulnerable to cache > > > > poisoning > > > > > > > > attacks. Each repository has a 5GB quota for writes. The > > entries > > > > are > > > > > > kept > > > > > > > > up to 7 days. > > > > > > > > The performance is fairly good. Loading docker images from the > > > > > > repository > > > > > > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s. > > (the > > > > > > > > performance of the GHA cache is most likely higher since this > > is > > > > the > > > > > > > > throughput for docker load / docker save) > > > > > > > > > > > > > > > > If a single repository has a lot of concurrent jobs, it could > > start > > > > > > > > evicting caches. > > > > > > > > However that isn't likely to happen with the way Pulsar is > > > > developed > > > > > > > since > > > > > > > > pull requests are created from personal forks. > > > > > > > > > > > > > > > > > I also think it could be helpful to explicitly document, or > > > > > reference > > > > > > > > > github documentation, on how failure will affect the DAG. I'm > > > > > > assuming > > > > > > > > that > > > > > > > > > if an action fails, its parallel peer actions will run to > > > > > completion, > > > > > > > and > > > > > > > > > that the rest of the remaining stages will get canceled, but > > I > > > > > > haven't > > > > > > > > > worked with github actions before. > > > > > > > > > > > > > > > > For matrix jobs, "fail fast" is the default, which cancels all > > jobs > > > > > in > > > > > > > the > > > > > > > > matrix if one fails. Other parallel flows would run to > > completion > > > > by > > > > > > > > default. > > > > > > > > In the prototype, I have added a Github script step to each > > job to > > > > > > cancel > > > > > > > > the complete workflow when a failure occurs. > > > > > > > > Here's an example: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289 > > > > > > > > > > > > > > > > The prototype follows a "fail fast" design. When a failure > > occurs, > > > > > fail > > > > > > > > fast and don't continue with other jobs. > > > > > > > > The benefit of this is that it reduces resource consumption. > > This > > > > > helps > > > > > > > > keep the build queue short. > > > > > > > > When the build queue is short, developers get quick feedback > > from > > > > CI. > > > > > > > > > > > > > > > > Documenting all details in the PIP document isn't practical. > > > > > > > > *I'm hoping to start a separate document on low level details > > when > > > > > > there > > > > > > > is > > > > > > > > a high level acceptance of the proposed "Changes to GitHub > > Actions > > > > > > based > > > > > > > > Pulsar CI".* > > > > > > > > Together we can make this happen. We need decisions too. This > > > > > proposal > > > > > > > > cannot stay as a draft forever. > > > > > > > > I'm looking forward to hearing from the Pulsar community, > > Pulsar > > > > > > > > committer and Pulsar PMC members how to take this forward. > > > > > > > > > > > > > > > > BR, > > > > > > > > -Lari > > > > > > > > > > > > > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall < > > > > > > mikemars...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > This will be a great improvement. I read through the PIP, and > > > > > > overall, > > > > > > > it > > > > > > > > > looks good to me. > > > > > > > > > > > > > > > > > > I left a question on the doc about how concurrent runs > > affect the > > > > > > > > > repository's 5 GB cache limit. > > > > > > > > > > > > > > > > > > I also think it could be helpful to explicitly document, or > > > > > reference > > > > > > > > > github documentation, on how failure will affect the DAG. I'm > > > > > > assuming > > > > > > > > that > > > > > > > > > if an action fails, its parallel peer actions will run to > > > > > completion, > > > > > > > and > > > > > > > > > that the rest of the remaining stages will get canceled, but > > I > > > > > > haven't > > > > > > > > > worked with github actions before. > > > > > > > > > > > > > > > > > > Thanks for all of the work you've put in so far. > > > > > > > > > > > > > > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uvar...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > > > > > This is great news. Thanks Hari , Mateo and pulsar > > community > > > > > > > > > > > > > > > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari < > > > > lari.hot...@sagire.fi > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Dear Pulsar community members, > > > > > > > > > > > > > > > > > > > > > > The work on "Changes to GitHub Actions based Pulsar CI" > > has > > > > > gone > > > > > > > > > forward > > > > > > > > > > > based on your feedback. Here are some updates about the > > work. > > > > > > > > > > > > > > > > > > > > > > The draft PIP proposal document is here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry > > > > > > > > > > > There's a *detailed status update in the document about a > > > > > > prototype > > > > > > > > for > > > > > > > > > > the > > > > > > > > > > > refactored Pulsar CI GitHub Actions based workflow*. > > > > > > > > > > > > > > > > > > > > > > Thanks for all the suggestions and feedback by now. A > > lot of > > > > > > > > > improvements > > > > > > > > > > > have been made by the Pulsar contributors to overcome the > > > > > > technical > > > > > > > > > > > obstacles. > > > > > > > > > > > Special thanks go to Matteo for reducing the sizes of > > docker > > > > > > > images. > > > > > > > > A > > > > > > > > > > lot > > > > > > > > > > > of small improvements have been made to the Pulsar maven > > > > build > > > > > to > > > > > > > > > enable > > > > > > > > > > > the new refactored GitHub Actions workflow. Thank you > > for all > > > > > PR > > > > > > > > > reviews > > > > > > > > > > > and feedback. > > > > > > > > > > > > > > > > > > > > > > The main goal of the "Changes to GitHub Actions based > > Pulsar > > > > > CI" > > > > > > > work > > > > > > > > > has > > > > > > > > > > > been to *reduce the resource consumption of the Pulsar CI > > > > build > > > > > > and > > > > > > > > to > > > > > > > > > > > speed up Pulsar development by improving the developer > > > > > > > productivity* > > > > > > > > > when > > > > > > > > > > > less time is wasted in waiting for Pulsar CI build > > feedback. > > > > > The > > > > > > > > > > prototype > > > > > > > > > > > demonstrates these improvements. > > > > > > > > > > > > > > > > > > > > > > As you can see from the email from Jan 28 below, *the > > > > resource > > > > > > > > > > consumption > > > > > > > > > > > was 19 hrs 36 minutes* for a single pull request that was > > > > > > observed > > > > > > > > when > > > > > > > > > > the > > > > > > > > > > > work began. > > > > > > > > > > > Now, with the prototype of the refactored Pulsar CI > > build, > > > > the > > > > > > > > resource > > > > > > > > > > > consumption is *7 hrs 9 minutes.* > > > > > > > > > > > *This is about 60% reduction in resource consumption.* > > The > > > > > whole > > > > > > > > > pipeline > > > > > > > > > > > completes in 75-100 minutes. > > > > > > > > > > > > > > > > > > > > > > Here's a breakdown of the duration (resource > > consumption) of > > > > > each > > > > > > > > build > > > > > > > > > > job > > > > > > > > > > > in the refactored workflow: > > > > > > > > > > > Workflow Job seconds h:mm:ss > > > > > > > > > > > Pulsar CI Changed files check 4 0:00:04 > > > > > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35 > > > > > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46 > > > > > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53 > > > > > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36 > > > > > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57 > > > > > > > > > > > Pulsar CI Build and License check 346 0:05:46 > > > > > > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23 > > > > > > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405 > > > > 0:06:45 > > > > > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20 > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968 > > 0:16:08 > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223 > > 0:37:03 > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32 > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16 > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42 > > > > > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31 > > > > > > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03 > > > > > > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54 > > > > > > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849 > > > > > 0:14:09 > > > > > > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50 > > > > > > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17 > > > > > > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48 > > > > > > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46 > > > > > > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02 > > > > > > > > > > > Pulsar CI CI - System - Function State 699 0:11:39 > > > > > > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59 > > > > > > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49 > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795 > > > > 0:29:55 > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312 > > > > > 0:38:32 > > > > > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57 > > > > > > > > > > > *Total resource consumption* > > > > > > > > > > > 7:08:57 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > GitHub Actions doesn't support restarting a single job ( > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234 > > > > > > > > > > > ). > > > > > > > > > > > However, this is not a showstopper since there are ways > > to > > > > > > address > > > > > > > > the > > > > > > > > > > > issues that cause flakiness. > > > > > > > > > > > There is a separate PIP for changing the way to handle > > flaky > > > > > > tests. > > > > > > > > You > > > > > > > > > > can > > > > > > > > > > > find the link to that in the "Changes to GitHub Actions > > based > > > > > > > Pulsar > > > > > > > > > CI" > > > > > > > > > > > document's header. > > > > > > > > > > > > > > > > > > > > > > *Some requests for the Pulsar community:* > > > > > > > > > > > > > > > > > > > > > > 1) *Please take a look at the updated PIP document*: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry > > > > > > > > > > > . *It also contains more details of the prototype that > > has > > > > been > > > > > > > > > > > successfully completed.* > > > > > > > > > > > > > > > > > > > > > > 2) *Please share your feedback and suggest a way > > forward.* > > > > > > > > > > > > > > > > > > > > > > *Thank you for your help!* > > > > > > > > > > > > > > > > > > > > > > BR, Lari > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari < > > > > > > lari.hot...@sagire.fi > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Dear Pulsar community members, > > > > > > > > > > > > > > > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are > > > > consuming > > > > > > the > > > > > > > > > > majority > > > > > > > > > > > > of the shared pool of resources allocated for > > > > > > github.com/apache > > > > > > > > > > > projects. > > > > > > > > > > > > Other Apache projects have been impacted and there is a > > > > > demand > > > > > > to > > > > > > > > > > improve > > > > > > > > > > > > the Pulsar CI > > > > > > > > > > > > < > > > > > > > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396 > > > > > > > > > > > > > > > > > > > > asap. > > > > > > > > > > > > > > > > > > > > > > > > In GitHub Actions Runners, the unit of resources is the > > > > time > > > > > > > that a > > > > > > > > > > > Runner > > > > > > > > > > > > is occupied. I observed the workflow runs for handling > > a > > > > > single > > > > > > > > Pull > > > > > > > > > > > > Request (in my personal fork) and these were the > > running > > > > > > > durations: > > > > > > > > > > > > Workflow name Duration > > > > > > > > > > > > CI - Build - MacOS 0:17:23 > > > > > > > > > > > > CI - Go Functions style check 0:02:38 > > > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40 > > > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28 > > > > > > > > > > > > CI - Misc 0:16:51 > > > > > > > > > > > > CI - Unit - Proxy 0:14:23 > > > > > > > > > > > > CI - Go Functions Tests 0:22:08 > > > > > > > > > > > > CI - CPP, Python Tests 0:23:30 > > > > > > > > > > > > CI - Unit 0:42:11 > > > > > > > > > > > > CI - Integration - Sql 1:00:13 > > > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18 > > > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13 > > > > > > > > > > > > CI - Integration - Function State 1:00:12 > > > > > > > > > > > > CI - Integration - Cli 1:10:22 > > > > > > > > > > > > CI - Integration - Transaction 1:16:34 > > > > > > > > > > > > CI - Integration - Process 1:11:23 > > > > > > > > > > > > CI - Shade - Test 1:15:45 > > > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13 > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05 > > > > > > > > > > > > CI - Integration - Standalone 0:45:29 > > > > > > > > > > > > CI - Integration - Messaging 1:00:23 > > > > > > > > > > > > CI - Integration - Thread 1:00:19 > > > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19 > > > > > > > > > > > > CI - Integration - Schema 1:00:19 > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31 > > > > > > > > > > > > TOTAL 19:36:50 > > > > > > > > > > > > > > > > > > > > > > > > *In this case, the total resource consumption of GitHub > > > > > Actions > > > > > > > > > Runners > > > > > > > > > > > is > > > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull > > request to > > > > > > > > > > > apache/pulsar.* > > > > > > > > > > > > > > > > > > > > > > > > Since GitHub Actions Runner resource pool utilization > > is > > > > very > > > > > > > high, > > > > > > > > > > this > > > > > > > > > > > > leads to the build queue to grow and take a long time > > to > > > > > > process. > > > > > > > > > > > > > > > > > > > > > > > > I have been looking for ways to improve the Pulsar CI > > for > > > > the > > > > > > > last > > > > > > > > 3 > > > > > > > > > > > > months. During this period I worked on a few > > experiments. > > > > The > > > > > > > > > learnings > > > > > > > > > > > > from the past experiments are documented at a high > > level in > > > > > the > > > > > > > > > > following > > > > > > > > > > > > draft PIP document. > > > > > > > > > > > > > > > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar > > CI" > > > > > > > document > > > > > > > > > is > > > > > > > > > > a > > > > > > > > > > > > Google doc:* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing > > > > > > > > > > > > > > > > > > > > > > > > *Please participate* so that we get the plan adjusted > > based > > > > > on > > > > > > > the > > > > > > > > > > > > feedback asap. If there's already a similar effort > > > > ongoing, I > > > > > > > hope > > > > > > > > we > > > > > > > > > > can > > > > > > > > > > > > join efforts. > > > > > > > > > > > > > > > > > > > > > > > > *Let's fix Pulsar CI!* > > > > > > > > > > > > > > > > > > > > > > > > BR, Lari > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
