Hey folks, I think we have enough in place now to start testing out the Github Actions CI more broadly. For now, the new CI is opt-in for each PR.
*To enable the new Github Actions workflow on your PR, use a branch name starting with "gh-"* Here's the current state of things: * Each PR, regardless of name, will run the "compile and check" jobs. You probably have already noticed these * If a PR's branch name starts with "gh-", the JUnit tests will be run with Github Actions * Trunk is already configured to run the new workflow alongside the existing Jenkins CI * PRs from non-committers must be manually approved before the Github Actions will run -- this is due to a default ASF Infra policy which we can relax if we want Build scans to ge.apache.org are working as expected on trunk. If a committer wants their PR to publish a build scan, they will need to push their branch to apache/kafka rather than their fork. One important note is that the Gradle cache has been enabled with the Actions workflows. For now, each trunk build will populate the cache and PRs will read from the cache. Thanks to Chia-Ping Tsai for all the reviews so far! -David On Thu, Aug 22, 2024 at 3:04 PM David Arthur <mum...@gmail.com> wrote: > The Github public runners (which we are using) only offer windows, mac, > and linux (x86_64). It is possible to set up dedicated "self-hosted" > runners for a project (or org) which would allow whatever architecture is > desired. Looks like someone has done this before for ppc64le > https://medium.com/@mayurwaghmode/github-actions-self-hosted-runners-on-ppc64le-architectures-902b8f826557. > Personally, I have done this for a Raspberry Pi on a different project. > There's a lot of flexibility with self-hosted. > > There has been some discussion of Infra setting up "self-hosted" runners > to supplement the existing Github runners. I'm not sure what the concrete > plans are, if any. > > So, to answer your specific question > > > I'm wondering if we also get access to other architectures via GitHub > actions? > > Yes, but only if someone sets up a self-hosted runner with that > architecture > > Cheers, > David > > On Thu, Aug 22, 2024 at 5:45 AM Mickael Maison <mickael.mai...@gmail.com> > wrote: > >> Hi David, >> >> Thanks for taking a look at this. Anything that can improve the >> feedback loop and ease of use is very welcome. >> >> One question I have is about the supported architectures. For example >> a while back we voted KIP-942 to add ppc64le to the Jenkins CI. Due to >> significant performance issues with the ppc64le environments this is >> still not properly enabled yet. See >> https://ci-builds.apache.org/job/Kafka/job/Kafka%20PowerPC%20Daily/ >> and https://issues.apache.org/jira/browse/INFRA-26011 if you are >> interested in the details. >> >> I'm wondering if we also get access to other architectures via GitHub >> actions? >> >> Thanks, >> Mickael >> >> On Fri, Aug 16, 2024 at 6:02 PM David Arthur <mum...@gmail.com> wrote: >> > >> > Josep, >> > >> > > By having CI commenting on the PR >> > everyone watching the PR (author and reviewers) will get notified when >> it's >> > done. >> > >> > Faster feedback is an immediate improvement I'd like to pursue. Even >> having >> > a separate PR status check for "compile + validate" would save the >> author a >> > trip digging through logs. Doing this with GH Actions is pretty >> > straightforward. >> > >> > David, >> > >> > 1. I will bring this up with Infra. They probably have some idea of my >> > intentions, due to all my questions, but I'll raise it directly. >> > >> > 2. I can think of two approaches for this. First, we can write a script >> > that produces the desired output given the junit XML reports. This can >> then >> > be used to leave a comment on the PR. Another is to add a summary block >> to >> > the workflow run. For example in this workflow: >> > https://github.com/mumrah/kafka/actions/runs/10409319037?pr=5 below the >> > workflow graph, there are summary sections. These are produced by steps >> of >> > the workflow. >> > >> > There are also Action plugins that render junit reports in various ways. >> > >> > --- >> > >> > Here is a PR that adds the action I've been experimenting with >> > https://github.com/apache/kafka/pull/16895. I've restricted it to only >> run >> > on pushes to branches named "gh-" to avoid suddenly overwhelming the ASF >> > runner pool. I have split the workflow into two jobs which are reported >> as >> > separate status checks (see https://github.com/mumrah/kafka/pull/5 for >> > example). >> > >> > >> > >> > On Fri, Aug 16, 2024 at 9:00 AM David Jacot <dja...@confluent.io.invalid >> > >> > wrote: >> > >> > > Hi David, >> > > >> > > Thanks for working on this. Overall, I am supportive. I have two >> > > questions/comments. >> > > >> > > 1. I wonder if we should discuss with the infra team in order to >> ensure >> > > that they have enough capacity for us to use the action runners. Our >> CI is >> > > pretty greedy in general. We could also discuss with them whether they >> > > could move the capacity that we used in Jenkins to the runners. I >> think >> > > that Kafka was one of the most, if not the most, heavy users of the >> shared >> > > Jenkins infra. I think that they will appreciate the heads up. >> > > >> > > 2. Would it be possible to improve how failed tests are reported? For >> > > instance, the tests in your PR failed with `1448 tests completed, 2 >> > > failed`. First it is quite hard to see it because the logs are long. >> Second >> > > it is almost impossible to find those two failed tests. In my >> opinion, we >> > > can not use it in the current state to merge pull requests. Do you >> know if >> > > there are ways to improve this? >> > > >> > > Best, >> > > David >> > > >> > > On Fri, Aug 16, 2024 at 2:44 PM 黃竣陽 <s7133...@gmail.com> wrote: >> > > >> > > > Hello David, >> > > > >> > > > I find the Jenkins UI to be quite unfriendly for developers, and the >> > > > Apache Jenkins instance is often unreliable. >> > > > On the other hand, the new GitHub Actions UI is much more appealing >> to >> > > me. >> > > > If GitHub Actions proves to be more >> > > > stable than Jenkins, I believe it would be a worthwhile change to >> switch >> > > > to GitHub Actions. >> > > > >> > > > Thank you. >> > > > >> > > > Best Regards, >> > > > Jiunn Yang >> > > > > Josep Prat <josep.p...@aiven.io.INVALID> 於 2024年8月16日 下午4:57 寫道： >> > > > > >> > > > > Hi David, >> > > > > One of the enhancements we can have with this change (it's easier >> to do >> > > > > with GH actions) is to write back the result of the CI run as a >> comment >> > > > on >> > > > > the PR itself. I believe not needing to periodically check CI to >> see if >> > > > the >> > > > > run finished would be a great win. By having CI commenting on the >> PR >> > > > > everyone watching the PR (author and reviewers) will get notified >> when >> > > > it's >> > > > > done. >> > > > >> > > > >> > > >> > >> > >> > -- >> > David Arthur >> > > > -- > David Arthur > -- David Arthur
