> One of the biggest ways that Traffic Control benefits from GH Actions is > > that workflows run on forks, which only counts towards that user's 20 > > runner limit. As a result, our contributors have less cleanup work to do > by > > the time they open a PR. > > > > How come? Most people in Airlfow simply open PR to Airflow and it counts > against the Airflow (thus Apache) queue. > How do you do that in Traffic Control ? I'd love to learn that.
Oh, I was talking about before the contributor opens a PR at all. Both the Apache repo and the forked repo run each workflow separately once a PR _is_ opened (although only jobs started by the Apache repo count against the Apache concurrency limit). I'd be interested to see whether workflows running on forks PRing into the Apache repo can at all overcome the Apache org's "180 concurrent jobs" limit. >VAST majority of our traffic comes from PRs from forks. Same for us. -Zach On Fri, Jan 8, 2021 at 3:12 PM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > On Fri, Jan 8, 2021 at 10:45 PM Zach Hoffman <zrhoff...@apache.org> wrote: > > > One of the biggest ways that Traffic Control benefits from GH Actions is > > that workflows run on forks, which only counts towards that user's 20 > > runner limit. As a result, our contributors have less cleanup work to do > by > > the time they open a PR. > > > > How come? Most people in Airlfow simply open PR to Airflow and it counts > against the Airflow (thus Apache) queue. > How do you do that in Traffic Control ? I'd love to learn that. > > > > > > > Since running on all forks is not an option with Jenkins, that's where my > > preference comes from. Jenkins is still useful for jobs that don't need > to > > run on forks, (e.g., periodically checking for Go version updates and > > opening a PR if a minor version update is found) > > > VAST majority of our traffic comes from PRs from forks. > > > > > -Zach > > > > On Fri, Jan 8, 2021 at 2:34 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > Let me just answer from my side. > > > > > > Personally - I hate and love Jenkins at the same time. I am a Jenkins > > user > > > and developer for 15 years or so. I even developed mobile app plugins > > for > > > jenkins and we run our own jenkins farm for mobile app CI. but some 5 > > years > > > ago we switched to GitLab which was so much modern then. It has not > > changed > > > since even after jenkins introduced pipelines. > > > > > > To be perfectly honest - Java and even (once beloved by me) Gradle > simply > > > sucks for python and (yes!) javascript developers when it comes to > > > productivity. > > > > > > But It has nothing to do with love/hate. > > > > > > We moved to GA when we had exactly the same troubles with > > > Apache-organisation run Travis. more than a year ago we got in very > much > > > the same situation and GA seemed like an easy win for us. > > > > > > So, we pioneered GA use in Apache > 1 year ago and it worked well for a > > > year or so. > > > > > > We have a rather complex ( > > > https://github.com/apache/airflow/blob/master/CI.rst) system built on > > top > > > of it that has a great integration with GitHub repo and allows us to > do a > > > number of things: > > > - images are build and share in Github Packages during the build > > > - we can directly make comments and push changes to Github Branches > > > - we do security analysis with CodeQL > > > - we read and update label on PRs that serve us to determine scope of > > tests > > > ("full tests neeed" = all tests, otherwise only one matrix combination) > > > - we push cache images to Github Packages so that our builds run > faster > > > - we update and modify content of Github Issues to keep track of > > > "quarantined" issues - flaky issues that cannot be fixed immediately > > > - we implemented a sophisticated 'cancel workflow" (based on my GH > > Action) > > > that cancels duplicate workflows but also workflows that failed > important > > > jobs > > > - we implemented "selective" checks that determine which of the > multitude > > > kind of tests (unit, cli, www, heisentests, integration, core, API) > > should > > > be tested depending on which files changed. > > > > > > > > > All this uses GA features and we implemented it over last year or so. > > > Reimplementing it for Jenkins/whatever is probably months of > engineering > > > time. > > > > > > And a lot of that requires a lot of custom java/gradle plugins which we > > > have pretty much 0 capacity at Airflow. We are Python devs and we can > do > > > some javascript. I personally did the whole set of gradle plugins for > > > mobile app development ( > > > https://mvnrepository.com/artifact/com.apphance.ameba/Ameba/0.99.4) > but > > > that was > 6 years ago and I moved on to Python/Javascript and never > look > > > back. > > > > > > J. > > > > > > > > > > > > > > > On Fri, Jan 8, 2021 at 9:49 PM Chris Lambertus <c...@apache.org> wrote: > > > > > > > > > > > > > > > > On Jan 8, 2021, at 12:11 PM, Kamil Breguła < > > kamil.breg...@polidea.com> > > > > wrote: > > > > > > > > > I would be happy if we could get a tip on what we can do next to > > > improve > > > > > our CI as well as keep our community happy. Is there any solution > > that > > > > > complies with the Apache policy and is more stable? In Airflow, > > we've > > > > > already tried several solutions, starting with the Travis CI, then > > > trying > > > > > to use GitLab Ci, and now we're trying to use Github Action. Each > > > > solution > > > > > has some kinds of problems and prevents us from working freely. > Does > > > > > anyone have a hint on what we can use now? > > > > > > > > Have you considered the internal and fully supported ASF Jenkins > and/or > > > > Buildbot infrastructure? Infra has little control over the free open > > > source > > > > offerings, but we have significantly more resources we can bring to > > bear > > > on > > > > own on CI systems. > > > > > > > > What are the gaps in the ASF CI systems that are pushing people onto > > less > > > > viable platforms such as GA? > > > > > > > > > > > > -Chris > > > > ASF Infra > > > > > > > > > > > > -- > > > +48 660 796 129 > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
