I'm interested in participating. The Heron project is hitting limits with TravisCI, I'd like to find a way to make our process better whether it's with Jenkins/GA or something else. Next week would be better for me, if we can push it back.
On Mon, Nov 1, 2021 at 8:49 AM Jarek Potiuk <ja...@potiuk.com> wrote: > I updated the graphs. I think things have changed a lot since December last > year. > > There are no absurd-high (and way above the limits) peaks of workflows in > progress for some projects (e.g. Pulsar). However a number of apache repos > using GA grows steadily. Number of workflows "queued" fluctuates for many > projects and for some projects (spark) it seems it steadily grows. > > Observation from the Airflow side (and I think lack of "escalation" > indicates that other projects have the same) is that we stopped observing > prolonged periods where a lot of jobs are queued for a long time (many > hours). One thing is that we moved a lot of the workload to our self-hosted > runners - when it comes to public runners, there are short peaks of queued > jobs but they seem to catch-up relatively quickly (max few hours). > > When I see those graph, my interpretation is that Github got a lot better > queuing algorithms.Previously it seemed to be possible by a single project > to take over the queue and get the number of "in progress" way higher than > the 150 jobs limit, and I think that effectively blocked other projects for > a long time. Currently it seems that the algorithm they have balances out > the workload between projects much better - and if there are projects that > have more and more workloads, it impacts their own queue size much more > than queuing time of other projects - see Spark's average number of queued > workflows. > > The graphs IMHO are much healthier than they were a year ago. > > I wonder what is the perception of others :). > > BTW. It would be great if we could move it by a week - this week is shorter > for many and some have longer holidays (1st of Nov is a holiday in a number > of countries). > > On Mon, Nov 1, 2021 at 2:13 PM Gavin McDonald <gmcdon...@apache.org> > wrote: > > > Hi All, > > > > I think it is past time to have another meeting. > > The stresses and strains of our current CI makes me think we should have > > another discussion. > > > > Link to the cwiki page: > > > > > https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-04 > > > > If this week doesn't work then we can postpone until next week. > > > > HTH > > > > -- > > > > *Gavin McDonald* > > Systems Administrator > > ASF Infrastructure Team > > >
