Re: External CI Service Limitations

[Jarek Potiuk]

Fantastic! but the link does not work for me :(

On Fri, Jul 19, 2019 at 8:16 AM Myrle Krantz  wrote:

> Hey Jarek,
>
> Someone at Google was actually looking to get rid of about 11 GKE vouchers
> for $500 each; I don't think anyone has taken them up on that yet, if you
> need more.
>
>
> https://lists.apache.org/thread.html/d9cb36493744836edcd59eebba6aef16356a3d5fd5fa90af336a82e1@%3Cmentors.community.apache.org%3E
>
> Best,
> Myrle
>
> On Fri, Jul 19, 2019 at 8:00 AM Jarek Potiuk 
> wrote:
>
> > OK. We got some credits from Google :) so I will soon be testing out
> > GitLabCI + GKE cluster combination and will let you know the results :)
> >
> > J.
> >
> > On Wed, Jul 17, 2019 at 7:38 AM Jarek Potiuk 
> > wrote:
> >
> > > From Apache Airflow side - I am going to try out GitLab CI setup for
> the
> > > project and I also reached out to Google OSS team to donate a GKE
> > > auto-scaling cluster for our workloads. If it works (highly probable),
> we
> > > might be able to use even less of the free minutes from GitLab because
> we
> > > will have the cluster available to run our builds on. Maybe that's also
> > > something that other projects could look at if the 50K minutes is not
> > > enough.
> > >
> > > Suggestion:  maybe it would be a great idea for GitLab to treat the
> > Apache
> > > projects differently and have a special agreement at least for some of
> > the
> > > projects that cannot get regular donations from other parties easily. I
> > > think this is more of a strategic decision for GitLab to see if this
> > might
> > > be in line with their strategy?.
> > >
> > > For now I think I have everything to try it out for Airflow project and
> > > make POC working in this setup (while waiting for the GKE cluster
> > > donation). Once done I am happy to share our learnings and maybe
> provide
> > > some guidelines for other projects?
> > >
> > > J.
> > >
> > >
> > > On Wed, Jul 17, 2019 at 2:37 AM Greg Stein  wrote:
> > >
> > >> Ray,
> > >>
> > >> Thanks for the offer of 50k minutes/project. That will definitely work
> > >> for most projects.
> > >>
> > >> While we don't have precise measurements, some projects used *way*
> more
> > >> than that within Travis last month:
> > >>
> > >> flink: 350k minutes
> > >> arrow: 260k minutes
> > >> cloudstack: 190k minutes
> > >> incubator-druid: 96k
> > >> airflow: 77k
> > >> ... others: less than 50k
> > >>
> > >> I don't know what would be needed from Infra, to enable the use of
> > Gitlab
> > >> CI for our projects. ??
> > >>
> > >> Thanks,
> > >> Greg Stein
> > >> Infrastructure Administrator, ASF
> > >>
> > >>
> > >> On Tue, Jul 16, 2019 at 7:27 PM Raymond Paik  >
> > >> wrote:
> > >>
> > >>> Joan,
> > >>>
> > >>> The 50,000 minutes would be for each project (assuming individual
> > >>> projects
> > >>> will apply for separate GitLab licenses).
> > >>>
> > >>> When you reach the limit, you'll have an option to purchase
> additional
> > >>> minutes. More info. available at
> > >>>
> > >>>
> >
> https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#what-happens-when-my-ci-minutes-quota-run-out
> > >>> and
> > >>> here's the relevant issue
> > >>>  >.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Ray
> > >>>
> > >>> On Tue, Jul 16, 2019 at 2:33 PM Joan Touzet 
> wrote:
> > >>>
> > >>> > Hi Raymond,
> > >>> >
> > >>> > Would this 50,000 CI minutes per month be spread across the entire
> > ASF,
> > >>> > or just each project? With >300 projects here, that's potentially
> > >>> > 50,000 * 300 = 15 million minutes we're talking about.
> > >>> >
> > >>> > What happens when a project exceeds that amount of minutes? Busy
> > >>> > projects that build each PR, and the build/test cycle takes let's
> say
> > >>> 30
> > >>> > minutes * 3 configuration = ~100 minutes per PR, would consume
> these
> > >>> > minutes with just 100 PRs (or incremental pushes to each PR).
> That's
> > >>> not
> > >>> > much time.
> > >>> >
> > >>> > -Joan
> > >>> >
> > >>> > On 2019-07-16 16:20, Raymond Paik wrote:
> > >>> > > Jarek,
> > >>> > >
> > >>> > > You're not required to migrate your repo over to GitLab. We have
> > >>> other
> > >>> > > projects that keep their source code in GitHub, but are using
> > GitLab
> > >>> for
> > >>> > > CI. Hope this helps...
> > >>> > >
> > >>> > > Thanks,
> > >>> > >
> > >>> > > Ray
> > >>> > >
> > >>> > > On Tue, Jul 16, 2019 at 12:51 PM Jarek Potiuk <
> > >>> jarek.pot...@polidea.com>
> > >>> > > wrote:
> > >>> > >
> > >>> > >> Yep we use Git indeed but we have Github repo (
> > >>> > >> https://github.com/apache/airflow)  and I believe this is
> pretty
> > >>> much
> > >>> > >> standard for all Apache projects (adding Greg as well).
> > >>> > >>
> > >>> > >> I don't think (or am I wrong?) the open source program directly
> > >>> applies
> > >>> > in
> > >>> > >> this case because we would have to have GitLab Repo as well, but
> > in
> > >>> our
> > >>> > >> case w

Re: External CI Service Limitations

[Chesnay Schepler]

For reference, the Travis CI usage was also discussed in INFRA-18533.

Since July 8th the Flink project is no longer running PR builds on ASF 
resources.Instead we mirror pull requests into an external repository 
and run these builds in a sponsored Travis account. If anyone is 
interested, check out the above JIRA.


The reduction in ASF resource usage is less than expected because we're 
currently in the process of creating a new release, which usually 
doubles our CI usage temporarily (since many fixes are merged to master 
and a release branch).


Nevertheless, looking at the resource usage since then (via 
https://kibble.dev), we are now in 4th place, behind incubator-druid.


In the future we also plan to migrate our push/cron builds to the 
external repository, but I can't provide an ETA for this.


On 2019/07/19 03:51:08, David Nalley  wrote:
> On Tue, Jul 16, 2019 at 5:37 PM Greg Stein  wrote:
> >
> > Ray,
> >
> > Thanks for the offer of 50k minutes/project. That will definitely 
work for

> > most projects.
> >
> > While we don't have precise measurements, some projects used *way* more
> > than that within Travis last month:
> >
> > flink: 350k minutes
> > arrow: 260k minutes
> > cloudstack: 190k minutes
> > incubator-druid: 96k
> > airflow: 77k
> > ... others: less than 50k
> >
> > I don't know what would be needed from Infra, to enable the use of 
Gitlab

> > CI for our projects. ??
> >
> > Thanks,
> > Greg Stein
> > Infrastructure Administrator, ASF
> >
>
> It's also worth noting that those numbers are likely constrained by
> our capacity. IIRC Humbedooh said we'd need something close to double
> our currently capacity to satiate the backlog of build requests that
> we had.
>
> Our current travis load should be something like 1.72MM minutes per 
month.

>
> --David
>

Re: [NOTICE] Jenkins GHPRB deprecated, please switch :)

[Udi Meiri]

Michał's investigation came up with 2 features that only GHPRB seems to have:
1. DSL configuration
2. Phrase hooks (being able to comment on a PR to trigger specific tests 
(=Jenkins jobs)).

On 2019/07/11 08:40:14, Michał Walenia  wrote: 
> Hi, could you tell us which plugin do you plan to use as a replacement for
> GHPRB?
> I'd also like to investigate how to migrate Beam's Jenkins jobs to use it.
> 
> Thanks,
> Michal
> 
> On Thu, Jul 11, 2019 at 9:21 AM Daniel Gruno  wrote:
> 
> > As there has been quite a bit of unforeseen trouble here, we have added
> > the old plugin back in, while we evaluate how to best switch (or whether to
> > stick with the GHPRB instead). Needless to say, there is some conflicting
> > documentation we need to look at one more time.
> >
> > On 2019/07/11 03:14:17, Ismael Juma  wrote:
> > > Hi,
> > >
> > > A couple more issues:
> > >
> > > 1. If you configure multiple Jenkins jobs (eg one for Java 8 and one for
> > > Java 11), only 1 seems to update the commit status in the PR. The
> > previous
> > > plugin was able to show the status for multiple jobs.
> > > 2. It's unclear how to enable a job for certain branches only. With the
> > > previous plugin, we could configure the Java 11 job to only trigger for
> > the
> > > branches where Java 11 is supported.
> > >
> > > Am I missing something or is the "modern" plugin not a suitable
> > > replacement? Would you consider restoring the previous plugin?
> > >
> > > Thanks,
> > > Ismael
> > >
> > > On Wed, Jul 10, 2019 at 7:40 PM Ismael Juma  wrote:
> > >
> > > > Does this "modern" version provide a way to retest builds without
> > pushing
> > > > additional commits? This is very useful when tests fail due to
> > > > environmental issues or if they're flaky. I can't seem to find mention
> > of
> > > > it in the documentation. It would be a shame to lose this feature.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Jul 10, 2019 at 7:30 PM Ismael Juma  wrote:
> > > >
> > > >> Looks like it has. Apache Kafka PRs don't trigger Jenkins builds any
> > > >> longer. :( I hope the conversion is smooth, will try it now.
> > > >>
> > > >> Ismael
> > > >>
> > > >> On Wed, Jul 10, 2019 at 3:33 PM Kenneth Knowles 
> > wrote:
> > > >>
> > > >>> Has this support already been removed?
> > > >>>
> > > >>> All of Beam's jobs use this plugin. They are generated using the job
> > DSL,
> > > >>> but we have not ported to the other plugin.
> > > >>>
> > > >>> Kenn
> > > >>>
> > > >>> On Wed, Jul 10, 2019 at 1:33 PM Daniel Gruno 
> > > >>> wrote:
> > > >>>
> > > >>> > Hi folks,
> > > >>> > as part of some cleanup and consolidation (essentially we don't
> > want to
> > > >>> > maintain two different plugins that do the same thing), we have
> > removed
> > > >>> > support for the old GitHub PR Builder on Jenkins, and are focusing
> > on
> > > >>> the
> > > >>> > modern variant. If your build previously made use of the old one
> > (It's
> > > >>> > called "GitHub Pull Request Builder" in the job configuration), we
> > ask
> > > >>> that
> > > >>> > you please switch to the newer one (called "Build pull requests to
> > the
> > > >>> > repository" in the same config). There should be no other changes,
> > but
> > > >>> if
> > > >>> > your builds start acting up, do let infra know :).
> > > >>> >
> > > >>> > As an added bonus, you no longer need to contact infra about
> > webhooks
> > > >>> when
> > > >>> > setting up PR builds for new repositories, it should just work(tm).
> > > >>> >
> > > >>> > With regards,
> > > >>> > Daniel on behalf of ASF Infra.
> > > >>> >
> > > >>>
> > > >>
> > >
> >
> 
> 
> -- 
> 
> Michał Walenia
> Polidea  | Software Engineer
> 
> M: +48 791 432 002 <+48791432002>
> E: michal.wale...@polidea.com
> 
> Unique Tech
> Check out our projects! 
>