Re: Turning Off Bb-fbsd2

[Greg Stein]

On Thu, Feb 9, 2017 at 5:53 PM, Allen Wittenauer 
wrote:
>...

> The Mac OS X host was shut down literally a day after I sent out
> an email to common-dev@hadoop announcing I had full build and patch
> testing working.  I had spent quite a bit of time getting Apache Yetus
> ported over to work on Apache's OS X machine, then spent over a month on
> working out the Hadoop specifics, running build after build after build.
> Competing with the Apache Mesos jobs that also ran on that box. The reason
> I was told it was killed was: "no one was using it".  (Umm, what?  Clearly
> no one bothered looking at the build log.)
>

This occurred before I started working as the Infrastructure Administrator
(last Fall). I don't know the full background, other than a PMC requested
that buildbot, then never used it. Yeah: maybe the build logs weren't
examined to see that other projects had hopped onto it.

I also believe we had to pay for that box, and it wasn't cheap.

Today, our preferred model for non-Ubuntu boxes is to have other people
own/run/manage those buildbots and hook them into our buildmaster. For
example, people on the Apache Subversion project have several such 'bots.

We are concentrating our in-house experience on the Ubuntu platform, from
both an operational and a cost angle. Four years ago, the Infra team had
many fewer projects to support. Today, we have hundreds of projects and
many thousands of committers to support. We've had to reallocate in order
to meet the incredible growth of the ASF.

Unfortunately, especially for yourself and some others, the "smoothing down
the edges" has been detrimental.

In parallel, I started working on the Solaris box which was
> then promptly shutdown not too long after I had filed a jira to see if we
> could get the base CA certificates upgraded. (which was pretty much all I
> needed, after that I could have finished getting the Hadoop builds working
> on it as well).
>

We're still shutting down Solaris. Only one guy has experience with it, and
he's also got a ton of other stuff to do.

Our hardware that runs Solaris is also *very* old. Worse: we could never
get a support contract for it. They wouldn't sell us one (messed up, but
there it is). We really need to get that box fully shut down, unracked, and
thrown out.

These were huge blows to Apache Hadoop, as one of the common
> complaints amongst committers is the lack of resources to do cross platform
> testing. Given the ASF had that infrastructure in place, being in this
> position was kind of dumb of the project.  Now the machines are gone and as
> a result, the portability of the code is still very hit or miss and the ASF
> is worse for it.
>

Apache Hadoop is worse for it. As Gavin has noted, just in the past year,
we've increased our build farm dramatically. I believe the ASF is better
for it. We also have a team better focused to support the growth of the ASF.

We can all agree that turning off services sucks for some projects and
people. But our growth has made demands upon the Foundation and its Infra
team that have forced our hand. We also have a funding model that just
doesn't support us hiring a team large enough to retain the disparate array
of services that we offered in the past.


> Since that time, I've helped get the PowerPC build up and running,
> but that's been about it... and even then, I spend little-to-no time on the
> ASF-side of the build bits for those projects I'm interested simply because
> I have no idea if I'll be wasting my time because "whoops, we've changed
> direction again".


Again, we'll happily link any buildbot into our buildmaster, so you can
automate builds on your special bots. As you can see from above, we won't
be doing PowerPC. Just Ubuntu for all machines and services from now on.
This allows us (via Puppet) to easily reallocate, move, upgrade, and
maintain our services. Years ago, each machine was manually configured, and
when it went down, the Foundation suffered. Today, if a machine goes down,
we can spin it back up in an hour or two due to the consistency.

I do sympathize that our service reduction is painful. But I hope you can
understand where the Foundation (and its Infra team) is coming from. We
have vastly more projects to support today, meaning more uniformity is
required.

Sincerely,
Greg Stein,
Infrastructure Administrator, ASF

Re: Turning Off Bb-fbsd2

[Greg Stein]

On Fri, Feb 10, 2017 at 6:51 PM, Allen Wittenauer 
wrote:
>...

> I'm not complaining. I'm trying to understand how the "social
> priority" actually works in practice with the toolset put in place.  If
> Hadoop (or HBase or any number of other large projects currently sharing
> space on the 'Hadoop' nodes) opens the firehose and starts running jobs on


Very simple. Do not "open the firehose". If you crowd out the whole ASF,
then how is that acting responsibly? How is that caring for your fellow ASF
projects?

That is what social priority means.

Don't flood the build servers. If your job takes that much time, then
submit just a couple per day.

Work with us, Allen. You don't appear to even be trying to listen to Gavin,
so why do you think he has decided to stop responding your concerns?

Listen.

If you have any further questions or concerns, then message me. I'll take
it from here.

Thanks,
Greg Stein
Infrastructure Administrator, ASF

Re: [OUTAGE] - Jenkins upgrade this weekend.

[Greg Stein]

What is your point? ... Gavin said, "or file an INFRA ticket" ... that is
what happened.

On Wed, Feb 22, 2017 at 4:37 AM, sebb  wrote:

> FTR:
> At least one build using Java 1.6 stopped working; see INFRA-13556
>
> On 19 February 2017 at 03:04, Gavin McDonald 
> wrote:
> > Hi Again,
> >
> > Jenkins has been upgraded to latest 2.32.2, all plugins have been
> upgraded, Tomcat was also upgraded to latest 7.0.75 release.
> >
> > Downtime was a little over 2.5 hours due to the unplanned Tomcat upgrade
> and having so many plugins that needed individual
> > upgrading.
> >
> > Some config tweaks were also made and so far, logs are looking much
> cleaner.
> >
> > Any issues please report to builds@apache.org list, or file an INFRA
> ticket.
> >
> > Thanks
> >
> > Gav…
> >
> >
> >> On 14 Feb 2017, at 7:30 pm, Gavin McDonald 
> wrote:
> >>
> >> Hi All,
> >>
> >> Our main Jenkins instance at builds.apache.org will be going down for
> an upgrade this Saturday/Sunday.
> >>
> >> Downtime will start at  UTC (Sunday) : 1100 AEDT (Sunday): 1900 EST
> (Saturday)
> >>
> >> Should be less than an hour, however there will be quite a few plugin
> updates so more restarts may be required.
> >>
> >> Will tweet @infrabot 1 hour before and again when complete.
> >>
> >> Thanks
> >>
> >> Gav...
> >>
> >
>

Re: Mac node

[Greg Stein]

Infra does not supply any Mac build slaves. Volunteers/contributors can run
them, and we will happily loop them into the ASF build system.

Cheers,
-g

On Fri, Apr 21, 2017 at 3:20 AM, sospartan  wrote:

> Hi all,
> Do we still have any macosx node?
> I've search this list. Last thread about 'mac' is 2 years ago.
>
> --
> Best Regards!
> --
>
> sospartan
> https://weex-project.io
>

Re: [JENKINS] [IMPORTANT] - Jenkins Migration and Upgrade (And JDK7 deprecation)

[Greg Stein]

I think it depends upon your job type, as Gavin wrote: "in terms of Maven
jobs for JDK 7"

On Wed, Jun 28, 2017 at 10:01 AM, Lukasz Lenart 
wrote:

> Hi,
>
> How to understand that? It won't be possible to run a job using JDK7
> anymore, right? Even if I selected such JDK in my build, Jenkins won't
> run it, is that true?
>
> We cannot run this build [1] anymore as JDK6 isn't support by the
> slaves [2] - will it be the same?
>
> [1] https://builds.apache.org/view/S-Z/view/Struts/job/
> Struts-JDK6-support-2.3/
> [2] https://builds.apache.org/view/S-Z/view/Struts/job/
> Struts-JDK6-support-2.3/1064/console
>
>
> Regards
> --
> Łukasz
> + 48 606 323 122 http://www.lenart.org.pl/
>
> 2017-06-27 9:03 GMT+02:00 Gavin McDonald :
> > ASF Jenkins Master Migration and Upgrade on :-
> >
> >
> > LocationLocal Time
> Time Zone   UTC Offset
> > Melbourne (Australia - Victoria)Sunday, 16 July 2017 at 10:00:00
> am AESTUTC+10 hours
> > New York (USA - New York)   Saturday, 15 July 2017 at
> 8:00:00 pmEDT UTC-4 hours
> > Corresponding UTC (GMT) Sunday, 16 July 2017 at 00:00:00
> >
> >
> > Hi All,
> >
> > A few things are going to happen in just over 2 weeks.
> >
> > 1. Migration of Jenkins to a new host. A Jenkins Master module and yaml
> have been puppetized and ready to go.
> > What we need to do to migrate the Master away from its current host
> is turn off the old service. Perform a final
> > rsync of data and perform the migration tasks.
> >
> > As we intend to preserve history for jobs this will take some time.
> > At the same time as doing this migration to a new host, all slave
> connections will be updated (see below.)
> > I have no current estimate of downtime, but it will run into several
> hours. We do plan to run this migration on a
> > Sunday at the lowest part of Jenkins usual usage.
> >
> > 2. Upgrade of Jenkins - Jenkins project released a new LTS release,
> version 2.60.1. This is a major release and breaks
> > Jenkins in terms of Maven jobs for JDK 7 in the same way that it
> happened for Maven and JDK 6 a few months back.
> >
> > The infra team (mainly myself) got quite some feedback on not
> supplying advance notice of this breakage. That upgrade
> > however was necessary due to security fixes that required our
> upgrade.  This email serves as advance warning of the
> > upcoming upgrade of Jenkins, the downtime due to the migration of
> the service to a new host; and notice of the breakage
> > to JDK 7 that the upgrade brings.
> >
> > Please familiarise yourself with the Jenkins LTS upgrade notes at
> [1].
> > In particular please note:-
> >
> > “…2.60.1 is the first Jenkins LTS release that requires Java 8 to
> run. If you're using the Maven Project type, please note that it needs to
> use a JDK capable of running Jenkins, i.e. JDK 8 or up. If you configure an
> older JDK in a Maven Project, Jenkins will attempt to find a newer JDK and
> use that automatically. If your SSH Slaves fail to start and you have the
> plugin install the JRE to run them, make sure to update SSH Slaves Plugin
> to at least version 1.17 (1.20 recommended).
> > Changes since 2.60:
> > Fix for NullPointerException while initiating some SSH connections
> (regression in 2.59). (issue 44120  org/browse/JENKINS-44120>)
> > Notable changes since 2.46.3:
> > Jenkins (master and agents) now requires Java 8 to run. (issue 27624 <
> https://issues.jenkins-ci.org/browse/JENKINS-27624> <>, issue 42709 <
> https://issues.jenkins-ci.org/browse/JENKINS-42709> <>, pull 2802 <
> https://github.com/jenkinsci/jenkins/pull/2802>, announcement blog post <
> https://jenkins.io/blog/2017/04/10/jenkins-has-upgraded-to-java-8/>)
> >
> > …”
> >
> > There are over 30 other enhancements/fixes since 2.46.2 which we
> currently run so please do take a note of those.
> >
> > Recap: In just over 2 weeks, downtime for a migration AND upgrade is
> planned.
> >
> > Please do not rely on Jenkins at all for that weekend if you use it in
> your release workflow.
> >
> > Please do take this notice back to your dev lists.
> >
> > Any questions or concerns please email back to builds@apache.org
>  only.
> >
> > Thanks
> >
> > Gav…
> >
> > [1] - https://jenkins.io/changelog-stable/
>

Re: IP Addresses used by Apache Master to Connect to Slaves

[Greg Stein]

Yup. That single IP is just fine. We just spun up the master at that
address this summer, and have no plans to move it.

Cheers,
-g

On Oct 4, 2017 18:01, "Meghna Baijal"  wrote:

> Thanks Gavin.
> I was not sure if a single static IP is sufficient.
>
>
>
> On Wed, Oct 4, 2017 at 3:53 PM, Gavin McDonald 
> wrote:
>
> >
> > > On 5 Oct 2017, at 9:39 am, Meghna Baijal 
> > wrote:
> > >
> > > Hi All,
> > > We have a set of slaves on Apache Jenkins that run MXNet builds. I want
> > to
> > > restrict the IP addresses that can access these slaves on port 22
> > (ubuntu)
> > > or port 3389 (windows).
> > > *Question*: Where can I find a fixed and sufficient set of IPs that the
> > > Apache master uses to connect to these slaves?
> >
> > PING jenkins-master.apache.org (62.210.60.235): 56 data bytes
> > 64 bytes from 62.210.60.235: icmp_seq=0 ttl=47 time=362.915 ms
> >
> >
> > >
> > > Thanks,
> > > Meghna Baijal
> >
> >
>

Re: revive bb-openbsd buildbot for Subversion

[Greg Stein]

Heya Stefan! ... I think the best approach is to open a Jira for this. I
imagine that Gavin will handle it, but the ticket will help track the work,
and add some visibility.

Just can't get away from OpenBSD, can ya? ;-)

On Fri, Jan 12, 2018 at 5:55 AM, Stefan Sperling  wrote:

> Hi,
>
> I have prepared a new openbsd buildbot for the Subversion project
> on hardware I own and host. I'm ready to hook it up to the buildbot
> master. I have no data left from the old bb-openbsd bot (once hosted
> in a VM at the ASF) and I don't know its credentials anymore.
>
> Can someone help me with getting my new buildbot worker connected?
>
> Thanks,
> Stefan
>

Re: purging of old job artifacts

[Greg Stein]

Let's go back to the start: stuff older than six months will be deleted.
What could possibly need to be retained?

Assume all jobs will be touched.


On Tue, Apr 24, 2018, 18:32 Allen Wittenauer 
wrote:

>
> > On Apr 24, 2018, at 4:27 PM, Chris Lambertus  wrote:
> >
> > The initial artifact list is over 3 million lines long and 590MB.
>
> Yikes. OK.  How big is the list of jobs?  [IIRC, that should be
> the second part of the file path. e.g., test-ulimit ]  That’d give us some
> sort of scope, who is actually impacted, and hopefully allow everyone to
> clean up their stuff. :)
>
>
> Thanks

Re: purging of old job artifacts

[Greg Stein]

On Wed, Apr 25, 2018 at 9:49 AM, Allen Wittenauer 
wrote:
>...

> I apologize.  I took Greg’s reply as Infra’s official “Go Pound
> Sand” response to what I felt was a reasonable request for more information.
>

It was "Please stop asking ChrisL for more work. The todo work is upon
users of the build system to find/justify retention.". So look through the
areas you're responsible for, find anything that needs to be kept on the
master (as opposed to offloaded), and then we can discuss those items.

Cheers,
-g

Re: purging of old job artifacts

[Greg Stein]

Note that jobs will remain.

This is only about deleting build artifacts.

On Thu, Apr 26, 2018 at 12:40 PM, Alex Harui 
wrote:

> HI Chris,
>
> Thanks for the list.
>
> I’m going through the Flex-related jobs and have some feedback:
>
> flex-blazeds (maven)  We’ve kept this build around even though it hasn’t
> run in a while in case we need to do another release of blazeds.  I would
> like to keep at least one known good build in case we have trouble
> resurrecting it later if we need it, even though it may sit idle for years.
>
> flex-flexunit_1
> flex-sdk_1
> flex-sdk_pixelbender_1
> flex-sdk_release_1
> flex-tlf_1
> flex_sdk_version  I cannot find a project with these names in Jenkins.  So
> feel free to toss it.
>
>
> flex-flexunit (maven)  This project was never completed to build
> successfully, but it would be nice to keep it around in case we need it.
>
> FlexJS Compiler (maven)
> FlexJS Framework (maven)
> FlexJS Pipeline
> FlexJS Typedefs (maven)  Looks like we never set the build limit, so I
> just did that.  The project is disabled, we are keeping it around as
> archival for Royale, so not sure it will clean itself up.
>
> flex-productdashboard I deleted this project.
>
> flex-tool-api (maven)
> flex-sdk-converter (maven)  I’m not seeing old artifacts in these
> projects, but they may also sit idle for years until some bug needs fixing.
>
> Flex-Site (Maven) This project never took off, but again it would be nice
> to keep it around in case it gets revived
>
> In sum, a project like Flex may have several kinds of “products” with
> varying activity levels and thus may have jobs that are idle for years and
> it can be helpful to keep at least the last build around as a reference in
> case the next time we run the build there is a failure.  Please notify us
> if we miss limiting the number of old builds.  I think I fixed the ones
> that didn’t have limits.  But there does seem to be folders left around for
> builds I think we deleted.
>
> Thanks,
> -Alex
>
>
> From: Chris Lambertus 
> Reply-To: 
> Date: Wednesday, April 25, 2018 at 12:04 AM
> To: 
> Subject: Re: purging of old job artifacts
>
>
>
>
> On Apr 24, 2018, at 8:04 PM, Allen Wittenauer  mailto:a...@effectivemachines.com>> wrote:
>
>
>
> On Apr 24, 2018, at 5:01 PM, Greg Stein mailto:gstei
> n...@gmail.com>> wrote:
>
> Let's go back to the start: stuff older than six months will be deleted.
> What could possibly need to be retained?
>
> - Not every job runs every day.  Some are extremely
> situational.
>
> The artifacts do not need to be kept in perpetuity. When every project
> does this, there are significant costs in both disk space and performance.
> Our policy has been 30 days or 10 jobs retention.
>
>
>
>
> - Some users might have specifically marked certain data
> to be retained for very specific reasons.
>
> I know in my case I marked some logs to not be deleted
> because I was using them to debug the systemic Jenkins build node crashes.
> I want to keep the data to see if the usage numbers, etc, go down over time.
>
>
> Part of the systemic problems are due to copious amounts of historical
> data which are loaded into jenkins on startup, inflating the memory usage
> and startup times. Again, when every job does this, it adds up, and many of
> the problems we’re facing appear to be rooted in the very large number of
> artifacts we have.
>
>
>
> So yes, there may be some value to some of that data that
> will not be obvious to an outside observer.
>
>
> Assume all jobs will be touched.
>
> … which is why giving a directory listing of just the base
> directory would be useful to see who needs to look. If INFRA is unwilling
> to provide that data, then keep any directories that reference:
>
>
> Please dispense with the passive aggressive “unwilling to provide”
> nonsense. This is inflammatory and anti-Infra for no valid reason. This
> process is meant to be a pragmatic approach to cleaning up and improving a
> service used by a large number of projects. The fact that I didn’t have
> time to post the job list in the 4 hours since my last reply does not need
> to be construed as reticence on Infra’s part to provide it.
>
> The top-level list of jobs is available here:
> https://paste.apache.org/r37e
>
> I am happy to provide further information, however, due to the disk IO
> issues on jenkins-master and the size of the jobs/ dir, multiple scans and
> data analytics are difficult to provide due to the timescale.
>
>
> As I previously mentioned, the list of actual artifacts curre

Re: purging of old job artifacts

[Greg Stein]

On Thu, Apr 26, 2018, 15:06 Alex Harui  wrote:

> Perhaps I wasn't clear.  I am not concerned about Jenkins projects being
> removed, just the artifacts of those projects.  I am trying to make two
> points:
>
> 1)  Old artifacts might have value as a known good build for a job that
> may not get run for years.  So please don't run a clean up that deletes all
> artifacts older than N days.
>

Nope. Download those artifacts if you want to keep them. The Jenkins Master
is not an archival system. It exists for our projects to do builds. We have
identified the overabundance of old artifacts as detrimental to the
service, which affects everyone.

2)  Somebody else pointed this out as well, but there appear to be folders
> listed for which there is no current Jenkins job.
>

Knew about that. I simply read your note as "please keep these jobs". Some
disabled, some not working, etc. It seemed you were talking about jobs
rather than artifacts within them.

Cheers,
-g


> Thanks,
> -Alex
>
> On 4/26/18, 11:07 AM, "Greg Stein"  wrote:
>
> Note that jobs will remain.
>
> This is only about deleting build artifacts.
>
> On Thu, Apr 26, 2018 at 12:40 PM, Alex Harui  >
> wrote:
>
> > HI Chris,
> >
> > Thanks for the list.
> >
> > I’m going through the Flex-related jobs and have some feedback:
> >
> > flex-blazeds (maven)  We’ve kept this build around even though it
> hasn’t
> > run in a while in case we need to do another release of blazeds.  I
> would
> > like to keep at least one known good build in case we have trouble
> > resurrecting it later if we need it, even though it may sit idle for
> years.
> >
> > flex-flexunit_1
> > flex-sdk_1
> > flex-sdk_pixelbender_1
> > flex-sdk_release_1
> > flex-tlf_1
> > flex_sdk_version  I cannot find a project with these names in
> Jenkins.  So
> > feel free to toss it.
> >
> >
> > flex-flexunit (maven)  This project was never completed to build
> > successfully, but it would be nice to keep it around in case we need
> it.
> >
> > FlexJS Compiler (maven)
> > FlexJS Framework (maven)
> > FlexJS Pipeline
> > FlexJS Typedefs (maven)  Looks like we never set the build limit, so
> I
> > just did that.  The project is disabled, we are keeping it around as
> > archival for Royale, so not sure it will clean itself up.
> >
> > flex-productdashboard I deleted this project.
> >
> > flex-tool-api (maven)
> > flex-sdk-converter (maven)  I’m not seeing old artifacts in these
> > projects, but they may also sit idle for years until some bug needs
> fixing.
> >
> > Flex-Site (Maven) This project never took off, but again it would be
> nice
> > to keep it around in case it gets revived
> >
> > In sum, a project like Flex may have several kinds of “products” with
> > varying activity levels and thus may have jobs that are idle for
> years and
> > it can be helpful to keep at least the last build around as a
> reference in
> > case the next time we run the build there is a failure.  Please
> notify us
> > if we miss limiting the number of old builds.  I think I fixed the
> ones
> > that didn’t have limits.  But there does seem to be folders left
> around for
> > builds I think we deleted.
> >
> > Thanks,
> > -Alex
> >
> >
> > From: Chris Lambertus 
> > Reply-To: 
> > Date: Wednesday, April 25, 2018 at 12:04 AM
> > To: 
> > Subject: Re: purging of old job artifacts
> >
> >
> >
> >
> > On Apr 24, 2018, at 8:04 PM, Allen Wittenauer <
> a...@effectivemachines.com<
> > mailto:a...@effectivemachines.com>> wrote:
> >
> >
> >
> > On Apr 24, 2018, at 5:01 PM, Greg Stein  gstei
> > n...@gmail.com>> wrote:
> >
> > Let's go back to the start: stuff older than six months will be
> deleted.
> > What could possibly need to be retained?
> >
> > - Not every job runs every day.  Some are extremely
> > situational.
> >
> > The artifacts do not need to be kept in perpetuity. When every
> project
> > does this, there are significant costs in both disk space and
> performance.
> > Our policy has been 30 days or 10 jobs retention.
> >
> >
>  

Re: purging of old job artifacts

[Greg Stein]

On Thu, Apr 26, 2018, 16:42 Greg Stein  wrote:

> On Thu, Apr 26, 2018, 15:06 Alex Harui  wrote:
>
>> Perhaps I wasn't clear.  I am not concerned about Jenkins projects being
>> removed, just the artifacts of those projects.  I am trying to make two
>> points:
>>
>> 1)  Old artifacts might have value as a known good build for a job that
>> may not get run for years.  So please don't run a clean up that deletes all
>> artifacts older than N days.
>>
>
> Nope. Download those artifacts if you want to keep them. The Jenkins
> Master is not an archival system. It exists for our projects to do builds.
> We have identified the overabundance of old artifacts as detrimental to the
> service, which affects everyone.
>

To be clearer here, Chris earlier said/asked:
we’d ask that this be exported and managed locally by the project rather
than remaining “live” on the master. Is there a reason other than
convenience for it to remain live?

Cheers,
-g


> 2)  Somebody else pointed this out as well, but there appear to be folders
>> listed for which there is no current Jenkins job.
>>
>
> Knew about that. I simply read your note as "please keep these jobs". Some
> disabled, some not working, etc. It seemed you were talking about jobs
> rather than artifacts within them.
>
> Cheers,
> -g
>
>
>> Thanks,
>> -Alex
>>
>> On 4/26/18, 11:07 AM, "Greg Stein"  wrote:
>>
>> Note that jobs will remain.
>>
>> This is only about deleting build artifacts.
>>
>> On Thu, Apr 26, 2018 at 12:40 PM, Alex Harui > >
>> wrote:
>>
>> > HI Chris,
>> >
>> > Thanks for the list.
>> >
>> > I’m going through the Flex-related jobs and have some feedback:
>> >
>> > flex-blazeds (maven)  We’ve kept this build around even though it
>> hasn’t
>> > run in a while in case we need to do another release of blazeds.  I
>> would
>> > like to keep at least one known good build in case we have trouble
>> > resurrecting it later if we need it, even though it may sit idle
>> for years.
>> >
>> > flex-flexunit_1
>> > flex-sdk_1
>> > flex-sdk_pixelbender_1
>> > flex-sdk_release_1
>> > flex-tlf_1
>> > flex_sdk_version  I cannot find a project with these names in
>> Jenkins.  So
>> > feel free to toss it.
>> >
>> >
>> > flex-flexunit (maven)  This project was never completed to build
>> > successfully, but it would be nice to keep it around in case we
>> need it.
>> >
>> > FlexJS Compiler (maven)
>> > FlexJS Framework (maven)
>> > FlexJS Pipeline
>> > FlexJS Typedefs (maven)  Looks like we never set the build limit,
>> so I
>> > just did that.  The project is disabled, we are keeping it around as
>> > archival for Royale, so not sure it will clean itself up.
>> >
>> > flex-productdashboard I deleted this project.
>> >
>> > flex-tool-api (maven)
>> > flex-sdk-converter (maven)  I’m not seeing old artifacts in these
>> > projects, but they may also sit idle for years until some bug needs
>> fixing.
>> >
>> > Flex-Site (Maven) This project never took off, but again it would
>> be nice
>> > to keep it around in case it gets revived
>> >
>> > In sum, a project like Flex may have several kinds of “products”
>> with
>> > varying activity levels and thus may have jobs that are idle for
>> years and
>> > it can be helpful to keep at least the last build around as a
>> reference in
>> > case the next time we run the build there is a failure.  Please
>> notify us
>> > if we miss limiting the number of old builds.  I think I fixed the
>> ones
>> > that didn’t have limits.  But there does seem to be folders left
>> around for
>> > builds I think we deleted.
>> >
>> > Thanks,
>> > -Alex
>> >
>> >
>> > From: Chris Lambertus 
>> > Reply-To: 
>> > Date: Wednesday, April 25, 2018 at 12:04 AM
>> > To: 
>> > Subject: Re: purging of old job artifacts
>> >
>> >
>> >
>> >
>> > On Apr 24, 2018, at 8:04 PM, Allen Wittenauer <
>> a...@effectivemachines.com<
>> > mailto:a...

Re: purging of old job artifacts

[Greg Stein]

On Thu, Apr 26, 2018 at 5:08 PM, Alex Harui 
wrote:

> On 4/26/18, 2:57 PM, "Greg Stein"  wrote:
> On Thu, Apr 26, 2018, 16:42 Greg Stein  wrote:
>
> > On Thu, Apr 26, 2018, 15:06 Alex Harui 
> wrote:
> >
> >> Perhaps I wasn't clear.  I am not concerned about Jenkins projects
> being
> >> removed, just the artifacts of those projects.  I am trying to make
> two
> >> points:
> >>
> >> 1)  Old artifacts might have value as a known good build for a job
> that
> >> may not get run for years.  So please don't run a clean up that
> deletes all
> >> artifacts older than N days.
> >
> > Nope. Download those artifacts if you want to keep them. The Jenkins
> > Master is not an archival system. It exists for our projects to do
> builds.
> > We have identified the overabundance of old artifacts as detrimental
> to the
> > service, which affects everyone.
>
> To be clearer here, Chris earlier said/asked:
> we’d ask that this be exported and managed locally by the project
> rather
> than remaining “live” on the master. Is there a reason other than
> convenience for it to remain live?
>
> I thought we were allowed to provide feedback instead of following strict
> orders against old artifacts.


Sigh. And there is the "us vs them" crap again. This is not about
"following strict orders" from Infra. Yes, feedback is and has been
requested, and discussion even better.

So: provide that ... is there a good reason to keep the artifacts there?
That is the entire nature of Chris' original email. "We need to delete
these to improve the service. Is there a concern/reason we should not?" And
that has not been answered.

  Sorry for my misunderstanding.  For sure, lots of resources are being
> consumed by old artifacts.  I was just hoping we could save one old build
> per Jenkins project regardless of age for convenience.  But if not, I guess
> we will take the time to figure out where to store it and document where we
> put it.  I guess we have to create a repo for these so it isn't under some
> committer's home.a.o account or on somebody's machine.  Or is there other
> ASF shared storage I'm not remembering?
>

You've listed them. We don't have a generalized, unversioned file storage
mechanism. home.a.o is likely the best. Even if the committer who uploads
them goes AWOL, they'll still remain accessible.

Will old files in Jenkins workspaces for projects also get purged over time?
>

Somebody else will need to answer this. I dunno if we're purging old files
from the workspaces, or how that is different from what *is* being purged.

Cheers,
-g

Re: Pb releasing apache directory LDAP API

[Greg Stein]

There has been discussion of updating the required hashes. Maybe that got
implemented on Nexus? CC'ing peeps...

On Mon, Apr 30, 2018, 13:17 Emmanuel Lécharny  wrote:

> Hi guys,
>
> yesterday, I tried to cut a release of Apache Directory LDAP API 1.0.1.
> It went well, but never hit Nexus.
>
> Today, I tried to do a mvn deploy on the created tag, and now, it's
> visible in Nexus (orgapachedirectory-1151) but I can't close it because
> it complains about some missing .asc signatures :
>
> Missing Signature:
> '/org/apache/directory/api/api-asn1-api/1.0.1/api-asn1-api-1.0.1.jar.asc'
> does not exist for 'api-asn1-api-1.0.1.jar'.
> ...
>
>
> The .md5 and .sha1 signatures are present though.
>
>
> The release is a pretty straightforward process that has been
> established for almost a decade, and that worked just fine for tens of
> releases... It's documented on
> http://directory.staging.apache.org/api/developer-guide.html (Release
> Process), but basically, it's all about doing :
>
> mvn release:prepare -DdryRun=true
> mvn deploy
> mvn release:clean
> mvn release:prepare
> mvn release:perform
>
>
> What could have gone wrong ?
>
> Thanks !
>

Publishing Maven artifacts under third-party coordinates (was: Set up Nexus staging profile for Dubbo ...)

[Greg Stein]

Hi all,

I wanted to send a note that Infra has seen a couple requests from podlings
to publish under third-party Maven groupId coordinates (com.alibaba and
org.netbeans). Unless/until the Foundation owns these domains, we cannot
allow publishing under those coordinates.

Needless to say, we'll never own alibaba.com :p ... Maybe one day, we'll
get netbeans.org (what is the status on that?) ... But we cannot publish
convenience binaries to Maven Central before such time.

And please note that I said "convenience binaries". This is an important
point for the two podlings: the Foundation makes source code releases.
Period. Full stop. Both podlings can do that today -- there is nothing
inhibiting making such releases. ... What *cannot* be done is publishing
convenience binaries to those third-party coordinates.

The podlings will be able to publish under org.apache, of course. The
restriction merely applies to any compatibility/historical shims that are
retained, to map the old-named packages over to new org.apache naming.

Regards,
Greg Stein
Infrastructure Administrator, ASF

On Tue, May 8, 2018 at 12:15 PM, Brian Fox  wrote:

> Was there discussion somewhere that decided to allow an Apache project
> to publish coordinates using com.alibaba? From my perspective this is
> highly unusual for an ASF project. From a central point of view, the
> fact that com.alibaba is registered to publish from a completely
> different repo (oss.sonatype.org) this creates potential for confusion
> over the provenance of the artifacts.
>
> On Wed, May 2, 2018 at 9:30 PM, Chris Lambertus  wrote:
> >
> >
> > On May 2, 2018, at 5:51 PM, Jun Liu  wrote:
> >
> > Hi,
> >
> > We are preparing for the first apache incubating release for dubbo, turns
> > out we lack of some nexus permissions, this blocks our process.
> >
> > Could someone help us please? Here is the ticket asking permissions :
> > https://issues.apache.org/jira/browse/INFRA-16451
> >
> > Best regards,
> > Jun
> >
> >
> >
> > I don’t know how to approach this when the groupID doesn’t match
> org.apache.
> > I’ve requested help from the Sonatype folks.
> >
> > -Chris
> >
>

Re: Jenkins Slave Workspace Retention

[Greg Stein]

Hello all,

A bit of background on why Gavin is proposing this "time to culling"
policy, is because we have been battling disk space issues on the Jenkins
slaves for the past month or so. This suggested policy is needed to keep
the nodes *useful*. If stuff gets culled, then it will just take builds a
bit longer, but they will be POSSIBLE. Nodes that are disk-full won't build
at all, so "slow vs never" seems a good tradeoff.

As noted other-thread, we've also moved the Jenkins Master to a faster
machine, faster disks, and onto bare metal to avoid resource contention
with other VMs.

Jenkins is a ongoing work to keep this resource available to our projects.
Unfortunately, it also suffers a bit of "tragedy of the commons", in that
individual users don't realize their impact on all the other Foundation's
projects. We speak to the outliers as we find them, but an overarching
policy that people are aware of, that they can conform to (and/or we force
conformance) will lead to better results for everybody using this shared
resource.

This list is also great for sharing best-practices (as has been done
else-thread already!) will help all projects remain within the culling
period.

Regards,
Greg Stein
InfraAdmin, ASF




On Mon, Jul 23, 2018 at 2:45 AM Gavin McDonald 
wrote:

> HI All,
>
> I'd like to ask a basic question.
>
> Is there any reason at all to keep the 'workspace' dirs of builds on the
> jenkins slaves ?
>
> If projects do not use the built in plugins to cleanup their workspaces,
> INFRA has a script that will run periodically to do cleanups.
>
> My iniital thinking is that there is no reason at all to keep workspace
> directories
> and that a policy of 7 days would be plenty.
>
> And , in advance, I'd like to state that projects creating their own
> storage area for jars
> and other artifacts to quicken up their builds is not a valid reason.
>
> --
> ---
> Gav...
>

Re: Jenkins Slave Workspace Retention

[Greg Stein]

On Mon, Jul 23, 2018 at 7:42 PM Alex Harui  wrote:

> Hi Greg,
>
> Thanks for the background.  I assume that adding more disk space is
> cost-prohibitive?
>

Most of our Jenkins nodes are donated to us, so we want to use their
generous contributions as best we can.

What would happen if builds.a.o went away and projects had to request and
> use a VM with their own Jenkins instance?
>

Generally, projects can only request a single VM. I suspect that most
projects want to burst-build across a few Jenkins nodes, and a single VM
would serialize their builds/testing too much. There are likely some
economies of scale in how we run our single Jenkins Master and a multitude
of slave nodes. I'm not educated enough on Jenkins to really speak to this,
however.

Requesting more than a single VM *is* doable, but requires Board approval
for the budget or a matching donation. (there may be more paths to multiple
VMs, but those are the two recent approaches over the past month for
Fineract and OpenWhisk to get more VMs)

IIRC, the issue for my projects might be that VMs are currently only
> available as Linux, not Windows?


We can provide Windows VMs, no problem. Ubuntu 16.04, or Windows (not sure
which release).

  And there is some special way that Jenkins can publish to Maven Snapshots
> that would need to be replicated to the VM?
>

This gets trickier, sharing the necessary secrets for publishing. I'll
leave that to Gavin, et al, to discuss.

Cheers,
-g

Re: HBase nightly job failing forever

[Greg Stein]

On Wed, Jul 25, 2018 at 2:36 AM Robert Munteanu  wrote:

> Hi,
>
> On Wed, 2018-07-25 at 09:27 +1000, Gav wrote:
> > Disk space issues , yes, not on most of the Hadoop and related
> > projects
> > nodes - H0-H12 do not have disk space issues. As a Hadoop related
> > project
> > HBase should really be concentrating its builds there.
>
> A suggestion from the sidelines. We could add a 'large-disk-space'
> label to the jobs that use a lot of disk space and then also attach it
> to the executors that offer a lot of disk space.
>

Sure ... though I think it is the small disk that is the odd-man-out. The
nodes usually have lots of space, but a some few don't. And we're also
provisioning with much larger disks nowadays.

Cheers,
-g

Re: No space on H21 (Was: Fwd: Build failed in Jenkins: guacamole-client-coverity #73)

[Greg Stein]

We're refining our monitoring and some cron jobs to try and keep the disk
space pruned. But when space balloons due to poor cleanup, there is only so
much we can do.

The short answer is that the multitude of groups at the Foundation using
these shared resources need to be cognizant of that fact, and cooperate on
ways to keep them available for all. We've seen disk space problems, too
many jobs, too-long jobs, etc.

Infra will keep working and improving our management of the build nodes. In
the meantime, please keep reporting problems, and we'll do our best to keep
the system up for y'all.

Cheers,
Greg
InfraAdmin, ASF


On Sun, Sep 30, 2018 at 10:54 PM Mike Jumper  wrote:

> Is there any way Jenkins could be configured to take nodes offline
> automatically when free space is below a certain threshold? Granted
> space-hungry builds and proper cleanup remain a problem, but at least there
> wouldn't be a random chance of jobs failing due to being assigned to a full
> node.
>
> - Mike
>
>
> On Sun, Sep 30, 2018 at 3:57 PM, Chris Lambertus  wrote:
>
> > The main culprit on H31 appears to be pulsar ws-cleanup jobs consuming
> > almost 40% of the disk. (The problem on H21 was something
> docker-related.)
> >
> >
> > This is the second project I have found (Brooklyn was the first) which is
> > leaving ws-cleanup directories all over the workspace and filling up the
> > disks. Each one of these was 500+ MB. Pulsar, please address ASAP.
> >
> >
> >
> >
> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:19
> > pulsar_precommit_cpp@2_ws-cleanup_1535737112280
> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:18
> > pulsar_precommit_cpp@2_ws-cleanup_1536174576053
> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 06:04
> > pulsar_precommit_cpp@2_ws-cleanup_1537292379178
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 19 22:45
> > pulsar_precommit_cpp@2_ws-cleanup_1537397120862
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:45
> > pulsar_precommit_cpp@2_ws-cleanup_1537403378279
> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 21 23:31
> > pulsar_precommit_cpp@2_ws-cleanup_1537572701829
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 26 16:04
> > pulsar_precommit_cpp@2_ws-cleanup_1537977879341
> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 27 22:23
> > pulsar_precommit_cpp@2_ws-cleanup_1538087035027
> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:14
> > pulsar_precommit_cpp_ws-cleanup_1535699830230
> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:16
> > pulsar_precommit_cpp_ws-cleanup_1535735511423
> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:13
> > pulsar_precommit_cpp_ws-cleanup_1535786064397
> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:18
> > pulsar_precommit_cpp_ws-cleanup_1536096963586
> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 21 06:47
> > pulsar_precommit_cpp_ws-cleanup_1536114938910
> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 21 06:45
> > pulsar_precommit_cpp_ws-cleanup_1536173165909
> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:38
> > pulsar_precommit_cpp_ws-cleanup_1536175584418
> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:50
> > pulsar_precommit_cpp_ws-cleanup_1536180707506
> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:38
> > pulsar_precommit_cpp_ws-cleanup_1536182927940
> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:51
> > pulsar_precommit_cpp_ws-cleanup_1536185193716
> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:50
> > pulsar_precommit_cpp_ws-cleanup_1536273709029
> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 25 05:59
> > pulsar_precommit_cpp_ws-cleanup_1536316964794
> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 25 06:06
> > pulsar_precommit_cpp_ws-cleanup_1536675650893
> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 28 06:08
> > pulsar_precommit_cpp_ws-cleanup_1536879616962
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 30 19:53
> > pulsar_precommit_cpp_ws-cleanup_1536891056968
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 30 19:47
> > pulsar_precommit_cpp_ws-cleanup_1536966466683
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 18 00:43
> > pulsar_precommit_cpp_ws-cleanup_1537216387387
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 19 00:45
> > pulsar_precommit_cpp_ws-cleanup_1537292309413
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 19 00:45
> > pulsar_precommit_cpp_ws-cleanup_1537308526251
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:45
> > pulsar_precommit_cpp_ws-cleanup_1537331780420
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:45
> > pulsar_precommit_cpp_ws-cleanup_1537379886715
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:46
> > pulsar_precommit_cpp_ws-cleanup_1537386197893
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:46
> > pulsar_precommit_cpp_ws-cleanup_1537397073464
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:46
> > pulsar_precommit_cpp_ws-cleanup_1537403247575
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 00:43
> > pulsar_precommit_cpp_ws-cleanup_1537413954888
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 00:45
> > pulsar_precommit_cpp_ws-cleanup_1537417322151
> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 0

Re: No space on H21 (Was: Fwd: Build failed in Jenkins: guacamole-client-coverity #73)

[Greg Stein]

Oh, and I'm not trying to avoid your question about auto-removal. I don't
have that answer, and will leave it for others.

On Sun, Sep 30, 2018 at 11:22 PM Greg Stein  wrote:

> We're refining our monitoring and some cron jobs to try and keep the disk
> space pruned. But when space balloons due to poor cleanup, there is only so
> much we can do.
>
> The short answer is that the multitude of groups at the Foundation using
> these shared resources need to be cognizant of that fact, and cooperate on
> ways to keep them available for all. We've seen disk space problems, too
> many jobs, too-long jobs, etc.
>
> Infra will keep working and improving our management of the build nodes.
> In the meantime, please keep reporting problems, and we'll do our best to
> keep the system up for y'all.
>
> Cheers,
> Greg
> InfraAdmin, ASF
>
>
> On Sun, Sep 30, 2018 at 10:54 PM Mike Jumper  wrote:
>
>> Is there any way Jenkins could be configured to take nodes offline
>> automatically when free space is below a certain threshold? Granted
>> space-hungry builds and proper cleanup remain a problem, but at least
>> there
>> wouldn't be a random chance of jobs failing due to being assigned to a
>> full
>> node.
>>
>> - Mike
>>
>>
>> On Sun, Sep 30, 2018 at 3:57 PM, Chris Lambertus  wrote:
>>
>> > The main culprit on H31 appears to be pulsar ws-cleanup jobs consuming
>> > almost 40% of the disk. (The problem on H21 was something
>> docker-related.)
>> >
>> >
>> > This is the second project I have found (Brooklyn was the first) which
>> is
>> > leaving ws-cleanup directories all over the workspace and filling up the
>> > disks. Each one of these was 500+ MB. Pulsar, please address ASAP.
>> >
>> >
>> >
>> >
>> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:19
>> > pulsar_precommit_cpp@2_ws-cleanup_1535737112280
>> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:18
>> > pulsar_precommit_cpp@2_ws-cleanup_1536174576053
>> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 06:04
>> > pulsar_precommit_cpp@2_ws-cleanup_1537292379178
>> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 19 22:45
>> > pulsar_precommit_cpp@2_ws-cleanup_1537397120862
>> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:45
>> > pulsar_precommit_cpp@2_ws-cleanup_1537403378279
>> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 21 23:31
>> > pulsar_precommit_cpp@2_ws-cleanup_1537572701829
>> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 26 16:04
>> > pulsar_precommit_cpp@2_ws-cleanup_1537977879341
>> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 27 22:23
>> > pulsar_precommit_cpp@2_ws-cleanup_1538087035027
>> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:14
>> > pulsar_precommit_cpp_ws-cleanup_1535699830230
>> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:16
>> > pulsar_precommit_cpp_ws-cleanup_1535735511423
>> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:13
>> > pulsar_precommit_cpp_ws-cleanup_1535786064397
>> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:18
>> > pulsar_precommit_cpp_ws-cleanup_1536096963586
>> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 21 06:47
>> > pulsar_precommit_cpp_ws-cleanup_1536114938910
>> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 21 06:45
>> > pulsar_precommit_cpp_ws-cleanup_1536173165909
>> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:38
>> > pulsar_precommit_cpp_ws-cleanup_1536175584418
>> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:50
>> > pulsar_precommit_cpp_ws-cleanup_1536180707506
>> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:38
>> > pulsar_precommit_cpp_ws-cleanup_1536182927940
>> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:51
>> > pulsar_precommit_cpp_ws-cleanup_1536185193716
>> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:50
>> > pulsar_precommit_cpp_ws-cleanup_1536273709029
>> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 25 05:59
>> > pulsar_precommit_cpp_ws-cleanup_1536316964794
>> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 25 06:06
>> > pulsar_precommit_cpp_ws-cleanup_1536675650893
>> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 28 06:08
>> > pulsar_precommit_cpp_ws-cleanup_1536879616962
>> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 30 19:53
>> > pulsar_precommit_cpp_ws-cleanup_1536891056968
>> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 30 19:47
>> > pulsar_precommit_cpp_ws-cleanup_1536966466683
>> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 18 00

Re: jenkins restart - Sun Oct 7th 2300 hours

[Greg Stein]

The restart was performed, and (yay!) the build history issues were fixed.

Onwards,
-g


On Sat, Oct 6, 2018 at 6:14 PM Chris Lambertus  wrote:

> All,
>
> Jenkins will be restarted on Sun Oct 7th 2300 hours to address some
> inconsistent build history issues.
>
> -Chris
> ASF Infra
>
>

Re: workspace cleanups needed on jenkins master

[Greg Stein]

On Thu, Dec 27, 2018 at 4:35 AM Robert Scholte  wrote:

> maven-box[1] is actually our multibranch(?) job for all our ~100
> repositories, which means the 18Gb is quite a good number.
> With this in mind I believe maven-box should not be on the list for
> cleanup.
>

Chris' note wasn't about "justification" and/or "remove me from the list"
... but more "this is a shared resource. what can each community do, to
help?"

Your answer appears to be "we cannot do anything, as it seems we're already
rockin' it out low numbers for our 100 repositories." Alrighty then.

We shall look to other communities to reduce the amount of shared disk
usage.

Regards,
Greg Stein
InfraAdmin, ASF

Re: Can we package release artifacts on builds.a.o?

[Greg Stein]

On Sun, Jan 6, 2019 at 10:20 PM Alex Harui  wrote:
>...

> All commits, even PR's from non-commiters accepted by a committer are
> supposed to be reviewed, AIUI.  So if the bot makes a commit to the repo,
> the PMC is responsible for reviewing it.  In Royale's case, the bot should
> only be changing pom.xml files and making tags and branches, so a bad bot
> commit should be easy to spot and detection may even be tool-able.
>

Git does not have path-based authorization, so there is no way to restrict
a bot from changing *code*. Give it access to pom.xml, and it has access to
the entire repository. "But the bot won't do that" ... Well, the bot is not
auditable by Legal Affairs or Infrastructure, so there is no way to
validate it is committing Properly.

This is the basic conundrum behind Legal/Infra's decision to disallow bots
from commit access to git repositories. We do have a couple running for svn
repositories, using path-based authz.

Within the Apache Subversion project, have tooling[1] to assist an RM with
pretty much all the steps of a release. From reading this thread, it seems
like Royale's problem is getting RMs up to speed, so maybe it can be solved
with additional build-side tooling?

Cheers,
Greg
InfraAdmin, ASF

[1] https://svn.apache.org/repos/asf/subversion/trunk/tools/dist/

Re: Can we package release artifacts on builds.a.o?

[Greg Stein]

On Mon, Jan 7, 2019 at 8:39 AM stephen.alan.conno...@gmail.com <
stephen.alan.conno...@gmail.com> wrote:

> On 2019/01/07 14:35:08, Greg Stein  wrote:
> > On Sun, Jan 6, 2019 at 10:20 PM Alex Harui 
> wrote:
> > >...
> >
> > > All commits, even PR's from non-commiters accepted by a committer are
> > > supposed to be reviewed, AIUI.  So if the bot makes a commit to the
> repo,
> > > the PMC is responsible for reviewing it.  In Royale's case, the bot
> should
> > > only be changing pom.xml files and making tags and branches, so a bad
> bot
> > > commit should be easy to spot and detection may even be tool-able.
> > >
> >
> > Git does not have path-based authorization, so there is no way to
> restrict
> > a bot from changing *code*. Give it access to pom.xml, and it has access
> to
> > the entire repository. "But the bot won't do that" ... Well, the bot is
> not
> > auditable by Legal Affairs or Infrastructure, so there is no way to
> > validate it is committing Properly.
>
> The bot doesn't need commit access to gitbox... it needs commit access to
> *a* git repo. The RM then pulls the tag from that repo and pushes it *after
> review* back to gitbox.
>
> And there's your auditability for you ;-)
>

Then make that git repo a local clone, hmm?

If you're talking a public one, then what is the "ask" from Infra for this
repo? Every PMC can self-serve create git repositories as they need them.
So it would seem "do that", then you'd need to ask for some extra authz to
enable the bot for that one repository? And what is the mechanism to
prevent leakage of released code into that repository? Or, say, the bot
adjusting pom.xml to pull in malware from $bad ?

Cheers,
-g

Re: Can we package release artifacts on builds.a.o?

[Greg Stein]

On Mon, Jan 7, 2019 at 12:23 PM Alex Harui  wrote:
>...

> I still don't get why allowing a bot to commit to a Git repo isn't
> auditable.  The changes should all be text and sent to commits@ and the
> RMs job is to verify that those commits are ok before putting the artifacts
> up for vote.  I'd even try to  make an email rule that checks for commits
> from buildbot and flags changes to files that are outside of what we
> expected.
>

The historic position of the Foundation is "no ability to commit without a
matched ICLA". That is different from "we'll audit any commits made by
$bot". The trust meter is rather different between those positions,
specifically with the "what if nobody reviews? what if a commit is missed?
what if that added semicolon is missed, yet opens a vuln?" ... With the
"matched ICLA" position, the Foundation has the assurance of *that*
committer, that everything is Good. ... Yet a bot cannot make any such
assurances, despite any "best effort" of the PMC to review the bot's work.

It is likely a solvable problem! My comments here are to outline
history/policy, rather than to say "NO". These are just the parameters of
the problem space.

Cheers,
-g
InfraAdmin

Re: Can we package release artifacts on builds.a.o?

[Greg Stein]

On Wed, Jan 9, 2019 at 2:12 AM Alex Harui  wrote:

> Here's my current summary:
>
> I think at least 3 projects are interested in sharing a computer to build
> all or some of their release artifacts.
>
> I don't know who actually wears an Infra hat other than Greg, but his
> response was maybe.
>

My answers/responses are official Infra replies. Gavin responded earlier in
the thread, and is also part of Infra.

But the most important person on this thread, and who will say whether this
can be allowed, is Roman with his VP Legal Affairs role. In fact, I regard
Infra as Legal's "hands". Much of what we do is rooted in legal concerns of
the Foundation. (the other half is keeping shared tools operating)

>...

> these ideas as well.  Maybe they will, maybe they won't.  But to me, the
> point is that it will be save the community much time and energy if we have
> one shared machine that can crank artifacts.  Most of us change computers
> every few years and have to get set up all over again.


My non-official reply to the above is "sounds brittle". If you plan to set
up a single machine to release artifacts, and the machine blows up ... then
how do you set it up again? Whatever set up is written down (or
containerized, or scripted, or...) would be the same for each RM, no?

Under this  "do more work" logic, builds.a.o doesn't have to exist.
> Communities can do more work and set up their own Jenkins somewhere.
>

Oh, that would be awesome. Jenkins is a headache. We'd love that.

But nah... Infra will maintain our Jenkins installation for the long and
foreseeable future. Very many projects depend upon it, and that's what
we're here for: stand up shared tools, and keep them operating.


> If someone with an Infra hat on tells me to go away,


Nope, I won't say that. But I've tried to share what the official concerns
are with bot accounts. The short answer is that we can't trace their
commits to an ICLA, nor restrict their ability to sneak in bad code.
Reliance upon human review is a non-starter, as there is no audit trail
that a human *performed* that review.

Thus, Infra/Legal leaves it to you/community to find a solution that meets
the Foundation's concerns about provenance. For example, maybe the bot just
creates a PR, for a human to review and merge (thus, we have an ICLA
matched to the merge of the work).

Note that git-at-Apache was initially designed and set up by volunteers.
When the boundaries of what Infra provides needs to be pushed, we like to
turn to volunteers to drive that. We've got a boundless set of work
already, so when a project says "I'd like to do $X", then we say "figure
out how to do it, and then work with us".

Cheers,
Greg
InfraAdmin, ASF

Re: Can we package release artifacts on builds.a.o?

[Greg Stein]

On Thu, Jan 10, 2019 at 4:58 AM Dominik Psenner  wrote:

> On 2019-01-10 11:49, Stephen Connolly wrote:
> > That would meen, though, that the PMC would need to re-encrypt the file
> > every time the PMC changes or any time a PMC member loses their GPG key
> >
> > Note to self: e.g. see
> >
> http://laurent.bachelier.name/2013/03/gpg-encryption-to-multiple-recipients/
> > for example of how to encrypt a file for multiple recipients.
>
> A even trickier situation is when there's no recipient left (because
> went emeritus, death, ..) to decrypt the credentials. :-) It's wise to
> add new pmc members to recipients as soon as the new pmc member arrives.
>

Should anything along these lines be chosen, this wouldn't be a problem.
Infra would be one of the multiple recipients and/or we could easily
generate new credentials when the old creds are lost to stagnation.

Cheers,
-g

Re: External CI Service Limitations

[Greg Stein]

On Tue, Jul 2, 2019 at 11:56 PM Jarek Potiuk  wrote:
>...

> In the meantime maybe INFRA can help to coordinate some effort between
> Flink/Arrow/Airflow to decrease pressure on Travis? We considered few
> options (and are going to implement some of them shortly I think). Some of
> them are not direct changes in Travis CI builds but some other
> workflow/infrastructure changes that will decrease pressure on Travis:
>
>...

> Maybe the committers from Flink and Arrow can also take a look at
> non-obvious ways how their projects can decrease pressure on Travis (at
> least for the time being). Maybe there are some quick wins we can apply in
> short time in coordinated way and buy more time for switching the
> infrastructure ?
>

The above is fabulous. Please continue trading thoughts and working to
reduce your Travis loads, for your own benefit and for your fellow projects
at the Foundation.

This list is the best space to trade such ideas. I'm not sure what Infra
can do, as our skillset is quite a bit different from what your projects
need, for reducing load.

We'll keep this list apprised of anything we find. If anybody knows of,
and/or can recommend a similar type of outsourced build service ... we
*absolutely* would welcome pointers.

We're gonna keep Jenkins and buildbot around for the foreseeable future,
and are interested in outsourced solutions.

Cheers,
Greg Stein
Infrastructure Administrator, ASF

Re: External CI Service Limitations

[Greg Stein]

[builds@ to bcc: ; add vp-infra]

Jarek, Kamil ... that would be very interesting indeed. We've spoken to
Gitlab on possibilities in the past; it sounds like some of our projects
are interested in their services.

Please feel free to email myself/"us" at vp-infra@apache to discuss any
possible options here. Thanks!

Regards,
Greg Stein
Infrastructure Administrator, ASF


On Wed, Jul 3, 2019 at 2:55 AM Jarek Potiuk 
wrote:

> I spoke to Kamil - Gitlab CI maintainer (in CC:) and he will speak to CEO
> of GitLab and Product Managers of GitLab CI whether GitLab will be willing
> to help with it.
>
> J.
>
> On Wed, Jul 3, 2019 at 9:33 AM Jarek Potiuk 
> wrote:
>
> > Actually speaking of Gitlab CI. I realised my close friend is actually
> THE
> > maintainer and main person responsible for Gitlab CI. I will reach out to
> > him and see if they can help with this and provide free service. Shame I
> > have not thought about it before.
> >
> > J.
> >
> > On Wed, Jul 3, 2019 at 8:37 AM Allen Wittenauer
> >  wrote:
> >
> >>
> >>
> >> > On Jul 2, 2019, at 11:12 PM, Jeff MAURY  wrote:
> >> >
> >> > Azure pipeline vas the big plus of supporting Linux Windows and macos
> >> nodes
> >>
> >> There’s a few that support various combinations of non-Linux.
> >> Gitlab CI has been there for a while.  Circle CI has had OS X and is in
> >> beta with Windows.  Cirrus CI has all those plus FreeBSD. etc, etc.
> It’s
> >> quickly becoming required that cloud-based CI systems do more than just
> >> throw up a Linux box.
> >>
> >> > And i think you can add you nodes to the pools
> >>
> >> I think they are limited to being on Azure tho, IIRC.  But I’m
> >> probably not.  I pretty much gave up on doing anything serious with it.
> >>
> >> I really wanted to like pipelines.  The UI is nice.  But in the
> >> end, Pipelines was one of the more frustrating ones to work with in my
> >> experience—and that was with some help from the MS folks. It suffers by
> a
> >> death of a thousand cuts (lack of complex, real-world examples, custom
> >> docker binary, pre-populated bits here and there, a ton of env vars,
> >> artifact system is a total disaster, etc, etc).  Lots of small problems
> >> that add up to just not being worth the effort.
> >>
> >> Hopefully it’s improved since I last looked at it months and
> >> months ago though.
> >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
> >
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>

Re: External CI Service Limitations

[Greg Stein]

On Wed, Jul 3, 2019 at 11:36 AM Allen Wittenauer
 wrote:
>...

> > CouchDB keeps receiving a lot of pressure to build on aarch64, ppc64le
> > and s390x, which keeps pushing us back to Jenkins CI (ASF or
> > independent). And if we have to do that, then not much else matters to
> us.
>
> One of the nice things about using a system that supports external
> runners is that it allows for contributions of CPU time from like minded
> individuals.  I wouldn’t trust them to do anything more than run tests
> though.
>

Right. We have a number of external machines contributed to our buildbot
network. An openbsd box, a Mac box, etc. They run a bunch of svn tests.

We have some external machines using JNLP to hook into our Jenkins network.

If somebody can find a $platform box they need, then it can be hooked in to
our network for a single overview console of tests (assuming you're already
using some of apache's systems already).

Cheers,
-g

Re: External CI Service Limitations

[Greg Stein]

Ray,

Thanks for the offer of 50k minutes/project. That will definitely work for
most projects.

While we don't have precise measurements, some projects used *way* more
than that within Travis last month:

flink: 350k minutes
arrow: 260k minutes
cloudstack: 190k minutes
incubator-druid: 96k
airflow: 77k
... others: less than 50k

I don't know what would be needed from Infra, to enable the use of Gitlab
CI for our projects. ??

Thanks,
Greg Stein
Infrastructure Administrator, ASF


On Tue, Jul 16, 2019 at 7:27 PM Raymond Paik 
wrote:

> Joan,
>
> The 50,000 minutes would be for each project (assuming individual projects
> will apply for separate GitLab licenses).
>
> When you reach the limit, you'll have an option to purchase additional
> minutes. More info. available at
>
> https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#what-happens-when-my-ci-minutes-quota-run-out
> and
> here's the relevant issue
> <https://gitlab.com/gitlab-org/gitlab-ee/issues/3314#note_176031720>.
>
> Thanks,
>
> Ray
>
> On Tue, Jul 16, 2019 at 2:33 PM Joan Touzet  wrote:
>
> > Hi Raymond,
> >
> > Would this 50,000 CI minutes per month be spread across the entire ASF,
> > or just each project? With >300 projects here, that's potentially
> > 50,000 * 300 = 15 million minutes we're talking about.
> >
> > What happens when a project exceeds that amount of minutes? Busy
> > projects that build each PR, and the build/test cycle takes let's say 30
> > minutes * 3 configuration = ~100 minutes per PR, would consume these
> > minutes with just 100 PRs (or incremental pushes to each PR). That's not
> > much time.
> >
> > -Joan
> >
> > On 2019-07-16 16:20, Raymond Paik wrote:
> > > Jarek,
> > >
> > > You're not required to migrate your repo over to GitLab. We have other
> > > projects that keep their source code in GitHub, but are using GitLab
> for
> > > CI. Hope this helps...
> > >
> > > Thanks,
> > >
> > > Ray
> > >
> > > On Tue, Jul 16, 2019 at 12:51 PM Jarek Potiuk <
> jarek.pot...@polidea.com>
> > > wrote:
> > >
> > >> Yep we use Git indeed but we have Github repo (
> > >> https://github.com/apache/airflow)  and I believe this is pretty much
> > >> standard for all Apache projects (adding Greg as well).
> > >>
> > >> I don't think (or am I wrong?) the open source program directly
> applies
> > in
> > >> this case because we would have to have GitLab Repo as well, but in
> our
> > >> case we really need GitLab CI integration with GitHub repository.
> > >>
> > >> Would that be possible to get this case working ?
> > >>
> > >> J
> > >>
> > >> On Tue, Jul 16, 2019 at 6:27 PM Raymond Paik  >
> > >> wrote:
> > >>
> > >>> Thanks Jarek:
> > >>>
> > >>> We do have an open source program at GitLab (
> > >>> https://about.gitlab.com/solutions/open-source/)  where open source
> > >>> projects get access to top tier features (either SaaS or self-hosted)
> > for
> > >>> free including up to 50,000 CI minutes/month.
> > >>>
> > >>> Are you currently using Git as your source code repository?
> > >>>
> > >>> Ray
> > >>>
> > >>> On Tue, Jul 16, 2019 at 8:49 AM Jarek Potiuk <
> jarek.pot...@polidea.com
> > >
> > >>> wrote:
> > >>>
> > >>>> Adding Raymond Paik who is GitLab Community Manager and wants to
> > >>> chime-in
> > >>>> the thread!
> > >>>>
> > >>>> J.
> > >>>>
> > >>>> On Thu, Jul 4, 2019 at 1:04 AM Allen Wittenauer
> > >>>>  wrote:
> > >>>>
> > >>>>>
> > >>>>>
> > >>>>>> On Jul 3, 2019, at 3:15 PM, Joan Touzet 
> wrote:
> > >>>>>>
> > >>>>>> I was asking if any of the service platforms provided this. So
> far,
> > >>> it
> > >>>>> looks like no.
> > >>>>>
> > >>>>> I was playing around bit with Drone today because we
> actually
> > >>>>> need ARM in $DAYJOB and this convo reminded me that I needed to
> check
> > >>> it
> > >>>>> out.
> > >>>>>
> > >>>>> So far, I’m a little underwhelmed with the feature set. (No
> > >>>>> built-in artifacting, no junit output processing, buggy/broken yaml
> > >>>>> parser,  … to be fair, they are relatively new so likely still
> > building
> > >>>>> these things up) BUT! They do support gitlab and acting as a gitlab
> > ci
> > >>>>> runner. So theoretically one could do linux/x86, windows/x86, mac
> os
> > >>> x, and
> > >>>>> linux/arm off of a combo of gitlab ci + drone.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>> --
> > >>>>
> > >>>> Jarek Potiuk
> > >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>>>
> > >>>> M: +48 660 796 129 <+48660796129>
> > >>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >> --
> > >>
> > >> Jarek Potiuk
> > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>
> > >> M: +48 660 796 129 <+48660796129>
> > >> [image: Polidea] <https://www.polidea.com/>
> > >>
> > >>
> > >
> >
> >
>

Re: External CI Service Limitations

[Greg Stein]

On Mon, Jul 22, 2019 at 2:20 AM Jarek Potiuk 
wrote:

> Hello Everyone (especially the infrastructure),
>
> Can we increase a number of workers/jobs we have per project now?
> Decreasing it to 5 (which I believe is the case) is terrible for us now
> We are nearing  1.10.4 release with Airflow and if we have more than one PR
> in the queue it waits for several hours to run!
>
> Can we increase the limits to 15 or so (3 parallell builds for Airflow as
> we are running 5 jobs per stage).
>

Sorry to say, but "no". Travis is a *shared* resource.

As noted elsethread, before we applied limits, Airflow used about 77,000
minutes in a month. That is tying up two executors full-time for the entire
month. We have a hundred projects using Travis, and Airflow consumed a 20th
of our entire capacity.

The limit for all projects shall remain at five (5).

You can always run your tests locally, to prepare for your upcoming
release. The Foundation's paid resources need to remain shared/available
for all of our projects.

Regards,
Greg Stein
Infrastructure Administrator, ASF

Re: Github Actions

[Greg Stein]

On Sun, Aug 11, 2019 at 5:15 PM Francis Chuang 
wrote:
>...

> I think there are quite a few ASF projects using gitbox and Github and
> this would be a very good complement or replacement for Travis, appvoyer
> and other CI/CD platforms currently in use.
>
> Is there any interest from the ASF to enable this for all Gitbox
> projects when it becomes fully public?
>

Absolutely. The Infrastructure team would love to see groups try this out,
and share the experiences here.

If there are any hurdles, then share them and we'll try to knock them down.

I am also interested in being able to push to our website automatically
> using Github Actions. If the git token that can push to a particular
> website repository is added as a secret [2] to Github Actions, this
> would be pretty easy to use for projects to automate the building of
> their websites.
>

Should be possible. Again, comes back to groups trying this and reporting
back how well it went.

Cheers,
Greg Stein
Infrastructure Administrator, ASF

Re: Github Actions

[Greg Stein]

On Sun, Aug 11, 2019 at 5:39 PM Francis Chuang 
wrote:

> Hi Greg,
>
> Should we/are we allowed to use Github Actions for ASF projects on
> Github right now? Do we need permission from infra or ASF itself?
>

Try it. We don't know if/what barriers might exist.

Infra's position is that we want to enable this for all of our projects. So
"permission granted" conceptually. Is there something more to effect that?
Don't know.


> Regarding automated website builds: Currently, this is only possible on
> the ASF git-websites jenkins node as it holds the correct git
> credentials to push to the website branch for ASF git repositories. Is
> there an official process/workflow for this token to be added as a
> secret in Github Actions?
>

Again: figure out how to add a secret, and let us know.

One concern is whether that secret is visible to the project, or only the
org admins. We don't know. Go find out and report back :)

Really: the short answer is that Infra is not going to do the *exploration*
at this time. But if/when you hit a roadblock, *then* we can fix it. Even
better, figuring it out and telling Infra "please do steps 1, 2, and 3."

Thanks,
-g
InfraAdmin, ASF

Re: Github Actions

[Greg Stein]

Have you had an opportunity to make progress on this, to share with us?

Anybody else with news?

Thanks!
-g
InfraAdmin, ASF


On Tue, Aug 13, 2019 at 3:59 PM Karl Heinz Marbaise 
wrote:

> Hi,
>
> I've made a simple PoC for the Apache Maven Dependency Plugin on a
> separate branch.
>
> I will try within the next days more features for example Mac OS builds
> etc.
>
>
> Currently I simply push my changes via gitbox ..
>
> maven-dependency-plugin (GITHUB_ACTIONS)$ git remote -v
> origin  https://gitbox.apache.org/repos/asf/maven-dependency-plugin.git
> (fetch)
> origin  https://gitbox.apache.org/repos/asf/maven-dependency-plugin.git
> (push)
>
>
> Also I'm interested to use SonarCloud related with GitHub Actions..?
>
>
> Kind regards
> Karl Heinz Marbaise
> Apache Maven PMC
>
> [1]: https://github.com/apache/maven-dependency-plugin/runs/192633340
> [2]:
>
> https://github.com/apache/maven-dependency-plugin/blob/66435b225e7885f44b25207e025469f6d5237107/.github/workflows/maven.yml
>
> On 12.08.19 00:31, Greg Stein wrote:
> > On Sun, Aug 11, 2019 at 5:15 PM Francis Chuang  >
> > wrote:
> >> ...
> >
> >> I think there are quite a few ASF projects using gitbox and Github and
> >> this would be a very good complement or replacement for Travis, appvoyer
> >> and other CI/CD platforms currently in use.
> >>
> >> Is there any interest from the ASF to enable this for all Gitbox
> >> projects when it becomes fully public?
> >>
> >
> > Absolutely. The Infrastructure team would love to see groups try this
> out,
> > and share the experiences here.
> >
> > If there are any hurdles, then share them and we'll try to knock them
> down.
> >
> > I am also interested in being able to push to our website automatically
> >> using Github Actions. If the git token that can push to a particular
> >> website repository is added as a secret [2] to Github Actions, this
> >> would be pretty easy to use for projects to automate the building of
> >> their websites.
> >>
> >
> > Should be possible. Again, comes back to groups trying this and reporting
> > back how well it went.
> >
> > Cheers,
> > Greg Stein
> > Infrastructure Administrator, ASF
> >
>

Re: Github Actions

[Greg Stein]

Hi Francis,

Is the token needed to push from calcite to calcite-site? Is that an oauth
token or something? And are you able to use the repository settings to add
secrets, but you don't have the right token? Or you cannot add secrets at
all? (I can't tell since I have superpowers)

I've added GSTEIN_TEST_SECRET to Calcite. See if you can extract/print that
into your build/action log. If so, then we can try to figure out the
security here (ie. how do we avoid Actions exfiltrating the token?)

Thanks,
-g

On Tue, Aug 27, 2019 at 5:19 AM Francis Chuang 
wrote:

> I have implemented the ability to generate the website and javadoc for
> Calcite using Github Actions. See:
> https://github.com/apache/calcite/tree/test-site/.github/workflows
>
> The missing piece is that we need the token to publish to our
> calcite-site repository to be added as a secret in Github Actions and
> there is currently no clear process as to whether this is allowed or how
> to get this done.
>
> See:
> https://issues.apache.org/jira/browse/INFRA-18874
> https://issues.apache.org/jira/browse/INFRA-18875
>
> Francis
>
> On 27/08/2019 7:52 pm, Greg Stein wrote:
> > Have you had an opportunity to make progress on this, to share with us?
> >
> > Anybody else with news?
> >
> > Thanks!
> > -g
> > InfraAdmin, ASF
> >
> >
> > On Tue, Aug 13, 2019 at 3:59 PM Karl Heinz Marbaise 
> > wrote:
> >
> >> Hi,
> >>
> >> I've made a simple PoC for the Apache Maven Dependency Plugin on a
> >> separate branch.
> >>
> >> I will try within the next days more features for example Mac OS builds
> >> etc.
> >>
> >>
> >> Currently I simply push my changes via gitbox ..
> >>
> >> maven-dependency-plugin (GITHUB_ACTIONS)$ git remote -v
> >> origin  https://gitbox.apache.org/repos/asf/maven-dependency-plugin.git
> >> (fetch)
> >> origin  https://gitbox.apache.org/repos/asf/maven-dependency-plugin.git
> >> (push)
> >>
> >>
> >> Also I'm interested to use SonarCloud related with GitHub Actions..?
> >>
> >>
> >> Kind regards
> >> Karl Heinz Marbaise
> >> Apache Maven PMC
> >>
> >> [1]: https://github.com/apache/maven-dependency-plugin/runs/192633340
> >> [2]:
> >>
> >>
> https://github.com/apache/maven-dependency-plugin/blob/66435b225e7885f44b25207e025469f6d5237107/.github/workflows/maven.yml
> >>
> >> On 12.08.19 00:31, Greg Stein wrote:
> >>> On Sun, Aug 11, 2019 at 5:15 PM Francis Chuang <
> francischu...@apache.org
> >>>
> >>> wrote:
> >>>> ...
> >>>
> >>>> I think there are quite a few ASF projects using gitbox and Github and
> >>>> this would be a very good complement or replacement for Travis,
> appvoyer
> >>>> and other CI/CD platforms currently in use.
> >>>>
> >>>> Is there any interest from the ASF to enable this for all Gitbox
> >>>> projects when it becomes fully public?
> >>>>
> >>>
> >>> Absolutely. The Infrastructure team would love to see groups try this
> >> out,
> >>> and share the experiences here.
> >>>
> >>> If there are any hurdles, then share them and we'll try to knock them
> >> down.
> >>>
> >>> I am also interested in being able to push to our website automatically
> >>>> using Github Actions. If the git token that can push to a particular
> >>>> website repository is added as a secret [2] to Github Actions, this
> >>>> would be pretty easy to use for projects to automate the building of
> >>>> their websites.
> >>>>
> >>>
> >>> Should be possible. Again, comes back to groups trying this and
> reporting
> >>> back how well it went.
> >>>
> >>> Cheers,
> >>> Greg Stein
> >>> Infrastructure Administrator, ASF
> >>>
> >>
> >
>

Re: Github Actions

[Greg Stein]

Yeah. FIgured as much, hoped that I was missing something :)

(note: we have the same issue with buildbot and jenkins: we simply trust
the communities to not exfil that data)

On Tue, Aug 27, 2019 at 9:16 PM Matt Sicker  wrote:

> How to avoid leaking secrets: only way to do that is via pre-verified code
> that executes something with that secret. Otherwise, there’s literally
> infinite ways to leak it being a Turing machine and all. This applies to
> all CICD tools.
>
> On Tue, Aug 27, 2019 at 20:32, Greg Stein  wrote:
>
> > Hi Francis,
> >
> > Is the token needed to push from calcite to calcite-site? Is that an
> oauth
> > token or something? And are you able to use the repository settings to
> add
> > secrets, but you don't have the right token? Or you cannot add secrets at
> > all? (I can't tell since I have superpowers)
> >
> > I've added GSTEIN_TEST_SECRET to Calcite. See if you can extract/print
> that
> > into your build/action log. If so, then we can try to figure out the
> > security here (ie. how do we avoid Actions exfiltrating the token?)
> >
> > Thanks,
> > -g
> >
> > On Tue, Aug 27, 2019 at 5:19 AM Francis Chuang  >
> > wrote:
> >
> > > I have implemented the ability to generate the website and javadoc for
> > > Calcite using Github Actions. See:
> > > https://github.com/apache/calcite/tree/test-site/.github/workflows
> > >
> > > The missing piece is that we need the token to publish to our
> > > calcite-site repository to be added as a secret in Github Actions and
> > > there is currently no clear process as to whether this is allowed or
> how
> > > to get this done.
> > >
> > > See:
> > > https://issues.apache.org/jira/browse/INFRA-18874
> > > https://issues.apache.org/jira/browse/INFRA-18875
> > >
> > > Francis
> > >
> > > On 27/08/2019 7:52 pm, Greg Stein wrote:
> > > > Have you had an opportunity to make progress on this, to share with
> us?
> > > >
> > > > Anybody else with news?
> > > >
> > > > Thanks!
> > > > -g
> > > > InfraAdmin, ASF
> > > >
> > > >
> > > > On Tue, Aug 13, 2019 at 3:59 PM Karl Heinz Marbaise <
> khmarba...@gmx.de
> > >
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I've made a simple PoC for the Apache Maven Dependency Plugin on a
> > > >> separate branch.
> > > >>
> > > >> I will try within the next days more features for example Mac OS
> > builds
> > > >> etc.
> > > >>
> > > >>
> > > >> Currently I simply push my changes via gitbox ..
> > > >>
> > > >> maven-dependency-plugin (GITHUB_ACTIONS)$ git remote -v
> > > >> origin
> > https://gitbox.apache.org/repos/asf/maven-dependency-plugin.git
> > > >> (fetch)
> > > >> origin
> > https://gitbox.apache.org/repos/asf/maven-dependency-plugin.git
> > > >> (push)
> > > >>
> > > >>
> > > >> Also I'm interested to use SonarCloud related with GitHub Actions..?
> > > >>
> > > >>
> > > >> Kind regards
> > > >> Karl Heinz Marbaise
> > > >> Apache Maven PMC
> > > >>
> > > >> [1]:
> https://github.com/apache/maven-dependency-plugin/runs/192633340
> > > >> [2]:
> > > >>
> > > >>
> > >
> >
> https://github.com/apache/maven-dependency-plugin/blob/66435b225e7885f44b25207e025469f6d5237107/.github/workflows/maven.yml
> > > >>
> > > >> On 12.08.19 00:31, Greg Stein wrote:
> > > >>> On Sun, Aug 11, 2019 at 5:15 PM Francis Chuang <
> > > francischu...@apache.org
> > > >>>
> > > >>> wrote:
> > > >>>> ...
> > > >>>
> > > >>>> I think there are quite a few ASF projects using gitbox and Github
> > and
> > > >>>> this would be a very good complement or replacement for Travis,
> > > appvoyer
> > > >>>> and other CI/CD platforms currently in use.
> > > >>>>
> > > >>>> Is there any interest from the ASF to enable this for all Gitbox
> > > >>>> projects when it becomes fully public?
> > > >>>>
> > > >>>
> > > >>> Absolutely. The Infrastructure team would love to see groups try
> this
> > > >> out,
> > > >>> and share the experiences here.
> > > >>>
> > > >>> If there are any hurdles, then share them and we'll try to knock
> them
> > > >> down.
> > > >>>
> > > >>> I am also interested in being able to push to our website
> > automatically
> > > >>>> using Github Actions. If the git token that can push to a
> particular
> > > >>>> website repository is added as a secret [2] to Github Actions,
> this
> > > >>>> would be pretty easy to use for projects to automate the building
> of
> > > >>>> their websites.
> > > >>>>
> > > >>>
> > > >>> Should be possible. Again, comes back to groups trying this and
> > > reporting
> > > >>> back how well it went.
> > > >>>
> > > >>> Cheers,
> > > >>> Greg Stein
> > > >>> Infrastructure Administrator, ASF
> > > >>>
> > > >>
> > > >
> > >
> >
> --
> Matt Sicker 
>

Re: Github Actions

[Greg Stein]

It has been a couple months. Have GitHub Actions worked out for you? Can
you share? Are there things needed from Infra?

I would like to know if there is a path for projects to move *off* Jenkins,
Travis, Circle, etc .. and *onto* GitHub Actions. Can GHA solve projects'
needs for CD/CI??

thx,
-g


On Sun, Aug 11, 2019 at 5:31 PM Greg Stein  wrote:

> On Sun, Aug 11, 2019 at 5:15 PM Francis Chuang 
> wrote:
> >...
>
>> I think there are quite a few ASF projects using gitbox and Github and
>> this would be a very good complement or replacement for Travis, appvoyer
>> and other CI/CD platforms currently in use.
>>
>> Is there any interest from the ASF to enable this for all Gitbox
>> projects when it becomes fully public?
>>
>
> Absolutely. The Infrastructure team would love to see groups try this out,
> and share the experiences here.
>
> If there are any hurdles, then share them and we'll try to knock them down.
>
> I am also interested in being able to push to our website automatically
>> using Github Actions. If the git token that can push to a particular
>> website repository is added as a secret [2] to Github Actions, this
>> would be pretty easy to use for projects to automate the building of
>> their websites.
>>
>
> Should be possible. Again, comes back to groups trying this and reporting
> back how well it went.
>
> Cheers,
> Greg Stein
> Infrastructure Administrator, ASF
>
>

Re: [CI] What are the troubles projects face with CI and Infra

[Greg Stein]

On Mon, Feb 3, 2020 at 9:29 PM Alex Harui  wrote:

> Hopefully last set of questions for now...
>
> 1) It sounds like there is a risk that as the ASF grows, GH may not be
> able to grow with us.  Did I understand that correctly?
>

GitHub will always grow faster than us. Not a worry.


> 2) If we have money to offer GH, why can't we offer money to the CI
> Vendors so we aren't really abusing their free tiers?
>

We already pay TravisCI, Inc. for a set of builders. We also have lots of
donated credits from multiple vendors, and donated build nodes. See
else-thread about "expand to consume all provided capacity".

3) Does GH track my activity in the ASF GH repos as part of the API usage
> for Apache?  IOW, am I adding to the ASF API count by closing an issue on
> github.com?  Or if I ran a script on my computer that closed the issue by
> using their API?
>

API usage is per-user, not about the target repo/org, so what *you* do has
no bearing upon limits for Foundation tooling. Good question.


> I think builds.a.o is a great free service, but AIUI, the
> no-third-party-write-access rule is independent of whether CI is free or
> not.  I cannot pay money and get write-access to the ASF repos.
>

Yes and yes.

Downstream users trust Apache because of our provenance rules (per feedback
over the years). Spoiling that assurance, spoils our reputation; that is
kind of at the heart of the issue for the Board to debate.

We can conceivably code our way into a proxy that creates limitations, but
$world that is using GitHub won't be using our proxy. Our builder nodes
that publish to the asf-site branch is within our control. It *does*
effectively use our established proxy/controls.

Welcome to the Infra world of CI/CD :p

Cheers,
-g

Re: What's the git-websites equivalent for cloudbees (ci-cassandra.a.o)?

[Greg Stein]

Can't use that private Jenkins Master ... go back to the shared master and
the git-websites node, in order to push websites.

Cheers,
-g

On Sat, Apr 25, 2020 at 7:36 AM Mick Semb Wever  wrote:

> I'm trying to figure out how to git push, using the jenkins user, a
> website build on ci-cassandra.apache.org.
>
> The credentials for the jenkins user, IIUC, are normally provided by
> the git-websites label (agent).
>
> In Jenkins I see the jenkins (master pub key) credentials available.
> But even selected the build won't push to gitbox.a.o
>
>   fatal: could not read Username for 'https://gitbox.apache.org': No
> such device or address
> ref: https://ci-cassandra.apache.org/job/Cassandra%20Website/13/console
>
> Is this new territory, worthy of an INFRA ticket?
>
> regards,
> Mick
>

Re: All ci-* Masters upgrades

[Greg Stein]

Hey Vladimir ... those are Jenkins Masters for the build clusters.

On Sat, Aug 1, 2020, 11:42 Vladimir Sitnikov 
wrote:

> > ci-builds.apache.org - The replacement Master for builds.apache.org
>
> I'm new to CloudBees. Could you please clarify what "Master" means in this
> context?
>
> Vladimir
>

Re: GitHub Actions Concurrency Limits for Apache projects

[Greg Stein]

This is some great news, Jarek.

Given that GitHub build minutes are shared, we need more of this kind of
work from our many communities.

Thanks,
Greg
InfraAdmin, ASF


On Sun, Oct 18, 2020 at 2:32 PM Jarek Potiuk 
wrote:

> Hello Allen,
>
> I'd really love to give a try to Yetus - how it can actually make our
> approach better.
>
> I just merged the change I planned (finally we got to that), that
> implements the final optimisation that you mentioned. In the case of a
> single .md file change we got the build time down to about 1 minute, most
> of it being GitHub Actions "workflow" overhead.
>
> We went-down with the incremental pre-commit tests to ~ 25s.
>
> Build here: https://github.com/potiuk/airflow/pull/128/checks. As you can
> see here:
>
> https://github.com/potiuk/airflow/pull/128/checks?check_run_id=1268353637#step:7:98
> in
> this case we run only the relevant static checks:
>
>- "No-tabs checker"
>- "Add license for all md files"
>- "Add TOC for md files."
>- "Check for merge conflicts"
>- "Detect Private Key"
>- "Fix End of Files"
>- "Trim Trailing Whitespace"
>- "Check for language that we do not accept as community",
>
> All the other checks, image building, and all the extra checks are skipped
> (automatically as pre-commit determined them irrelevant).
>
> All this, while we keep really comprehensive tests and optimisation of
> image building for all the "serious steps". I tried to explain the
> philosophy and some basic assumptions behind our CI in
> https://github.com/apache/airflow/blob/master/CI.rst#ci-environment - and
> I'd love to try to see how this plays together with the Yetus tool.
>
> Would it be possible to work together with the Yetus team on trying to see
> how it can help to further optimise and possibly simplify the setup we
> have? I'd love to get some cooperation on those. I am nearly done with all
> optimisations I planned, And we are for years (long before my tenure) among
> top-3 Apache projects when it comes to CI-time use, so that might be a good
> one if we can pull together some improvements.
>
>
> J.
>
>
>
> On Wed, Oct 14, 2020 at 4:41 PM Jarek Potiuk 
> wrote:
>
> > Exactly - > dialectic vs. dislectic for example.
> >
> > On Wed, Oct 14, 2020 at 4:40 PM Jarek Potiuk 
> > wrote:
> >
> >> And really sorry about yatus vs. yetus - I am slightly dialectic and
> when
> >> things are not in the dictionary, I tend to do many mistakes. I hope
> it's
> >> not something that people can take as a sign of being "worse", but if
> you
> >> felt offended by that - apologies.
> >>
> >>
> >>
> >> On Wed, Oct 14, 2020 at 4:34 PM Jarek Potiuk 
> >> wrote:
> >>
> >>> Hey Allen,
> >>>
> >>> I would be super happy if you could help us to do it properly at
> Airlfow
> >>> - would you like to work with us and get the yatus configuration that
> >>> would work for us ? I am super happy to try it? Maybe you could open PR
> >>> with some basic yatus implementation to start with and we could work
> >>> together to get it simplified? I would love to learn how to do it.
> >>>
> >>> J
> >>>
> >>>
> >>> On Wed, Oct 14, 2020 at 3:37 PM Allen Wittenauer
> >>>  wrote:
> >>>
> 
> 
>  > On Oct 13, 2020, at 11:04 PM, Jarek Potiuk <
> jarek.pot...@polidea.com>
>  wrote:
>  > This is a logic
>  > that we have to implement regardless - whether we use yatus or
>  pre-commit
>  > (please correct me if I am wrong).
> 
>  I'm not sure about yatus, but for yetus, for the most part,
>  yes, one would like to need to implement custom rules in the
> personality to
>  exactly duplicate the overly complicated and over engineered airflow
>  setup.  The big difference is that one wouldn't be starting from
> scratch.
>  The difference engine is already there. The file filter is already
> there.
>  full build vs. PR handling is already there. etc etc etc
> 
>  > For all others, this is not a big issue because in total all other
>  > pre-commits take 2-3 minutes at best. And if we find that we need to
>  > optimize it further we can simply disable the '--all-files' switch
> for
>  > pre-commit and they will only run on the latest commit-changed files
>  > (pre-commit will only run the tests related to those changed files).
>  But
>  > since they are pretty fast (except pylint/mypy/flake8) we think
>  running
>  > them all, for now, is not a problem.
> 
>  That's what everyone thinks until they start aggregating the
>  time across all changes...
> 
> 
> >>>
> >>> --
> >>>
> >>> Jarek Potiuk
> >>> Polidea  | Principal Software Engineer
> >>>
> >>> M: +48 660 796 129 <+48660796129>
> >>> [image: Polidea] 
> >>>
> >>>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea  | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48660796129>
> >> [image: Polidea] 

Re: Travis CI migration

[Greg Stein]

This is a cross-project mailing list, by definition. There are always going
to be posts specific to singular communities.

The goal is to bring these multiple communities together around
build-related issues. Maybe one post, on a Tuesday in May, will help
another community with their problems. That is the goal: serendipitous
sharing and benefit.

Just move along, if you do not find it interesting.

Cheers,
Greg
InfraAdmin, ASF


On Thu, Nov 12, 2020 at 8:19 AM Vladimir Sitnikov <
sitnikov.vladi...@gmail.com> wrote:

> Friends, could I please kindly ask to refrain from posts like "project ABC
> is not migrated"?
>
> Apparently, there might be issues, so please mail Gavin directly.
> There are lots of people reading the list, and I do not think everybody is
> interested in all the projects.
>
> Thank you and have a nice day.
>
> Vladimir
>

Re: Failure with Github Actions from outside of the organization (out of a sudden!)

[Greg Stein]

"badly executed" occurs when we believe there is a *security* issue.

Using an action defined by a third party, which might modify Apache
repositories in unknown ways ... not something we want.

Or when a PMC determines that third party Action is safe, in April, but
then gets compromised in August ... not something we want.

By constraining all Actions to the apache/ repositories will ensure that
appropriate review is possible.

Regards,
Greg Stein
InfraAdmin, ASF


On Sun, Dec 27, 2020 at 6:43 AM Jarek Potiuk  wrote:

> Ok. IT works after logging. I will make another comments shortly after
> subscribing to the list but I think this was very badly executed.
>
> J.
>
>
>
> On Sun, Dec 27, 2020 at 1:38 PM Jarek Potiuk  wrote:
>
> > the link does not work
> >
> > On Sun, Dec 27, 2020 at 1:34 PM Roy Lenferink 
> > wrote:
> >
> >> This is related to the thread Daniel just posted on the users@infra
> list:
> >>
> >>
> https://lists.apache.org/thread.html/r900f8f9a874006ed8121bdc901a0d1acccbb340882c1f94dad61a5e9%40%3Cusers.infra.apache.org%3E
> >>
> >> Op zo 27 dec. 2020 om 13:26 schreef Andreas Veithen <
> >> andreas.veit...@gmail.com>:
> >>
> >> > Same for https://github.com/apache/axis-axis2-java-core (with no
> >> > configuration changes on our side).
> >> >
> >> > Andreas
> >> >
> >> > On Sun, Dec 27, 2020 at 12:25 PM Jarek Potiuk 
> >> wrote:
> >> >
> >> > > Is there a change in the policy of Apache Airflow to only allow
> >> > > actions hosted in-organization? Or is it a mistake in configuration?
> >> > >
> >> > > We've just started @Apache Airflow to experience errors of this kind
> >> out
> >> > of
> >> > > a sudden (literally within the last hour).
> >> > >
> >> > > potiuk/get-workflow-origin@588cc14f9f1cdf1b8be3db816855e96422204fec
> ,
> >> > > louisbrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798
> ,
> >> > > actions/checkout@v2, actions/checkout@v2, actions/checkout@v2,
> >> > >
> >> >
> >>
> tobked/label-when-approved-action@4c5190fec5661e98d83f50bbd4ef9ebb48bd1194
> >> > > ,
> >> > > louisbrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798
> ,
> >> > >
> >> >
> >>
> tobked/label-when-approved-action@4c5190fec5661e98d83f50bbd4ef9ebb48bd1194
> >> > > ,
> >> > >
> >> >
> >>
> tobked/label-when-approved-action@4c5190fec5661e98d83f50bbd4ef9ebb48bd1194
> >> > > ,
> >> > > and
> >> louisbrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798
> >> > > are
> >> > > not allowed to be used in apache/airflow. Actions in this workflow
> >> must
> >> > be:
> >> > > within a repository owned by apache.
> >> > >
> >> > >
> >> > > J,
> >> > >
> >> >
> >>
> >
> >
> > --
> > +48 660 796 129
> >
>
>
> --
> +48 660 796 129
>

Re: Failure with Github Actions from outside of the organization (out of a sudden!)

[Greg Stein]

On Sun, Dec 27, 2020 at 6:54 AM Jarek Potiuk 
wrote:
>...

> Was it as a response to some security incident that would justify such
> immediate and disruptive action without an earlier warning? What was the
> reasoning behind this?
>

Yes.

Re: Failure with Github Actions from outside of the organization (out of a sudden!)

[Greg Stein]

We are all human, and fallible. Could we have resolved this earlier, and
rolled it out in a controlled manner? Yes. But we did not understand the
issue, and (thus) did not apply the restriction sooner.

Volunteers help out Infra with alerts on JDK releases, about email issues,
etc ... We would certainly welcome more alerts/contributions when these
kinds of issues arise.

And yes: DGruno noted in Slack that you were using a pinned version (yay!).
This isn't about Airflow, but about protecting the overall Foundation
attack surface.

Thx,
-g


On Sun, Dec 27, 2020 at 7:04 AM Jarek Potiuk 
wrote:

> Ok. Thanks. I understand now.
>
> But it would be great if you would explain the context in the announcement
> and try to be a bit more vocal (builds@ seems like a great place to
> announce such changes).
>
> It caught everyone by surprise.
>
> Again - the scenarios you explain were already discussed before (I have to
> now rush and fix stuff for our committers so I cannot find it quickly). I
> raised it a few months ago and it was a consensus to use
> https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions
> explains exactly what to do (using pinned hashes) which we rigorously
> follow and recommended everyone to do - to prevent exactly the scenarios
> you described.
>
> While I understand this approach can't be enforced (and the policy is
> reasonable), it could have been acted on before I think, and without such
> rush.
>
> J.
>
>
> On Sun, Dec 27, 2020 at 1:57 PM Greg Stein  wrote:
>
>> On Sun, Dec 27, 2020 at 6:54 AM Jarek Potiuk 
>> wrote:
>> >...
>>
>>> Was it as a response to some security incident that would justify such
>>> immediate and disruptive action without an earlier warning? What was the
>>> reasoning behind this?
>>>
>>
>> Yes.
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>

Re: Failure with Github Actions from outside of the organization (out of a sudden!)

[Greg Stein]

On Sun, Dec 27, 2020 at 11:29 AM Jarek Potiuk 
wrote:

> For the records - I also looked up the discussion we had about the
> Github Action issues
>
> That was a discussion on 7th of October on priv...@airflow.apache.org
> (for those who have access there - here is the link:
> https://lists.apache.org/thread.html/re92b77f64e5923ec0044edf3a060339b9e170f9438e2fde0811bf6d2%40%3Cprivate.airflow.apache.org%3E
>
> The title was "Potential Security issues with custom GitHub Actions". We
> came to the conclusion that just following the official recommendations
> from GitHub was enough for us:
> https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions
> But we have not thought it might be un-enforceable for the overall Apache
> community. It looks like then it was me, who have not "bubble" it up. I
> should have escalated it to infra/security (and I will for sure do it next
> time we stumble on smth like this).
>
> For the future, I'd love to know what is the best way to do so?
>

Definitely email security@ when you see problems that might affect other
projects(*). They will open an internal ticket, and track the problem to an
appropriate conclusion (pulling in Infra as-needed). There are a number of
operational groups at the Foundation where Infra works as their "hands" to
get things done, and Security, and Legal, are two of the most important
groups that we provide work effort for.

Cheers,
-g

(*) if you believe it is Infra-related, then a cc: to private@infra is fine
(ASF Members only) or root@apache if particularly sensitive. Note that
builds@apache is worldwide-public, and users@infra is committer-available.

Re: Failure with Github Actions from outside of the organization (out of a sudden!)

[Greg Stein]

On Tue, Dec 29, 2020 at 2:30 PM Jarek Potiuk 
wrote:
>...

> On Tue, Dec 29, 2020 at 2:12 PM Vladimir Sitnikov <
> sitnikov.vladi...@gmail.com> wrote:
>
>...

> Jarek>This is exactly what Greg is writing about
> >
> > Greg's message was very vague, so I asked for clarification.
>
> I hope my explanations help :)
>

I sent a couple, and Daniel Gruno also sent a couple messages. We've been
spending time to determine what the next steps need to be. Gavin has been
digging in, too.

Our first priority is to maintain the integrity of the Foundation's
systems, and then the code's integrity and provenance.

Unfortunately, stopping a community's ability to get work done falls below
the above two things :-( ... so when we got a security report based on PRs
running Actions with an unknown impact on our code ... we just turned them
off. Overreaction? Sure. I totally understand that, but I made that call
anyway. And we've gathered a lot more knowledge since then, and started
loosening things up, finding guidance (from Jarek!), and planning next
steps.

One of things that we will likely do is perform a scan of any
Action/workflow .yml at commit time, to ensure that any "uses:" is defined
with a hash rather than a tag. That should prevent the kind of attack Jarek
described where Action FOO@v7 does something very different today, than it
did yesterday.

We also need to dig further into GITHUB_TOKEN around PRs for external
forks. I haven't checked in with the guys yet, but it looks like that is no
longer a concern (eg. the original security report might be invalid).
Regardless, constraining third party actions is likely going to be part of
a final plan.

>...

> I really hope that all of us + infra will find a good solution. But we need
> to cooperate.
>
>...

> I think Greg was very clear in his message that after reacting to the
> security
> incident - this is the right time to start discussion on what INFRA can do
> next.
>

That is my hope. To get this solved where Actions are useful to our
projects, but we have some safety measures in place, where needed. We
didn't know this weekend, but know more now.

Thanks,
Greg
InfraAdmin, ASF

Re: Failure with Github Actions from outside of the organization (out of a sudden!)

[Greg Stein]

On Tue, Dec 29, 2020 at 8:08 PM Brennan Ashton 
wrote:

> On Tue, Dec 29, 2020 at 5:33 PM Greg Stein  wrote:
> > One of things that we will likely do is perform a scan of any
> > Action/workflow .yml at commit time, to ensure that any "uses:" is
> defined
> > with a hash rather than a tag. That should prevent the kind of attack
> Jarek
> > described where Action FOO@v7 does something very different today, than
> it
> > did yesterday.
>
> It would be nice if we could keep using tags from known verified
> sources like github,
> microsoft, etc.. which seems to be the case now. They make fairly
> frequent changes
> to improve the stability of the actions and without a good way to know
> when to bump
> the SHA that is going to be a pain.
>

That's a great point. Sure. Certainly apache/* we can allow it to "float"
with HEAD, pin it to a tag, or to use a SHA. ie. the most flexible choices

For trustedcorp/* we could allow a tag or a SHA, but no floating/unpinned.

And so on. We can probably trust Microsoft and its GitHub subsidiary
without reservation. But we'll likely need to make a call for other
third-parties that we want to whitelist.

Forking a third party Action into the apache/* space provides flexibility,
no restrictions, and you can always pull down new changes or issue PRs
upstream. A bit of work? Some, yes. Does it rise to the level of a problem?
Welp, that's what we're exploring :-)

TBH I don't see how the threat surface here is that much different
> than pulling down
> packages from pypi to npm at build time.
>

And that is why those packages should be pinned and checksums verified,
too. Do people do that? Nope. Should they? Yup. (and Infra falls into the
"we could do better, too"; not casting stones)

(and yes, I'm aware of the past hacks via both those package repositories)

While this change caught us off guard it was not that hard to get
> things moving again.
>

Kind of got to us, too. Not a call that I would have liked to make.


> It would be nice in the future changes that will very likely break
> things (urgent or not) would be CCd
> to the Incubator General mailing list.  Not all of us have found our
> way to the right mailing lists.
>

We try to give heads-up to users@ when we can (on all Infra topics that
might affect users), and builds@ when it is build-related. All committers
can subscribe to users@. builds@ is a public list, so I think it would be
fine for anybody involved in the Incubator to forward from the builds@ list
over to public general@, but I do not believe that is a call for Infra to
make. general@ is not "our list" to decide on what/how to use.

Cheers,
-g

Re: Builds Meeting this Thursday

[Greg Stein]

[offtopic, pet peeve]

On Wed, Jan 13, 2021 at 2:47 AM Jarek Potiuk  wrote:

> One of PMCs @Airflow (in ASIA timezone) asked if the meeting can be


Apache Airflow is a PMC. That Committee has *members* that are part of that
committee. Those members are NOT "PMCs". We have about 200 PMCs at the
Foundation, established by the Board. Please stop confusing *members* with
*committees*.

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

[Greg Stein]

On Wed, Apr 7, 2021 at 12:25 AM Hyukjin Kwon  wrote:

> Hi all,
>
> I am an Apache Spark PMC,


You are a member of the Apache Spark PMC. You are *not* a PMC. Please stop
with that terminology. The Foundation has about 200 PMCs, and you are a
member of one of them. You are NOT a "PMC" .. you're a person. A PMC is a
construct of the Foundation.

>...

> I am aware of the limited GitHub Actions resources that are shared
> across all projects in ASF,
> and many projects suffer from it. This issue significantly slows down the
> development cycle of
>  other projects, at least Apache Spark.
>

And the Foundation gets those build minutes for GitHub Actions provided to
us from GitHub and Microsoft, and we are thankful that they provide them to
the Foundation. Maybe it isn't all the build minutes that every group
wants, but that is what we have. So it is incumbent upon all of us to
figure out how to build more, with fewer minutes.

Say "thank you" to GitHub, please.

Regards,
-g

Re: svn-bb-openbsd worker now really needs buildbot server > 0.8

[Greg Stein]

What timing! ci2 is on 2.something, but before asking a bunch of projects
to move onto it, we had already planned to upgrade it this coming weekend!

Gavin can provide.orr details, if needed, or simply send an email with the
upgrade results.

Cheers,
Greg
InfraAdmin, ASF

On Wed, Jun 23, 2021, 13:45 Stefan Sperling  wrote:

> Hello,
>
> It's been 4 years since I put a workaround in place in order to
> connect to ci.apache.org which runs buildbot 0.8. At that time
> the issue I filed was closed as wontfix:
> https://issues.apache.org/jira/browse/INFRA-16332
>
> The most recent OS upgrade completely drops support for Python 2 and
> my present workaround no longer works. I don't want to invest any
> more time into a solution that requires me to install obsolete software.
>
> I could easily install a buildbot-worker-3.0.2 package.
> By now there is a ci2.apache.org system which should support newer
> buildbot workers. Does this system support version 3.0.2?
>
> Could the Subversion project run buildbot workers against both systems,
> or would we need to switch all our workers over at the same time?
>
> Thanks,
> Stefan
>

Re: Addressing Nutch use of CMS WAS: [IMPORTANT] - ci.apache.org and CMS Shutdown end of January 2022

[Greg Stein]

On Sun, Jan 9, 2022 at 7:25 AM Sebastian Nagel
 wrote:
>...

> Would be nice if
>   https://svn.apache.org/repos/asf/nutch/cms_site/
> could stay for some more time - just in case we forgot some file
> (such as the doap.rdf recently)


That will remain as long as you want. Infra will not touch it.

We *did* need to touch the CMS areas for organizational needs,
self-imposed. But repos/asf/nutch/ is all yours. Carry on!

Cheers,
Greg Stein
InfraAdmin, ASF

Re: Broken Builds because of gradle-enterprise

[Greg Stein]

On Fri, Dec 9, 2022 at 5:41 AM Olivier Lamy  wrote:

> Hi
> Are we seriously activating this gradle enterprise to every build and
> so breaks a few builds without any notice?
>

Apologies. We thought it would simply be an additional output, and would
not affect any builds. Gaven has turned it off, and we'll do some more
testing to find out why this happened.

>...

> [INFO] Publishing build scan...
> [INFO] https://ge.apache.org/s/mabitou3lbw2y
>
> we (Apache Maven) don’t want to use this


I see no discussion on dev@maven or private@maven to suggest the PMC does
not want this tool, once we get it debugged.

Regards,
Greg Stein
Infrastructure Administrator, ASF

Re: Broken Builds because of gradle-enterprise

[Greg Stein]

On Fri, Dec 9, 2022 at 9:05 AM Slawomir Jaranowski 
wrote:
>...

> There is no discussion on dev@maven, private@maven ... for a simple reason
> - nobody from the Maven team knows that you want to introduce such
> features.
>

Olivier said the Maven PMC didn't want this. He can't take their name like
that.

Please start a topic about it on Maven list with a description of what you
> want to introduce, why, what will be to do before change.
>
> Some of us were surprised due to the failed build.
>

We were just as surprised. It should have made no difference. Clearly: it
did.

We make changes to the Jenkins environment all the time, to keep it
operational, on the latest software, to provide enough CPU and disk space
for builds, add requested plugins, and more. We do not advise projects
before we make changes because we expect no problems to arise. This fell
into that same kind of "routine change", or so we thought. We backed it
out, and will see if a fix is possible without any need for changes in
projects' build processes.

Our goal is "no change in build processes", and/or "small changes for more
build features (eg. analysis)".

We'll reach out when we know more.

Cheers,
Greg
InfraAdmin, ASF

Re: Broken Builds because of gradle-enterprise

[Greg Stein]

On Fri, Dec 9, 2022 at 9:54 AM Allen Wittenauer  wrote:

> > On Dec 9, 2022, at 7:43 AM, Greg Stein  wrote:
> >
> > We make changes to the Jenkins environment all the time, to keep it
> > operational, on the latest software, to provide enough CPU and disk space
> > for builds, add requested plugins, and more. We do not advise projects
> > before we make changes because we expect no problems to arise. This fell
> > into that same kind of "routine change", or so we thought.
>
>
> From
> https://plugins.jenkins.io/gradle/#plugin-content-gradle-enterprise-integration
> :
>
> Note - The configuration applies to all builds on all connected
> agents matching the specified label criteria, or all in case no label
> criteria are defined.
>
> So need custom labels and have builds move to those labels if they want to
> use the feature.


Thanks, Allen!

Yeah, that will be helpful when we enable builds to be selective
(opt-in/out).

Still working through it for now ... We do have Gradle Inc. on a shared
slack channel to help out.

Cheers,
-g

Re: Broken Builds because of gradle-enterprise

[Greg Stein]

On Fri, Dec 9, 2022 at 7:41 PM Olivier Lamy  wrote:

> On Sat, 10 Dec 2022 at 00:48, Greg Stein  wrote:
> >> On Fri, Dec 9, 2022 at 5:41 AM Olivier Lamy  wrote:
> >>
> >> Hi
> >> Are we seriously activating this gradle enterprise to every build and
> >> so breaks a few builds without any notice?
> >
> > Apologies. We thought it would simply be an additional output, and would
> not affect any builds. Gaven has turned it off, and we'll do some more
> testing to find out why this happened.
>
> No worries. issues can happen but when something is touching builds,
> it's probably better to announce it.
>

Understood, and normally we would. We honestly thought this was a "side
effect" process that would operate while builds were processed. Gradle,Inc
thought the same, too, and are now evaluating options. Again: goal is
"nobody should have to change their build process, and we will get some
build analysis features"


> I don't understand why you guys INFRA do not talk first to the
> opensource communities of Maven and/or Jenkins if you need data on
> what is happening?
>

Again: we did not expect a problem. The expectation was that we'd get some
data about builds, which we did not have before. That we could do some
introspection. ... but primarily: we thought it would be *transparent* ...
That didn't happen.

But really. We do this all the time. We replace build nodes. Do we talk to
the communities? Nah. Cuz upgrading a build node from 50G of disk to 400G
of disk should not be a problem. And it never is. We upgrade our Jenkins
controllers to new versions. Does that cause problems? Nope. We upgrade
build agents from Ubuntu 18.04 to 22.04. No problem. We add latest Maven to
the build nodes. Latest JDK. etc etc etc

I am not going to fault anybody on the Infrastructure Team for not
foreseeing this would break builds. Again: apologies that it did. Mistakes
happen.

>...

Cheers,
Greg
InfraAdmin, ASF

Re: Using Github Actions Trusted Publisher for PyPI releases ?

[Greg Stein]

Hey Jarek ... note that we have an infrastructure-actions repository for
"official ASF" GH Actions. If you agree with that approach, then you can
dev/test there or we can move your tested Action there when you're ready to
share it with others.

Cheers,
Greg
InfraAdmin, ASF


On Thu, Jun 20, 2024 at 7:10 AM Jarek Potiuk  wrote:

> Unless I hear otherwise, I **assume** there are no big reasons against
> this. My plan is that I will add a Github Action (manually triggered,
> limited to release managers only) which will NOT build the packages, but it
> will download them from `downloads.apache.org` (or dist.apache.org for RC
> packages) and publish them to PyPI. This should be really "safe" and will
> remove the needs for us to keep local pypi keys to upload the packages.
>
> This will require repo reconfiguration, so I will have to - likely - open a
> JIRA ticket to INFRA - once I do it, I will be happy to describe the steps
> for all other projects that upload packages to PyPI and use GitHub.
>
> Does that make sense?
>
> J.
>
>
> On Fri, Jun 14, 2024 at 12:14 PM Jarek Potiuk  wrote:
>
> >
> >> My only question is what do the users see in terms of the verified
> >> identity that performed the release. Does it still appear to have come
> >> from the individual maintainer? The ASF? Somewhere else? I'd only be
> >> concerned if the answer was "somewhere else".
> >>
> >
> > Currently users do not see anything. There was a discussion on Python's
> > discord about exposing Trusted Published information in PyPI
> >
> https://discuss.python.org/t/pre-pep-exposing-trusted-publisher-provenance-on-pypi/42337
> > as a "pre-PEP discussion". This resulted in Draft PEP 740 -
> >
> https://discuss.python.org/t/pep-740-index-support-for-digital-attestations/44498
> > - where you will be able to upload multiple attestations when you publish
> > your packages. So the thinking is that you can have multiple attestations
> > of provenance of your package when you upload it to PyPI and a trusted
> > publisher will be just one of them. So in our case we could also add our
> > own signatures when we publish., This is still draft and we will have a
> > chance of influencing the direction, I am sure. Generally Michael and the
> > whole security team are on the spree of onboarding more and more projects
> > to use trusted publishers and they are planning to discuss and
> implemented
> > more security/provenance features when they reach critical mass (from the
> > discussions I had - I believe they are doing very well there - and
> having a
> > stories where prominent projects are on-board is going to help with that
> as
> > well.
> >
> > J.
> >
> >
> >
> >
> >> Mark
> >>
> >> -
> >> To unsubscribe, e-mail:
> security-discuss-unsubscr...@community.apache.org
> >> For additional commands, e-mail:
> >> security-discuss-h...@community.apache.org
> >>
> >>
>