Re: Pushing beam.apache.org updates from Jenkins

[Udi Meiri]

Bumping this, in case it got lost during the Jenkins outage last week.
+Ahmet Altay 
+Mikhail Gryzykhin 
+Alan Myrvold 

On Wed, Aug 29, 2018 at 2:28 PM Udi Meiri  wrote:

> Hi,
> I filed https://issues.apache.org/jira/browse/INFRA-16967 and was told to
> inquire at this address about how to best update Beam's website from a
> Jenkins job.
>
> Thanks,
> - Udi
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [NOTICE] Jenkins GHPRB deprecated, please switch :)

[Udi Meiri]

Michał's investigation came up with 2 features that only GHPRB seems to have:
1. DSL configuration
2. Phrase hooks (being able to comment on a PR to trigger specific tests 
(=Jenkins jobs)).

On 2019/07/11 08:40:14, Michał Walenia  wrote: 
> Hi, could you tell us which plugin do you plan to use as a replacement for
> GHPRB?
> I'd also like to investigate how to migrate Beam's Jenkins jobs to use it.
> 
> Thanks,
> Michal
> 
> On Thu, Jul 11, 2019 at 9:21 AM Daniel Gruno  wrote:
> 
> > As there has been quite a bit of unforeseen trouble here, we have added
> > the old plugin back in, while we evaluate how to best switch (or whether to
> > stick with the GHPRB instead). Needless to say, there is some conflicting
> > documentation we need to look at one more time.
> >
> > On 2019/07/11 03:14:17, Ismael Juma  wrote:
> > > Hi,
> > >
> > > A couple more issues:
> > >
> > > 1. If you configure multiple Jenkins jobs (eg one for Java 8 and one for
> > > Java 11), only 1 seems to update the commit status in the PR. The
> > previous
> > > plugin was able to show the status for multiple jobs.
> > > 2. It's unclear how to enable a job for certain branches only. With the
> > > previous plugin, we could configure the Java 11 job to only trigger for
> > the
> > > branches where Java 11 is supported.
> > >
> > > Am I missing something or is the "modern" plugin not a suitable
> > > replacement? Would you consider restoring the previous plugin?
> > >
> > > Thanks,
> > > Ismael
> > >
> > > On Wed, Jul 10, 2019 at 7:40 PM Ismael Juma  wrote:
> > >
> > > > Does this "modern" version provide a way to retest builds without
> > pushing
> > > > additional commits? This is very useful when tests fail due to
> > > > environmental issues or if they're flaky. I can't seem to find mention
> > of
> > > > it in the documentation. It would be a shame to lose this feature.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Jul 10, 2019 at 7:30 PM Ismael Juma  wrote:
> > > >
> > > >> Looks like it has. Apache Kafka PRs don't trigger Jenkins builds any
> > > >> longer. :( I hope the conversion is smooth, will try it now.
> > > >>
> > > >> Ismael
> > > >>
> > > >> On Wed, Jul 10, 2019 at 3:33 PM Kenneth Knowles 
> > wrote:
> > > >>
> > > >>> Has this support already been removed?
> > > >>>
> > > >>> All of Beam's jobs use this plugin. They are generated using the job
> > DSL,
> > > >>> but we have not ported to the other plugin.
> > > >>>
> > > >>> Kenn
> > > >>>
> > > >>> On Wed, Jul 10, 2019 at 1:33 PM Daniel Gruno 
> > > >>> wrote:
> > > >>>
> > > >>> > Hi folks,
> > > >>> > as part of some cleanup and consolidation (essentially we don't
> > want to
> > > >>> > maintain two different plugins that do the same thing), we have
> > removed
> > > >>> > support for the old GitHub PR Builder on Jenkins, and are focusing
> > on
> > > >>> the
> > > >>> > modern variant. If your build previously made use of the old one
> > (It's
> > > >>> > called "GitHub Pull Request Builder" in the job configuration), we
> > ask
> > > >>> that
> > > >>> > you please switch to the newer one (called "Build pull requests to
> > the
> > > >>> > repository" in the same config). There should be no other changes,
> > but
> > > >>> if
> > > >>> > your builds start acting up, do let infra know :).
> > > >>> >
> > > >>> > As an added bonus, you no longer need to contact infra about
> > webhooks
> > > >>> when
> > > >>> > setting up PR builds for new repositories, it should just work(tm).
> > > >>> >
> > > >>> > With regards,
> > > >>> > Daniel on behalf of ASF Infra.
> > > >>> >
> > > >>>
> > > >>
> > >
> >
> 
> 
> -- 
> 
> Michał Walenia
> Polidea  | Software Engineer
> 
> M: +48 791 432 002 <+48791432002>
> E: michal.wale...@polidea.com
> 
> Unique Tech
> Check out our projects! 
> 

Website publishing Jenkins job using Docker issue

[Udi Meiri]

Hi builds,

I'm trying to debug a website publishing issue for the Beam project.
We use a ruby:2.5 based docker image to generate the website. Here's a
successful website test job running on one of Beam's own Jenkins machines
(with extra debugging information):
https://builds.apache.org/job/beam_PreCommit_Website_Commit/3575/

The /repo container directory is using a "bind mount" mapped to the git
repo
(/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Website_Commit/src).
On this successful run the ownership is the uid:gid of the host's jenkins
user:

drwxrwxr-x 18 1017 1018 4096 Apr 14 23:54 /repo


I believe this used to also work on "websites" (1-2 months ago), but is no
longer: https://builds.apache.org/job/beam_PostCommit_Website_Test_PR/17/

Specifically, I believe the issue is with the ownership of /repo, which is:

drwxr-xr-x 18 root root 4096 Apr 14 23:54 /repo


I would appreciate any help in this.
Is the Docker configuration on websites more strict? Perhaps a recent
change?
Is it using AppArmor or SELinux, and could that be the issue?

Re: docker permissions on websites node

[Udi Meiri]

Hi,

Could you share what changes were made?

Beam is having issues since we use bind mounts to write output to the host.
Setting UID:GID on the image gives permission errors in the container when
accessing the mount (can't write).
Not setting it causes files to be created with the wrong (root:root)
permissions, causing permission errors on the host (and having to manually
ssh into Jenkins machines to clear these files as root).

On 2020/04/10 19:22:57, Chris Lambertus  wrote:
> All,>
>
> Infra has implemented some permissions changes to the docker installation
on the websites node, and all containers now map to the jenkins UID on the
host. If you have specified a UID on the command line previously to avoid
permissions issues, your build may now be broken. Please remove any docker
run parameters you have which force a UID. The docker daemon will now
ensure that files created on the host are owned by the jenkins user.>
>
> If you continue to have docker build troubles after removing those
parameters, please let us know.>
>
> -Chris>
>
>

Re: docker permissions on websites node

[Udi Meiri]

Thanks!
That makes sense, and a welcome change to limit root access.


On Thu, Apr 16, 2020 at 2:46 PM Chris Lambertus  wrote:

> Specifically, docker on the websites node (and ONLY the websites node) now
> uses userns-remap to the jenkins UID so you no longer need to pass a -u
> 910:910 parameter to 'docker run’. Unfortunately, the nature of
> userns-remap means that if you DO specify -u 910:910, things fail in a
> weird way. Sorry about that, but we needed the docker daemon to not be able
> to arbitrarily write files as root, as it was causing problems. This is a
> change that will make its way to all the build nodes eventually.
>
> For normal docker builds, you don’t need to specify anything. The default
> UID 0 mapping inside the container will now allow for the container to
> write to your jenkins workspace directory as the jenkins user. If you are
> specifying -u 910:910, remove that.
>
> If you are using the jenkins docker pipeline plugin, you will need to add
> "args '-u root’” to the dockerfile blocks of your jenkinsfile. (INFRA-20116)
>
> Technically speaking, these are the changes made:
>
>
> /etc/docker/daemon.json:
>
> {
>   "userns-remap": "jenkins"
> }
>
>
> cml@jenkins-websites-he-de:~$ cat /etc/subuid
> lxd:10:65536
> root:10:65536
> dockremap:165536:65536
> jenkins:910:1
> jenkins:165536:65536
>
> cml@jenkins-websites-he-de:~$ cat /etc/subgid
> lxd:10:65536
> root:100000:65536
> dockremap:165536:65536
> jenkins:910:1
> jenkins:165536:65536
>
>
>
> -Chris
>
>
>
>
>
>
> > On Apr 16, 2020, at 2:30 PM, Udi Meiri  wrote:
> >
> > Hi,
> >
> > Could you share what changes were made?
> >
> > Beam is having issues since we use bind mounts to write output to the
> host.
> > Setting UID:GID on the image gives permission errors in the container
> when
> > accessing the mount (can't write).
> > Not setting it causes files to be created with the wrong (root:root)
> > permissions, causing permission errors on the host (and having to
> manually
> > ssh into Jenkins machines to clear these files as root).
> >
> > On 2020/04/10 19:22:57, Chris Lambertus  wrote:
> >> All,>
> >>
> >> Infra has implemented some permissions changes to the docker
> installation
> > on the websites node, and all containers now map to the jenkins UID on
> the
> > host. If you have specified a UID on the command line previously to avoid
> > permissions issues, your build may now be broken. Please remove any
> docker
> > run parameters you have which force a UID. The docker daemon will now
> > ensure that files created on the host are owned by the jenkins user.>
> >>
> >> If you continue to have docker build troubles after removing those
> > parameters, please let us know.>
> >>
> >> -Chris>
> >>
> >>
>
>