Re: [Hosting] Downtime for VMs on gprod5 7/18
Move completed, all VMs restarted successfully. Users of mysql-vip2 may have seen a brief outage believed to be due to a saturated network switch. We've move gprod5 to a separate switch to alleviate this. Apologies! -- Justin Dugger Senior Systems Administrator OSU Open Source Lab On Thu, Jul 13, 2017 at 3:03 PM, Justin Dugger wrote: > At 10am on July 18th, all VMs running on gprod5 will be shut down. > > As part of the OSL's ongoing rack rotation datacenter efficiency > project, VMs hosted on gprod5 will be offline for the duration of the > move: > > bro-try.osuosl.org > darcs.osuosl.org > jenkins.inkscape.org > > Based on past experience, I expect the outage to last about an hour. > > Note this is the rescheduling of a move originally planned for 6/28. > > -- > Justin Dugger > Senior Systems Administrator > OSU Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.osuosl.org/mailman/listinfo/hosting
Re: jenkins emergency restart notice - Tues 00:00 UTC
Have the changes resulted in any noticeable improvements? Is there a JIRA interested folks can follow along on? On Tue, Jul 18, 2017 at 6:40 PM, Chris Lambertus wrote: > We’re trying a suggested downgrade of the credentials binding plugin which > may be related to this. No guarantees, though. I will be restarting again > tonight at 0100 UTC 19 July. > > -Chris > > > > > > On Jul 18, 2017, at 9:00 AM, Chris Lambertus wrote: > > > > This appears to be a bug in one of the new versions of either jenkins or > a plugin. We are still looking into the issue. The limit was raised > yesterday by a factor of 4. > > > > > > > > > >> On Jul 18, 2017, at 4:08 AM, Geoff Macartney < > geoff.macart...@cloudsoft.io> wrote: > >> > >> We're seeing them in Apache Brooklyn builds too, e.g. > >> https://builds.apache.org/job/brooklyn-server-master/664. > >> > >> regards > >> Geoff > >> > >> > >> > >> On Tue, 18 Jul 2017 at 12:03 Ismael Juma wrote: > >> > >>> Hi, > >>> > >>> Thanks for the heads up. Was this done? We are still seeing "too many > open > >>> files" issues in Kafka builds. One example of many: > >>> > >>> https://builds.apache.org/job/kafka-trunk-jdk7/ > lastCompletedBuild/console > >>> > >>> Ismael > >>> > >>> On Mon, Jul 17, 2017 at 2:53 PM, Chris Lambertus > wrote: > >>> > > Jenkins will be restarted at midnight tonight UTC to address problems > related to ‘too many open files’ and to attempt to correct a problem > with > some windows agents. I will be stopping new builds so that the queue > can > empty over the next 2 hours. > > -Chris > > > >>> > > > >
Re: jenkins emergency restart notice - Tues 00:00 UTC
I filed https://issues.apache.org/jira/browse/INFRA-14628 which is probably related to this since the exception message says "Too many open files". 2017-07-19 17:01 GMT+02:00 Mike Drob : > Have the changes resulted in any noticeable improvements? Is there a JIRA > interested folks can follow along on? > > On Tue, Jul 18, 2017 at 6:40 PM, Chris Lambertus wrote: > > > We’re trying a suggested downgrade of the credentials binding plugin > which > > may be related to this. No guarantees, though. I will be restarting again > > tonight at 0100 UTC 19 July. > > > > -Chris > > > > > > > > > > > On Jul 18, 2017, at 9:00 AM, Chris Lambertus wrote: > > > > > > This appears to be a bug in one of the new versions of either jenkins > or > > a plugin. We are still looking into the issue. The limit was raised > > yesterday by a factor of 4. > > > > > > > > > > > > > > >> On Jul 18, 2017, at 4:08 AM, Geoff Macartney < > > geoff.macart...@cloudsoft.io> wrote: > > >> > > >> We're seeing them in Apache Brooklyn builds too, e.g. > > >> https://builds.apache.org/job/brooklyn-server-master/664. > > >> > > >> regards > > >> Geoff > > >> > > >> > > >> > > >> On Tue, 18 Jul 2017 at 12:03 Ismael Juma wrote: > > >> > > >>> Hi, > > >>> > > >>> Thanks for the heads up. Was this done? We are still seeing "too many > > open > > >>> files" issues in Kafka builds. One example of many: > > >>> > > >>> https://builds.apache.org/job/kafka-trunk-jdk7/ > > lastCompletedBuild/console > > >>> > > >>> Ismael > > >>> > > >>> On Mon, Jul 17, 2017 at 2:53 PM, Chris Lambertus > > wrote: > > >>> > > > > Jenkins will be restarted at midnight tonight UTC to address > problems > > related to ‘too many open files’ and to attempt to correct a problem > > with > > some windows agents. I will be stopping new builds so that the queue > > can > > empty over the next 2 hours. > > > > -Chris > > > > > > >>> > > > > > > > > -- Dominik Psenner
Re: jenkins emergency restart notice - Tues 00:00 UTC
Apache Brooklyn builds seem to be ok now, thanks. On Wed, 19 Jul 2017, 16:09 Dominik Psenner, wrote: > I filed https://issues.apache.org/jira/browse/INFRA-14628 which is > probably > related to this since the exception message says "Too many open files". > > 2017-07-19 17:01 GMT+02:00 Mike Drob : > > > Have the changes resulted in any noticeable improvements? Is there a JIRA > > interested folks can follow along on? > > > > On Tue, Jul 18, 2017 at 6:40 PM, Chris Lambertus wrote: > > > > > We’re trying a suggested downgrade of the credentials binding plugin > > which > > > may be related to this. No guarantees, though. I will be restarting > again > > > tonight at 0100 UTC 19 July. > > > > > > -Chris > > > > > > > > > > > > > > > > On Jul 18, 2017, at 9:00 AM, Chris Lambertus wrote: > > > > > > > > This appears to be a bug in one of the new versions of either jenkins > > or > > > a plugin. We are still looking into the issue. The limit was raised > > > yesterday by a factor of 4. > > > > > > > > > > > > > > > > > > > >> On Jul 18, 2017, at 4:08 AM, Geoff Macartney < > > > geoff.macart...@cloudsoft.io> wrote: > > > >> > > > >> We're seeing them in Apache Brooklyn builds too, e.g. > > > >> https://builds.apache.org/job/brooklyn-server-master/664. > > > >> > > > >> regards > > > >> Geoff > > > >> > > > >> > > > >> > > > >> On Tue, 18 Jul 2017 at 12:03 Ismael Juma wrote: > > > >> > > > >>> Hi, > > > >>> > > > >>> Thanks for the heads up. Was this done? We are still seeing "too > many > > > open > > > >>> files" issues in Kafka builds. One example of many: > > > >>> > > > >>> https://builds.apache.org/job/kafka-trunk-jdk7/ > > > lastCompletedBuild/console > > > >>> > > > >>> Ismael > > > >>> > > > >>> On Mon, Jul 17, 2017 at 2:53 PM, Chris Lambertus > > > wrote: > > > >>> > > > > > > Jenkins will be restarted at midnight tonight UTC to address > > problems > > > related to ‘too many open files’ and to attempt to correct a > problem > > > with > > > some windows agents. I will be stopping new builds so that the > queue > > > can > > > empty over the next 2 hours. > > > > > > -Chris > > > > > > > > > >>> > > > > > > > > > > > > > > > > -- > Dominik Psenner >
Re: jenkins emergency restart notice - Tues 00:00 UTC
I just commented on that ticket. Looks like this was primarily due to the switch to systemd startup in ubuntu 16.04 and the need to update DefaultLimitNOFILE in /etc/systemd/system.conf (grr) to affect processes started via systemctl. Thanks Joan Touzet for pointing this out. -Chris > On Jul 19, 2017, at 8:47 AM, Geoff Macartney > wrote: > > Apache Brooklyn builds seem to be ok now, thanks. > > On Wed, 19 Jul 2017, 16:09 Dominik Psenner, wrote: > >> I filed https://issues.apache.org/jira/browse/INFRA-14628 which is >> probably >> related to this since the exception message says "Too many open files". >> >> 2017-07-19 17:01 GMT+02:00 Mike Drob : >> >>> Have the changes resulted in any noticeable improvements? Is there a JIRA >>> interested folks can follow along on? >>> >>> On Tue, Jul 18, 2017 at 6:40 PM, Chris Lambertus wrote: >>> We’re trying a suggested downgrade of the credentials binding plugin >>> which may be related to this. No guarantees, though. I will be restarting >> again tonight at 0100 UTC 19 July. -Chris > On Jul 18, 2017, at 9:00 AM, Chris Lambertus wrote: > > This appears to be a bug in one of the new versions of either jenkins >>> or a plugin. We are still looking into the issue. The limit was raised yesterday by a factor of 4. > > > > >> On Jul 18, 2017, at 4:08 AM, Geoff Macartney < geoff.macart...@cloudsoft.io> wrote: >> >> We're seeing them in Apache Brooklyn builds too, e.g. >> https://builds.apache.org/job/brooklyn-server-master/664. >> >> regards >> Geoff >> >> >> >> On Tue, 18 Jul 2017 at 12:03 Ismael Juma wrote: >> >>> Hi, >>> >>> Thanks for the heads up. Was this done? We are still seeing "too >> many open >>> files" issues in Kafka builds. One example of many: >>> >>> https://builds.apache.org/job/kafka-trunk-jdk7/ lastCompletedBuild/console >>> >>> Ismael >>> >>> On Mon, Jul 17, 2017 at 2:53 PM, Chris Lambertus wrote: >>> Jenkins will be restarted at midnight tonight UTC to address >>> problems related to ‘too many open files’ and to attempt to correct a >> problem with some windows agents. I will be stopping new builds so that the >> queue can empty over the next 2 hours. -Chris >>> > >>> >> >> >> >> -- >> Dominik Psenner >> signature.asc Description: Message signed with OpenPGP
