Hi all - Finally! builds.apache.org is all the way back up and running. It's feeling at least a bit peppier than it did before, and startup time was significantly faster. Sorry again for the inconvenience of the extended downtime - getting off that problematic ZFS volume that made the backup so slow is exactly why we wanted to do this! I'd expect to see a little weirdness with most recent builds in some jobs, but nothing that shouldn't clear up after a new build runs. If you see anything more strange, please let me know and I'll investigate. Thanks again!
A. On Fri, Jul 3, 2015 at 6:29 PM, Andrew Bayer <andrew.ba...@gmail.com> wrote: > Hi all - > > Regrettably, the restore from backup is taking dramatically longer than > expected - we're now not even halfway through, so I don't expect to be back > online until Saturday morning UTC. > > A. > > On Fri, Jul 3, 2015 at 9:35 AM, Andrew Bayer <andrew.ba...@gmail.com> > wrote: > >> Hi all - >> >> Sorry for the delay in updating - we were finally able to get the OS >> installed and booting properly, and are now in the process of restoring >> Jenkins from backup. This'll take a few hours at least - sorry for the >> inconvenience, but we should be back up this evening Europe time at the >> outside. >> >> A. >> >> On Thu, Jul 2, 2015 at 5:17 PM, Andrew Bayer <andrew.ba...@gmail.com> >> wrote: >> >>> Hi all - >>> >>> builds.apache.org is still down - we're having problems rebuilding it. >>> We don't yet have a timeframe for when it'll be back up, but once it's been >>> reinstalled, I'll send another update. Thanks. >>> >>> A. >>> >>> >>> On Thu, Jul 2, 2015 at 1:28 PM, Andrew Bayer <andrew.ba...@gmail.com> >>> wrote: >>> >>>> Hooray - a correction. =) I've been able to get all the projects to >>>> backup in a reasonable timeframe, so we should hopefully not be missing >>>> builds. Sorry for the false alarm! >>>> >>>> A. >>>> >>>> On Thu, Jul 2, 2015 at 12:37 PM, Andrew Bayer <andrew.ba...@gmail.com> >>>> wrote: >>>> >>>>> builds.apache.org has been down for the last two hours as we work on >>>>> finalizing the backup - due to the absolutely atrocious performance we've >>>>> seen on this box with ZFS on Linux and directory traversal, we may not be >>>>> able to back up Maven project type builds from around 5pm UTC yesterday >>>>> until Jenkins was brought offline at around 8:30am UTC today. To get all >>>>> of >>>>> those backed up, it might take another 4 hours of rsyncing. The same build >>>>> numbers won't be repeated - just that any builds run in that time may not >>>>> be preserved. This will only be the case for Maven project type builds, >>>>> not >>>>> freestyle jobs, and all previous builds and job configuration etc will >>>>> still be there regardless. More updates when we're further along. >>>>> >>>>> A. >>>>> >>>>> On Tue, Jun 30, 2015 at 11:52 AM, Andrew Bayer <aba...@apache.org> >>>>> wrote: >>>>> >>>>>> A reminder - builds.apache.org will be down on July 2nd, starting >>>>>> around 9am UTC/11am CEST/10am BST/5am EDT/2am PDT. We'll be taking down >>>>>> Jenkins, backing it up, rebuilding the machine, and restoring Jenkins. >>>>>> Hopefully this will be done within 5-6 hours, but could take longer. >>>>>> We'll >>>>>> send updates on the status as we go. Thanks. >>>>>> >>>>>> If you have any questions please contact infrastruct...@apache.org. >>>>>> >>>>>> A. >>>>>> >>>>> >>>>> >>>> >>> >> >
