On Wednesday 18 Feb approximately 1800 hours UTC, Infra was notified that the ci-builds Jenkins service was offline. We contacted our service provider, and they restarted the system. After this restart, Infra started receiving notifications of missing job configurations.
Upon beginning the investigation, the system again went offline, and required another forced restart. Infra's initial investigation shows a number of missing config.xml files which we are in the process of restoring from backup. Additionally, Infra is evaluating the existing hardware for issues, and beginning a migration process to new hardware. Please bear with us as we restore the config.xml files, and investigate the root cause of their disappearance. As part of this recovery, we are exploring additional avenues to improve the backup methodologies for job configurations. We will provide another update once we are further along in the restoration process, likely within 24 hours. -Chris ASF Infra > On Feb 18, 2021, at 1:20 AM, Gavin McDonald <gmcdon...@apache.org> wrote: > > Hi All, > > No need for any more messages, all of it is known. > > On Thu, Feb 18, 2021 at 10:15 AM P. Ottlinger <pottlin...@apache.org> wrote: > >> Hi, >> >> Am 18.02.21 um 06:05 schrieb Gavin McDonald: >>> Whilst the service is back, there are quite a few jobs 'missing' since >> the >>> machine went offline. >> >> Creadur is missing completely: >> https://ci-builds.apache.org/job/Creadur/ >> >> :( >> >> Thanks, >> Phil >> > > > -- > > *Gavin McDonald* > Systems Administrator > ASF Infrastructure Team
