On Wednesday 18 Feb approximately 1800 hours UTC, Infra was notified that the 
ci-builds Jenkins service was offline. We contacted our service provider, and 
they restarted the system. After this restart, Infra started receiving 
notifications of missing job configurations.

Upon beginning the investigation, the system again went offline, and required 
another forced restart.

Infra's initial investigation shows a number of missing config.xml files which 
we are in the process of restoring from backup.

Additionally, Infra is evaluating the existing hardware for issues, and 
beginning a migration process to new hardware.

Please bear with us as we restore the config.xml files, and investigate the 
root cause of their disappearance. As part of this recovery, we are exploring 
additional avenues to improve the backup methodologies for job configurations.

We will provide another update once we are further along in the restoration 
process, likely within 24 hours.

-Chris
ASF Infra



> On Feb 18, 2021, at 1:20 AM, Gavin McDonald <gmcdon...@apache.org> wrote:
> 
> Hi All,
> 
> No need for any more messages, all of it is known.
> 
> On Thu, Feb 18, 2021 at 10:15 AM P. Ottlinger <pottlin...@apache.org> wrote:
> 
>> Hi,
>> 
>> Am 18.02.21 um 06:05 schrieb Gavin McDonald:
>>> Whilst the service is back, there are quite a few jobs 'missing' since
>> the
>>> machine went offline.
>> 
>> Creadur is missing completely:
>> https://ci-builds.apache.org/job/Creadur/
>> 
>> :(
>> 
>> Thanks,
>> Phil
>> 
> 
> 
> -- 
> 
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team

Reply via email to