> On Aug 30, 2018, at 9:46 AM, Chris Lambertus <c...@apache.org> wrote:
> 
> 
> 
>> On Aug 29, 2018, at 2:35 PM, Chris Lambertus <c...@apache.org 
>> <mailto:c...@apache.org>> wrote:
>> 
>> 
>> 
>>> On Aug 29, 2018, at 7:36 AM, Chris Lambertus <c...@apache.org 
>>> <mailto:c...@apache.org>> wrote:
>>> 
>>> 
>>> 
>>>> On Aug 28, 2018, at 8:39 PM, Chris Lambertus <c...@apache.org 
>>>> <mailto:c...@apache.org>> wrote:
>>>> 
>>>> If you have not been following status.a.o or the infrabot twitter, our 
>>>> Jenkins master has suffered a critical disk controller failure. We are 
>>>> working with our datacenter colocation provider to identify and repair the 
>>>> problem, but we do not currently have an estimated time of repair. Please 
>>>> keep an eye on status.apache.org <http://status.apache.org/> for further 
>>>> updates.
>>> 
>>> 
>>> Jenkins remains down. Our hosting provider eventually identified and 
>>> replaced a failed backplane. Sadly, the nature of the backplane failure 
>>> corrupted both OS drives (which were mirrored.) Out of an abundance of 
>>> caution, we have decided to migrate the jenkins data to a different server, 
>>> one with a higher degree of fault tolerance and ability for us to monitor 
>>> the underlying hardware for predictive failure (as well as slightly higher 
>>> storage capacity.) Unfortunately, this operation will be time consuming, as 
>>> the jenkins data is over three terabytes. We will provide an additional 
>>> update in an hour or two as to the status of the copy and the updated ETR.
>>> 
>> 
>> Apologies for the delay in updating.
>> 
>> We are currently waiting for our colocation provider to provision the new 
>> server. We expect that to be done in the EU morning (approx 8-10 hours from 
>> now.) Once the new box is provisioned we will initiate the data copy to the 
>> new system. We will provide another update once the new box has been 
>> provisioned.
>> 
> 
> 
> The new box has been provisioned and the data synchronization is underway. 
> Our current estimate is 10-12 hours to complete the copy and restore service.
> 

Jenkins has been returned to service. Thanks for your patience in bearing with 
us through this outage.

-Chris


Attachment: signature.asc
Description: Message signed with OpenPGP

Reply via email to