Re: The War on Orange Needs YOU

2013-03-07 Thread Ed Morley
Just to follow up on this - yesterday the metrics Elastic Search cluster experienced some data loss, with OrangeFactor being amongst the services affected. It's not yet clear if there is a backup available [1][2], so for now the stats shown on http://brasstacks.mozilla.com/orangefactor/ are mi

The War on Orange Needs YOU

2013-03-01 Thread Clint Talbert
The rate of intermittent failures (which is called the orange factor, and the effort to diminish it is called The War on Orange) in our automation has skyrocketed. Starting around February 17 on mozilla-inbound, we took off on an exponential curve that is making it extremely hard to sheriff the