Scheduled : [TCW] Scheduled Tree Closing Maintenance Window

tl;dr - HARD tree closure from 0800PT - 1230PT Builds in progress may burn - it's up to developers to re-trigger anything they need. -- Forwarded message -- From: Date: Tue, Feb 7, 2017 at 11:18 AM Subject: [Mozilla status] Scheduled : [TCW] Scheduled Tree Closing Maintenance W

FYI: Tree Closing Window (TCW) this Saturday, March 25 0500-0930 PT

This will be a SOFT Tree Closure. As a reminder, that means: - jobs may burn - You are responsible for sheriffing your own jobs [TCW] Tree Closing Maintenance Window Mar 25, 05:00-09:30 PDT Trees will be closed for the duration of this ma

Fwd: [Planned] Scheduled Tree Closing and IT Maintenance Window, Sat August 29 2015, 0500-1700 PT

FYI - all trees closed, and hg.mozilla.org will be unavailable for commits as well. Tracking bug is https://bugzil.la/1197981 if you want to follow along -- Forwarded message -- From: Date: Tue, Aug 25, 2015 at 8:46 AM Subject: [Planned] Scheduled Tree Closing and IT Maintenance W

Fwd: [Planned] TCW 10-10-2015 6a - 2p PDT

-- Forwarded message -- From: Date: Mon, Oct 5, 2015 at 12:55 PM Subject: [Planned] TCW 10-10-2015 6a - 2p PDT To: all-moco-m...@mozilla.com Issue Status: Upcoming Short Summary: We will have our next tree closing window on 10/10/2015 starting at 6a to perform the following work

Update: CANCELLED: TCW 10-10-2015 6a - 2p PDT

Happy uninterrupted coding, everyone! -- Forwarded message -- From: m...@mozilla.com Date: Fri, Oct 9, 2015 at 11:15 AM Subject: Fwd: [Planned] TCW 10-10-2015 6a - 2p PDT To: all-moco-m...@mozilla.com The TCW scheduled for tomorrow has been canceled - please have a great weeken

Fwd: [Planned] Bugzilla Failover Test 2015-11-14 09:00 PST 120 mins

FYI: As has been announced for this week via a banner on bmo. -- Forwarded message -- From: Date: Wed, Nov 11, 2015 at 9:49 AM Subject: [Planned] Bugzilla Failover Test 2015-11-14 09:00 PST 120 mins To: all-moco-m...@mozilla.com Issue Status: Upcoming Short Summary: On Saturday

Fwd: [Planned] Tree Closing Window 2015-11-21 0600-1630 PST 2015-11-21 06:00 PST 330 mins

FYI: Trees will be closed during our normal Tree Closing Window. However, some of the work mentioned below has been deferred, so trees should be reopened by 11:30 PT. -- Forwarded message -- From: Date: Mon, Nov 16, 2015 at 8:54 AM Subject: [Planned] TCW 2015-11-21 0600-1630 PST 2

FYI: Tree Closing Maintenance Window, Sat January 9 2016, 0600-1330 PST 2016-01-09 06:00 PST 450 mins

Tracker bug: https://bugzil.la/1230205 -- Forwarded message -- From: Date: Mon, Jan 4, 2016 at 3:03 PM Subject: [Planned] Scheduled Tree Closing Maintenance Window, Sat January 9 2016, 0600-1330 PST 2016-01-09 06:00 PST 450 mins Issue Status: Upcoming Short Summary: IT will be

TCW Soft Close: Tree Closing Maintenance Window, Sat February 13 2016, 06:00-10:00a PST

FYI. We do not expect any significant impact to platform operations. "Soft Close" means we'll leave the trees open, but devs who push: - can expect issues - are personally responsible for managing their job (retries, etc.) -- Forwarded message -- From: Date: Tue, Feb 9, 2016 at

FYI: Emergency TCW this Saturday, February 20, 0600-1000 PT

tl;dr: this will be a "Soft Close" to deal with glibc updates As most of you know, upgrading glibc requires a reboot of each Linux host. With so many hosts needing to be rebooted, there will be impacts on the build and test machinery. We fully expect the system to self-recover, but jobs in progre

Fwd: [TCW] Scheduled Tree Closing Maintenance Window 2016-03-26 06:00a PDT 10 hours

As usual, we have a Tree Closing Window (TCW) this Saturday from 1300 UTC to 2300 UTC. This will be a "soft close" window: we expect some minor delays and recoverable outages. We recommend enjoying the first full weekend of spring/fall. If you do start jobs, you will need to monitor them yourself,

Fwd: [TCW] Scheduled Tree Closing Maintenance Window 2016-06-25 06:00hrs PDT (1.5 Hours)

NOTE: we do not expect any significant disruption from this activity, but a few jobs may burn. Developers are responsible for sheriffing their own jobs, and re-triggering as needed. Thanks! -- Forwarded message -- From: Date: Thu, Jun 23, 2016 at 11:09 AM Subject: [TCW] Scheduled

Fwd: PLANNED week ending 2016-08-21 people.m.o (Tue) and Tree Closing Window (Sat)

FYI, TCW Next Saturday, Aug 20, 1300 UTC -> 2000 UTC Correct tracking bug number is https://bugzilla.mozilla.org/show_bug.cgi?id=1289548 -- Forwarded message -- From: m...@mozilla.com Date: Mon, Aug 15, 2016 at 8:53 AM Subject: PLANNED week ending 2016-08-21 people.m.o (Tue) and

Fwd: URGENT PLANNED Tree Closing Window for firewall maintenance 2016-08-27 06:00a PDT 2 hours

-- Forwarded message -- From: m...@mozilla.com Date: Fri, Aug 26, 2016 at 11:59 AM Subject: URGENT PLANNED Tree Closing Window for firewall maintenance 2016-08-27 06:00a PDT 2 hours To: Cc: moc bcc: all@ *Service and Impact:* A bug has been discovered in the code running the Co

Fwd: [TCW] Scheduled Tree Closing Maintenance Window 2016-10-08 06:00a PDT 9 hours

Our usual Tree Closing Window is Saturday, Oct 8. This time, there is some major work, so trees will be closed hard at the start at 0600 PT. We hope to be able to switch to "soft close" around 1100 PT. "Soft Close" means the trees are open, but there may be possible hickups, and it is up to devs t

FYI: SOFT tree close for TCW Scheduled Tree Closing Maintenance Window 2016-11-19 07:00a PST 5 hours

The tree closing this weekend will be a SOFT CLOSE -- there will be period of time where new commits to hg.mozilla.org will be disabled, but the rest of the automation infrastructure will be operational. Some minor work might cause an unlucky job to burn - devs will need to retrigger those jobs. T

Completed: Tree closure for infrastructure work, Sunday 21April2013 10:00-14:00PT

This work has been completed, and trees reopened as of 12:06 PT (19:06 UTC). Thanks for everyone's help & cooperation. --Hal On 2013-04-18 17:22 , John O'Duinn wrote: > hi; > > We will have a treeclosing maintenance window Sunday, April 21 from > 10:00-14:00PT (17:00-21:00UTC). This is for chang

[Planned Maintenance Notification] - Tree Closure, 2013-10-12 @ 1200 PDT

All trees closed Saturday, Oct 12, from 12:00-16:30 PDT Original Message Subject:[Planned Maintenance Notification] - Tree Closure, 2013-10-12 @ 1200 PDT Date: Thu, 10 Oct 2013 13:51:48 -0700 From: Mozilla IT Operations To: every...@mozilla.org Short Summary

Fwd: [Planned Maintenance Notification] Tree Closure and MTV1 Office Downtime Nov 22 1815 - Nov 23 0100 PT

Original Message Subject:[Planned Maintenance Notification] Tree Closure and MTV1 Office Downtime Date: Fri, 22 Nov 2013 19:34:55 - From: notificati...@mozilla.com Short Summary : Due to a planned power outage in the building that hosts the Mountai

[Planned Maintenance Notification] Tree Closure, 2014-01-11

Short Summary : This is our once in every six weeks planned maintenance window, where IT and Release Engineering perform maintenance work on critical parts of Mozilla Infrastructure. As part of this window, we will begin the move of the tegra infrastructure from Mountain View to the Santa Clar

FYI: Try repo will be reset Thur, Jan 22, 0500 PT

As usual, previous try pushes will be discarded. After the reset, try push times should be improved. Two improvements since last try reset: - tbpl has been enhanced to maintain the links to your recent try pushes that will be lost - going forward, this will be a standard part of the once per cyc

Fwd: [Planned Maintenance Notification] fw1.releng.scl3 hardware/software upgrade Mon Feb 3, 9am-10am PT

This work is to further protect against a major cause of tree closures over the last several weeks. Original Message Subject:[Planned Maintenance Notification] fw1.releng.scl3 hardware/software upgrade Date: Fri, 31 Jan 2014 22:50:38 - From: net...@mozilla.com To

Fwd: [Planned Maintenance Notification] Tree Closure & Try Reset

Please see below for tree closing activities on Sat, Feb 22 -- including a reset of the try repo. Original Message Subject:[Planned Maintenance Notification] Tree Closure Date: Wed, 19 Feb 2014 23:20:05 - From: notificati...@mozilla.com To: infra-...@mozilla.

Consensus sought - when to reset try repository?

tl;dr: what is the balance point between pushes to try taking too long and loosing repository history of recent try pushes? Summary: As most developers have experienced, pushing to try can sometimes take a long time. Once it takes "too long" (as measured by screams of pain in #releng)

Re: Consensus sought - when to reset try repository?

On 2014-02-28 17:32 , L. David Baron wrote: > On Friday 2014-02-28 17:24 -0800, Hal Wine wrote: >> So, the question is - what say developers -- what's the balance point >> between: >> - too often, making collaborating on try pushes hard >> - too infrequent, i

Decision reached on: Consensus sought - when to reset try repository?

On 2014-02-28 17:24 , Hal Wine wrote: > tl;dr: what is the balance point between pushes to try taking too long > and loosing repository history of recent try pushes? Based on the responses to this specific question, we'll go back to waiting for developers to notify IT when ther

FYI I you have a slave currently borrowed from Releng (Maint 3/28 2200 PT)

NOTE: if RelEng has loaned you a slave, you may be impacted by the VPN work below happening this Friday night Original Message Subject:[Planned Maintenance Notification] Mozilla VPN maintenance Date: Wed, 26 Mar 2014 18:44:23 - From: notificati...@mozilla.com To:

Fwd: [Planned Maintenance Notification] 4/5 Tree Closing Maintenance Window

FYI: (bcc: release@, sheriffs@) Original Message Subject:[Planned Maintenance Notification] 4/5 Tree Closing Maintenance Window Date: Tue, 01 Apr 2014 20:32:56 - Short Summary: Mozilla IT will have a tree closing maintenance window on Saturday, 4/5, from 0700

URGENT FYI: git.mozilla.org will be offline at 1000 PT today for approx 1 hr - Tree Closure

As some of you are aware, we've been having some capacity issues with git.mozilla.org of late, which has impacted build times and caused tree closures. While we work on longer term fixes, we're going to take the server down to add addition RAM at 1000PT today. This will require closing all trees

Successful: git.mozilla.org will be offline at 1000 PT today for approx 1 hr - Tree Closure

This work has been completed, and we're back on line with maxed out RAM for the box! Special thanks to fubar in dev services and van in dcops for making this happen so quickly! Go TEAM! --Hal On 2014-04-23 08:46 , Hal Wine wrote: > As some of you are aware, we've been having some ca

New consensus sought - when to reset try repository?

On 2014-04-30 16:52 , Daniel Holbert wrote: > On 03/07/2014 02:41 PM, Hal Wine wrote: >> On 2014-02-28 17:24 , Hal Wine wrote: >>> tl;dr: what is the balance point between pushes to try taking too long >>> and loosing repository history of recent try pushes? >&g

Current action plan for try server issues

[including dev.platform this time as originally intended] tl;dr: there was a 4h "outage" on Wed, here's our plan if it happens again. Active bug for next reset: bug 1001735 (https://bugzil.la/1001735) Action Summary: - if we get another multi-hour (>2) "outage", we will immediately do a hard

FYI: intermittent build farm capacity reductions during data center move starts May 19

As you heard in today's project meeting, IT's consolidation of data centers will save us $900K/yr!! Yay! There will be some impact to the build farm as the move actually happens, and this posting outlines them. The impact to engineering will be a slight capacity reduction in the build far for the

FYI: intermittent build farm capacity reductions during data center move starts May 19

As you heard in today's project meeting, IT's consolidation of data centers will save us $900K/yr!! Yay! There will be some impact to the build farm as the move actually happens, and this posting outlines them. The impact to engineering will be a slight capacity reduction in the build far for the

UPDATE: Current action plan for try server issues

es history We have some confidence that (b) will be successful, and could be automated with minimal try closure time. I'll get a new bug open on that within a few days. --Hal On 2014-05-01, 11:26 , Hal Wine wrote: > [including dev.platform this time as originally intended] > &g

FYI: [Planned Maintenance Notification] Tree closing maintenance window, Sat May 17, 0900-1600 PT

Original Message Subject:[Planned Maintenance Notification] Tree closing maintenance window Date: Thu, 15 May 2014 21:28:41 - From: notificati...@mozilla.com To: every...@mozilla.org Short Summary: Mozilla IT will have a tree closing maintenance window o

FYI: [Planned Maintenance Notification] August 9th - Tree Closing Maintenance Window

FYI - some significant network work will be performed during our regular "Tree Closing Window" (TCW) Saturday, Aug 9, 0900-1200 PT (details below) We are going to not formally close the trees, but there will likely be delays (from auto retry) and some jobs might need to be manually retriggered. P

FYI: Emergency try reset in process - https://bugzil.la/1053558

tl;dr: try repo is being reset - read on for ways to access old push data Today we experienced another hg.m.o event that is not recovering similar to past events. After 2-1/2 hours with no signs of recovery, and with the recommendation of the sheriffs, we are resetting the try repo now. While you

Reminder: Tree Closing Window, Sat Sep 20, 0900-1300 PT

See below for details - to check status, please visit #moc in IRC or https://status.mozilla.org/en-US/calendar/ Main bug for whistlepig: 1065514 Short Summary: Mozilla IT will have a infrastructure maintenance window on Saturday, September 20th, from 0900-1300 PDT. During this window we will

FYI: developer services components moving in Bugzilla next week.

tl;dr: Where you file bugs for version control systems, etc. will be changing. Developer Services (https://wiki.mozilla.org/DeveloperServices) are the folks who support the version control systems, source code indexing, etc. These systems have been handled by various groups in the past, and the bu

FYI: developer services components moving in Bugzilla Oct 1

These changes (below) will be made on Oct 1 around 2100 PT The complete set of changes are listed at: https://bugzilla.mozilla.org/show_bug.cgi?id=1071332#user_story_header On 2014-09-26 16:35 , Hal Wine wrote: > tl;dr: Where you file bugs for version control systems, etc. will be > ch

FYI:

tl;dr: trees and buildfarm will be bumpy this Saturday, November 15, from 0300-1400 PT (1100-2200 UTC) devs will need to monitor their builds, and trigger rebuilds via self serve as needed. There may be several short time frames when

Reminder: Tree Closing Window, Sat Dec 20, 0900-1730 PT

tracker bug with details: https://bugzil.la/702 status at: https://treestatus.mozilla.org/ NOTE: the trees will be left open; devs must monitor their own builds, and retrigger as needed. IT will be performing maintenance work this Saturday, primarily in SCL3. That will cause intermittent

Reminder: Tree Closing Window, Sat Jan 24, 0600-1400 PT

tracker bug with details: https://bugzil.la/1123733 status at: https://treestatus.mozilla.org/ NOTE: the trees will be left open; devs must monitor their own builds, and retrigger as needed. IT will be performing maintenance work this Saturday, primarily in SCL3. That will cause intermittent

Partial restoration: gecko-dev and Git replication will be broken for a little while

Date: Fri, 30 Jan 2015 22:30:18 -0800 From: Hal Wine To: dev-g...@lists.mozilla.org, dev-...@lists.mozilla.org, VCS devs , dev-planf...@lists.mozilla.org, Bhavana

Completed & FYI: Re: Current recovery plan for gecko-dev and git (Was: Re: gecko-dev and Git replication will be broken for a little while)

All work was completed on Monday and services fully restored. If you are experiencing any issues, please contact #vcs. NOTE: if you use the remote git.mozilla.org/integration/gecko-dev.git please keep reading. If you use this particular remote, and pulled during the problem window (roughly J

FYI - gitmirror.m.o duplicate repositories will be removed Feb 16

tl;dr: if you reference git://gitmirror.mozilla.org/mozilla-b2g/gaia, please change to git://git.mozilla.org/releases/gaia.git gitmirror.mozilla.org gets overloaded from time to time. To reduce load, we will be removing dead repositories and repositories already mirrored to git.mozilla.org. (s

fyi, temporary degraded performance in syncing to gecko-dev.git

Status: gecko-dev.git updating normally at present. tl;dr: there were some significant conversion delays Friday morning PT (approx 0500-1040PT). Service has been restored. One additional (much shorter) delay will be needed for final fix,

Reminder: Tree Closing Window, Sat March 14, 0600-1400 PT

tracker bug with details: https://bugzil.la/1141402 status at: https://treestatus.mozilla.org/ IT and Release Operations will be performing maintenance work this Saturday. The work is expected to be invasive, and we will close the trees for the duration of the window, 8 hours from 0600 PDT to 1

Reminder: Tree Closing Window, Sat April 18, 0600-1800 PT

tracker bug with details: https://bugzil.la/1151645 status at: https://treestatus.mozilla.org/ for trees status at: https://whistlepig.mozilla.org/en-US/detail/302/ for event IT will be performing maintenance work this Saturday. The work is expected to be invasive, and we will close the trees fo

[Planned] Tree Closing Maintenance Window, Sat May 23 2015, 0800-1800 PT

Trees will be closed, and other services (such as bugzilla) unavailable for periods of time. Forwarded Message Subject: [Planned] Tree Closing Maintenance Window, Sat May 23 2015, 0800-1800 PT Date: Wed, 20 May 2015 19:19:11 - From: m...@mozilla.com To: Issu

Reminder: Tree Closing Window (TCW), Sat June 20, 0700-0900 PDT

tracker bug with details: https://bugzil.la/1175630 status at: https://whistlepig.mozilla.org/en-US/detail/368/ NOTE: the trees will be left open; devs must monitor their own builds, and retrigger as needed. IT will be performing maintenance work this Saturday, primarily in SCL3. That will ca

REMINDER: [Planned ] Scheduled Tree Closing and disruptive IT Maintenance Window, Sat July 18 2015, 0700-1400 PT

If you want to follow along, the tracking bug is https://bugzil.la/1182561 -- Forwarded message -- From: Date: Mon, Jul 13, 2015 at 10:32 AM Subject: [Planned ] Scheduled Tree Closing and disruptive IT Maintenance Window, Sat July 18 2015, 0700-1400 PT To: all-moco-m...@mozilla.co