Re: "Hypothetical" Datacenter Overheating

- Original Message - > From: "Tom Beecher" >> It's certainly one of many possible root causes which someone doing an >> AAR on an event like this should be thinking about, and looking for in >> their evaluation of the data they see. > > And I'm sure they are and will. > > By the time th

Re: "Hypothetical" Datacenter Overheating

disagree. On Sun, Jan 21, 2024 at 1:22 AM Jay R. Ashworth wrote: > - Original Message - > > From: "Tom Beecher" > > To: "Lamar Owen" > > Cc: nanog@nanog.org > > Sent: Wednesday, January 17, 2024 8:06:07 PM > > Subject: Re: "

Re: "Hypothetical" Datacenter Overheating

- Original Message - > From: "Tom Beecher" > To: "Lamar Owen" > Cc: nanog@nanog.org > Sent: Wednesday, January 17, 2024 8:06:07 PM > Subject: Re: "Hypothetical" Datacenter Overheating >> If these chillers are connected to BACnet

Re: "Hypothetical" Datacenter Overheating

I'm actually referring to something like, I've not yet had a system where they have made sense, I mostly deal with either places where I have no say in the hvac or very small server rooms, but I've thought these were an interesting concept since I first saw them years ago. https://www.chiltrix

Re: "Hypothetical" Datacenter Overheating

On 1/15/24 10:14, sro...@ronan-online.com wrote: I’m more interested in how you lose six chillers all at once. According to a post on a support forum for one of the clients in that space: "We understand the issue is due to snow on the roof affecting the cooling equipment." Never overlook the

Re: "Hypothetical" Datacenter Overheating

On 1/17/24 20:06, Tom Beecher wrote: If these chillers are connected to BACnet or similar network, then I wouldn't rule out the possibility of an attack. Don't insinuate something like this without evidence. Completely unreasonable and inappropriate. I wasn't meaning to insinuate an

Re: "Hypothetical" Datacenter Overheating

Hammett" < na...@ics-il.net > Cc: sro...@ronan-online.com , "NANOG" < nanog@nanog.org > Sent: Thursday, January 18, 2024 9:00:34 AM Subject: Re: "Hypothetical" Datacenter Overheating and none in the other two facilities you operate in that same building h

Re: "Hypothetical" Datacenter Overheating

tix> > The Brothers WISP <http://www.thebrotherswisp.com/> > <https://www.facebook.com/thebrotherswisp> > <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> > -- > *From: *"Tom Beecher" > *To: *"Mike Hammett"

Re: "Hypothetical" Datacenter Overheating

.com, "NANOG" Sent: Thursday, January 18, 2024 9:00:34 AM Subject: Re: "Hypothetical" Datacenter Overheating and none in the other two facilities you operate in that same building had any failures. Quoting directly from their outage ticket updates : CH2 does not have c

Re: "Hypothetical" Datacenter Overheating

brotherswisp.com/> > <https://www.facebook.com/thebrotherswisp> > <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> > ------ > *From: *sro...@ronan-online.com > *To: *"Mike Hammett" > *Cc: *"NANOG" > *Sent: *Monday, January 15, 2024

Re: "Hypothetical" Datacenter Overheating

> > If these chillers are connected to BACnet or similar network, then I > wouldn't rule out the possibility of an attack. > Don't insinuate something like this without evidence. Completely unreasonable and inappropriate. On Wed, Jan 17, 2024 at 8:31 AM Lamar Owen wrote: > >This sort of mass fa

Re: "Hypothetical" Datacenter Overheating

Great question. Stop powering off the non-essential equipment and get natural air going in a flow. And then start praying and watching the utility's GIS outage map. lol -- Later, Joe On Mon, Jan 15, 2024 at 2:26 PM Mike Hammett wrote: > Let's say that hypothetically, a datacenter you're in had

Re: "Hypothetical" Datacenter Overheating

On Wed, Jan 17, 2024 at 12:07:42AM -0500, Glenn McGurrin via NANOG wrote: > Free air cooling loops maybe? (Not direct free air cooling with air > exchange, the version with something much like an air handler outside with a > coil and an fan running cold outside air over the coil with the water/glyc

Re: "Hypothetical" Datacenter Overheating

>This sort of mass failure seems to point >towards either design issues (like equipment >selection/configuration vs >temperature range for the location), systemic maintenance issues, or >some sort of single failure point that could take all the chillers out, >none of which I'd be happy to see in

Re: "Hypothetical" Datacenter Overheating

On Wed, 17 Jan 2024 at 03:18, wrote: > Others have pointed to references, I found some others, it's all > pretty boring but perhaps one should embrace the general point that > some equipment may not like abrupt temperature changes. Can you share them? Only one I've found is: https://www.ashrae.o

Re: "Hypothetical" Datacenter Overheating

Free air cooling loops maybe? (Not direct free air cooling with air exchange, the version with something much like an air handler outside with a coil and an fan running cold outside air over the coil with the water/glycol that would normally be the loop off of the chiller) the primary use of th

Re: "Hypothetical" Datacenter Overheating

For completeness' sake at the first commercial ISP to sell individual dial-up to the public, The World, we had six of those typical desktop 2400bps modems (I forget the brand tho I still have them, a photo also I think) sitting on a file cabinet in an office space in Brookline, MA plugged into a

Re: "Hypothetical" Datacenter Overheating

Others have pointed to references, I found some others, it's all pretty boring but perhaps one should embrace the general point that some equipment may not like abrupt temperature changes. But phones (well, modern mobile phones) don't generally have moving parts. So the issue is more likely wit

Re: "Hypothetical" Datacenter Overheating

On Tue, 2024-01-16 at 10:44 -0800, Jay Hennigan wrote: > We made our own. And then we had to deal with all the wall warts. We > rigged up a power supply with a big snake of barrel jacks. Luxury. We had a hamster in a hamster wheel for each modem. Ah, the old days. Regards, K. -- ~

Re: "Hypothetical" Datacenter Overheating

350 Cermak Chicago is a "historic" building which means you can't change the visible outside. Someone had long discussions about the benefits of outside air economizers, but can't change the windows. Need to hide HVAC plant (as much as possible). I would design all colos to look like 375

Re: "Hypothetical" Datacenter Overheating

On 1/16/24 10:33, Shawn L via NANOG wrote: I remember those days --- I think we bought cages from someone and pulled the boards out of the modems to mount them. We made our own. And then we had to deal with all the wall warts. We rigged up a power supply with a big snake of barrel jacks. Liv

Re: "Hypothetical" Datacenter Overheating

On Tue, Jan 16, 2024 at 10:30 AM Chris Adams wrote: > The back-room ISP I started at was at least owned by a company with > their own small machine shop, so we had them make plates we could mount > two Sportsters (sans top) to and slide them into card cages. 20 modems > in a 5U cage! The ISP wher

Re: "Hypothetical" Datacenter Overheating

On Tue, Jan 16, 2024 at 08:37:09AM -0800, Warren Kumari wrote: > ISP/Colo provider The good ole days. When one stacked modems with two pencils in between them and box fans blew through the gaps. -- . ___ ___ . . ___ . \/ |\ |\ \ . _\_ /__ |-\ |-\ \__

RE: "Hypothetical" Datacenter Overheating

[linkedin] <https://www.linkedin.com/company/next-dimension-inc.> [instagram] <https://www.instagram.com/next.dimension.inc> -Original Message- From: NANOG On Behalf Of Saku Ytti Sent: January 16, 2024 2:09 AM To: b...@theworld.com Cc: NANOG Subject:

Re: "Hypothetical" Datacenter Overheating

On Mon, Jan 15, 2024 at 9:55 AM, William Herrin wrote: > On Mon, Jan 15, 2024 at 6:08 AM Mike Hammett wrote: > > Let's say that hypothetically, a datacenter you're in had a cooling > failure and escalated to an average of 120 degrees before mitigations > started having an effect. What should be

Re: "Hypothetical" Datacenter Overheating

On Mon, Jan 15, 2024 at 10:14:49AM -0500, sro...@ronan-online.com wrote: > I’m more interested in how you lose six chillers all at once. Because you're probably mistaking an air handling unit for a chiller. I usually point people at this to get us on the same page: https://www.youtube.com/watch?v

Re: "Hypothetical" Datacenter Overheating

On 16/01/2024 01:32, Mike Hammett wrote: Someone I talked to while on scene today said their area got to 130 and cooked two core routers. We've lost one low-end switch. I'm very glad it wasn't two core routers! We're still looking into what recourse we have against the datacenter operator.

Re: "Hypothetical" Datacenter Overheating

On Tue, 16 Jan 2024 at 12:22, Nathan Ward wrote: > Here’s some manufacturer specs: > https://www.dell.com/support/manuals/en-nz/poweredge-r6515/per6515_ts_pub/environmental-specifications?guid=guid-debd273c-0dc8-40d8-abbc-be059a0ce59c&lang=en-us > > 3rd section, “Maximum temperature gradient”. T

Re: "Hypothetical" Datacenter Overheating

On 16/01/2024 at 10:50:13 PM, Saku Ytti wrote: > On Tue, 16 Jan 2024 at 11:00, William Herrin wrote: > > You have a computer room humidified to 40% and you inject cold air > > below the dew point. The surfaces in the room will get wet. > > > I think humidity and condensation is well understood a

Re: "Hypothetical" Datacenter Overheating

On Tue, 16 Jan 2024 at 11:00, William Herrin wrote: > You have a computer room humidified to 40% and you inject cold air > below the dew point. The surfaces in the room will get wet. I think humidity and condensation is well understood and indeed documented but by NEBS and vendors as verboten.

Re: "Hypothetical" Datacenter Overheating

On 1/15/24 23:11, Martin Hannigan wrote: On Mon, Jan 15, 2024 at 4:10 PM Jay Hennigan > wrote: On 1/15/24 10:37, Pennington, Scott wrote: > yes but it has been -8 in Chicago plenty of times before this. >   Very interested in root cause... Absolu

Re: "Hypothetical" Datacenter Overheating

On Mon, Jan 15, 2024 at 11:08 PM Saku Ytti wrote: > On Tue, 16 Jan 2024 at 08:51, wrote: > > A rule of thumb is a few degrees per hour change but YMMV, depends on > > the equipment. Sometimes manufacturer's specs include this. > > Is this common sense, or do you have reference to this, like paper

Re: "Hypothetical" Datacenter Overheating

Good thing there are no windows at this “hypothetical” location :) > On Jan 16, 2024, at 1:51 AM, b...@theworld.com wrote: > >  > Something worth a thought is that as much as devices don't like being > too hot they also don't like to have their temperature change too > quickly. Parts can expand/

Re: "Hypothetical" Datacenter Overheating

On Tue, 16 Jan 2024 at 08:51, wrote: > A rule of thumb is a few degrees per hour change but YMMV, depends on > the equipment. Sometimes manufacturer's specs include this. Is this common sense, or do you have reference to this, like paper showing at what temperature change at what rate occurs wha

Re: "Hypothetical" Datacenter Overheating

Something worth a thought is that as much as devices don't like being too hot they also don't like to have their temperature change too quickly. Parts can expand/shrink variably depending on their composition. A rule of thumb is a few degrees per hour change but YMMV, depends on the equipment. S

Re: "Hypothetical" Datacenter Overheating

: Monday, January 15, 2024 8:08:25 AM Subject: "Hypothetical" Datacenter Overheating Let's say that hypothetically, a datacenter you're in had a cooling failure and escalated to an average of 120 degrees before mitigations started having an effect. What are normal QA procedu

Re: "Hypothetical" Datacenter Overheating

On Mon, 2024-01-15 at 08:08 -0600, Mike Hammett wrote: > Let's say that hypothetically, a datacenter you're in had a cooling > failure and escalated to an average of 120 degrees Major double-take there for this non-US reader, until I realised you just had to mean Fahrenheit. Regards, K. --

Re: "Hypothetical" Datacenter Overheating

On Mon, Jan 15, 2024 at 7:14 AM wrote: >> I’m more interested in how you lose six chillers all at once. >Extreme cold. If the transfer temperature is too low, they can reach a >state where the refrigerant liquifies too soon, damaging the compressor. >Regards, >Bill Herrin Our 70-ton Tranes here h

Re: "Hypothetical" Datacenter Overheating

On Mon, Jan 15, 2024 at 7:14 AM wrote: > I’m more interested in how you lose six chillers all at once. Extreme cold. If the transfer temperature is too low, they can reach a state where the refrigerant liquifies too soon, damaging the compressor. Regards, Bill Herrin -- William Herrin b...@he

Re: "Hypothetical" Datacenter Overheating

On Mon, Jan 15, 2024 at 4:10 PM Jay Hennigan wrote: > On 1/15/24 10:37, Pennington, Scott wrote: > > yes but it has been -8 in Chicago plenty of times before this. > > Very interested in root cause... > > Absolutely. My point was that claiming "Global warming" isn't going to > fly as an exc

Re: "Hypothetical" Datacenter Overheating

My sarcasm generator is clearly set incorrectly :) -mel > On Jan 15, 2024, at 10:33 AM, Jay Hennigan wrote: > > On 1/15/24 07:21, Mel Beckman wrote: >> Easy. Climate change. Lol! > > It was -8°F in Chicago yesterday. > On Jan 15, 2024, at 7:17 AM, sro...@ronan-online.com wrote: >>> >>

Re: "Hypothetical" Datacenter Overheating

ly as an excuse. *From:* NANOG on behalf of Jay Hennigan *Sent:* Monday, January 15, 2024 1:31 PM *To:* nanog@nanog.org *Subject:* Re: "Hypothetical" Datacenter Overheating On 1/15/24 07:21, Mel Beckman wrote: Easy. Climate change. Lol! It was -8°F in Chicago yes

Re: "Hypothetical" Datacenter Overheating

On 1/15/24 07:21, Mel Beckman wrote: Easy. Climate change. Lol! It was -8°F in Chicago yesterday. On Jan 15, 2024, at 7:17 AM, sro...@ronan-online.com wrote:  I’m more interested in how you lose six chillers all at once. -- Jay Hennigan - j...@west.net Network Engineering - CCIE #7880 50

Re: "Hypothetical" Datacenter Overheating

Exactly. Perhaps they weren’t all online to begin with…On Jan 15, 2024, at 10:18 AM, Mike Hammett wrote:and none in the other two facilities you operate in that same building had any failures.-Mike HammettIntelligent Computing SolutionsMidwest Internet ExchangeThe Brothers WISPFrom: sro...@ro

Re: "Hypothetical" Datacenter Overheating

Easy. Climate change. Lol! -mel On Jan 15, 2024, at 7:17 AM, sro...@ronan-online.com wrote:  I’m more interested in how you lose six chillers all at once. Shane On Jan 15, 2024, at 9:11 AM, Mike Hammett wrote:  Let's say that hypothetically, a datacenter you're in had a cooling failure an

Re: "Hypothetical" Datacenter Overheating

t;NANOG" Sent: Monday, January 15, 2024 9:14:49 AM Subject: Re: "Hypothetical" Datacenter Overheating I’m more interested in how you lose six chillers all at once. Shane On Jan 15, 2024, at 9:11 AM, Mike Hammett wrote: Let's say that hypothetically, a

Re: "Hypothetical" Datacenter Overheating

I’m more interested in how you lose six chillers all at once.ShaneOn Jan 15, 2024, at 9:11 AM, Mike Hammett wrote:Let's say that hypothetically, a datacenter you're in had a cooling failure and escalated to an average of 120 degrees before mitigations started having an effect. What are normal QA

Re: "Hypothetical" Datacenter Overheating

hers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> *From: *"Clayton Zekelman" *To: *"Mike Hammett" , "

Re: "Hypothetical" Datacenter Overheating

*From: *"Clayton Zekelman" *To: *"Mike Hammett" , "NANOG" *Sent: *Monday, January 15, 2024 8:23:37 AM *Subject: *Re: "Hypothetical" Datacenter Overheating At 09:08 AM 2024-01-15, Mike Hammett wrote: >Let's say that hypothetically, a datacenter you&

Re: "Hypothetical" Datacenter Overheating

On Mon, Jan 15, 2024 at 6:08 AM Mike Hammett wrote: > Let's say that hypothetically, a datacenter you're in had a cooling failure > and escalated to an average of 120 degrees before mitigations started > having an effect. What should be expected in the aftermath? Hi Mike, A decade or so ago I m

Re: "Hypothetical" Datacenter Overheating

AM Subject: Re: "Hypothetical" Datacenter Overheating At 09:08 AM 2024-01-15, Mike Hammett wrote: >Let's say that hypothetically, a datacenter you're in had a cooling >failure and escalated to an average of 120 degrees before >mitigations started having an effect.

Re: "Hypothetical" Datacenter Overheating

At 09:08 AM 2024-01-15, Mike Hammett wrote: Let's say that hypothetically, a datacenter you're in had a cooling failure and escalated to an average of 120 degrees before mitigations started having an effect. What are normal QA procedures on your behalf? What is the facility likely to be doi

"Hypothetical" Datacenter Overheating

Let's say that hypothetically, a datacenter you're in had a cooling failure and escalated to an average of 120 degrees before mitigations started having an effect. What are normal QA procedures on your behalf? What is the facility likely to be doing? What should be expected in the aftermath?