Re: facebook outage
> On Oct 4, 2021, at 4:30 PM, Bill Woodcock wrote: > > > >> On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: >> >> >> >>> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: >>> >>> They’re starting to pick themselves back up off the floor in the last two >>> or three minutes. A few answers getting out. I imagine it’ll take a while >>> before things stabilize, though. >> >> nd we’re back: >> >> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9 > > So that was, what… 15:50 UTC to 21:05 UTC, more or less… five hours and > fifteen minutes. > > That’s a lot of hair burnt all the way to the scalp, and some third-degree > burns beyond that. > > Maybe they’ll get one or two independent secondary authoritatives, so this > doesn’t happen again. :-) > DNS was a victim in this outage, not the cause. >-Bill
Re: Facebook post-mortems...
> On Oct 5, 2021, at 10:32 AM, Jean St-Laurent via NANOG > wrote: > > If you have some DNS working, you can point it at a static “we are down and > we know it” page much sooner, At the scale of facebook that seems extremely difficult to pull off w/o most of their architecture online. Imagine trying to terminate >billion sessions. When they started to come back up and had their "We're sorry" page up- even their static png couldn't make it onto the wire.
Re: Observations of an Internet Middleman (Level3)
On 5/15/14, 11:58 AM, Joe Greco wrote: 2) Netflix purchases 5Mbps "fast lane" I appreciate Joe's use of quotation marks here.A lot of the dialog has included this 'fast lane' terminology, yet all of us know there's no 'fast lane' being constructed, rather just varying degrees of _slow_ applied to existing traffic. It's a shame the use of 'fast lane' is ubiquitous in this argument. If the local distribution networks would like to actually build something fast, then this would be a different story. -Ryan Brooks
Re: Got a call at 4am - RAID Gurus Please Read
Zfs on BSD or a Solaris like OS > On Dec 11, 2014, at 10:06 AM, Bacon Zombie wrote: > > Are you running ZFS and RAIDZ on Linux or BSD? >> On 10 Dec 2014 23:21, "Javier J" wrote: >> >> I'm just going to chime in here since I recently had to deal with bit-rot >> affecting a 6TB linux raid5 setup using mdadm (6x 1TB disks) >> >> We couldn't rebuild because of 5 URE sectors on one of the other disks in >> the array after a power / ups issue rebooted our storage box. >> >> We are now using ZFS RAIDZ and the question I ask myself is, why wasn't I >> using ZFS years ago? >> >> +1 for ZFS and RAIDZ >> >> >> >>> On Wed, Dec 10, 2014 at 8:40 AM, Rob Seastrom wrote: >>> >>> >>> The subject is drifting a bit but I'm going with the flow here: >>> >>> Seth Mos writes: >>> Raid10 is the only valid raid format these days. With the disks as big as they get these days it's possible for silent corruption. >>> >>> How do you detect it? A man with two watches is never sure what time it >>> is. >>> >>> Unless you have a filesystem that detects and corrects silent >>> corruption, you're still hosed, you just don't know it yet. RAID10 >>> between the disks in and of itself doesn't help. >>> And with 4TB+ disks that is a real thing. Raid 6 is ok, if you accept rebuilds that take a week, literally. Although the rebuild rate on our 11 disk raid 6 SSD array (2TB) is less then a day. >>> >>> I did a rebuild on a RAIDZ2 vdev recently (made out of 4tb WD reds). >>> It took nowhere near a day let alone a week. Theoretically takes 8-11 >>> hours if the vdev is completely full, proportionately less if it's >>> not, and I was at about 2/3 in use. >>> >>> -r >>
Re: iOS 7 update traffic
Sounds like a great plan. You could do it for Netflix, Hulu, amazon, Walmart, etc. Get a piece of the action.Am I talking to Verizon? On 9/19/13 1:46 PM, Warren Bailey wrote: A line, is a line, is a line, is a line. There's no difference. Updates are available to all devices on a "download day", and providers networks are drastically reduced in capacity as a result. Apple does not cut them checks to serve it up, why should that traffic be more important than anything else? I'd DSCP updates to best effort hell and tell Apple I'd like a small share of the revenue they've gained from all the devices *I* am responsible for updating. They're not getting these updates OTA often, they actually advocate (shocking, AT&T wanting to save bandwidth) using your home Wi-Fi to download it. Providers can handle peaks, but SURGES begin to cause problems quickly. On narrowband pipes, we actually KILL updates.. They screw us that hard.
Re: Redundant Data Center Architectures
Roland Dobbins wrote: On Oct 28, 2009, at 8:26 PM, Stefan Fouant wrote: I'm wondering what are the growing trends in connecting Data Centers for redundancy in DR/COOP environments. 'DR' is an obsolete 40-year-old mainframe concept; it never works, as funding/testing/scaling of the 'backup' systems is never adequate and/or allowed. Layer-2 between sites is evil, as well. Layer-3-independence and active/active/etc. is where it's at in terms of high availability in the 21st Century. GSLB, et. al. And that's about all you need to know. Never heard it put so succinctly - thanks, -Ryan Brooks
Re: I don't need no stinking firewall!
On 1/5/10 3:24 PM, Robert Brockway wrote: On Tue, 5 Jan 2010, Dobbins, Roland wrote: The problem is that your premise is wrong. Stateful firewalls (hereafter just called firewalls) offer several advantages. This list is not necessarily exhaustive. Great advantages list, but where's the disadvantages list? Here's mine: 1..n) Stateful firewalls go down. It's the very nature of what they do. If you haven't had this problem, then your application is small. Everyone needs to listen to Roland's mantra: "stateless ACLs in hardware than can handle Mpps". It's more than just a hint.
Re: DDoS mitigation recommendations
On 1/26/10 11:56 AM, Gerald Wluka wrote: I am new to this mailing list We can tell. - this should be a response to an already started thread that I cannot see:
