[lopsa-tech] Anybody else seeing this? (Amazon AWS problem)

2015-07-02 Thread Edward Ned Harvey (lopser)
About an hour ago, our alert systems started spamming us. It seems like a problem with Amazon AWS, US East. I'm able to access at least two of the systems in US East via https - but one of them is not responding to ssh - So I figured I would reboot it via AWS control panel - And when I login to

Re: [lopsa-tech] Anybody else seeing this? (Amazon AWS problem)

2015-07-02 Thread Edward Ned Harvey (lopser)
Yup, something weird on their firewalls. Right now, I have two machines in our colo, with different externally facing IP's that are both in the same network segment, both continuously pinging a machine in Amazon. As I sit here, intermittently for no apparent reason, the amazon machine stops repl

Re: [lopsa-tech] Anybody else seeing this? (Amazon AWS problem)

2015-07-02 Thread Mike Robinson
Saw mention of leap second (yesterday midnight UTC) and of a bgp route leak causing AWS troubles on the NANOG list. Likely one or both are related to your issue, I'd already deleted the NANOG digest, not sure if there was much more detail. m. Sent from my iPhone > On Jul 2, 2015, at 8:43 AM,

Re: [lopsa-tech] Anybody else seeing this? (Amazon AWS problem)

2015-07-02 Thread Edward Ned Harvey (lopser)
> From: Mike Robinson [mailto:mrobin...@nortoncottage.com] > > Saw mention of leap second (yesterday midnight UTC) and of a bgp route > leak causing AWS troubles on the NANOG list. > > Likely one or both are related to your issue, I'd already deleted the NANOG > digest, not sure if there was much

Re: [lopsa-tech] Anybody else seeing this? (Amazon AWS problem)

2015-07-02 Thread john boris
It must be a bad day for the East Coast. While in Physical Therapy (Haddonfield nj) lights and AC blinked. On the way home Dunking Donuts (next town over) lost their computers as the power blinked. (Lucky I had cash). I get home and the wife is bitching that the power flickered and she can't get an

Re: [lopsa-tech] Anybody else seeing this? (Amazon AWS problem)

2015-07-02 Thread Paul Graydon
With the amount of redundancy in the power systems in a typical DC, power drops would be extremely unlikely to cause any impact. On 07/02/15 08:52, john boris wrote: It must be a bad day for the East Coast. While in Physical Therapy (Haddonfield nj) lights and AC blinked. On the way home Dunkin

Re: [lopsa-tech] Anybody else seeing this? (Amazon AWS problem)

2015-07-02 Thread Paul Graydon
That was only causing problems for a few minutes on Tuesday evening, it certainly wouldn't be causing an impact now. On 07/02/15 06:58, Mike Robinson wrote: Saw mention of leap second (yesterday midnight UTC) and of a bgp route leak causing AWS troubles on the NANOG list. Likely one or both a

Re: [lopsa-tech] Anybody else seeing this? (Amazon AWS problem)

2015-07-02 Thread Edward Ned Harvey (lopser)
> From: john boris [mailto:jbori...@gmail.com] > > It must be a bad day for the East Coast. While in Physical Therapy > (Haddonfield nj) lights and AC blinked. On the way home Dunking Donuts > (next town over) lost their computers as the power blinked. (Lucky I had > cash). I get home and the wife

Re: [lopsa-tech] Anybody else seeing this? (Amazon AWS problem)

2015-07-02 Thread Edward Ned Harvey (lopser)
I don't know anything about how BGP works, or even, if BGP is what's currently used on the Internet. I found it very weird, that two machines in the same LAN (with public IP's), pinging the same target IP address far away, one of them got a response while the other didn't. And it was even more

Re: [lopsa-tech] Anybody else seeing this? (Amazon AWS problem)

2015-07-02 Thread Ross West
Almost all network circuit load balancing systems use some kind of (src/dst) hash in order to attempt to keep end-to-end packet ordering the same. What the hash is built upon (ip, tcp/udp port, etc) is up to the local admin. So if one path is borked and sending packets to a blackhole (but the

Re: [lopsa-tech] Anybody else seeing this? (Amazon AWS problem)

2015-07-02 Thread Edward Ned Harvey (lopser)
> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] > On Behalf Of Ross West > > Almost all network circuit load balancing systems use some kind of > (src/dst) hash in order to attempt to keep end-to-end packet ordering > the same. What the hash is built upon (ip, tcp/udp p