Zach,
As mentioned before, I am open to peering (where possible) but have not
received a response.
My goal is to connect with someone at Amazon and work with them on a
technical solution, which is why I posted asking for that here. Various
factors mean that we can't just upgrade our way out of this one, and
manual whac-a-mole on the sources has shown to have limited use.
These attacks also impact Amazon and the networks in between the sources
and targets, and they take time to handle by the abuse teams, so there
are good reasons to investigate them further and find better ways to
mitigate or prevent them.
-John
On 11/8/2018 1:12 PM, Zach Puls wrote:
Makes sense, that's understandable. Do you peer with AWS? If not, maybe opening
up a peering agreement will give you a better contact, and a bit more pull when
attacks occur? I know someone with a peering agreement with AWS, and they have
been able to get resolutions fairly quickly when issues arise.
Other than that, I'm not sure of a solution other than more IP transit.
Thanks,
Zach Puls
Network Engineer | MEF-CECP
KsFiberNet
-----Original Message-----
From: John Weekes <j...@nuclearfallout.net>
Sent: Thursday, November 08, 2018 15:03
To: Zach Puls <zp...@ksfiber.net>
Cc: nanog@nanog.org
Subject: Re: Amazon network engineering contact? re: DDoS traffic
Zach,
Yes, RTBH is used to distribute the null-routes that I mentioned.
Unfortunately, even brief saturation events lasting just 5-10 seconds (a
typical amount of time to detect the loss, issue the null-route, and see the
traffic start to fall off as it is distributed upstream) can cause real damage
to those customers who are sensitive to latency and packet loss. So while
null-routes limit the duration of the impact, they can't eliminate it entirely.
And, of course, the actual target of the attack
-- the now-null-routed IP address -- becomes unreachable, which was presumably
the goal of the attacker.
-John
On 11/8/2018 12:54 PM, Zach Puls wrote:
No idea about an Amazon abuse contact, but do you have RTBH communities enabled
with your upstream provider(s)? As a general practice, when you detect a (D)DoS
attack in progress, it would help to automatically advertise that prefix to
your upstream(s) with the black-hole community. This would at least help
mitigate the effects of the attacks when they do occur, even if they come from
a different source than AWS.
Thanks,
Zach Puls
Network Engineer | MEF-CECP
KsFiberNet
-----Original Message-----
From: NANOG <nanog-boun...@nanog.org> On Behalf Of John Weekes
Sent: Thursday, November 08, 2018 14:44
To: nanog@nanog.org
Subject: Amazon network engineering contact? re: DDoS traffic
We've been seeing significant attack activity from Amazon over the last two
months, involving apparently compromised instances that commonly send 1-10G of
traffic per source and together generate Nx10G of total traffic. Even when our
overall upstream capacity exceeds an attack's overall size, the nature of
load-balancing over multiple 10G upstream links means that an individual link
can be saturated by multiple large flows, forcing our systems to null-route the
target to limit impact.
We've sent an abuse notification about every traffic source to Amazon, and
specific sources seem to stop their involvement over time (suggesting that
abuse teams are following up on them), but there is an endless parade of new
attackers, and each source participates in many damaging attacks before it is
shut down.
Is there anyone at Amazon who can help with an engineering solution in terms of
programmatically detecting and rate-limiting attack traffic sources, to our
networks or overall? Or applying the kludge of a rate-limit for all Amazon
traffic to our networks? Or working with us on some other option?
At least one other large cloud provider has an automatic rate-limiting system
in place that is effective in reducing the damage from repeat high-volume
attacks.
Emails to the Amazon NOC, peering contacts (since that would be another
possible solution), and abuse department have not connected me with anyone.
Thanks,
John
