On 2020-03-14 23:50, Damian Menscher <damian at google.com> wrote: > I don't recommend filtering the SYN-ACK packets. That's what Octolus did, > and the result was leaving half-open SYN_RECV connections on all the nodes > used for reflection. That has two downsides: > > - the reflectors will retry the SYN-ACK (several times), which increases > your PPS load (amplifying the attack) > - the providers may notice the large number of SYN_RECV connections from > your network and put you on a blacklist
I work at Path Network; we're providing DDoS mitigation services for Octolus. I wanted to follow up your message with a couple of points from our POV. The filter we have running for Octolus is a more generic stateful TCP filter, designed primarily to curb spoofed TCP floods. The bigger attacks we see are on the magnitude of 100s of Gbps/Mpps, and a simple fact is that outright dropping an out-of-state packet is multiple times less expensive for us than creating and responding with a RST. We see spoofed TCP attacks much more frequently than a SYN-ACK reflection, and in fact from our automation's point of view this looks exactly like a SYN-ACK flood from randomized source addresses. Aside from the resources cost, sending back RSTs for non-reflected attacks of respectable volume might also not be appreciated by some networks. Knowing the specifics of this long-running reflection attacks, we're considering how to reply with RSTs so to not leave lingering SYN_RECVs on reflectors' side. On 2020-03-15 16:36, Jean | ddostest.me <jean at ddostest.me> wrote: > I believe that Oculus blocked the RST and not the SYN/ACK. The SYN-ACKs are dropped; letting them reach the end servers and dropping outgoing RSTs instead would make for poor mitigation. :) On 2020-03-14 23:50, Damian Menscher <damian at google.com> wrote (cont.): > I don't want to leave you with the impression that it's hopeless... these > attacks aren't impossible to stop --- it just requires convincing the > transit providers to care. > > Damian > > On Sat, Mar 14, 2020 at 1:31 PM Jean | ddostest.me via NANOG < > nanog at nanog.org> wrote: > > Hi Bill, > > thanks for sharing the data. Indeed, I can't offer you a way to backtrack > the spoofed packets. > > Anyway, I'm not sure what could you do legally as there is a very high > chance that these people are not in the USA and the CFAA won't apply to > them. > > Here is what I would do if I was in your situation. > > Since these packets are spoof and malformed, I would block all SYN/ACK > based on the length. > > Depending on your hardware, it's very easy to inspect *only the SYN/ACK > by length* if you have modern hardware. On linux/unix, it's also very > straightforward. I'm not sure for windows though. > > Here is the details of the analysis: > > Today, all the SYN and SYN/ACK includes a minimum of options like MSS, WS, > SACK, NOP (Only a spacer, sometimes 2) and extended TS. There might be > others, but let's use the basic one. > > In your case, there are none. There is only MSS and the SYN length is 44 > bytes. These are spoof packets maybe generated by either TCP-AMP like > reported earlier. > > I would try to block all SYN/ACK coming toward your network with a length > of 44 bytes or lower. But, this is weird because it should be 54 bytes. > Maybe there is some offloading of some sort in your gear. > > Now depending on your hardware, it could work or it could kill your router > as it will punt the cpu. I guess you have some modern gear. > > What I do when I am not sure about the length, I start to accept and log > at 60 bytes, then 58, 56, 54... 44 until I catch the gremlins. > > Once you found the sweet spot, you drop all SYN/ACK toward your /23 lower > than X bytes. It won't kill or block anything legitimate if you do it > properly. :) > > What will happen is that you will not reply to these spoof SYN/ACK with a > RST and still allowing RST for legit SYN and SYN/ACK. Akamai and your > service providers will be happy and should not penalize you. > > I'm pretty sure that it will help you as it did for me in the past. > > Let me know if it's not clear and/or which part is foggy and I'll try to > give more details and better explanation. > > Regards, > > Jean St-Laurent > On 2020-03-14 11:46, William Herrin wrote: > > On Sat, Mar 14, 2020 at 4:02 AM Jean | ddostest.me via NANOG<nanog at nanog.org> <nanog at nanog.org> wrote: > > can you post some forged packets please? You can send them offlist if > you prefer. > > Hi Jean, > > Here are a couple examples (PDT this morning): > > 08:22:43.413250 IP (tos 0x0, ttl 55, id 10108, offset 0, flags [none], > proto ICMP (1), length 56) > 45.89.93.26 > 199.33.225.218: ICMP host 45.89.93.26 unreachable - > admin prohibited filter, length 36 > IP (tos 0x0, ttl 69, id 10108, offset 0, flags [DF], proto TCP > (6), length 40) > 199.33.225.218.9851 > 45.89.93.26.443: [|tcp] > 0x0000: 4500 0038 277c 0000 3701 28da 2d59 5d1a > 0x0010: c721 e1da 030d 4b61 0000 0000 4500 0028 > 0x0020: 277c 4000 4506 dae4 c721 e1da 2d59 5d1a > 0x0030: 267b 01bb a057 e903 > > 08:25:47.787326 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto > TCP (6), length 44) > 104.87.78.95.80 > 199.33.225.143.8667: Flags [S.], cksum 0xc97a > (correct), seq 1216155085, ack 11765035, win 29200, options [mss > 1156], length 0 > 0x0000: 4500 002c 0000 4000 3606 e564 6857 4e5f > 0x0010: c721 e18f 0050 21db 487d 0dcd 00b3 852b > 0x0020: 6012 7210 c97a 0000 0204 0484 > > I have observed no consistency in the remote IP addresses. I receive > no more than a few of each and they don't line up with particular > networks. Remote ports are heavily 80, 443, 22, 25, etc. but a > smattering of less common ports too. I'm not seeing any RSTs at all > nor any port-unreachables. Lots of syn/acks and a few time exceeded > and host unreachables. I don't know what to make of that. > > > On Sat, Mar 14, 2020 at 1:46 AM Andrew Smith<andrew.william.smith at gmail.com> <andrew.william.smith at gmail.com> wrote: > > Look inside the ICMP Unreachable backscatter at the truncated original packet that caused the unreachable message. > > Clever! I wouldn't have thought of that. Unfortunately as in the > example above, the TTLs in the packets encapsulated in ICMP are not > especially close to one of the common boundaries. > > Regards, > Bill Herrin > > -- > William Herrinbill at herrin.ushttps://bill.herrin.us/ > >
