You are welcome and thank you for the usage/testing ! :) And thank you for the bug report too - I made a fix for the stateful ICMP handling in the master, and added a few testcases to catch it in the future:
https://gerrit.fd.io/r/#/c/15004/ --a On 9/26/18, Rubina Bianchi <r_bian...@outlook.com> wrote: > Thanks for your following up. π > ________________________________ > From: Andrew π½ Yourtchenko <ayour...@gmail.com> > Sent: Wednesday, September 26, 2018 12:31 AM > To: Rubina Bianchi > Cc: vpp-dev@lists.fd.io > Subject: Re: [vpp-dev] Odd problem in adding new session on vpp when its > session table is full > > Dear Rubina, > > On 9/25/18, Rubina Bianchi <r_bian...@outlook.com> wrote: >> Dear Andrew, >> >> Actually it's not a RFC standard , or known attack, we extracted the >> restriction using experiment on linux netfiltter and tried to implement >> its >> behavior. > > hmm okay. so my position about implementing it still stands then :-) > > Also, another point - if you send two echo requests at once with the > same code (which is fine, because they are differentiated by ICMP ID > and sequence number), then the second request will overwrite bihash > values on activation (which is fine) and then the first reply that > comes will erase those upon deactivation, so the second request will > be dropped. This can make the troubleshooting harder in some cases > since this is confusing. One way to deal with that is to keep some > sort of reference counter, or extend the bihash key, but again in the > absence of real-world threat model this looks like shooting pigeons > from ICBMs... (the only attack I know for this scenario is smurf and > the existing implementation covers it fine) > >> >> I checked your patch, throughput is fine. After about 2000 seconds my >> throughput was still maximum and there was no drop-rate on trex. Also, >> rx-miss was very little in this patch. > > Excellent, thanks a lot! > >> However, one thing is attracting for >> me is that, in the same scenario in previous discussion, vpp served this >> traffic with 3.6 million sessions, but in current patch it serves with >> 3.9 >> million sessions. Is this related to your last changes? > > Yeah, it could be... the new change make the expiry somewhat longer: > the session in any given timeout list is checked twice per timeout, > rather than every X seconds (X being the shortest list timeout). So if > e.g. a UDP session timeout is 100 seconds, and there is a packet on it > 10 seconds before the session is checked, the next check will be 50 > seconds later, at which point the idle time on the session (60 > seconds) is still smaller than 100, so the session will be expunged > only 50 seconds later, with the idle time of 110 seconds. > > Contrast this with the previous behavior where all the lists might be > checked as often as 2 seconds - this would have resulted in a more > precise timing out, but at an expense of a lot of CPU usage. > > But then all of the TCP sessions will transition into the tcp > transient state upon closure, so the only impact in real world will be > with the UDP connections, which should be relatively small. > > All that said: I am considering to move the timeout infra to the > tw_timers (which weren't available back when I was writing the code) - > then in theory we can get both the precise expiry and the efficiency. > But that is a larger change, and I am not sure of it yet, so I wanted > to get this simpler solution in first. > > --a > >> >> Thanks, >> Sincerely >> ________________________________ >> From: Andrew π½ Yourtchenko <ayour...@gmail.com> >> Sent: Monday, September 24, 2018 6:49 PM >> To: Rubina Bianchi >> Cc: vpp-dev@lists.fd.io >> Subject: Re: [vpp-dev] Odd problem in adding new session on vpp when its >> session table is full >> >> Dear Rubina, >> >> On 9/24/18, Rubina Bianchi <r_bian...@outlook.com> wrote: >>> Dear Andrew, >>> >>> It's hardcoded as it was simple and fast solution for our default >>> scenario >>> implementation. >>> As you correctly mentioned in previous email one of the bug fixes was >>> the >>> restriction. Also another one is preventing reply packets pass through >>> even >> >> ok. that's more a "featurette" - I deliberately did not attempt to >> implement the "strict checking" because I had difficult time finding >> the attack vector. (rather than maybe some kind of "compliance" checks >> for the reasons of compliance ?) >> >>> if those packets are matched with an acl rule. In another word these >>> reply >>> packets are not belong to any echo request in reverse direction. >> >> hmm so you are sort-of making a "protocol inspection engine" there ? :-) >> >> Anyway, so far I haven't managed to recreate this condition - though >> if you were running the 18.07 rather than 18.07.1 code, then the bug >> related to hash acl manipulation on ACL changes might have caused that >> effect... I will experiment a bit more, though. >> >> Also, remember the other thread we discussed a while ago about the >> throughput getting lower over time.. I have made >> https://gerrit.fd.io/r/#/c/14821/ which should significantly reduce >> the amount of session list shuffling work in normal case scenarios. >> Before I commit it, could you give it a shot to see if it indeed >> behaves as I would expect it to behave ? Thanks a lot! >> >> --a >> >>> Thanks, >>> Sincerely >>> ________________________________ >>> From: Andrew π½ Yourtchenko <ayour...@gmail.com> >>> Sent: Tuesday, September 18, 2018 4:06 PM >>> To: Rubina Bianchi >>> Cc: vpp-dev@lists.fd.io >>> Subject: Re: [vpp-dev] Odd problem in adding new session on vpp when its >>> session table is full >>> >>> Dear Rubina, >>> >>> On 9/18/18, Rubina Bianchi <r_bian...@outlook.com> wrote: >>>> Dear Andrew, >>>> >>>> Our changes is provided to you by creating a patch which is attached to >>>> this >>>> email. >>>> I didn't commit it to gerrit due to our specific scenario >>>> (permit+reflect >>>> on >>>> all inputs, permit+reflect or deny on all outputs). >>> >>> Why do you hardcode it as opposed to making it part of configuration ? >>> permit+reflect in one direction and deny except established sessions >>> is a fairly standard config. >>> >>>> In addition to ICMP timeout handling, our code fixes some ICMP bugs. >>> >>> Do you mean the "strict" enforcement of the one-request-one-response >>> policy for ICMP that this code does ? >>> >>> --a >>> >>>> Although, I think code is clear for you, I can explain it in details if >>>> you >>>> ask. >>>> >>>> Thanks, >>>> Sincerely >>>> ________________________________ >>>> From: Andrew π½ Yourtchenko <ayour...@gmail.com> >>>> Sent: Tuesday, September 18, 2018 11:27 AM >>>> To: Rubina Bianchi >>>> Cc: vpp-dev@lists.fd.io >>>> Subject: Re: [vpp-dev] Odd problem in adding new session on vpp when >>>> its >>>> session table is full >>>> >>>> >>>> >>>> >>>> Hi Rubina, >>>> >>>> On 18 Sep 2018, at 11:14, Rubina Bianchi >>>> <r_bian...@outlook.com<mailto:r_bian...@outlook.com>> wrote: >>>> >>>> Hi Dear Andrew >>>> >>>> 1) I just attached my init.conf to this email. As you guessed session >>>> table >>>> size is 1000000. This problem is occurred on vpp stable/1807. >>>> >>>> Ah, cool, that helps, thanks! >>>> >>>> >>>> 2) Yes, there is 6 timeout list. We added a list for handling icmp >>>> timeouts. >>>> >>>> That is not the stable/1807, then βΊοΈ would you mind submitting the >>>> change >>>> to >>>> gerrit so we could take a look at it and ideally incorporate into the >>>> master >>>> ? >>>> >>>> βa >>>> >>>> >>>> ________________________________ >>>> From: Andrew π½ Yourtchenko >>>> <ayour...@gmail.com<mailto:ayour...@gmail.com>> >>>> Sent: Monday, September 17, 2018 8:03 PM >>>> To: Rubina Bianchi >>>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> >>>> Subject: Re: [vpp-dev] Odd problem in adding new session on vpp when >>>> its >>>> session table is full >>>> >>>> Dear Rubina, >>>> >>>> looking at the outputs, there are a few anomalies that hopefully you >>>> can clarify: >>>> >>>> 1) the max session count is 1000000. The latest master has the default >>>> limit of 500000, and I do not see any startup config parameters >>>> changing that. Which version are you testing with/building off ? >>>> >>>> 2) there are 6 fa_conn_list_head elements in each worker for your >>>> outputs. That number was initially 3, and in the early spring when I >>>> introduced the purgatory list and the reserved unused list this number >>>> has increased to 5. The vectors are initialized at a start with a >>>> constant, so I am wondering why your outputs have a different number. >>>> >>>> Would be able to comment on these observations ? >>>> >>>> Thank you! >>>> >>>> --a >>>> >>>> On 9/17/18, Rubina Bianchi >>>> <r_bian...@outlook.com<mailto:r_bian...@outlook.com>> wrote: >>>>> * Dear VPP >>>>> >>>>> I ran a test on VPP configured with permit+reflect ACl rules with >>>>> t-rex. >>>>> In >>>>> this test, I put two interfaces on one bridge-domain and had an ACL on >>>>> all >>>>> of its input and output interfaces. The ACL had just one rule which >>>>> was >>>>> allowing any traffic. I ran my test until VPP's session table was >>>>> full. >>>>> I >>>>> run t-rex whith following command: >>>>> >>>>> "./t-rex-64 -f cap2/sfr.yaml -m 10 -d 10000" >>>>> >>>>> >>>>> After a couple of days, I took another test on VPP. I tried to >>>>> establish >>>>> a >>>>> ssh session between two clients via my VPP. But session could not be >>>>> established. When I checked VPP trace, All of my ssh packets where >>>>> dropped >>>>> due to following error: >>>>> >>>>> "acl-plugin-in-ip4-l2: too many sessions to add new" >>>>> >>>>> when I checked VPP's session table, I realized that it was full. No >>>>> session >>>>> where deleted since my previous test and no session where going to be >>>>> added >>>>> to session table.I also checked my /var/log/hawk.log file and saw >>>>> following >>>>> error: >>>>> >>>>> "acl_fa_node_fn:516: BUG: session LSB16(sw_if_index) and 5-tuple >>>>> collision!" >>>>> >>>>> I could not fix this problem so I restarted my VPP service. After >>>>> that, >>>>> I could not reproduce this state again. Does anyone have any idea on >>>>> what my problem on VPP was? >>>>> >>>>> I attached my hawk.log, vpp trace, "vppctl sh acl-plugin sessions" >>>>> output >>>>> and startup.conf file to this email. >>>>> >>>>> >>>>> >>>>> >>>>> >>>> <init.conf> >>>> >>> >> >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10676): https://lists.fd.io/g/vpp-dev/message/10676 Mute This Topic: https://lists.fd.io/mt/25722080/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
