Are you using latest master ? I fixed a couple of issues in bihash last week related to memory usage... if it’s the latest master, the output of used vs available looks weird... - so please let me know...
As for the “general” growth - basically what happens is bihash doubles each bucket size whenever there is a collision on insert, and then converts the bucket into linear lookup whenever there is still a collision after that growth. Then the only time the shrinkage/reset is happening is when the bucket is completely free - which with long living sessions with overlapping lifetimes might mean never. So one approach to this is to increase the number of buckets. Then they will be smaller and have higher probability of being freed. This is assuming there is nothing else “funny” going on. You can do “show acl-plugin sessions verbose 1” via vppctl (It will take forever to complete and needs pager disabled since it dumps the entire bihash) to inspect the way the buckets are filled... --a > On 25 Sep 2018, at 12:12, khers <s3m2e1.6s...@gmail.com> wrote: > > It's amazing!!! > > IPv4 Session lookup hash table: > Hash table ACL plugin FA IPv4 session bihash > 968086 active elements 65536 active buckets > 13 free lists > [len 16] 1 free elts > [len 32] 1 free elts > [len 256] 10669 free elts > [len 512] 36768 free elts > [len 1024] 4110 free elts > [len 2048] 156 free elts > [len 4096] 4 free elts > 844 linear search buckets > arena: base 7fe912320000, next 2680ca780 > used 10335594368 b (9856 Mbytes) of 10000000000 b (9536 Mbytes) > > >> On Tue, Sep 25, 2018 at 1:39 PM khers <s3m2e1.6s...@gmail.com> wrote: >> Yes, that's right. I think is completely another issue from the patch you >> sent >> >>> On Tue, Sep 25, 2018 at 1:35 PM Andrew 👽 Yourtchenko <ayour...@gmail.com> >>> wrote: >>> Excellent, thanks! >>> >>> Memory usage - you mean in bihash arena ? >>> >>> --a >>> >>>> On 25 Sep 2018, at 11:38, khers <s3m2e1.6s...@gmail.com> wrote: >>>> >>>> Throughput and session add/del is stable as rock. The only danger i see is >>>> growing memory usage. >>>> look at this >>>> >>>> >>>>> On Tue, Sep 25, 2018 at 11:31 AM khers <s3m2e1.6s...@gmail.com> wrote: >>>>> Of course, I test your patch, there is no slowdown with my scenario. I >>>>> need more time to test other >>>>> scenarios and make sure. >>>>> >>>>> >>>>>> On Mon, Sep 24, 2018 at 3:11 PM Andrew 👽 Yourtchenko >>>>>> <ayour...@gmail.com> wrote: >>>>>> Cool. Then it is probably indeed the session requeues that are not yet >>>>>> efficient... I have been looking at optimizing that. >>>>>> >>>>>> I have a draft in the works which should have less session requeues - I >>>>>> have just added you to it, could you give it a shot and see if it makes >>>>>> things better ? >>>>>> >>>>>> --a >>>>>> >>>>>>> On 24 Sep 2018, at 12:55, khers <s3m2e1.6s...@gmail.com> wrote: >>>>>>> >>>>>>> yes, I confirm >>>>>>> >>>>>>>> On Mon, Sep 24, 2018 at 2:08 PM Andrew 👽 Yourtchenko >>>>>>>> <ayour...@gmail.com> wrote: >>>>>>>> Okay, so what I think I am hearing - the gradual slowdown is/was >>>>>>>> always there, and is somewhat more pronounced in master, right ? >>>>>>>> >>>>>>>> --a >>>>>>>> >>>>>>>>> On 24 Sep 2018, at 11:49, khers <s3m2e1.6s...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> I allways get SIGSEGV or 'worker thread dead lock' In 1804 with 1 or >>>>>>>>> more worker thread and 1 main, >>>>>>>>> but when vpp using one cpu I hadn't any problem. In the 1807 multi >>>>>>>>> core is stable i didn't see any of those >>>>>>>>> problem but throughput is declining slowly. >>>>>>>>> I ran another test with same version of last email, which vpp is >>>>>>>>> configured with one core and throughput is declining slower than >>>>>>>>> master >>>>>>>>> second 200 >>>>>>>>> second 5900 >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Sun, Sep 23, 2018 at 6:57 PM Andrew 👽 Yourtchenko >>>>>>>>>> <ayour...@gmail.com> wrote: >>>>>>>>>> Interesting - but you are saying in 1804 this effect is not observed >>>>>>>>>> ? There was no other notable changes with regards to session >>>>>>>>>> management - but maybe worth it to just do hit bisect and see. >>>>>>>>>> Should be 4-5 iterations. Could you verify that - if indeed this is >>>>>>>>>> not seen in 1804. >>>>>>>>>> >>>>>>>>>> --a >>>>>>>>>> >>>>>>>>>>> On 23 Sep 2018, at 16:42, khers <s3m2e1.6s...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> I checked out the version before the gerrit 12770 is merged to >>>>>>>>>>> master. >>>>>>>>>>> 2371c25fed6b2e751163df590bb9d9a93a75a0f >>>>>>>>>>> >>>>>>>>>>> I got SIGSEGV with 2 workers, so I repeat the test with one worker. >>>>>>>>>>> Throughput is going down like the latest version. >>>>>>>>>>> >>>>>>>>>>>> On Sun, Sep 23, 2018 at 4:55 PM Andrew 👽 Yourtchenko >>>>>>>>>>>> <ayour...@gmail.com> wrote: >>>>>>>>>>>> Would you be able to confirm that it changes at a point of >>>>>>>>>>>> https://gerrit.fd.io/r/#/c/12770/ ? >>>>>>>>>>>> >>>>>>>>>>>> --a >>>>>>>>>>>> >>>>>>>>>>>>> On 23 Sep 2018, at 13:31, emma sdi <s3m2e1.6s...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Dear Community >>>>>>>>>>>>> >>>>>>>>>>>>> I have simple configuration as following: >>>>>>>>>>>>> >>>>>>>>>>>>> startup.conf >>>>>>>>>>>>> simple_acl >>>>>>>>>>>>> >>>>>>>>>>>>> I used Trex packet generator with following command: >>>>>>>>>>>>> ./t-rex-64 --cfg cfg/trex_config.yaml -f cap2/sfr.yaml -m 5 -c 2 >>>>>>>>>>>>> -d 6000 >>>>>>>>>>>>> The Total-RX gradually decrease, here is output of Trex in second >>>>>>>>>>>>> 200, and 5900. >>>>>>>>>>>>> >>>>>>>>>>>>> I did not saw this problem in 18.04. I think session_cleaner >>>>>>>>>>>>> thread make so many >>>>>>>>>>>>> interrupt, do you have any idea? >>>>>>>>>>>>> >>>>>>>>>>>>> Regards >>>>>>>>>>>>> >>>>>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=- >>>>>>>>>>>>> Links: You receive all messages sent to this group. >>>>>>>>>>>>> >>>>>>>>>>>>> View/Reply Online (#10615): >>>>>>>>>>>>> https://lists.fd.io/g/vpp-dev/message/10615 >>>>>>>>>>>>> Mute This Topic: https://lists.fd.io/mt/26145401/675608 >>>>>>>>>>>>> Group Owner: vpp-dev+ow...@lists.fd.io >>>>>>>>>>>>> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub >>>>>>>>>>>>> [ayour...@gmail.com] >>>>>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10647): https://lists.fd.io/g/vpp-dev/message/10647 Mute This Topic: https://lists.fd.io/mt/26145401/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
