Hi, I have this memory issue again.
The status now is that I compiled kamailio as followed here and use mem_join=1. I reload my data from DB every minute! I know i reload a lot, will take care of that (will update you all on April :-)). When kamailio starts the shmem is about 17% of 4Gb and after 7 days (600 reloads per day) the shmem is about 24%. The traffic is not the issue. Still, can defrag of memory improve? Ideas for defrag? Thanks, Uri On Tue, Nov 27, 2012 at 12:37 PM, Uri Shacked <ushac...@gmail.com> wrote: > > OK > >>ok - it's 56sec or 36sec? > It was 56 sec. > > Now, after recompile, when cfg mem_join=0 the reload takes 8 sec and the > shmem real used is 25% of 4Gb. > When mem_join=, the reload take 30 sec and the shmem real used is 17%. > > > Looks good with mem_join=1. > > Thanks, > Uri > > > On Mon, Nov 26, 2012 at 5:43 PM, Daniel-Constantin Mierla < > mico...@gmail.com> wrote: > >> Hello, >> >> >> On 11/22/12 8:51 AM, Uri Shacked wrote: >> >> >> Hi, >> >> >> Btw, is the reload time constant now? Even if you run couple of times? >> >> yes, the reload time is constant - 56 sec. i tested for 100 reloads in an >> hour and it was OK. >> >> >> ok - it's 56sec or 36sec? >> >> >> >> >> What are the values for 'kamctl mi get_statistics shmem:'? >> >> i configured kamailio to start with 4Gb and after reload the shmem >> (real_used) take around 30% of it. >> but, after 20 reloads it grows in 1%. so, after the 100 reloads the >> real_used take around 34%-35% of shmem. >> i made the choise to compile again with f_malloc and not use mem_join. >> the reloads are faster, it uses less real_size (12% and not 30%) and the >> increasment of it is around 1% for 5 reloads (i do 5 reloads a week). i >> will keep track on it and update. >> >> thanks a lot. >> >> Btw, what do you think? would you use f_malloc with no mem_join or >> q_malloc with join? >> >> >> Can you try with master branch? I switched to q_malloc with no debug >> info. That means the overhead should be lower and the joining faster. >> >> You can do it also with 3.3, you have to edit Makefile.defs: >> - be sure MEMDBG=0 >> - replace the next block: >> >> ifeq ($(MEMDBG), 1) >> C_DEFS+= -DDBG_QM_MALLOC >> C_DEFS+= -DMEM_JOIN_FREE >> else >> C_DEFS+= -DF_MALLOC >> C_DEFS+= -DMEM_JOIN_FREE >> endif >> >> with: >> >> ifeq ($(MEMDBG), 1) >> C_DEFS+= -DDBG_QM_MALLOC >> C_DEFS+= -DMEM_JOIN_FREE >> else >> C_DEFS+= -DMEM_JOIN_FREE >> endif >> >> Practically is removal of line C_DEFS+= -DF_MALLOC >> >> Then recompile and reinstall.. >> >> Cheers, >> Daniel >> >> >> BR, >> Uri >> >> >> On Wed, Nov 21, 2012 at 8:55 PM, Daniel-Constantin Mierla < >> mico...@gmail.com> wrote: >> >>> Hello, >>> >>> >>> On 11/21/12 1:33 PM, Uri Shacked wrote: >>> >>> Hi, >>> >>> I recompiled with MEMDBG=1 and installed. >>> here are the results for reloading 5 million rows with MTREE: >>> >>> mem_join=1 -->takes 56 seconds and the real_used_size of shmem is around >>> 1.2Gb. >>> mem_join=0 --> takes 10 seconds and the real_used_size of shmem is >>> around 2Gb >>> does it seems normal? >>> 56 seconds is a lot of time...... >>> >>> the join is done for the free operation, meaning that most of the time >>> is spent when freeing the old tree from memory. The new values will be used >>> after loading the database records, then the old tree is destroyed (this >>> involves the join operation). Also, the sip routing is not affected, >>> loading the new records and destroying old memory tree is done in the >>> MI/RPC process. >>> >>> In other words, while the MI/RPC process takes care of loading new data >>> and destroying the old one, the SIP routing is not affected at all. >>> >>> Even when the reload command is executed, the old tree is used until all >>> the records are loaded in a new tree. At that moment, the pointer to the >>> active tree is changed from the old tree to the new tree (a very simple >>> sequence of assignments, very fast). Routing will use the new tree and the >>> Mi/RPC process starts destroying the old tree. >>> >>> >>> >>> by the way, when the f_malloc was used, the size of the real_used shmem >>> was twice smaller. >>> >>> >>> The overhead when storing small values is significant for q_malloc, >>> each fragment keeping references (pointers) to file name and line where it >>> was allocated and freed. In addition it keeps information to get to >>> previews and next fragment, resulting in faster join. >>> >>> It is some space to improve, in order to make less overhead (like a >>> compile time option not to keep info about file and line of malloc/free). I >>> will think about what can be done for the next release. >>> >>> Btw, is the reload time constant now? Even if you run couple of times? >>> >>> What are the values for 'kamctl mi get_statistics shmem:'? >>> >>> Cheers, >>> Daniel >>> >>> >>> Thanks. >>> On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla < >>> mico...@gmail.com> wrote: >>> >>>> Hello, >>>> >>>> >>>> On 11/20/12 7:34 PM, Uri Shacked wrote: >>>> >>>> Hi, >>>> >>>> can you be a litle more specific of the steps of the install and where >>>> do i make the changes? >>>> >>>> >>>> in the source tree, edit the file Makefile.defs and set: >>>> >>>> MEMDBG=1 >>>> >>>> then run: >>>> >>>> make all >>>> make install >>>> >>>> >>>> some words of what is the diff between f_malloc and q_malloc will be >>>> great :-). >>>> >>>> >>>> q_malloc is more debugging purposes, keeping more information for each >>>> chunk, therefore the overhead is a bit higher than with f_malloc, but >>>> because keeps more details, it is faster to find the fragments that can be >>>> joined. >>>> >>>> Cheers, >>>> Daniel >>>> >>>> >>>> >>>> thanks, >>>> Uri >>>> >>>> On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla < >>>> mico...@gmail.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> ok, I will look over it. At this moment the f_malloc (which is enabled >>>>> for 3.3) has a pretty inefficient mem join implementation, can you try >>>>> with >>>>> q_malloc? Edit Makefile.defs and set: >>>>> >>>>> MEMDBG=1 >>>>> >>>>> Then compile and install. >>>>> >>>>> The join operation should be faster, let's see if you get blocking >>>>> issues with this one. >>>>> >>>>> Cheers, >>>>> Daniel >>>>> >>>>> >>>>> On 11/20/12 2:57 PM, Uri Shacked wrote: >>>>> >>>>> Daniel hi, >>>>> >>>>> I attached 2 txt files. >>>>> One with mem_join=1, the other with mem_join=0, and the info you asked >>>>> for. >>>>> Let me know if it is OK. >>>>> >>>>> Thanks, >>>>> Uri >>>>> >>>>> On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla < >>>>> mico...@gmail.com> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> if you set memjoin to 0, do you see any difference? >>>>>> >>>>>> Can you try again (with memjoin 1 as well as 0) and send the output >>>>>> of: >>>>>> >>>>>> kamctl mi get_statistics shmem: >>>>>> >>>>>> before executing the reload commands? >>>>>> >>>>>> When it gets to 100%, can you see which process is using the cpu and >>>>>> attach to it with: >>>>>> >>>>>> gdb /path/to/kamailio PID >>>>>> >>>>>> then do: >>>>>> >>>>>> bt full >>>>>> >>>>>> and send output here? >>>>>> >>>>>> Cheers, >>>>>> Daniel >>>>>> >>>>>> >>>>>> On 11/18/12 4:09 PM, Uri Shacked wrote: >>>>>> >>>>>> After some testing I notice the following: >>>>>> First reload of 5 million records after kamailio started took about 9 >>>>>> sec. >>>>>> Second reload (4 minutes after the first one) took 60 sec. >>>>>> The third one (again about 4 minutes after the secind) got kamailio >>>>>> to use 100% cpu and after 13 minutes! i killed it..... >>>>>> >>>>>> I can understand that the memory manger works harder, still, any >>>>>> ideas on how to use mem_join and keep on reloading data. >>>>>> (in real life our data loads 5 million records once a day when almost >>>>>> no traffic. still after a few days it stops...) >>>>>> >>>>>> Thanks, >>>>>> Uri >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked <ushac...@gmail.com>wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am using MTREE and DIALPLAN modules to load lots of info to >>>>>>> kamailio. (6 million rows). >>>>>>> >>>>>>> When kamailio was running with 3.2.1 (no mem_join=1 option), the >>>>>>> used size was increasing but the process of loading the data was fast >>>>>>> eanough. >>>>>>> >>>>>>> I upgraded to 3.3.2 and set mem_join=1. Now the loading process take >>>>>>> about 10 time longer and sometimes stops kamailio from responding to >>>>>>> traffic. >>>>>>> >>>>>>> Any ideas? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Uri >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing >>>>>> listsr-us...@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>>>>> >>>>>> >>>>>> -- >>>>>> Daniel-Constantin Mierla - >>>>>> http://www.asipto.comhttp://twitter.com/#!/miconda - >>>>>> http://www.linkedin.com/in/miconda >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Daniel-Constantin Mierla - >>>>> http://www.asipto.comhttp://twitter.com/#!/miconda - >>>>> http://www.linkedin.com/in/miconda >>>>> >>>>> >>>> >>>> -- >>>> Daniel-Constantin Mierla - >>>> http://www.asipto.comhttp://twitter.com/#!/miconda - >>>> http://www.linkedin.com/in/miconda >>>> >>>> >>> >>> -- >>> Daniel-Constantin Mierla - >>> http://www.asipto.comhttp://twitter.com/#!/miconda - >>> http://www.linkedin.com/in/miconda >>> >>> >> >> -- >> Daniel-Constantin Mierla - >> http://www.asipto.comhttp://twitter.com/#!/miconda - >> http://www.linkedin.com/in/miconda >> >> >
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
