I have not experience with RabbitMQ but I've seen this guy around on high volume data transfer scenarios. A message queuing approach sounds more resilient to me but this will need consideration at a later point. Thanx though for the thoughts. What I currently do is: iptables -> uacctd -> mysql -> processing of mysql and further aggregation -> json file -> push to central hub (rsync) -> import to DB -> visualize... (I've been playing like this for at least 3 years without issues) It is interesting to hear how you tackle such issues.
Alex On Mon, Jun 25, 2018 at 7:51 PM, Dariush Marsh-Mossadeghi < [email protected]> wrote: > Rasto's comments got me thinking… > > Not being privy to your application and it’s architecture this may not > work for you at all. > > I’ve had success in the past using rabbitmq to offload flow logs. > We used it to deliver netflow to ELK stacks running on AWS, and it worked > really well. > It has the major advantage of operational resilience. > > Something like this: > > netflow sources -> nfacctd -> rabbitmq -> big long unreliable network path > across public internet -> rabbitmq -> logstash -> elasticsearch > > as ever YMMV :-) > Dariush > > > On 25 Jun 2018, at 16:53, Alex K <[email protected]> wrote: > > > > On Mon, Jun 25, 2018 at 6:49 PM, Rasto Rickardt <[email protected]> wrote: > >> Hello, >> >> If there is no more RAM available, i would test sqlite3 plugin, as >> sqlite3 is more suited for limited resources usage. Youl will possibly >> need to change your workflow to export sqlite3 files and load it >> somewhere else, but it should be a lot cheaper memory wise. >> >> Thanx. Seems that I have to stick with mysql at least at this phase. > > This one is swapping pretty heavily, i would suggest to set to around 10 >> for 4GB RAM. >> >> sysctl vm.swappiness=10 so it will start using swap when available RAM >> is around 400MB. >> > This is already done. > > >> r. >> >> On 06/25/2018 05:35 PM, Alex K wrote: >> > Thanks Dariush. Appreciate your feedback. >> > >> > I was testing several stripped down kernels by compiling and removing >> > most of unused modules. The gain I had was in the range of few MB. >> > Seems that I have to find a more elegant approach at uacctd >> > configuration since with current setup I am loading several mysql >> > plugins so as to be able to filter traffic with directions, networks and >> > ports. >> > I was testing an edge case where these plugins may rise up to 210 >> > instances thus leading to this memory usage. Normally this will not be >> > done so the normal case will be using 2.5 GB of RAM instead of 3.5 GB. >> > Thus in normal cases seems that I am ok. >> > >> > Cheers >> > >> > >> > On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> > OK, so it’s effectively an embedded system scenario, with a fixed >> > config hardware, or similar >> > No silver bullets or one-liner fixes here :-\ >> > >> > You’ve a number of options to slim down your memory footprint >> > - Strip Debian of all the packages you don’t need, learn a lot about >> > kernel tuning, and tune the kernel to your needs. >> > - Start looking at one of the more embedded systems oriented >> > distros, rather than OOTB Debian. Although Debian is skinnier than >> > most, there’s a lot that can be done to minimise footprint. The fact >> > that you’ve got a 64bit Debian9 running on it makes me fairly >> > optimistic that you could run pretty much any Debian derivative. >> > - Profiling your userspace software stack and see if there are any >> > dependencies you don’t need, strip them out. Whether you can do this >> > will very much depend on the modularity of the components. >> > >> > All of the above lead you down the path of maintaining forks and >> > compiling your own packages, to a greater or lesser extent. That’s a >> > maintenance overhead you want to avoid unless you have to. >> > Having said all that, if you’re looking at deploying thousands of >> > units, the economics of software maintenance may stack up for you. >> > >> > HTH >> > Dariush >> > >> > >> >> On 25 Jun 2018, at 14:32, Alex K <[email protected] >> >> <mailto:[email protected]>> wrote: >> >> >> >> Let me change the posting style... :) >> >> >> >> On Mon, Jun 25, 2018 at 3:51 PM, Dariush >> >> Marsh-Mossadeghi <[email protected] >> >> <mailto:[email protected]>> wrote: >> >> >> >> OK, so we’re moving from bottom-posting to top-posting… >> >> that’ll make it interesting for other readers ;-) >> >> >> >> The output of free doesn’t look desperate, but it is starting >> >> to look a bit tight. >> >> You’ve got about a gig of buffer/cache, which the kernel will >> >> evict if it needs it. >> >> You’ve got 200M of genuinely free memory. >> >> What is potentially a little concerning is the gig of swap in >> >> use, that may or may not be a problem depending on what else >> >> is running on the box and how it’s memory use varies over time. >> >> >> >> I’m not intimatley familiar with the internals of uacctd’s >> >> mysql plugins, but my advice would be: >> >> >> >> - Monitor the memory usage. If it doesn’t vary much over time >> >> you could go on for years and be just fine. Keep a close watch >> >> on swap usage, if that varies a lot or grows over time you’ll >> >> want to do something about it. >> >> >> >> Monitoring of last 2 weeks is showing that free mem is fluctuating >> >> form 50M to 220 MB and swaps remains steady. Thats why I am trying >> >> to find a solution. >> >> >> >> >> >> - Read up on OOM Killer, what it does and how it behaves. If >> >> you see OOM Killer entries in your logs, it’s time to think >> again >> >> >> >> I am aware of that. thanx >> >> >> >> >> >> - Can you put some more memory in the box? 8GB of RAM will >> >> cost you about £60. How much is your time worth ? >> >> >> >> This is not an option. What I have is 4 GB. This is not just a >> >> personal project. There are going to be thousands of such >> >> installations >> >> >> >> HTH >> >> Dariush >> >> >> >> >> >>> On 25 Jun 2018, at 12:32, Alex K <[email protected] >> >>> <mailto:[email protected]>> wrote: >> >>> >> >>> Thanx for the reply. >> >>> >> >>> The output of free is the following: >> >>> >> >>> free >> >>> total used free shared >> >>> buff/cache available >> >>> Mem: 4046572 2832576 152012 784240 >> >>> 1061984 204248 >> >>> Swap: 3906556 1086080 2820476 >> >>> >> >>> While is as below when stopped: >> >>> >> >>> free >> >>> total used free shared >> >>> buff/cache available >> >>> Mem: 4046572 520352 3223884 14916 >> >>> 302336 3286952 >> >>> Swap: 3906556 485040 3421516 >> >>> >> >>> Seems that mysql plugins are reserving quite some memory as they >> >>> list first in htop when sorted with memory. >> >>> >> >>> Thanx, >> >>> Alex >> >>> >> >>> >> >>> On Mon, Jun 25, 2018 at 2:22 PM, Dariush >> >>> Marsh-Mossadeghi <[email protected] >> >>> <mailto:[email protected]>> wrote: >> >>> >> >>> >> >>>> On 25 Jun 2018, at 11:54, Alex K <[email protected] >> >>>> <mailto:[email protected]>> wrote: >> >>>> >> >>>> Hi all, >> >>>> >> >>>> I have a setup with uacctd monitoring traffic of several >> >>>> interfaces through NFLOG. >> >>>> With uacctd stopped I see that the server (a relatively >> >>>> small device with 4 GB of RAM) consumes 450MB of RAM. Once I >> >>>> start uacctd the mem usage goes up to 3.5 GB. I am using >> >>>> mysql plugin and this is running on Debian9 64 bit. >> >>>> >> >>>> Is there any tweaks I can use to put a limit on the memory >> >>>> usage of uacctd. >> >>>> >> >>>> Thanx, >> >>>> Alex >> >>>> _______________________________________________ >> >>>> pmacct-discussion mailing list >> >>>> http://www.pmacct.net/#mailinglists >> >>>> <http://www.pmacct.net/#mailinglists> >> >>> >> >>> >> >>> tldr; post the output of the free command when uacctd is >> >>> running and I’ll do my best to interpret it for you :-) >> >>> >> >>> Is it _really_ using it, or is it just not completely unused? >> >>> >> >>> The linux kernel generally does pretty good job of keeping >> >>> stuff that might be useful in memory, but getting rid of it >> >>> very very quickly if the space is needed for something else. >> >>> The challenge you face is that most of the userspace tools >> >>> come with a long list of caveats about what they appear to >> >>> report vs what’s really happening, this is mainly due to the >> >>> way the kernel shares memory between processes, and its use >> >>> of as much memory as possible for various caches. >> >>> >> >>> Some threads worth reading if you’re not familiar with the >> >>> in’s and out’s of linux kernel memory management... >> >>> >> >>> https://www.linuxatemyram.com/ >> >>> https://stackoverflow.com/questions/4802481/how-to-see-top- >> processes-sorted-by-actual-memory-usage >> >>> <https://stackoverflow.com/questions/4802481/how-to-see-top >> -processes-sorted-by-actual-memory-usage> >> >>> https://stackoverflow.com/questions/3784974/want-to-know- >> whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/ >> >>> <https://stackoverflow.com/questions/3784974/want-to-know- >> whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/> >> >>> >> >>> HTH >> >>> Dariush >> >> >> > >> > >> > _______________________________________________ >> > pmacct-discussion mailing list >> > http://www.pmacct.net/#mailinglists >> > <http://www.pmacct.net/#mailinglists> >> > >> > >> > >> > >> > _______________________________________________ >> > pmacct-discussion mailing list >> > http://www.pmacct.net/#mailinglists >> > >> >> _______________________________________________ >> pmacct-discussion mailing list >> http://www.pmacct.net/#mailinglists >> > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists > > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists >
_______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
