On Mon 13 Nov 2006 08:57, Chris Wilson wrote: > Hi Peter, Paolo and all, > > On Sat, 11 Nov 2006, Peter Nixon wrote: > > This is something I have discussed with Paolo on a number of occasions > > also. I see no practical reason why pmacct cannot store all flows with > > src and dest port and ip for a low speed link (<10Mbit) yet 256K of > > traffic manages to overwhelm an opteron DB server. > > Ouch, I didn't know that the performance was that bad, thanks for the > warning. Do you mean 256 kbits or kbytes? For what it's worth I have > pmacct monitoring an 8mbit/384kbit link which is often full, on a Celeron > 366, and it "normally" runs OK, but it has brought down that box on one > occasion due to the number of database threads spawned, and the resulting > system load.
I mean 256kbps (< 30KB/sec). As I said this is when running a config like the following: aggregate[all]: proto,src_host,dst_host,src_port,dst_port Now, in my case the src side is a single class B so its not quite as bad as it looks either. This is an ambitious setting I know, but pmacct should be the one blowing up, NOT the database. (At least there should be a config option to force the load onto pmacct instead of the database) Actually to be honest I am using postgresql, and I have not actually seen it die, but it does get sooo many sockets open (500+) that things run so slowly that pmacct starts to give up on the db handles (And postgres obviously starts doing deadlock detection which makes it even slower).. it pmacct hit it with a single thread it would be committing is sub 100ms range instead of >10 sec range and the problem would basically disapear.. > > This is purely because of the WAY that pmacct access the database, not > > the amount of data (At least from the testing I have done), but it would > > require a change to the way DB access is handled to fix. Basically any > > query over threadlimit (which should default to 2 or 3) should be queued > > instead of dropped...This obviously requires someone write (or borrow) a > > decent queuing system for db access.. > > I don't think it's as hard as all that. The OS will happily queue UDP and > pcap packets for us while we write to the database. So I'd just drop the > separate thread entirely, and write to the database in the main thread. > Under normal conditions this will not cause any problems at all, and it > will degrade gracefully under load, unlike the current situation. OK. Sounds like its worth a try. I have since moved to a pretag config as the only way to keep my system reliable and get the data I want, but if someone writes the code I will test it with the old config. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
