Hi Paolo, On Sat, 13 Jun 2009, Paolo Lucente wrote:
> > minb = 10000, zero_dstip, minb = 10000, zero_dstport, minb = 10000, > > zero_srcport, minb = 10000, zero_srcip > > > > Then any flows which together do not add up to enough bytes to pass > > the minb filters, even after aggregation, end up in a record where all > > the selector fields are zeroed out. Since there is no final minb > > condition, this row would always be added to the database, never > > rejected, so SUM(bytes) would again equal the interface counters for > > any given time range. > > I explored this valid approach some time ago (years!); by zeroing some > aggregation primitives previously selected, duplicates are likely to be > created. The trick is to "resolve" such duplicates before offering them > to the SQL database - via a sub-aggregation operation. The cache is not > sorted - making any sub-aggregation operation very expensive (scaling > linearly with the number of aggregated being offered); the idea here is > to index the cache, perform the sub-aggregation and offer the result of > this to the SQL database. I agree that merging duplicate records would produce the most useful results for us. > In summary, it's not something quick to do but it can be done - maybe > something good for inclusion within the 0.12 trunk later in the year. At > this stage, this feature can't be included in the first pre-release > version (0.12.0p1) but I can plan it along the rocky way to the first > official release, 0.12.0. Maybe already in 0.12.0p2. How does it sound? That sounds great! I was not expecting you to offer to implement it so quickly. I understand that it's difficult and may conflict with your other priorities. > Let me spend a couple of words on a different aspect: the above approach > implies everything ends in the same SQL table - which can have pros and > cons; the pro is simplicity (one table for everything); the con is that > might want to have sub-aggregated data clearly separated into a > different table to, say, apply different policies. This is something can > be done today with pmacct as 'sql_preprocess' offers also the "max" > version of the "min" features you are using. It means having, for > example, two SQL plugins, writing to different SQL tables, aggregating > data differently and using complementary sql_preprocess features (so > that at the end by summing data in both tables one ends with the full > picture). Would this be a feasible approach to you? We are only interested in a single table. We can show "0.0.0.0" as "Aggregated out" in the pmGraph user interface. I'd rather that we didn't have to query five separate tables to get the results at different levels of aggregation, and merge them all together in our code. However I can see that some people would prefer to keep them in separate tables. Cheers, Chris. -- Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES Aptivate is a not-for-profit company registered in England and Wales with company number 04980791. _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
