Victor Duchovni:
> On Tue, Jan 18, 2011 at 07:06:29AM -0500, Wietse Venema wrote:
>
> > > > There are many ways to arrive at a moving average. Where do these
> > > > magic numbers come from?
> > >
> > > The 0.05, 0.95 magic numbers give you an average over a few tens of sample
> > > points, that is not overly sensitive to a single spike, and purge stale
> > > state reasonably quickly. TCP alpha smoothing uses 0.1 IIRC, I found
> > > this to be a bit too fast.
> >
> > Taking 100ms as the threshold, this would limit postscreen to 10
> > database operations/second. A moving average over 20 samples would
> > cover a few seconds of traffic. This would be sufficient to get
> > rid of the false alarms that are now logged as isolated events.
>
> An "alpha" value of 0.05 gives you effectively a sample size of O(20),
> without having to store the past 19 values with the average to compute
> updates. This is an exponentially decaying weighted moving average,
> 0.95^20 ~ 1/e, so approximately 1/3 of the moving average is from sample
> points 20 or more in the past.
You are describing basic math to someone with a master's degree in
roughly 50% math and 50% physics :-) (My PhD was physics only).
When I wrote the SATAN security scanner's UDP port scanner in 1995,
I smoothed the latency measurements with this simple routine:
double average(new, old)
double new;
double old;
{
if (new > old) { /* quick rise */
return ((new + old) / 2);
} else { /* slow decay */
return (0.1 * new + 0.9 * old);
}
}
This is basically the same averager that you describe, invented
independently, with a quick-rise feature added for something that
is not applicable for postscreen.
With postscreen there are multiple tables (the permanent access
table, the temporary whitelisting table, and perhaps other tables).
Each table requires its own latency average, but all tables can
share the "don't warn" timer to avoid spamming the logfile when
the entire system becomes slow.
Wietse
