On Wed, Apr 29, 2009 at 7:56 PM, Adam Katz <antis...@khopis.com> wrote: >> I guess it depends what you mean by "enormous". A sought rule update is >> 135k. > > And 135k doesn't add up to a lot of bandwidth? I suppose it depends > on the number of users, and I'm figuring worst-case scenario, e.g. > when/if it ships enabled in the default SA install.
Well, it depends what you're measuring. :) The update itself isn't large, it's just 135k, which is the not "enormous" bit. 135k in and of itself is a pretty tiny file, but I'm not sure what "enormous" means in this context -- megs? gigs? The aggregate bandwidth could very well be large, depending on update publish frequency, client update frequency, number of clients, client bandwidth, etc. From what I've seen, the standard SA updates w/ the same ~130k size and the current number of users ... isn't a lot of bandwidth. There are some pretty standard ways to deal with this issue though, such as: a) have lots of mirrors, same idea as your P2P idea though less dynamic (oh, that was another thought I had ... go short of using torrents since they're resource heavy and instead make our own P2P protocol doing a dynamic http/mirrored.by system) b) split the channel into a frequent / not frequent channel (or stable / testing, or split based on content, or ...) for patterns which don't change often, there's no reason to keep sending them out. same idea I mentioned before. c) shrink or hold update size steady in face of updates. hard. d) make updates less frequently. defeats the purpose? clearly every 15m is different than every day is different than weekly ... To be perfectly honest, I really don't worry about the "omg, update bandwidth" issue right now. I worry that there aren't enough updates right now. The only auto-generated one, sought, is daily, and the manual ones now are more than weekly on average. I don't know if sought could even be produced faster, you need a certain amount of incoming ham and spam to sample and produce test rules, and enough diversity of mails to test against to avoid "obvious" bad rules...
