Tomasz, thanks for your detailed response. I'll follow-up off-list with
some of the other sig writers and see what their thoughts are on bug #781.
Best regards,
Bill
> On Tue, 30 Jun 2009 11:26:25 -0700
> "Bill Landry" <b...@inetmsg.com> wrote:
>
>> So if I were to include a signature in my 3rd party database, and then a
>> few days later ClamAV adds the same signature to the official signature
>> database, that is not your problem, but rather my problem? Seems like
>> if
>> you (ClamAV) is providing the means for including 3rd party databases,
>> then wouldn't you agree that it really is ClamAV's responsibility to
>> make
>> sure that duplicate signatures do not get loaded and used?
>
> Hi Bill,
>
> taking care about duplicates in the engine doesn't make sense (see below).
> Without a centralized system for signature maintenance we offered to 3rd
> parties, it's not possible to avoid duplicates. Having said that, even if
> there
> were a few thousands of duplicated sigs, it shouldn't cause any
> significant
> slowdown to the engine.
>
>> > We had an idea to allow 3rd party signature
>> > creators to use our mechanisms for signature maintenance ([1], easy
>> > checking for FPs, dups, name collisions) and also our network
>> > infrastructure and freshclam to make everything more smooth but
>> > unfortunately this idea didn't get much interest.
>>
>> Hmmm, first I've heard of this. Why was there a lack of interest?
>
> Well, I don't know why.. AFAIK, only Securiteinfo was interested in using
> that solution. And in my opinion it would only have advantages - all the
> mechanisms we developed for the last 7 years, including the mirror
> infrastructure, could be used to maintain and distribute the 3rd party
> sigs making all processes much more efficient!
>
>> > It would be inefficient (and could be even unsafe in some cases) to do
>> > such things in the engine.
>>
>> Why is that? If ClamAV sorts all signatures when reloading, and ignores
>> duplicate signatures, why would that be dangerous in the engine?
>
> Because detecting duplicated signatures is not that easy and must be
> done with a great care so that we don't incorrectly skip some unique sigs!
>
> Eg. the following logical sigs are all duplicates:
>
> Sig1;Target:0;0&1&(2|3);dead;beef;feed;face
> Sig2;Target:0;0&((1&2)|(1&3));dead;beef;feed;face
> Sig3;Target:0;0&1&(2|3);dead;beef;face;feed
> Sig4;Target:0;(0|1)&2&3;feed;face;dead;beef
>
> but this one is not (and still is very similar):
>
> Sig5;Target:0;(0|1)&2&3;feed;dead;face;beef
>
> Even for some very simple hex signatures there may be cases where
> it's not easy to detect dups, eg. dead{3}beef is in practice a duplicate
> of dead??????beef but since the engine handles these signatures
> differently, the situation complicates again. So in the engine we could
> only implement some very limited checks, but then the other day
> someone would open a bug report that this "feature" doesn't work
> nicely for some sigs... (take the issue with local.ign for example)
>
> The centralized system for signature development eliminates the
> problem because one can easily see that a sample is already detected
> (such samples automatically get "closed"). It could also provide some
> detection of duplicates which could be later handled manually. It's
> working really great for us that's why we made that offer to 3rd party
> signature developers. Hopefully, we will close the bug #781 some day...
>
> But as I said, duplicated sigs don't make much harm after all.
>
> Regards,
> Tomasz
>
> --
> oo ..... Tomasz Kojm <tk...@clamav.net>
> (\/)\......... http://www.ClamAV.net/gpg/tkojm.gpg
> \..........._ 0DCA5A08407D5288279DB43454822DC8985A444B
> //\ /\ Tue Jun 30 23:03:52 CEST 2009
> _______________________________________________
> Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
> http://www.clamav.net/support/ml
>
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml
