Hi Please excuse the long composite response:
On 30/05/12 12:17, G.W. Haywood wrote: > On Wed, 30 May 2012, Cedric Knight wrote: > >> What I'm looking for is a way to avoid having to report new malware >> variants so frequently. > > You need iptables, a long GreetPause, a multi-line 'HELO' greeting, > greylisting, and maybe a few milters. Drop connections with dodgy > SMTP greetings, mail from weird senders and any sender from freemail > domains that you've never heard of. Firewall the IPs for a few days. ... All good advice, thank you, but I think you misunderstand me. I *do* catch a fair amount of it using custom regexes, but am trying to understand why AV detection - by all AVs, not just ClamAV - is apparently so patchy for the type of malware that gets through ClamAV scanning of email. Typically detection is by 4 of 40 scanners listed in VirusTotal after 3 hours (eg https://www.virustotal.com/file/92281ac5b621fd5329b183b5496173a85c577aaa7f0c2ee0673c883c8a80c8f8/analysis/). I'm not sure how often VT updates its sigs. After a week, the same sample is typically caught by half the scanners, after 3 weeks most of them (although sometimes not ClamAV, eg https://www.virustotal.com/file/e4d9b77dc0c6a90f27442a8a817593aced65e274cfecdedff1ffb4a6f118ff49/analysis/), but by then several new variants have been built. So in a way, my own detection doesn't worry me because it uses unpublished, albeit crude, techniques; the concern that prompted this thread is that McAfee and F-Prot and Avast and ClamAV is doing as much as it can to protect other internet users in the way one idealistically hopes for. The type of malware I'm talking about gets through greylisting and most RBLs miss it and GreetPause. This is largely because part of the malware "lifecycle" is a script on a compromised server to send out the stuff through an ordinary mail queue; the botnets I think are used to compromise FTP and other accounts rather than send mail directly. (Personally I don't like that GreetPause because it ties up more smtp daemons at sending end than necessary.) I'd like to automate my sample submissions at some point, but it's work and I would assume my spamtraps have a much lower coverage than most RBLs. >> Wouldn't more spamtraps that feed virus samples directly to AV >> analysts help? > > Are you new to this? :) Just think about the numbers involved. For > every person trying to stop spam and malicious mail there are a couple > of thousand trying to make a fast buck. Those 'analysts' are doing all > they can; they're doing a tremendous job but on its own it isn't enough > and, while ever people are the way they are, it never will be enough. I agree, and would like some idea of what the numbers are, to get some understanding about how current evasion techniques work. > When you're a mail administrator you're always on the back foot, and > nobody loves you. You have to learn to live with that. :) Sure :). Maybe remind them on Sysadmin day 27 July. On 31/05/12 00:29, Jason Haar wrote: > Sounds like a good idea to me - I know most AV companies do just that so > I'm sort of surprised that ClamAV isn't (is that true?) I definitely don't want to cast any aspersions on ClamAV and the excellent work the team does. I'm just wondering why, 6 hours after I've caught several samples of the same malware, that malware apparently hasn't been seen by Kaspersky or Sophos or ClamAV. > > Tell you what - could this be turned into a "community" project? What if > instead of ClamAV staff running spampots, all of us out there running > edge mail protection ran (hourly) cronjobs to parse our quarantines for > spam that contained Window binaries, run clamav over it (probably for a > 2nd time) just in case it's already been picked up - and then feed new > files back to Clamav.net for further analysis? Really quite doable at > the "client" end - but could cause some serious load on Clamav.net if it > was popular... We'd have to have individual accounts so that dickheads > uploading notepad.exe can be tracked and blocked, maybe a "must be seen > by >10 clients" rule needs to be in place to reduce the FPs too, but of > course they'd need to be reviewed by someone eventually. Could we even > cross-check checksums against (say) virustotal.com in an automated > fashion so that only files marked as malware by another product end up > in the final human-facing queue? It's uncommon to see 0/42 on VirusTotal, but that happens too, at least for a few hours. I like Jason's thinking-through of this, and guess it would help ClamAV gain from other AVs. We would want to avoid submitting known good files that would trigger FPs of course; but also ideally avoid submitting submitting the same file twice. Are there (a) any facilities for checking whether a given sha256sum has already been submitted; and (b) standard ways of removing variable/polymorphic sections? Or is that sensitive information or better done at ClamAV? > > I'm sure ClamAV staff would like a "too large" corpus of malware than > "too little"? At the moment the ClamAV submission page asks for at most two a day (or to get in touch for special arrangement). I respect that, but send things to other AV vendors in any case, trusting that samples do get shared. On 31/05/12 01:59, Matt Watchinski wrote: > We have spamtraps and tons of other collection mechanisms, which bring > in a little over 100k samples a day. While obviously not perfect, as > pointed out in the beginning of this thread, is pretty good. If > anyone is interested in automating up sample submission of stuff we > missed like Jason is suggesting, please feel free to contact me > offlist and I'll provide automated ways for sending us samples. That's good to hear. How many new sigs is that per day? On 31/05/12 04:00, Jason Haar wrote: > I'll take a stab. I'm the author of Qmail-Scanner, so I could naturally > create a script that feeds such data back to you guys Great. > Glad to hear you were already doing this - it did seem obvious. I bet > you're main problem is not having enough staff to actually check the new > stuff? (downside to open source ;-) As I say, this isn't just a problem with ClamAV. Maybe Sourcefire will be able to put more resources in and develop better automated analysis? On 31/05/12 12:59, G.W. Haywood wrote: > Hi there, > Perhaps you're right. Apologies, it had been a bad day (and this is > another one). No offence taken at all. I think it's useful to compare notes, although our scenarios may be rather different, where I have no influence over how end users may choose to access email or what they might use it for. > But it really seems to me like this is (a) making rods > for your own backs and (b) addressing a non-problem. Executables in > mail are the least of my problems. The vast majority of malicious > executables that I see reach their target get in through other means, > mostly by browsing. In my experience, far more links to malicious > executables appear in mail than do the actual executables themselves. This was certainly true with the Storm/Dorf botnet, but I think the pendulum may have swung back. I'm not sure how easy it is to quantify the ratio exactly. [snip CNET] > But look at it another way then: there's an inexhaustible supply of > malware, and a relatively limited set of useful executable stuff. If > you try the 'accept everything that is not bad' approach, you *will* > be exhausted and you *will* miss something which *will* cause trouble. Yes, but there are lots of new unannounced builds of useful executable stuff, and I can't see a cleanlist working. > If you're wedded to the idea of permitting executables in mail (which > I personally think is insane if you have any Windows boxes inside your > firewalled network) then does it not make sense to simply get in touch > with the presumably relatively small number of people who are likely > to be sending these executables as and when they want to send them, > and arrange something simple (such as a particular 'Subject:' line) > which will flag to your mail filters that this message is NOT to go > into the bit-bucket with all the other executables, but into some > quarantine place where a human can give it a once over? > > It will be a lot less work. I've tried rejecting all zipped EXEs in zipext.zmd and had complaints within a few hours from legitimate senders, despite the fact that Gmail also has such a rejection policy. Perhaps I should have kept the ban and cleanlisted the senders or recipients, but with amavis I think that requires a separate instance of ClamAV (you don't want to put them in virus_lovers). > > You might be asking yourself "Then why does he use ClamAV?", which is > a perfectly fair question. I use it for all these sigs: [snip clamav-unofficial-sigs] > Some of them 'kick serious butt' as I think someone on here put it a > while ago. But if anything looks even vaguely like an executable it > doesn;'t get scanned by ClamAV here. It gets a 5.7.1. and an iptables > entry all of its own. I can't remember the last time a virus actually > got as far as being scanned by one of my ClamAV daemons. Malware links are a different issue, and I think can some of the .ndb files you mention have sufficiently high false-positive rates that you might not want to use them unless that possibility is made clear to users. > > Finally there's an alternative, that is to ditch all the Windows boxes. > My experience of doing that is, let's say, patchy. But I'm working on > it, one step at a time. The first step was easy, there are no Windows > boxes in my own business network. The second step hasn't been so easy > and I've been on the journey for over ten years. It's taken that long > to replace two out of about thirty Windows machines in one customer's > business, but the two Directors using those machines now haven't had a > virus on their machines since they started using Linux. Previously it > was approaching one a week. A good argument for a non-standard OS, but also perhaps also reminding us that AVs aren't perfect and user education is important? All best wishes Cedric Knight _______________________________________________ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
