Hi David,
Yes, I had sent you and Urban a patch last week to correct this problem.
This patch had been confirmed yesterday by Urban as working. This fix will
be included in the upcoming 0.98.5 release. Please confirm if possible.
Thanks,
Steve
On Tue, Sep 9, 2014 at 12:54 PM, J. David Rye of Roadtech <
d....@roadtech.co.uk> wrote:
> On Tuesday 02 September 2014 23:12, Steven Morgan wrote:
> > Hi J. David,
> >
> > Thanks for the additional analysis and information. I've been looking at
> > this for a bit today. I have opened a ticket in the ClamAV bugzilla
> system
> > to track the issue. The ticket number is 11089. Hope to have an answer
> > soon.
> >
>
> Have been doing a bit of hacking around with the code.
> Have not been able to get the logging functions built in to libclamd
> functions such as "cli_dbgmsg" to log reliably.
>
> freshcam and clamd appear to use a call back to intercept and route
> their output via their own function "logg". However clamav-milter
> seeems to be missing the necessary bits to do this.
>
> I ended up putting direct calls in to fprintf and fflush.
> problem is definitely in cl_hash_data. As a work round i have patched
> the function cli_md5buff in libclamav/others_common.c to use the
> random data directly to create the file name when the call to
> cl_hash_data fails.
>
> [root@mailhost-c6 BUILD]# diff -Naur
> clamav-0.98.4.orig/libclamav/others_common.c
> clamav-0.98.4.rt/libclamav/others_common.c
> --- clamav-0.98.4.orig/libclamav/others_common.c 2014-05-21
> 16:25:05.000000000 +0100
> +++ clamav-0.98.4.rt/libclamav/others_common.c 2014-09-08
> 19:27:00.458404432
> +0100
> @@ -825,21 +825,41 @@
> {
> unsigned char digest[16];
> char *md5str, *pt;
> + const unsigned char *pt2;
> int i;
>
> - cl_hash_data("md5", buffer, len, digest, NULL);
> + if(!(md5str = (char *) cli_calloc(32 + 1, sizeof(char)))) {
> + cli_dbgmsg("cli_md5buff: out of memory\n");
> + return NULL;
> + }
>
> - if(dig)
> - memcpy(dig, digest, 16);
> + if ( cl_hash_data("md5", buffer, len, digest, NULL) == NULL ){
> + cli_dbgmsg("cli_md5buff: Call to cl_hash_data failed to return
> hash\n");
> + fprintf(stderr, "cli_md5buff:fprint Call to cl_hash_data failed to
> return hash\n");
> +
> +/* In the absence of hash use random data from buffer directly */
> +/* this is a nasty cludge based onthe fact that we are on called from
> cli_gentmp */
> +/* with 16 byes saved from last digest, and 32 bytes of fresh random data
> */
> + pt = md5str;
> + pt2 = buffer + 16 ;
> + for(i = 0; i < 16; i++) {
> + sprintf(pt, "%02x", *pt2);
> + pt += 2;
> + pt2 += 2;
> + }
>
> - if(!(md5str = (char *) cli_calloc(32 + 1, sizeof(char))))
> - return NULL;
> + } else { /* successful hash */
> + fprintf(stderr, "cli_md5buff:fprint using hash returned from
> cl_hash_data\n");
>
> - pt = md5str;
> - for(i = 0; i < 16; i++) {
> + if(dig)
> + memcpy(dig, digest, 16);
> +
> + pt = md5str;
> + for(i = 0; i < 16; i++) {
> sprintf(pt, "%02x", digest[i]);
> pt += 2;
> - }
> + }
> + };
>
> return md5str;
> }
> @@ -889,11 +909,14 @@
>
> if(!tmp) {
> free(name);
> - cli_dbgmsg("cli_gentemp('%s'): out of memory\n", mdir);
> + cli_dbgmsg("cli_gentemp('%s'): error from cli_md5buff\n", mdir);
> return NULL;
> }
>
> snprintf(name, len, "%s"PATHSEP"clamav-%s.tmp", mdir, tmp);
> +/* log all names */
> + cli_dbgmsg("cli_gentemp: filename %s"PATHSEP"clamav-%s.tmp\n",
> mdir,
> tmp);
> + fprintf(stderr, "cli_gentemp: filename %s"PATHSEP"clamav-%s.tmp\n",
> mdir, tmp);
> free(tmp);
>
> return(name);
> @@ -915,7 +938,9 @@
> free(*name);
> *name = NULL;
> return CL_ECREAT;
> - }
> + } else {
> + cli_errmsg("cli_gentempfd: Createed temporary file %s: %i\n",
> *name,
> *fd);
> + };
>
> return CL_SUCCESS;
> }
> [root@mailhost-c6 BUILD]#
>
> Diagnostic output from my current version below.
> Note it is the call to EVP_get_digestbyname from cli_hash_data that fails.
>
> Reading the manual page on EVP_get_digestbyname, it will only work if
> OpenSSL_add_all_digests() or OpenSSL_add_all_algorithms() has been called
> first.
>
> It looks like these are only called from cl_initialize_crypto().
>
> Snag as far as I can see cl_initialize_crypto is not called from
> clamav-milter
>
>
> LibClamAV Info: cli_hash_data: Called
> LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
> cli_md5buff:fprint Call to cl_hash_data failed to return hash
> cli_gentemp: filename /tmp/clamav-cc4ad42fc23e93aaf82a1acc428307d9.tmp
> LibClamAV Error: cli_gentempfd: Createed temporary
> file /tmp/clamav-cc4ad42fc23e93aaf82a1acc428307d9.tmp: 11
> LibClamAV Info: cli_hash_data: Called
> LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
> cli_md5buff:fprint Call to cl_hash_data failed to return hash
> cli_gentemp: filename /tmp/clamav-6296cff0b0f8e0d462e2a82ede5885ac.tmp
> LibClamAV Error: cli_gentempfd: Createed temporary
> file /tmp/clamav-6296cff0b0f8e0d462e2a82ede5885ac.tmp: 13
> LibClamAV Info: cli_hash_data: Called
> LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
> cli_md5buff:fprint Call to cl_hash_data failed to return hash
> cli_gentemp: filename /tmp/clamav-40f9e43c4dc6f6213d604a6c642bb738.tmp
> LibClamAV Error: cli_gentempfd: Createed temporary
> file /tmp/clamav-40f9e43c4dc6f6213d604a6c642bb738.tmp: 11
> LibClamAV Info: cli_hash_data: Called
> LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
> cli_md5buff:fprint Call to cl_hash_data failed to return hash
> cli_gentemp: filename /tmp/clamav-45633b0b40207939c2acdc5d752f0862.tmp
> LibClamAV Error: cli_gentempfd: Createed temporary
> file /tmp/clamav-45633b0b40207939c2acdc5d752f0862.tmp: 12
> LibClamAV Info: cli_hash_data: Called
> LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
> cli_md5buff:fprint Call to cl_hash_data failed to return hash
> cli_gentemp: filename /tmp/clamav-876a73b889eb4308c4773efa6818933d.tmp
> LibClamAV Error: cli_gentempfd: Createed temporary
> file /tmp/clamav-876a73b889eb4308c4773efa6818933d.tmp: 12
> LibClamAV Info: cli_hash_data: Called
> LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
> cli_md5buff:fprint Call to cl_hash_data failed to return hash
> cli_gentemp: filename /tmp/clamav-6ed0e7970718b603f4125c9c920aecee.tmp
> LibClamAV Error: cli_gentempfd: Createed temporary
> file /tmp/clamav-6ed0e7970718b603f4125c9c920aecee.tmp: 11
> LibClamAV Info: cli_hash_data: Called
> LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
> cli_md5buff:fprint Call to cl_hash_data failed to return hash
> cli_gentemp: filename /tmp/clamav-0e1024e48ddf5d9245ec42cc366bba55.tmp
> LibClamAV Error: cli_gentempfd: Createed temporary
> file /tmp/clamav-0e1024e48ddf5d9245ec42cc366bba55.tmp: 12
>
>
>
> > Steve
> >
> > On Sun, Aug 31, 2014 at 5:52 AM, J. David Rye <d....@roadtech.co.uk>
> wrote:
> > > On Thu, 2014-08-21 at 19:22 -0400, Steven Morgan wrote:
> > > > Hi Urban,
> > > >
> > > > I took a look at this code. The real problem is the inability to
> > > > create a
> > > > temporary file. The second message just results from the return code
> > > > of the
> > > > function that attempts to create the temp file. We need to find out
> > > > why the
> > > > temp file creation fails. There should also be a clamav error message
> > > > written from: cli_errmsg("cli_gentempfd: Can't create temporary file
> > > > %s:
> > > > %s\n", *name, strerror(errno)); Can you find this message?
> > > >
> > > > Otherwise, it is a memory allocation failure for space for the temp
> > > > file
> > > > name, which seems unlikely.
> > > >
> > > > Steve
> > >
> > > I am also seeing this issue. Mostly intermitant but see further down.
> > >
> > > cli_errmsg wont work if clamav-milter has daemonezed.
> > > it only writes to STDERR and the function daemonize closes standard
> > > error even if you recompile with CL_DEBUG set.
> > >
> > > Only way to get is to get the error messages from cli_gentempfd seams
> > > to be to uncomment the line
> > >
> > > "#Foreground yes"
> > >
> > > In clamav-milter.conf, then run in foreground from command line.
> > >
> > > As an aside I wonder why cli_gentempfd does not use the function logg()
> > > and output to file or syslog depending on configuration file.
> > >
> > > I am running clamav-milter on a VM. OS is CentoOS 6.5
> > > VM has 4 vcpu, and 2GB RAM
> > > clamav-milter is version 0.98.4-1.el6.rf installed from rpmforge
> > > repository.
> > >
> > > Looking at he logs if time stamps in syslog for calls to clamav-milter
> > > are two seconds or more apart the problem never shows.
> > >
> > > However if 4 or more messages arrive in two seconds problem always
> shows
> > > up, the failure to create temp file is usually time stamped 2 seconds
> > > after the first message in the burst that triggered it.
> > >
> > > On a sustained burst of traffic pretty much all the messages trip the
> > > issue.
> > > In a 1 hour period last week when I had a lot of messages due to a
> > > different issue. I had 20,000 temp file failures, and 23 messages
> > > delivered.
> > >
> > > [root@mailhost-c6 etc]# clamav-milter
> > > --config-file=/etc/clamav-milter.conf.foreground
> > > Local socket unix:/var/run/clamav/clamd.sock added to the pool (slot 1)
> > > Probe for slot 1 returned: success
> > > LibClamAV Error: cli_gentempfd: Can't create temporary
> > > file /tmp/clamav-0000000000000000626683ff3a000000.tmp: File exists
> > > ERROR: Failed to create temporary file
> > > ERROR: Failed to initiate streaming/fdpassing
> > > LibClamAV Error: cli_gentempfd: Can't create temporary
> > > file /tmp/clamav-0000000000000000626683ff3a000000.tmp: File exists
> > > ERROR: Failed to create temporary file
> > > ERROR: Failed to initiate streaming/fdpassing
> > > LibClamAV Error: cli_gentempfd: Can't create temporary
> > > file /tmp/clamav-0000000000000000626683ff3a000000.tmp: File exists
> > > ERROR: Failed to create temporary file
> > > ERROR: Failed to initiate streaming/fdpassing
> > > LibClamAV Error: cli_gentempfd: Can't create temporary
> > > file /tmp/clamav-0000000000000000626683ff3a000000.tmp: File exists
> > > ERROR: Failed to create temporary file
> > > ERROR: Failed to initiate streaming/fdpassing
> > > LibClamAV Error: cli_gentempfd: Can't create temporary
> > > file /tmp/clamav-0000000000000000626683ff3a000000.tmp: File exists
> > > ERROR: Failed to create temporary file
> > > ERROR: Failed to initiate streaming/fdpassing
> > > Message from <n0r3ply812...@scotland117.wanadoo.co.uk> to
> > > <stevensonbros> infected by Heuristics.Phishing.Email.SpoofedDomain
> > > Message from <n0r3ply620...@aughamullan.dungannon.ni.sch.uk> to
> > > <brett01> infected by Heuristics.Phishing.Email.SpoofedDomain
> > > Probe for slot 1 returned: success
> > >
> > >
> > > I think
> > >
> > > clamav-milter does a lot of initialization, including setting up a
> > > structure with a list of function entry points it then calls smfi_main.
> > >
> > > smfi_main in turn forks one thread for each message, and calls the
> entry
> > > points in the context of the thread.
> > >
> > > call back to function clamfi_header
> > > that calls sendchunk
> > > which calls nc_connect_rand
> > > which calls cli_gentempfd which prints the EEXISTS errors to stderr
> > > shown above.
> > >
> > > File name looks like it is supposed to be based on a 16 byte MD5 digest
> > > printed in hex.
> > >
> > > cli_gentempfd builds the name by calling cli_gentemp
> > > which adds 32 bytes of random data from cli_rndnum to a 16 byte
> > > seed
> > > passes a pointer the the 48 buffer to cli_md5buff
> > > which in turn calls cl_hash_data
> > > digest returned is used to create the file name,
> > > and update the seed for the next name.
> > >
> > >
> > > Note cli_gentemp is common to all programs in the clam set.
> > >
> > > Note cl_hash_data is calling openssl library functions to calculate the
> > > digest.
> > >
> > > Note cli_md5buff does not check for an error in cl_hash_data
> > >
> > > _______________________________________________
> > > Help us build a comprehensive ClamAV guide:
> > > https://github.com/vrtadmin/clamav-faq
> > > http://www.clamav.net/support/ml
> >
> > _______________________________________________
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> > http://www.clamav.net/support/ml
>
_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
