You seem to be running into two main problems:
1) Your mail traffic is extremely bursty (it all comes at once!).
2) Your MTA doesn't seem to do anything to limit resource consumption. (I'm
not familiar with exim as to whether this is an option or not -- by default,
sendmail will just queue mail once load goes over 12, and 4xx mail with load
over 25).
Some suggestions:
1) Run fetchmail as a daemon (or cron) throughout the day. You don't say
whether or not you're on dialup, but if you have a persistent connection, this
is your best bet for smoothing out your mail flow.
2) fetchmail's 'fetchlimit' option. man fetchmail for more details.
3) As you mentioned, spamc/spamd. It makes a huge difference as spawning a new
perl interpreter for each message is a huge resource hit.
4) Your setup would be just fine removing the MTA altogether. Use
fetchmail's 'mta' option and point it directly at procmail. As fetchmail
executes the "mta" synchronously, you will only have one copy of procmail and
one copy of spamassassin running at a time. No need to get a real MTA involved
at all.
-- Jeff
Quoting Jan Korger <[EMAIL PROTECTED]>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi,
> I installed SA on Debian woody to filter my POP3 email. Therefore
> fetchmail is used, forwarding fetched mail to localhost:smtp. Thus, exim
> will be run through inetd. Exim will then find .procmailrc in my home
> directory and run procmail. Procmail will (besides some very basic
> filtering) invoke SA and pgpenvelope_decrypt (for verifying PGP sigs with
> gpg) on any message received.
>
> I understand that this is not an optimal setup for receiving huge amounts
> of mail, but
> a) my mail traffic is very low (100/day) -- just that all of them will be
> received after booting the machine all at once
> b) this is the default setup for debian *stable* and what everyone will
> try first
> Please do always consider people using SA cannot know SA is likely to
> overload their machine and noone has any control over the amount of email
> received. What I want to tell you is that distributing SA as it is now
> causes a lot of trouble and should be discontinued -- even though SA is a
> great tool. It's too dangerous. (The warning that SA can overload a
> machine with lots of mail traffic is not good enough as ppl such as me
> don't consider their personal email traffic a lot and ping (through a dDoS
> attack) can also overload a machine/network, but most ppl accept pings)
>
> So, what happened?
> Well, I started my machine after ~20h of not using it nor reading mail
> through any other box. I'm starting pine on one VT, top on another (too
> watch SA working). Starting pine works (no new messages), top only gives
> me one line with uptime and "average load 74.74 35.?? ??.??" Seconds later
> the screen gets cluttered up with messages like
>
> Oct 25 21:15:33 jan kernel: Out of Memory: Killed process 1224 (mysqld).
> Oct 25 21:15:33 jan kernel: Out of Memory: Killed process 1224 (mysqld).
> Oct 25 21:15:39 jan kernel: Out of Memory: Killed process 409
> (spamassassin).
> Oct 25 21:19:11 jan kernel: Out of Memory: Killed process 1219 (apache).
> Oct 25 21:19:24 jan kernel: Out of Memory: Killed process 1219 (apache).
>
> I don't know who writes them there. There is no "/dev/console" in
> /etc/syslog.conf. They are supposed to be logged to /dev/tty12, which
> works.
> Thus, I tried stopping apache, mysqld, fetchmail and inetd. Otherwise it
> would have been impossible to use the machine. (OK, I could have waited,
> but I had waited and it seemed forever and I wanted to stop that strange
> automated killing of processes. I didn't know linux would do that.) That
> didn't not help at once but as more and more SA/procmail/exim processes
> termitated you use the box and finally everything seemed fine. But with
> only 20 messages in INBOX plus 5-10 new messages in my incomming spam
> folder. But according to syslog fetchmailed received and forwarded 60
> messages from my main account plus some from the other mailboxes. ==> A
> minimum of 40 _important_ messages lost. That is quite an issue to me as
> ppl expect me to read my email.
>
> fetchmail flushed the messages on the POP3 server. Is there any chance in
> restoring the lost mail or finding out _who_ emailed me? I guess they got
> lost by this strange "killing processes" "feature" of linux as fetchmail
> does only flush messages successfully forwarded. This "feature" is not
> your fault but you need to take it into consideration.
>
> I understand that I can significantly reduce the load by using spamc/spamd
> but for small sites (especially single user workstations) that kind of a
> setup is sub-optimal (and BTW not recommend by any SA documentation) as it
> requires an additional deamon to be running and waste (virtual) memory,
> which won't have a lot to do. Besides, running spamc will probably cause
> the same problems as spamassassin on a different scale, which is requiring
> to fork/exec on any message received and many messages being processed at
> the same time possibly using up all of (virtual) memory.
>
> Using nice priorities does not help at all, I did run procmail (nice -n 5)
> and fetchmail (6) nicely.
>
> I suggest for the short-term, _everyone_ should be using spamd, running
> spamassassin directly form procmail should not be possible/recommend for
> new user who didn't study SA and their system VERY carefully before
> trying to do so.
>
> Some further thoughts on running SA:
>
> 1. Is there a way to limit the number of messages processed by an MTA or
> fetchmail. (This is not what you want on a mail hub but this is exactly
> what you want on a machine which's main purpose is NOT mail processing)
>
> 2. Limit the number of SA processes (and therefore the number of mails
> processed) to 1 [or very low]. This could be done by writing any mail to a
> named pipe (or maybe regular file mailbox; called "to_be_checked") and
> make sure one instance of a program (called "spamd2") runs. This could be
> achieved by filtering a carbon copy of any mail to another program
> ("spamc2") which would check whether spamd2 was running and start if not.
> Spamd2 would then read the to_be_checked and process any message through
> SA, wait for SA to terminate and write to final destination mailbox (or
> send to the user if procmail will only call spamc2 for mails with no
> X-Spam.... stuff in the header.
>
> 3. spamd can be configured to limit the number of messages processed at a
> time (GOOD) but only a certain number of messages can be queued (VERY BAD)
> due to some system restrictions concerning sockets. Think of a different
> way of queing?
>
>
> Jan
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.0.6 (GNU/Linux)
> Comment: pgpenvelope 2.10.2 - http://pgpenvelope.sourceforge.net/
>
> iD8DBQE9ua1MY6Nk2Nv6ZRcRAk0EAJ9wOtZv9G0F1ATIHu2+FgPBsHXRIQCeIJsJ
> Hiddo983kDAfxCYhiwyHfr8=
> =fPT1
> -----END PGP SIGNATURE-----
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by: Influence the future
> of Java(TM) technology. Join the Java Community
> Process(SM) (JCP(SM)) program now.
> http://ads.sourceforge.net/cgi-bin/redirect.pl?sunm0004en
> _______________________________________________
> Spamassassin-talk mailing list
> [EMAIL PROTECTED]
> https://lists.sourceforge.net/lists/listinfo/spamassassin-talk
>
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
Spamassassin-talk mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/spamassassin-talk
