look for ISOQLOG nice
----- Original Message -----
From: pop corn <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Thursday, June 28, 2001 10:00 AM
Subject: Integrating the logs
> I have my different daemons logging into the various log subdirectories
via
> multilog.
>
> My problem now is integrating them so that I have a continuous line of
> activity from the beginning to end for a given email.
>
> For example, I can do a "tail -f current" log for qmail-pop3 while running
> tests. However, I would like to know what related activities are occurring
> in other logs for this same email test.
>
> I have pulled the following info about qmail-analog from the following
> length thread in the archives. It includes an example script. I cut/paste
> quickly, so not everyone gets the credit they deserve for their posts in
> this thread. I have at minimum two questions after reading all of the info
> below:
>
> 1) what are all the z... files in the example script?
> for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids zrxdelay;
>
> 2) where is a real working example of qmail-mrtg?
>
> ==========
>
> I want to know how many messages were sent/failed etc. for a given period
of
> time (say the last three days).
> I have done the following in both /var/log/qmail/qmail-send and
> /var/log/qmail/qmail-smtpd (I'll admit my ignorance and say that I don't
> know the difference between the two. Is qmail-send local deliveries and
> qmail-smtpd remote deliveries?):
> 1) Ran "matchup" on /var/log/qmail/qmail-send(smtpd)/current
> 2) Converted the "matchedup" version of "current" into human readable
> format using tai64nlocal
> 3) Pulled out dates for which I want to see log results from the file
> created above
> 4) Convert the data above to tai64 format using tai64n
> 5) Ran this data through zoverall to see qmailanalog results
> Regardless of whether I run it against /var/log/qmail/qmail-send or
> /var/log/qmail/qmail-smtpd I get the following:
> ++++++++++++
> Completed messages: 0
> Total delivery attempts: 0
> ++++++++++++
> Am I anywhere near doing this right?
> ++++++++++++
> Here are my actual commands
> 1) cat /var/log/qmail/qmail-smtpd/current |
> /usr/local/qmailanalog/bin/matchup > /var/log/qmail/qmail-smtpd/matchedup
> 2) cat /var/log/qmail/qmail-smtpd/matchedup | /usr/local/bin/tai64nlocal
>
> human_readable_current
> 3) vi human_readable_current (remove all unneeded data)
> 4) cat /var/log/qmail/qmail-send/human_readable_current |
> /usr/local/bin/tai64n > tai64_current
> 5) cat ./tai64_current | /usr/local/qmailanalog/bin/zoverall >
overall_log
> No. qmail-smtpd is incoming mail via SMTP. qmail-send is all deliveries,
> local and remote.
> No. Instead of converting the tai64n timestamps to human-readable, you
need
> to convert them to the fractional seconds (tai) that qmail-analog expects.
> You can do this with tai64n2tai, included in Bruce Guenter's qlogtools
> package if I remember correctly. His software is at untroubled.org.
Thanks
> for the info Charles, but I'm confused. How do most of you folks pull out
> information from your logs? Log files generated by qmail are
> unreadable/unusable in the current (multilog) format. In order for them
to
> make sense to me, and in order to sift them for specific dates I have to
> convert them to human readable format. I can do this with tai64nlocal.
Once
> I have removed data that is not pertinent I then have to change them back
> into multilog format using tai64n, and then convert them into the older
> TAI64 format that qmailanalog understands, then run them through the
> qmailanalog scripts.
> Wow, that's a convoluted process using tools that until now had worked
> together to provide a graceful solution to my email needs.
> >Thanks for the info Charles, but I'm confused. How do most of you folks
> >pull out information from your logs?
> With qmail-analog, tai64nlocal, and "less", in my case. Most people here
> probably use something similar.
> >Log files generated by qmail are unreadable/unusable in the current
> >(multilog) format.
> tai64n timestamps aren't supposed to be human readable. They're supposed
to
> be easily parsable by programs. That's the whole point of tai64nlocal --
> you log with tai64n timestamps, and if you want to read the log with
> human-readable timestamps, you do:
> tai64nlocal < log | pager_of_choice
> Don't run the logs through tai64nlocal before they hit the disk.
> >In order for them to make sense to me, and in order to sift them for
> >specific dates I have to convert them to human readable format.
> No, it's much simpler than that. A program to filter a log with
tai64nlocal
> timestamps for particular dates is trivial; Bruce's qlogtools probably
> includes one (though I haven't checked). After you've filtered them, you
> run it through tai64nlocal before reading it.
> >Once I have removed data that is not pertinent I then have to change them
> >back into multilog format using tai64n, and then convert them into the
> >older
> >TAI64 format that qmailanalog understands, then run them through the
> >qmailanalog scripts.
> Don't remove any data. What isn't pertinent? qmail-analog needs all of
the
> various data that qmail-send logs to be able to accurately summarize it. I
> have a script that runs every night to give me a summary of the day's
> activity on each mail server. There's a slightly different version that
> does it at the end of the month for a month's logs. This script may have
> bash-specific constructions, it's not optimized, and it uses tools from
> Bruce Guenter's qlogtools package in addition to daemontools and
> qmail-analog. Pick up Bruce's software at untroubled.org.
> #!/bin/bash
> set -e
> HOST=`hostname -f`
> MAILTO=admin-mailstats@your_domain
> export PATH="$PATH:/root/bin:/usr/bin/qmailanalog"
> tmpdir=/tmp/qmail-cron.$$.$RANDOM
> mkdir $tmpdir
> pushd $tmpdir >/dev/null
> s_year=`date -d '1 day ago' +%Y`
> s_month=`date -d '1 day ago' +%m`
> s_day=`date -d '1 day ago' +%d`
> e_year=`date +%Y`
> e_month=`date +%m`
> e_day=`date +%d`
> start="$s_year-$s_month-$s_day"
> end="$e_year-$e_month-$e_day"
> LOGDIR=/var/log/qmail
> cat "$LOGDIR"/{"@",cur}* \
> | tai64n2tai \
> | qlogselect start $start end $end \
> | matchup >logfile 5>/dev/null
> for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids zrxdelay;
do
> $ana 2>/dev/null <logfile >$ana
> attach="$attach -a $ana"
> done
> rm -f logfile
> mutt -s "$HOST: mailstats for $s_year-$s_month-$s_day" \
> -x $attach $MAILTO </dev/null
> popd >/dev/null
> rm -rf $tmpdir
> Change the list list of reports you want as appropriate. You'll also
likely
> have to change the PATH setting, etc in the script.
> >For Charles sake, I don't want to simply look at the log files. I want a
> >qmailanalog-style report on a subset of the information contained within
my
> >"current" file.
> Just like the above? :)
> Charles
> P.S. Russell, if there's any interest in putting this up on your site,
feel
> free to make a local copy. I don't have it on any ftp or http servers at
> the moment.
> >This script may have
> >bash-specific constructions,
> >s_year=`date -d '1 day ago' +%Y`
> >s_month=`date -d '1 day ago' +%m`
> >s_day=`date -d '1 day ago' +%d`
> This works only with GNU date - the original date doesn't have -d. Your
> script is nice, though.
> >for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids zrxdelay;
do
> Well, for mailservers beeing somehow busy I'd _really_ avoid at least
> zrhosts and zrxdelay - these lists become ___very___ long. If you aren't
> running a virtual user setup I'd also avoid zsuids.
> I found qmail-mrtg (the version that doesn't use qmail-analog) very
helpful.
>
> ==========
>
>
>
> _________________________________________________________________
> Get your FREE download of MSN Explorer at http://explorer.msn.com
>
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
