This site is now 100% read-only, and retired.

XML logo

Awstats + Postfix: per domain mail statistics
Posted by mar on Sat 6 Jun 2009 at 15:48

Summary

Recently I was forced to compute mail bandwith per domain. Our MTA is Postfix, reporting tool is Awstats. After some googling I found some hints but they have to be slightly modified to present agregated per domain statistics.

Resources and software

  1. Awstats -- especially the FAQ: http://awstats.sourceforge.net/docs/awstats_faq.html#MAIL
  2. Prepflog -- no debian package: http://web.tiscali.it/postfix/prepflog.html
  3. Postfix of course

Comments on debian setup

These comments are not step by step how-to; just hints to someone like me, who tries to force Awstat show per domain mail statistics.

Software

First thing to note -- Debian installs awstats scripts into /usr/share/doc/awstats/examples. Beacause of this, you have some trouble to generate the default Awstats config, but you dont have to do that -- you can simply copy the default and modify it (dont forget to remove/modify the Include directive if you copy the default one).

Prepflog is probably a good idea, though i just believe to what is said on the product page -- it should narrow the log, so no duplicate entries are present.

Principle

You use your standard /var/log/mail.info.0 Postfix log, you pipe it through several usefull perl one liners producing CLF (common logging format) file for Awstats. Thankfully, Awstats is capable of proccessing a piped input directly.

Settings

Generate and modify the config for awstats

Create a config according to
http://awstats.sourceforge.ne/docs/awstats_faq.html#MAIL

It is important to set all the options as required -- with two notable exceptions -- LogFile and SiteDomain. For example my config looks like:

LogFile="/root/hampejz/awstats_mail_filter_by_domain.sh |"
SiteDomain="domain.mail.example.com"
Include "/etc/awstats/awstats.0.mail.example.com.common"

Note that LogFile points to my custom filter. Include just includes standard Awstats file according to URL above.

Prepare the custom filter

#! /bin/sh

# filters the postfix mail log to be used by awstats
# if the generation takes long, switch off DNS lookup in awstats config        
# filtering is preproccessed by perl to collect per-domain statistics

cat /var/log/mail.info.0 | \
    perl /root/hampejz/prepflog.pl | \
    perl /usr/share/doc/awstats/examples/maillogconvert.pl standard | \
    perl -pe 's/<>/user\@unknown/g' | \
    perl -pe 's/(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} )((\S+)(@\S+ ))((\S+)(@\S+ ))/\1\4\7/g' | \
    perl -pe 's/(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} )((\S+)(@\S+ ))((\S+ ))/\1\4\@localhost /g'

Some comments:

  1. We use /var/log/mail.info.0 -- my system is configured with delayed compression of logfiles -- so I can always use static log files to generate statistics (they are modified no more by daemons).
  2. We run the prepflog (I just believe the agenda :)
  3. We run the Awstats filter, which produces the CFL file. From now on, the Awstats is capable of processing the result, though it would show the mail by user, not by domain.
  4. We handle the mail with no sender (some kind of spam and some locally generated messages from ugly crontabs)
  5. 's/(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} )((\S+)(@\S+ ))((\S+)(@\S+ ))/\1\4\7/g' -- regexp guru could make it nicer, but I just want to make it readable. This changes

    DATE TIME sender@somewhere1 recepient@somewhere2 ANYTHING

    into

    DATE TIME @somewhere1 @somewhere2 ANYTHING.

    This leaves all the mail from sending and receiving domain to be agregated (it is taken as a mail from single user).
  6. 's/(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} )((\S+)(@\S+ ))((\S+ ))/\1\4\@localhost /g'

    handles locally generated mail for local delivery (ie. no domain) as a mail for @localhost.

Crontab

All of this should be run from crontab. I believe you can handle that, so let it be the homework for a reader -- my file in /etc/cron.hourly:


#! /bin/sh

# updates awstats files

files=`ls /etc/awstats/awstats.*.conf`

for i in $files; do
    domain=`echo $i | sed -e 's/\/etc\/awstats\/awstats\.\(.*\)\.conf/\1/g'`
    # generate statistics and build static pages in one command
    mkdir -p /var/www/awstats/$domain
    perl /usr/share/doc/awstats/examples/awstats_buildstaticpages.pl \
        -config=$domain \
        -configdir=/etc/awstats \
        -update \
        -output \
        -staticlinks \
        -awstatsprog=/usr/lib/cgi-bin/awstats.pl \
        -dir=/var/www/awstats/$domain \
        -diricons=/awstats-icon
    chown www-data.www-data -R /var/www/awstats/$domain
done
chown www-data.www-data /var/lib/awstats/*

 

Comments on this Entry

Re: Awstats + Postfix: per domain mail statistics
Posted by Wayne (82.144.xx.xx) on Sun 7 Jun 2009 at 12:35
[ View Weblogs ]
Nice

I would like to see this as a full howto

[ Parent ]

Re: Awstats + Postfix: per domain mail statistics
Posted by mar (83.208.xx.xx) on Sun 7 Jun 2009 at 14:04
[ View Weblogs ]

Thanks :), if you try to set up something simillar, i can provide more info, but the idea is simple -- use tools already available and post-process the resulting CLF file to ignore individual users and replace them with single domain name. Once done, the Awstat will compute the statistics as usual.

[ Parent ]

Re: Awstats + Postfix: per domain mail statistics
Posted by Anonymous (88.169.xx.xx) on Wed 21 Oct 2015 at 08:07
Thank you, i'm looking for this feature since a long time !
Here my modification for a system using iRedmail :

cat /var/log/maillog | \
perl /usr/bin/maillogconvert.pl standard | \
perl -pe 's/(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} )((\S+)(@\S+ ))((\S+)(@\S+ ))/\1\4\7/g' | \
perl -pe 's/(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} )((\S+)(@\S+ ))((\S+ ))/\1\4\@localhost /g'

[ Parent ]