This site is now 100% read-only, and retired.

Logging the right way

Posted by jeld on Mon 24 Oct 2005 at 14:09

Every system administrator worth his pay knows, that one's system should log as much as possible and that logs should be checked for errors, warnings, suspicious activities and other things. Unfortunately, by default, logging is set up in a way that makes this job difficult, bordering on impossible.

There is an old problem with the default system log daemon syslogd. It can only log things according to two parameters passed to the syslog() function call by the program doing the logging.

These parameters are called facilities and priorities. There are eight standard priorities defined: emergency, alert, critical, error, warning, notice, info and debug.

This generally should give any program a sufficient control over how badly we want to pass our message to the human in charge. But priority at which a particular message is logged, determined by the creator(s) of the program in question and their opinion of seriousness of the situation is not always the same as that of the system administrator (albeit arguably if the developer of the program considers a particular message critical you might want to see it at least once).

This is far from ideal. On the other hand, facilities have been inherited from the olden UNIX times and for reasons unknown have been passed down the generations. There are twelve standard facilities (auth, authpriv, cron, daemon, ftp, kernel, lpr, mail, news, syslog, user, uucp), plus eight of so called local facilities ranging from local0 to local7.

Just reading the list of default facilities you might see the problem. Where my samba server would log things? Daemon? What about SSH? Auth or user? What about my LDAP server? How many people on earth are still running UUCP that there is a separate facility for it? How do I make my DNS server log into a separate file?

Obviously a lot of programs provide their own logging and do not use syslog for anything except startup and shutdown messages (if that). A good example would be Apache web server. But centralized logging control is a nice thing to have and not everything in this world has its own logging facilities.

What can we do? One of the solutions would be to setup the logging according to priorities and meticulously check anything that gets logged with priority higher then, say, error. The problem with this approach is that not every message that goes there will really be important to you and you can still miss things that you wanted to know and the developers didn't think were important enough.

The solution presented here involves three distinct steps.

  1. Replacing syslogd and setting up log files in a logical fashion.
  2. Setting up log monitoring.
  3. Customizing log monitor for the particulars of the system.

Replacing syslogd

There are a few different flavors of syslogd replacements out there, you can find them by running:

apt-cache search syslog daemon

For our purposes we will use the one called syslog-ng (ng stands for new generation). So lets start by installing it.

dpkg -P sysklogd klogd
apt-get install syslog-ng

By default syslog-ng will remove sysklogd and the kernel logger, but we do not want their configs to sit in our /etc till the underverse comes and therefore we purge them explicitly (personally I added APT::Get::Purge "true" to my apt.conf so that anything I uninstall gets purged, but apt settings are not for this article).

Customizing syslog-ng config

The default config file for the syslog-ng is called, quite unsurprisingly, in /etc/syslog-ng/syslog-ng.conf. Take a look at it. Scared? Its OK, I was scared too the first time around. OK, lets stop the daemon and customize the file.

/etc/init.d/syslog-ng stop
cp /etc/syslog-ng/syslog-ng.conf /etc/syslog-ng/syslog-ng.original
vim /etc/syslog-ng/syslog-ng.conf
The basic rules of the syntax for the syslog-ng.conf are that everything ends in semicolon and everything is enclosed in curly braces. So, the first thing of order is to have everything log into a single file, then we can look at the result and split the logs into more files. Mind you, considering that we are about to setup a logging monitor a single file would do just as well, but just to show you how much nicer syslog-ng is, compared to sysklogd we will do some log splitting.

Also, once in a while (generally when there is a problem) you might want to look at complete logs, and then it is very useful to have them sorted in a nice logical fashion. The initial file will look like this:

options {
        chain_hostnames(0);
        time_reopen(10);
        time_reap(360);
        log_fifo_size(2048);
        create_dirs(yes);
        owner(root);
        group(root);
        perm(0644);
        dir_perm(0755);
        use_dns(no);
};
source int { internal(); };
source main { unix-stream("/dev/log"); };
source kernel { file("/proc/kmsg" log_prefix("kernel: ")); };

destination messages { file("/var/log/messages"); };

log { source(main); source(int); source(kernel); destination(messages); };
Confused? In syslog-ng we define four main things. The sources of log messages, destinations for the log messages (for example log files), filters defining what messages out of the source actually go to a particular destination and log definitions where we put those three together to form a logging mechanism. In this case we don't have any filters (everything goes into /var/log/messages) we only have one destination and we get messages from three standard sources. We can start the syslog-ng daemon back and watch it dump all the stuff into the messages file.
/etc/init.d/syslog-ng start
tail -f /var/log/messages

Now we have to fix something. You see, by default our logs get rotated by logrotate program running from a cronjob every night. If logrotate misses a few files it will still run, but it will complain.

So, lets go and edit /etc/logrotate.d/syslog-ng. Just remove everything and leave the messages file. Be careful to leave around the postrotate script.

The resulting file should look something like this:

/var/log/messages {
   rotate 4
   weekly
   missingok
   notifempty
   compress
   postrotate
      /etc/init.d/syslog-ng reload >/dev/null
   endscript
}

OK, now that we have logging and rotation under control lets split the logs into several files.

This is where personal taste and system functionality plays the role. For example, on my systems I found that kernel messages are generally something I actually want to see separately and that information from the mail server clutter the messages file a lot. So, for that I add a filter separating mail messages from the rest and setup a couple of new destinations. So the end of the file now looks like this:

filter mail { facility(mail); };

destination mail { file("/var/log/mail.log"); };
destination kernel { file("/var/log/kernel.log"); };
destination messages { file("/var/log/messages"); };

log { source(kernel); destination(kernel); };
log { source(main); filter(mail); destination(mail); flags(final); };
log { source(main); source(int);  destination(messages); };

Notice the final flag on the mail log? That insures, that the message processing actually stops at that entry of the log message matches the filter. This way although everything still goes into messages, mail logs only go into mail log file.

As you can see, I use the standard mail facility to filter out the mail logs and you might be thinking, why would he go into the trouble of replacing sysklogd if he still uses standard filtering?

Here are a few things you cannot do with standard sysklogd. Since my systems are serving web content, there is no reason for anyone except me to log into them. On the other hand I know there are people out there who will constantly try. Therefore anything that SSH daemon has to tell me I consider important.

destination sshd { file("/var/log/ssh.log"); };
filter sshd { program("ssh");
log { source(main); filter(sshd); destination(sshd); flags(final); }

Lets say that you are running a very active mail server and even daily rotation of logs makes them impossible to parse. How about rotating logs when they reach certain size? Sure, there is a program in apache2-utils package called rotatelogs. So, all we have to do, to make our mail logs rotate when they reach 500MB in size is to change our mail destination to something like

destination mail { program("/usr/sbin/rotatelogs /var/log/mail 500M"); };

Get the idea? And this is not all. There are a lot of useful options in syslog-ng that might make your log files more comprehensible. To see more things you can do with syslog-ng, check out its web site. Remember, that every time you change one of the standard log definitions you might want to check the logrotate config and add or remove appropriate entries. And obviously, before attempting doing any of these steps you want to RTFM, so

man syslog-ng
man syslog-ng.conf
man logrotate
man logrotate.conf

Setting up log monitoring

OK, so all your logs go into appropriate places and don't mix with one another and rotate properly and all the other nice things from the previous chapter.

You might even wade through the important ones every morning looking for anything that might be out of order.

That process quickly gets tedious. And what if something bad happens that you would want to know right away or at least in a matter of minutes, not the next morning?

Yes, there is a solution. First, I will digress a little. A few weeks ago a rather controversial article was published by a security professional Marcus Ranum. The title of the article was The Six Dumbest Ideas in Computer Security. The article discusses some common approaches to computer and network security and discusses (in rather harsh terms) why is it not a good idea to follow them. In my opinion Mr. Ranum, being a very intelligent person, is very convincing, but not always very realistic. But one of the discussed practices struck me as very true and very bad. Marcus calls it "enumerating badness". In this particular case, topic of this article being logging, I was looking at different programs to monitor logs for signs of bad things. As I soon found out, most log watcher programs out there are doing exactly what Marcus Ranum warns us against. The want you to have a list of patterns of bad stuff and they will tell you when they see such a pattern appear in one of your logs.

Why is it bad? Because there is a lot of bad stuff that can possibly happen and you cannot be sure to include every possible pattern in your watch lists. The right approach is of course weed out good stuff. That is if you see a message and you know that it is not important, filter it out.

Soon, the only messages you get are the one worth paying attention. Enter logcheck. Logcheck is a bunch of clever shell scripts that do exactly that. Even better, Debian users are blessed, since a lot of Debian packages provide logcheck pattern files with patterns for common unimportant messages. So, as usual the first step, is to install:

apt-get install logcheck

Now we need edit a few things. First lets define what files we want to watch. This is defined in /etc/logcheck/logcheck.logfiles.

My logcheck.logfiles looks like this:

/var/log/messages
/var/log/kernel.log
/var/log/mail.log

Now lets look at the actual config file. Generally speaking the only thing you need to specify here is the address where you would like to receive the alerts, but there are a few things here that might warrant changing. One of them is the report level.

Logcheck maintains three distinct sets of patterns, designed for different system and different administrators. These sets are called "server", "workstation" and "paranoid".

It is easy to guess, that workstation is most liberal at weeding out things, server is a bit more conservative and paranoid will start sending alerts at the slightest provocation.

Choosing a particular level is up to you, but I found myself quite comfortable using the default server setting. The file is very well commented and should give you no trouble. Here is what I did:

INTRO=0
REPORTLEVEL="server"
SENDMAILTO="myemail@company.com"
ADDTAG="yes"
ATTACKSUBJECT="Attack Alerts"
SECURITYSUBJECT="Security Events"
EVENTSSUBJECT="System Events"

The ADDTAG allows me to easily filter logcheck alerts to a separate folder, and INTRO=0 makes logcheck skip silly message headers and footers.

Now you want to make sure that your MTA is actually capable of delivering mail (the default setup of exim should be enough for Debian users, personally I replaced it with postfix, but this is again out of the scope of this article).

And there you go. Logcheck runs from cron. The file controlling its frequency is located in /etc/cron.d/logcheck. By default it runs every hour.

I found that every hour is enough for me to keep track of potential issues. One thing about logcheck and syslog-ng are permissions on the log files. If you look at my samples of syslog-ng.conf again, you will notice (or already notices) that I set default log file permissions to 644. This is rather important, since logcheck doesn't run as root (it creates its own logcheck user) and with default permissions of 640 it is unable to access the log files.

Another way of solving this is to set the default group ownership of log files to the logcheck group.

Customizing logcheck rules

So, now you open your mail and see all the interesting log messages neatly sitting in your mail. No more wading through endless log files and no more scrolling through thousands of similar unimportant log entries. What next?

Right. Albeit default logcheck rules weed out the most common stuff, in the real world and on a real system there are still a lot of repeating messages that you consider to be non-critical.

This is where things get tricky. Logcheck uses grep (or to be precise egrep) to weed out log messages, so logcheck pattern files are actually just lists of extended grep regular expressions.

If you are good with grep and something like the following makes perfect sense to you, you are in luck.

^\w{3} [ :0-9]{11} [._[:alnum:]-]+ jabberd/s2s\[[0-9]+\]: \[[0-9]+\] \[[0-9.]{7,15}, port=[0-9]+\]

If not, remember Douglas Adams. Do Not Panic.

Log entries produced by syslog-ng (and by sysklogd and by some other logger daemons) all start more or less the same. They have date and time, name of the host system usually continued by the name of the program doing the logging and by the message itself.

The pattern files for logcheck are located in /etc/logcheck/ignore.d.[report level] directories.

If you followed my lead and didn't change the default report level, the patterns used will be in /etc/logcheck/ignore.d.server. Since we do not want to interfere with software packaging, we might want to create a separate file for our custom patterns. I use /etc/logcheck/ignore.d.server/local.

Since describing proper writing of regular expressions is a big job, I will give you two examples. Soon after setting up logcheck, I found that due to some reasons, my system is attempting to send mail messages with no recipients.

After checking out the source of the problem, I found that nothing sinister is going on and decided to filter these postfix messages out. The message in question looks like this (here and in other sample logs and patterns I inserted line breaks for better readability. In the actual files these are single lines):

Oct 19 13:08:19 hostname postfix/cleanup[6560]: E366374E63: 
to=, relay=none, delay=1, status=bounced 
(No recipients specified)

Since I didn't want to come up with a regular expression for this from scratch, I looked at the existing pattern list for postfix. A quick grep gave me a sample:

grep clean postfix
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postfix/cleanup\[[0-9]+\]: [[:alnum:]]+: message-id=.*$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postfix/cleanup\[[0-9]+\]: [[:alnum:]]+: resent-message-id=<.+>$

Now all I have to do is copy the start of these patterns and add something that would be unique for the message I want to filter. The result I came up with looks like this: ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postfix/cleanup\[[0-9]+\]: [[:alnum:]]+: [[:print]]+ \(No recipients specified\)

After I added this line to my custom patterns logcheck stopped bothering me about mails with no recipients being bounced.

After that, I installed the shorewall firewall on some of my system and immediately started receiving a lot of messages about packets being dropped by the firewall. A sample message looks like this:

Oct 21 11:05:24 hostname kernel: Shorewall:net2all:DROP:IN=eth0 OUT= 
MAC=00:06:5b:0f:3e:ba:00:0e:38:57:d1:80:08:00 SRC=[Source IP] DST=[My IP] 
LEN=1067 TOS=0x00 PREC=0x00 TTL=114 ID=11979 PROTO=UDP SPT=50606 DPT=80 LEN=1047

Now, if the packet was dropped, I think I am reasonably safe. So I don't want logcheck bother me every hour with loads of these lines.

I looked at the set of kernel patterns, and came up with the following:

^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel: Shorewall:net2all:DROP:IN=eth0

This was sufficient and so far I am a happy system administrator, knowing what is going on on his systems and not having to go too far to obtain that knowledge.

Obviously this is not the only measure that I took on my way to true enlightenment, but it is one of the important ones.

If this article is well received, I will continue to document some of the methods I have come up with to keep my systems safe and secure. Till then, good luck to you all.

Dmitriy Kropivnitskiy
Happy System Administrator

 

 


Re: Logging the right way
Posted by bosteen (213.235.xx.xx) on Mon 24 Oct 2005 at 15:21
This article popped up in my rss tracker, just at the point that I was considering installing/running heavyweight log file analyzers for my lone home server. Just what I was looking for: A simple and fairly elegant solution! Thanks!

[ Parent ]

Re: Logging the right way
Posted by Anonymous (213.164.xx.xx) on Mon 24 Oct 2005 at 18:26
> Obviously a lot of programs provide their own logging and
> do not use syslog for anything except startup and shutdown
> messages (if that). A good example would be Apache web server.

Nice article, but this part is completely untrue. Apache will log to syslog, you just have to tell it to. syslog is really for logging of important messages, if you wanted to log your Apache access_log to it, you can call logger from Apache.

Which other programs have you had problems logging to syslog with?

It would also be good if you mentioned the relative benefits of tcp versus udp.

[ Parent ]

Re: Logging the right way
Posted by Anonymous (68.173.xx.xx) on Mon 24 Oct 2005 at 21:26
Yes, you can make apache log to syslog, but in most default setups it will do its own logging.

[ Parent ]

Re: Logging the right way
Posted by jeld (68.173.xx.xx) on Mon 24 Oct 2005 at 21:47
The above comment is mine. Forgot to login :)

[ Parent ]

Re: Logging the right way
Posted by chris (80.203.xx.xx) on Mon 24 Oct 2005 at 18:39
[ View Weblogs ]
This does look interesting :) I'm going to have to spend some time testing this. Things like changing munin scripts to read the correct log files etc spring to mind.

I've never really got to grips with logcheck as yet - but perhaps now is the time :)

[ Parent ]

Re: Logging the right way
Posted by simonw (84.45.xx.xx) on Mon 24 Oct 2005 at 19:03
[ View Weblogs ]
Excellent article.

One mystery of Debian I am pondering, the default syslog config creates too many "mail" log files, is there some deep reason for this, or is this an Exim thing (I to usually install Postfix).

[ Parent ]

Re: Logging the right way
Posted by philcore (70.161.xx.xx) on Tue 25 Oct 2005 at 00:58
[ View Weblogs ]
I have to agree. I've spent a lot of time trimming down syslog to what I consider a sane value. mail.log, mail.info, daemon, syslog, messages, argh. That's why $god made grep. I imagine every log entry is created at least four times with default debian [ks]yslog.

[ Parent ]

Re: Logging the right way
Posted by DaveV (24.8.xx.xx) on Mon 24 Oct 2005 at 19:23
Very well done, I look forward to your next article.
Thanks.

[ Parent ]

Re: Logging the right way
Posted by Anonymous (67.104.xx.xx) on Mon 24 Oct 2005 at 22:46
Nice article, I enjoy your writing style. Keep it up.

[ Parent ]

Re: Logging the right way
Posted by linulin (195.131.xx.xx) on Tue 25 Oct 2005 at 06:41
"The right approach is of course weed out good stuff. That is if you see a message and you know that it is not important, filter it out."

Nothing is perfect. This approach could also lead to missed alerts. It is quite easy to make a mistake writing regular expression. (Especially if you're not a regexp guru.) Clearly, this way one might filter out some important stuff along with insignificant log entries.

I would use a combination of alert + filter patterns. I.e. filter out only those log entries that do not match the alert patterns. Don't know whether it is supported by logcheck though.

--
...Bye..Dmitry.

[ Parent ]

Re: Logging the right way
Posted by Anonymous (24.229.xx.xx) on Tue 25 Oct 2005 at 10:04

Yes, logcheck supports alert patterns. See README.logcheck-database

Also, It will soon be easier to craft regular expressions :)

[ Parent ]

Re: Logging the right way
Posted by jeld (24.39.xx.xx) on Tue 25 Oct 2005 at 18:17
Yes, of course you can make mistakes in regular expressions and filter out stuff you would want to see. That is why I tried to make my example regexes as precise as I could. Logcheck includes a set of patterns in a directory called violations.d where log entries matching these patterns will be reported no matter what (unless that is they are contradicted in violations.ignore.d pattern set). I haven't used these as of yet.

You are off the edge of the map, mate. Here there be monsters!

[ Parent ]

automatic entry classification in syslog-ng.conf
Posted by Anonymous (195.202.xx.xx) on Tue 25 Oct 2005 at 09:17
Here is a little time-saver. It automatically classifies all log entries by date, hostname, and program name. Very nice for a central log server taking logs from many machines.

destination alllog { file("/var/log/$YEAR-$MONTH-$DAY/$FULLHOST/$PROGRAM.log" owner("root") group("adm") perm(0640)); };
filter f_alllog { host(".*"); };
log { source(src); filter(f_alllog); destination(alllog); };

[ Parent ]

Re: Logging the right way
Posted by th0re (83.243.xx.xx) on Wed 26 Oct 2005 at 11:16
[ View Weblogs ]
I've carefully copied your settings and mail seems to be logged to /var/log/mail.N is a large number. Any ideas on how to get the logging to /var/log/mail.log?

--
th0re

[ Parent ]

Re: Logging the right way
Posted by jeld (24.39.xx.xx) on Wed 26 Oct 2005 at 15:15
Obviously what happened, is that you copied my sample mail log setup with rotatelogs. Rotatelogs program will name logs automatically using numbers, as you mentioned. In most cases you do not need to use a special log rotation program to rotate your logs (standard logrotate setup should be good enough for most anything). So use the setting I give for the mail log that doesn't use rotatelogs. If you read the article carefully you would notice, that I didn't intend the rotatelogs line as default setting and only gave it as an example of what could be done with rotatelogs. And don't forget to read your man pages before changing configs.

You are off the edge of the map, mate. Here there be monsters!

[ Parent ]

Re: Logging the right way
Posted by lpenz (200.102.xx.xx) on Thu 27 Oct 2005 at 14:49
Does logwatch also include any messages it doesn't know how to classify?

[ Parent ]

Re: Logging the right way
Posted by jeld (24.39.xx.xx) on Thu 27 Oct 2005 at 15:43
That is basically what logwatch does. It will check for messages it is supposed to include and make sure those are not removed. It will check for messages it is supposed to ignore and remove them. The rest goes into the report. So two kinds of things end up in the report: the stuff we want to know about, the stuff we don't know about.

You are off the edge of the map, mate. Here there be monsters!

[ Parent ]

Re: Logging the right way - file permissions
Posted by shufla (83.22.xx.xx) on Sat 5 Nov 2005 at 14:11
Hi,


Well, you can use acl (apt-cache show acl) to get more secure log files. Chmod them to 640, and then setfacl logcheck:r-- :) I do not know, if that is supported by logrotate.


Luke

[ Parent ]

Re: Logging the right way
Posted by Anonymous (82.96.xx.xx) on Mon 14 Nov 2005 at 23:47
I might be mistaken but i recon theres a typo in the sshd logging example,
the second line " filter sshd { program("ssh"); " seems to be missing a " };" at the end.

The article is great, keep 'em coming!

[ Parent ]

Re: Logging the right way
Posted by Nikem (194.126.xx.xx) on Fri 2 Dec 2005 at 08:18
Couldn't you help me with the following error:
I have installed logcheck as described here. I have tried run it by hand. All went Ok, I have got my report. But all following runs started by cron job failed with the following error:

Details:
Could not run logtail or save output

strace gives me the answer:
socket(PF_FILE, SOCK_DGRAM, 0) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
connect(3, {sa_family=AF_FILE, path="/dev/log"}, 16) = -1 EPROTOTYPE (Protocol wrong type for socket)
close(3) = 0
socket(PF_FILE, SOCK_STREAM, 0) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
connect(3, {sa_family=AF_FILE, path="/dev/log"}, 16) = 0
send(3, "<38>Dec 2 10:10:45 su[11173]: ("..., 75, MSG_NOSIGNAL) = 75


What should I do?

[ Parent ]

Re: Logging the right way
Posted by Nikem (194.126.xx.xx) on Fri 2 Dec 2005 at 09:54
Well, it seems that apt-proxy.log file was the foe. Removing it from logcheck.logfiles has solved the problem.

[ Parent ]

Re: Logging the right way
Posted by Anonymous (68.173.xx.xx) on Fri 2 Dec 2005 at 13:51
In general doing an strace on logcheck is not that interesting, since logcheck is a shell script. In this case, it looks like a real permission issue, just check that the file you were having troubles with (apt-proxy) is readable by logcheck user.

[ Parent ]

Re: Logging the right way
Posted by Nikem (194.126.xx.xx) on Fri 2 Dec 2005 at 14:07
Bingo! Thanks a lot!

[ Parent ]

Good but no prize money..
Posted by listmail (150.101.xx.xx) on Wed 14 Dec 2005 at 21:58
Thanks for the guide> I appreciate the effort however in my n00b eyes it is not complete as there is no testing.

We never actually generate and examine an email of logs. We just sit around hoping one will come. I've complete all the steps above and see things logging to /var/log/messages but I have yet to see an email.

I like complete How Tos aimed at dummies :-)

TH

Debian stable 2.6.8-2-k7

[ Parent ]

Latest sid update required a config change
Posted by chris (217.8.xx.xx) on Tue 7 Mar 2006 at 08:39
[ View Weblogs ]

The latest apt-get update here broke syslog-ng's config.

I have (taken from this article) amongst other things

source kernel { file("/proc/kmsg" log_prefix("kernel: ")); };
destination kernel { file("/var/log/kernel.log"); };
log { source(kernel); destination(kernel); };

Syslog startup gives syntax errors on the source and destination lines - the following works:

source krnl { file("/proc/kmsg" log_prefix("kernel: ")); };
destination krnl { file("/var/log/kernel.log"); };
log { source(krnl); destination(krnl); };

I've not had time to investigate this further (seems like kernel has become a reserved word or maybe it's automatically defined or something) - but this change got me back to exactly where I was before the update.

Just thought I'd mention it in case people were having the same issue.

[ Parent ]

Re: Logging the right way
Posted by tmetro (66.166.xx.xx) on Wed 23 May 2007 at 19:54
There are 4 problems with the default logging setup on Debian:

1. It comes with a poorly designed syslog.conf. The messages aren't logically sorted into separate files. It contains entries for obsolete facilities like uucp.

2. syslog doesn't use logrotate. Instead it uses its own shell script. (I remedied this by relocating all the syslog managed log files to /var/log/syslog/ and replacing the shell script (/etc/cron.{daily,weekly}/sysklogd) with a wild card logrotate configuration.) Using syslog-ng also obsoletes the shell script, but I notice from the approach recommended in the article, doesn't solve the maintenance hassle of keeping the logrotate configuration in sync with the syslog-ng configuration. (I've used syslog-ng years ago, but haven't bothered with it on more recently set up systems as I don't think it offers enough benefit compared to a well tuned syslog.conf. Also, more and more applications do their logging outside of syslog by default.)

3. logrotate is a poor solution for modern day log rotation. For example, it doesn't support rolling logs to filenames incorporating a timestamp. (There have been proposed patches to do this, but they haven't made it into the main code yet, as far as I know.) logrotate also can't run an external script to determine the list of files to rotate (and thus the shell script approach used by syslog). logrotate is also notorious for producing poor diagnostic information when something fails during a rotation. About every 6 months I do a search to see if any suitable alternatives to logrotate have turned up. The ideal solution should be backwards compatible with the logrotate configuration files, which are supplied by many packages.

4. A tool like logcheck should be part of the base install. Otherwise logs will mostly just be ignored by casual users.

Even logcheck doesn't sound ideal. Based on this article, it doesn't sound like it can handle the detection of states - such as flagging or ignoring the appearance of a line matching pattern X, only after seeing pattern Y.

-Tom

[ Parent ]

Re: Logging the right way
Posted by Anonymous (66.166.xx.xx) on Thu 7 Jun 2007 at 23:47
>> logrotate is a poor solution ... it doesn't support rolling logs to filenames incorporating a timestamp. <<

Correction: as of Debian Etch the patch to implement filenames with timestamps is incorporated. Ubuntu Feisty likewise includes it.

But logrotate still has other problems, which could be fairly easily solved it if was written in a flexible, high-level scripting language instead of a rigid, low-level compiled language. C is the wrong tool for this job.

-Tom

[ Parent ]

Re: Logging the right way
Posted by Anonymous (78.83.xx.xx) on Tue 25 Dec 2007 at 09:52
All these articles are GREAT !Great Work keep on going!

[ Parent ]

Re: Logging the right way
Posted by Anonymous (69.70.xx.xx) on Tue 25 Nov 2008 at 15:37
Great article!

[ Parent ]

Re: Logging the right way
Posted by sixerjman (64.252.xx.xx) on Sat 23 May 2009 at 21:45
Great article. I have implemented syslog-ng and logcheck based on this guide.

The current Debian logger is rsyslog. Have you or anyone migrated from syslog-ng to rsyslog? If so, could you post a sample rsyslog.conf here? Thanks again for the great article.

[ Parent ]

Re: Logging the right way
Posted by sixerjman (64.252.xx.xx) on Wed 27 May 2009 at 01:51
I have this rsyslog config set up, it seems to be working. Steep learning curve for 7 lines, but it looks logical, if not 'good' :p

# /etc/rsyslog.conf Configuration file for rsyslog v3.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html


#################
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

#
# Include all config files in /etc/rsyslog.d/
#
#$IncludeConfig /etc/rsyslog.d/*.conf


###############
#### RULES ####
###############

#
# Emergencies are sent to everybody logged in.
#
*.emerg *

#
# Secure shell, kernel and mail messages
# are routed to their respective destinations
# and subsequently discarded. Whatever is
# left goes to the messages file.
#
# This is kind of ugly because the syslogfacility
# property has to be numeric per the rsyslog-doc
# package. For details on the facility numbers see the
# BSD logging protocol RFC 3164
#

:programname, contains, "ssh" /var/log/ssh.log
:programname, contains, "ssh" ~
:syslogfacility, isequal, "0" /var/log/exim4/mainlog
:syslogfacility, isequal, "0" ~
:syslogfacility, isequal, "2" /kern.log
:syslogfacility, isequal, "2" ~
*.* /var/log/messages

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole

[ Parent ]