Log files are very usefully for troubleshooting a problem that has displayed symptoms and gained attention. They are even more useful, though, for preventing problems. Log files can indicate a program having intermittent errors, a shortage of disk space, addition of users, attempts to break into a system or scan a system. Most systems automatically log information for administrators. That administrator, however, still needs to look at the log files and evaluate the contents and how to read those log entries. This can be a very tedious task.
There are a variety of tools available to assist with monitoring log files. RedHat Linux (and several other distributions) include a utility called LogWatch. RedHat even sets this up with a default installation. The utility consists of a series of perl scripts that are run once a day. These scripts toss out items it recognizes, group the remaining lines of the log files, and sends an email to root. While reviewing this email message is better than nothing, it is not a real time monitoring approach.
One of the most popular real time monitors in the Unix/Linux world is SWATCH "The Simple WATCHdog". Like LogWatch, this utility is also a series of perl scripts, however, unlike LogWatch, SWATCH has two additional major features
the ability to monitor in real time
the ability to perform an action when a trigger is found
Swatch is a available at sourceforge.net/projects/swatch
and the homepage for the project is at swatch.sourceforge.net
Swatch was originally written to actively monitor messages as
they are written to a log file via the UNIX syslog utility. It is
provided as source code and is dependent on
perl modules and the UNIX command tail.
There is no cost
for SWATCH, though donations to the developer are welcome with
information provided in the README file.
SWATCH comes in the form of a compressed tar file. After extracting the files, there is an INSTALL file with instructions. The overall steps are:
Confirm that all perl modules are present (script provided to help)
I had to install several of these modules - some were available from RedHat, others had to be obtained (and compiled) from search.cpan.org
Use make commands to compile and install
Once installed, SWATCH still needs to be configured. A .swatchrc file needs to be created in the home directory of the user that will run swatch. This file must contain:
what patterns to "watch for"
what action to take when pattern is found
Optionally, SWATCH can be configured to:
tell swatch not to repeat the action for a time period, even if another pattern is found.
tell swatch when to perform an action, such as writing to the console always, but only emailing on weekends.
There is little documentation for SWATCH. Sources for usage and configuration ideas include:
Paranoid Penguin: swatch: Automated Log Monitoring for the Vigilant but Lazy Posted Linux Journal Wednesday, August 01, 2001 by Mick Bauer www.linuxjournal.com/article.php?sid=4776
Watching your Logs by Lance Spitzner July 19, 2000 www.spitzner.net/swatch.html
The README file from the developer provided with the source code.
I set up several triggers and found that they performed as
expected.
A swatchrc file that will email a user (could be a
pager address) when a user is added or deleted to the system would
be:
watchfor /useradd|userdel/
mail addresses=susan,
subject=UseraddedThis would be run with the following command:
watch -t /var/log/secure &
When a user is added or deleted from the system, an email is sent with the body of the message containing the line from the log file.
Swatch only monitors the file that is indicated when it is started. Some documentation refers to this limiting swatch to only watch a single file, however, the README from the developer indicates that this is no longer a limitation. I had success monitoring multiple files with the following command:
swatch --tail-file="/var/log/secure /var/log/messages" &
Multiple instances of swatch can also be running so that they may use
different swatchrc files or monitor different files.
Finally,
SWATCH writes out watcher process scripts so in addition to space
being available in the user's home directory (or directory specified
with command line options) it is recommended that the directory not
be world accessible.
Obviously, some knowledge of Unix or Linux command line and
software installation is required. It took some searching to find out
if RedHat had package files for the perl modules that I was missing.
Then some more searching to find the correct modules from the CPAN
site. If I knew more about perl and perl modules it may not have
taken so long.
Experience with script writing is helpful but
articles are available to help with any setup of the swatchrc file.
Learning Regular Expressions will help most with developing
search criteria.
Most important, is the need to know what to look
for in the log files.
I would use this product for high priority alerts, but would probably also use other software that generates more generic reports on a regular basis (like LogWatch). I would only recommend it to users that are already aware of what needs to be looked at. This can automate parts of the process, but not until a process is defined and only if the process is regularly reviewed.
A utility like LogWatch is a negative match utility - it lets you know everything that does NOT match in a log file. This is a better choice for new security administrators so that they can learn what is unusual. SWATCH is a positive match utility - it is triggered by what DOES match. This is a better tool once you know what is unusual and know that you need to be altered promptly.