So before I begin (or technically, just after I’ve begun), let me remind you that my perl skills are shockingly bad.  All my perl scripts are written in the same style as the script I was copying from at the time I wrote them.

Introduction

I’ve recently set up an IRC server, partly to mess about with it and partly to consider using it to keep in touch with friends.  I’m acutely aware that it’s the kind of thing that gets attacked, so I’ve made sure ngIRCd (the daemon I chose) is logging everything, and then I started looking for a logwatch (homepage) script to monitor the logs and alert me of anything suspicious going on.

Sadly, I couldn’t find one, so I decided to do the only sensible thing and write my own, which is fine, but as you’ll see if you search the web for ‘writing custom logwatch scripts’, it’s sort of both easy and hard.  It’s easy once all the bits fall into place, but sometimes the terminology gets in the way.  So, here’s how I did it.

Detail

You absolutely need two files, one which describes which logs you’re going to handle, and another which is the script which does the handling.  You should name them in some way which makes sense (after the service you’re monitoring for example).  Once you put them in the right place, logwatch will execute your script and you’re away.  There are some optional files, if you want to do some logfile pre-processing (I think) but as I never used those, I can’t comment.

So, I want to monitor ngIRCd which on my server logs everything it does to /var/log/messages under the service name ngircd.  Here’s an example line,

[sourcecode language=”plain” gutter=”false”]
Aug 13 08:01:54 hostname ngircd[10898]: User "bob!~ident@some.machine" registered (connection 8).
[/sourcecode]

The first thing I did was create a file describing which logs to monitor and how to filter the data, and I stole various bits of information from the other files distributed with logwatch.  I called my file ngircd.conf and place it in,

[sourcecode language=”plain” gutter=”false”]
/etc/logwatch/conf/services/
[/sourcecode]

That’s the default location on Debian.  Here’s the content of my file with some comments,

[sourcecode language=”bash”]
# set the title for the reports
Title = "ngIRCd"
# set the logfile to the messages log file *group*
LogFile = messages
# only return entries made by ngIRCd which reduces our effort in the script
*OnlyService = ngircd
# remove the date / time stamp, hostname, service name, etc.
*RemoveHeaders
[/sourcecode]

Line 3 is important and took me a little while to work out.  In the config file for your service, you describe the log file group that is used, which in turn tells logwatch which file in the /logfiles/ directory structure describes the actual log files which are scanned.  So the above line tells logwatch (in the case of Debian) to use the log files described in /usr/share/logwatch/default.conf/logfiles/messages.conf.  That file handles the log file names, how to deal with date/time stamps, archived logs, etc.

If the log files for your new service don’t already have a matching log file group configuration file, you should create one in /etc/logwatch/conf/logfiles, using an example from /usr/share/logwatch/default.conf/logfiles.  Anyway, in my case, since I was using /var/log/messages which is already described in /usr/share/logwatch/default.conf/logfiles/messages.conf I didn’t need to create one.

Now that you’ve got the service configuration covered, you need a script, and it needs to be named after the config file (so if you call your config file foo.conf, then your script needs to be called foo).  You can write this script in any language that can read from STDIN and write to STDOUT, but like other folk before me I made the joyful error of sticking to perl.

You place this file in

[sourcecode language=”plain” gutter=”false”]
/etc/logwatch/scripts/services
[/sourcecode]

The important things to remember are,

1. your script will receive the content of the appropriate logs via STDIN
2. it should write output to STDOUT and should use the environment variable LOGWATCH_DETAIL_LEVEL to determine the detail level passed to the logwatch program
3. the output should be tidy and should avoid being verbose
4. if you’ve configured the service conf script correctly you won’t need to worry about parsing dates, stripping headers, or other rubbish.  This does depend on the log file in question though and the application.
5. To keep in line with other scripts, you should capture everything you know you don’t care about and ignore it, process stuff you do care about, and report stuff you don’t recognise.

The link below is the script I cobbled together to handle ngIRCd so far.  At the moment, I ignore my own advice and don’t check the detail level, I just wanted initially to get my data out.  I have no idea if the regexp’s are correct or efficient, but at present, it displays what I care about. Is that enough caveats?  I’m not looking for feedback on the quality of my perl! I’m just trying to show how it can be done.

ngircd

[sourcecode language=”perl”]
#!/usr/bin/perl
##########################################################################
# ngircd
##########################################################################

use Logwatch ‘:all’;

my $Detail = $ENV{‘LOGWATCH_DETAIL_LEVEL’} || 0;
my $Debug = $ENV{‘LOGWATCH_DEBUG’} || 0;

my %FailedLogin = ();
my %FailedOpers = ();
my $FailedOpCommands;
my %TriedConnections = ();
my %GoodConnectionsi = ();
my %GoodOper = () ;
my %BadOpCommands = ();
my %OtherList = ();

if ( $Debug >= 5 ) {
print STDERR "\n\nDEBUG: Inside ngircd Filter \n\n";
$DebugCounter = 1;
}

while (defined(my $ThisLine = <STDIN>)) {
if ( $Debug >= 5 ) {
print STDERR "DEBUG($DebugCounter): $ThisLine";
$DebugCounter++;
}

chomp($ThisLine);
if ( # We don’t care about these
( $ThisLine =~ m/connection .* shutting down / ) or
( $ThisLine =~ m/^New TLSv1 connection using cipher/ ) or
( $ThisLine =~ m/^Now listening on/ ) or
( $ThisLine =~ m/^IO subsystem: epoll/ ) or
( $ThisLine =~ m/^Reading configuration from/ ) or
( $ThisLine =~ m/^ngircd .* started/ ) or
( $ThisLine =~ m/^Created pre-defined channel/ ) or
( $ThisLine =~ m/^Not running with changed root directory/ ) or
( $ThisLine =~ m/^Notice: Can’t change working directory to/ ) or
( $ThisLine =~ m/^getnameinfo: Can’t resolve address/ ) or
( $ThisLine =~ m/^Shutting down all listening sockets/ ) or
( $ThisLine =~ m/^ServerUID must not be 0, using/ ) or
( $ThisLine =~ m/^OpenSSL .* initialized/ ) or
( $ThisLine =~ m/^Configuration option .* not set/ ) or
( $ThisLine =~ m/^User .* unregistered/ ) or
( $ThisLine =~ m/^Server restarting NOW/ ) or
( $ThisLine =~ m/^Server going down NOW/ ) or
( $ThisLine =~ m/^Shutting down connection .* \(Got QUIT command\.\)/ ) or
( $ThisLine =~ m/^Connection .* with .* closed / ) or
( $ThisLine =~ m/^Running as user/ ) or
( $ThisLine =~ m/^Shutting down connection .* \(Server going down/ ) or
( $ThisLine =~ m/^Shutting down connection .* \(Socket closed/ ) or
( $ThisLine =~ m/^Shutting down connection .* \(Ping timeout/ ) or
( $ThisLine =~ m/is closing the connection/ ) or
( $ThisLine =~ m/^ngircd done/ ) or
( $ThisLine =~ m/^Client unregistered/ ) or
( $ThisLine =~ m/^Client .* unregistered/ ) or
( $ThisLine =~ m/^User .* changed nick/ )
) {
# We don’t care, do nothing
} elsif ( my ($Host) = ($ThisLine =~ /Accepted connection .* from ([\d\.]+)/ )) {
$TriedConnections{$Host}++;
} elsif ( my ($User,$Connection) = ($ThisLine =~ /^User \"([^ ]+)!([^ ]+)\" registered /)) {
$GoodConnections{$Connection}++;
} elsif ( my ($User,$Connection) = ($ThisLine =~ /^Got invalid OPER from \"([^ ]+)!([^ ]+)\": / )) {
$FailedOpers{$Connection}++;
} elsif ( my ($User,$Connection) = ($ThisLine =~ /^No privileges: client \"([^ ]+)!([^ ]+)\", command / )) {
$BadOpCommands{$Connection}++;
} elsif ( my ($Host) = ($ThisLine =~ /^Shutting down connection .* \(Bad password\) with ([^ ]*):/)) {
$FailedLogin{$Host}++;
} elsif ( my ($User,$Connection) = ($ThisLine =~ /^Got valid OPER from \"([^ ]+)!([^ ]+)\", user is an IRC operator now/ )) {
$GoodOper{$Connection}++;
} else {
# Report any unmatched entries…
$OtherList{$ThisLine}++;
}
}

#######################################################

if (keys %BadOpCommands) {
print "\nIRCOp commands from regular users:\n";
foreach my $key (keys %BadOpCommands) {
my $totcount = 0;
$totcount += $BadOpCommands{$key};
my $plural = ($totcount > 1) ? "s" : "";
print " $key: $totcount time$plural\n";
}
}

if (keys %FailedLogin) {
print "\nFailed logins from:\n";
foreach my $key (keys %FailedLogin) {
my $totcount = 0;
$totcount += $FailedLogin{$key};
my $plural = ($totcount > 1) ? "s" : "";
print " $key: $totcount time$plural\n";
}
}

if (keys %FailedOpers) {
print "\nFailed attempts to become IRCOps from:\n";
foreach my $key (keys %FailedOpers) {
my $totcount = 0;
$totcount += $FailedOpers{$key};
my $plural = ($totcount > 1) ? "s" : "";
print " $key: $totcount time$plural\n";
}
}

if (keys %GoodOper) {
print "\nGood attempts to become IRCOps from:\n";
foreach my $key (keys %GoodOper) {
my $totcount = 0;
$totcount += $GoodOper{$key};
my $plural = ($totcount > 1) ? "s" : "";
print " $key: $totcount time$plural\n";
}
}

if (keys %TriedConnections) {
print "\nAttempted connections from:\n";
foreach my $ip (sort SortIP keys %TriedConnections) {
my $name = LookupIP($ip);
my $totcount = 0;
$totcount += $TriedConnections{$ip};
my $plural = ($totcount > 1) ? "s" : "";
print " $name: $totcount time$plural\n";
}
}

if (keys %GoodConnections) {
print "\nGood connections from:\n";
foreach my $key (keys %GoodConnections) {
my $totcount = 0;
$totcount += $GoodConnections{$key};
my $plural = ($totcount > 1) ? "s" : "";
print " $key: $totcount time$plural\n";
}
}

if (keys %OtherList) {
print "\n**Unmatched Entries**\n";
foreach $line (sort {$OtherList{$b}<=>$OtherList{$a} } keys %OtherList) {
print " $line: $OtherList{$line} Time(s)\n";
}
}

exit(0);
[/sourcecode]

If I have the inclination, I plan to update this to display different levels of detail based on the logwatch detail option, format the output a little nicer, handle some different bits of information and split the input lines up into more fields.  But you know, now it does 90% of what I want, that might never happen.

Summary

• Pick a name (based on the service you’re reporting on)
• Create /etc/logwatch/conf/services/myname.conf and describe the log file group to use, and any other options
• Create a script /etc/logwatch/scripts/services/myname in your favourite language and parse STDIN, sending useful information to STDOUT
• Bingo