Category Archives: Internet

Orange is a state of mind

Posted on by

Finally settled on a new theme (the excellent Suffusion) which comes with a bunch of colour schemes, and I’ve picked Orange on Black (for now).  I know it makes little difference since the 6 people who actually read the blog do so via RSS so never see the colours anyway, and the other hits are all chinese ‘bots trawling data to try and gimp Google out of some cash.  Some folk are tired of dark themes, but I still like them.

The Orange on White version is pretty good as well and if I get bored with dark I might go for that.

I’ve had to work on some new headers, since the old ones were too tall and too narrow.  Managed to salvage the Matrix and floaty text ones, and added a couple of others.  Was very excited to find an Atari Space Invaders icon font!

Haven’t been doing much stuff out of work other than playing Dragon Age: Origins for the second time.  Tried playing it through a second time after originally completing it but it was too soon.  A lengthy break has done the trick and it’s almost as engaging the second time around.  Grete’s also playing it, so we’re taking turns like well behaved kids.  First time through I played as a warrior and was hugely frustrated by all the chests you can’t open until you find your first rogue companion.  So this time, started as a rogue!  Very different feel to the game, can’t go rushing into fights, need to somehow get the others engaged first, and if the warriors die, I’m not likely to survive (where-as when you’re the warrior you can hold on for quite a while with just Wynne).

Playing through again reminds me how good BioWare are at dialog and storyline.

Grete’s doing well at pretending I’m not annoying, when she plays and I shout ‘no, no, do Sten’s quest first’, or whatever.  So I thank her for that.

We bought a couple of the little add-ons, well bought one, got one free.  I’ve also bought Dragon Age: Awakening which was good but no where near worth the price.

I’ve also signed up to the PlayStation Plus service thing.  Got a few free games so far (Zen Pinball being the best).  Hopefully there’ll be enough free content in the next 10 months to justify the cost.

Custom Logwatch script for ngIRCd

Posted on by

So before I begin (or technically, just after I’ve begun), let me remind you that my perl skills are shockingly bad.  All my perl scripts are written in the same style as the script I was copying from at the time I wrote them.

Introduction

I’ve recently set up an IRC server, partly to mess about with it and partly to consider using it to keep in touch with friends.  I’m acutely aware that it’s the kind of thing that gets attacked, so I’ve made sure ngIRCd (the daemon I chose) is logging everything, and then I started looking for a logwatch (homepage) script to monitor the logs and alert me of anything suspicious going on.

Sadly, I couldn’t find one, so I decided to do the only sensible thing and write my own, which is fine, but as you’ll see if you search the web for ‘writing custom logwatch scripts’, it’s sort of both easy and hard.  It’s easy once all the bits fall into place, but sometimes the terminology gets in the way.  So, here’s how I did it.

Detail

You absolutely need two files, one which describes which logs you’re going to handle, and another which is the script which does the handling.  You should name them in some way which makes sense (after the service you’re monitoring for example).  Once you put them in the right place, logwatch will execute your script and you’re away.  There are some optional files, if you want to do some logfile pre-processing (I think) but as I never used those, I can’t comment.

So, I want to monitor ngIRCd which on my server logs everything it does to /var/log/messages under the service name ngircd.  Here’s an example line,

[sourcecode language=”plain” gutter=”false”]
Aug 13 08:01:54 hostname ngircd[10898]: User "bob!~ident@some.machine" registered (connection 8).
[/sourcecode]

The first thing I did was create a file describing which logs to monitor and how to filter the data, and I stole various bits of information from the other files distributed with logwatch.  I called my file ngircd.conf and place it in,

[sourcecode language=”plain” gutter=”false”]
/etc/logwatch/conf/services/
[/sourcecode]

That’s the default location on Debian.  Here’s the content of my file with some comments,

[sourcecode language=”bash”]
# set the title for the reports
Title = "ngIRCd"
# set the logfile to the messages log file *group*
LogFile = messages
# only return entries made by ngIRCd which reduces our effort in the script
*OnlyService = ngircd
# remove the date / time stamp, hostname, service name, etc.
*RemoveHeaders
[/sourcecode]

Line 3 is important and took me a little while to work out.  In the config file for your service, you describe the log file group that is used, which in turn tells logwatch which file in the /logfiles/ directory structure describes the actual log files which are scanned.  So the above line tells logwatch (in the case of Debian) to use the log files described in /usr/share/logwatch/default.conf/logfiles/messages.conf.  That file handles the log file names, how to deal with date/time stamps, archived logs, etc.

If the log files for your new service don’t already have a matching log file group configuration file, you should create one in /etc/logwatch/conf/logfiles, using an example from /usr/share/logwatch/default.conf/logfiles.  Anyway, in my case, since I was using /var/log/messages which is already described in /usr/share/logwatch/default.conf/logfiles/messages.conf I didn’t need to create one.

Now that you’ve got the service configuration covered, you need a script, and it needs to be named after the config file (so if you call your config file foo.conf, then your script needs to be called foo).  You can write this script in any language that can read from STDIN and write to STDOUT, but like other folk before me I made the joyful error of sticking to perl.

You place this file in

[sourcecode language=”plain” gutter=”false”]
/etc/logwatch/scripts/services
[/sourcecode]

The important things to remember are,

  1. your script will receive the content of the appropriate logs via STDIN
  2. it should write output to STDOUT and should use the environment variable LOGWATCH_DETAIL_LEVEL to determine the detail level passed to the logwatch program
  3. the output should be tidy and should avoid being verbose
  4. if you’ve configured the service conf script correctly you won’t need to worry about parsing dates, stripping headers, or other rubbish.  This does depend on the log file in question though and the application.
  5. To keep in line with other scripts, you should capture everything you know you don’t care about and ignore it, process stuff you do care about, and report stuff you don’t recognise.

The link below is the script I cobbled together to handle ngIRCd so far.  At the moment, I ignore my own advice and don’t check the detail level, I just wanted initially to get my data out.  I have no idea if the regexp’s are correct or efficient, but at present, it displays what I care about. Is that enough caveats?  I’m not looking for feedback on the quality of my perl! I’m just trying to show how it can be done.

ngircd

[sourcecode language=”perl”]
#!/usr/bin/perl
##########################################################################
# ngircd
##########################################################################

use Logwatch ‘:all’;

my $Detail = $ENV{‘LOGWATCH_DETAIL_LEVEL’} || 0;
my $Debug = $ENV{‘LOGWATCH_DEBUG’} || 0;

my %FailedLogin = ();
my %FailedOpers = ();
my $FailedOpCommands;
my %TriedConnections = ();
my %GoodConnectionsi = ();
my %GoodOper = () ;
my %BadOpCommands = ();
my %OtherList = ();

if ( $Debug >= 5 ) {
print STDERR "\n\nDEBUG: Inside ngircd Filter \n\n";
$DebugCounter = 1;
}

while (defined(my $ThisLine = <STDIN>)) {
if ( $Debug >= 5 ) {
print STDERR "DEBUG($DebugCounter): $ThisLine";
$DebugCounter++;
}

chomp($ThisLine);
if ( # We don’t care about these
( $ThisLine =~ m/connection .* shutting down / ) or
( $ThisLine =~ m/^New TLSv1 connection using cipher/ ) or
( $ThisLine =~ m/^Now listening on/ ) or
( $ThisLine =~ m/^IO subsystem: epoll/ ) or
( $ThisLine =~ m/^Reading configuration from/ ) or
( $ThisLine =~ m/^ngircd .* started/ ) or
( $ThisLine =~ m/^Created pre-defined channel/ ) or
( $ThisLine =~ m/^Not running with changed root directory/ ) or
( $ThisLine =~ m/^Notice: Can’t change working directory to/ ) or
( $ThisLine =~ m/^getnameinfo: Can’t resolve address/ ) or
( $ThisLine =~ m/^Shutting down all listening sockets/ ) or
( $ThisLine =~ m/^ServerUID must not be 0, using/ ) or
( $ThisLine =~ m/^OpenSSL .* initialized/ ) or
( $ThisLine =~ m/^Configuration option .* not set/ ) or
( $ThisLine =~ m/^User .* unregistered/ ) or
( $ThisLine =~ m/^Server restarting NOW/ ) or
( $ThisLine =~ m/^Server going down NOW/ ) or
( $ThisLine =~ m/^Shutting down connection .* \(Got QUIT command\.\)/ ) or
( $ThisLine =~ m/^Connection .* with .* closed / ) or
( $ThisLine =~ m/^Running as user/ ) or
( $ThisLine =~ m/^Shutting down connection .* \(Server going down/ ) or
( $ThisLine =~ m/^Shutting down connection .* \(Socket closed/ ) or
( $ThisLine =~ m/^Shutting down connection .* \(Ping timeout/ ) or
( $ThisLine =~ m/is closing the connection/ ) or
( $ThisLine =~ m/^ngircd done/ ) or
( $ThisLine =~ m/^Client unregistered/ ) or
( $ThisLine =~ m/^Client .* unregistered/ ) or
( $ThisLine =~ m/^User .* changed nick/ )
) {
# We don’t care, do nothing
} elsif ( my ($Host) = ($ThisLine =~ /Accepted connection .* from ([\d\.]+)/ )) {
$TriedConnections{$Host}++;
} elsif ( my ($User,$Connection) = ($ThisLine =~ /^User \"([^ ]+)!([^ ]+)\" registered /)) {
$GoodConnections{$Connection}++;
} elsif ( my ($User,$Connection) = ($ThisLine =~ /^Got invalid OPER from \"([^ ]+)!([^ ]+)\": / )) {
$FailedOpers{$Connection}++;
} elsif ( my ($User,$Connection) = ($ThisLine =~ /^No privileges: client \"([^ ]+)!([^ ]+)\", command / )) {
$BadOpCommands{$Connection}++;
} elsif ( my ($Host) = ($ThisLine =~ /^Shutting down connection .* \(Bad password\) with ([^ ]*):/)) {
$FailedLogin{$Host}++;
} elsif ( my ($User,$Connection) = ($ThisLine =~ /^Got valid OPER from \"([^ ]+)!([^ ]+)\", user is an IRC operator now/ )) {
$GoodOper{$Connection}++;
} else {
# Report any unmatched entries…
$OtherList{$ThisLine}++;
}
}

#######################################################

if (keys %BadOpCommands) {
print "\nIRCOp commands from regular users:\n";
foreach my $key (keys %BadOpCommands) {
my $totcount = 0;
$totcount += $BadOpCommands{$key};
my $plural = ($totcount > 1) ? "s" : "";
print " $key: $totcount time$plural\n";
}
}

if (keys %FailedLogin) {
print "\nFailed logins from:\n";
foreach my $key (keys %FailedLogin) {
my $totcount = 0;
$totcount += $FailedLogin{$key};
my $plural = ($totcount > 1) ? "s" : "";
print " $key: $totcount time$plural\n";
}
}

if (keys %FailedOpers) {
print "\nFailed attempts to become IRCOps from:\n";
foreach my $key (keys %FailedOpers) {
my $totcount = 0;
$totcount += $FailedOpers{$key};
my $plural = ($totcount > 1) ? "s" : "";
print " $key: $totcount time$plural\n";
}
}

if (keys %GoodOper) {
print "\nGood attempts to become IRCOps from:\n";
foreach my $key (keys %GoodOper) {
my $totcount = 0;
$totcount += $GoodOper{$key};
my $plural = ($totcount > 1) ? "s" : "";
print " $key: $totcount time$plural\n";
}
}

if (keys %TriedConnections) {
print "\nAttempted connections from:\n";
foreach my $ip (sort SortIP keys %TriedConnections) {
my $name = LookupIP($ip);
my $totcount = 0;
$totcount += $TriedConnections{$ip};
my $plural = ($totcount > 1) ? "s" : "";
print " $name: $totcount time$plural\n";
}
}

if (keys %GoodConnections) {
print "\nGood connections from:\n";
foreach my $key (keys %GoodConnections) {
my $totcount = 0;
$totcount += $GoodConnections{$key};
my $plural = ($totcount > 1) ? "s" : "";
print " $key: $totcount time$plural\n";
}
}

if (keys %OtherList) {
print "\n**Unmatched Entries**\n";
foreach $line (sort {$OtherList{$b}<=>$OtherList{$a} } keys %OtherList) {
print " $line: $OtherList{$line} Time(s)\n";
}
}

exit(0);
[/sourcecode]

If I have the inclination, I plan to update this to display different levels of detail based on the logwatch detail option, format the output a little nicer, handle some different bits of information and split the input lines up into more fields.  But you know, now it does 90% of what I want, that might never happen.

Summary

  • Pick a name (based on the service you’re reporting on)
  • Create /etc/logwatch/conf/services/myname.conf and describe the log file group to use, and any other options
  • Create a script /etc/logwatch/scripts/services/myname in your favourite language and parse STDIN, sending useful information to STDOUT
  • Bingo

Hosting Changes

Posted on by

Hopefully no one’s noticed that I’ve moved my websites (including this one) to a new hosting provider (well, new for this site, but not new to me in general).  When I first decided to move away from shared hosting at Gradwell, I wanted to find a VPS solution to give me more control over the sites.  At the time, I was hosting an EverQuest guild website which got a reasonable amount of traffic, and so I chose a VPS solution that would definitely have enough oomph and bandwidth to deliver that.  Not a huge amount, but enough.  It was my first time in the VPS market and I wasn’t really sure who were best, what was good value and what the different offerings resulted in.

The 1and1 service I went with was pretty good, but not what I would call cheap.  Over the 18 months or so I’ve had the service it’s been pretty reliable, a few unexplained outages, and a couple of periods of downtime that were longer than I would have liked.  But the VPS was powerful, had plenty of memory and lots of network capacity and easily delivered the 7 or so domains I hosted on it.

Not long after I moved the sites though, the EverQuest guild site dropped off dramatically in terms of load (lots of people left the game), and after 18 months it’s become apparent that the 5 or 6 vanity domains I host really don’t justify the cost and performance of the 1and1 VPS.

I looked again at shared hosting, because I’m a pretty good example of who should use it.  I tried tsohosts and while they’re excellent value and I have nothing bad to say about them, I really don’t get on with cPanel and the shared hosting mentality, especially after running my own VPS for so long.  I wanted to get into the config files and set things up ‘just so’, and after only 2 or 3 days fighting with cPanel I gave up and bought another VPS from Gandi.

I love the Gandi system (I already have another VPS from them hosting a usenet server), and although a single share VPS is pretty low resource you can deliver quite a lot from a Linux machine with not much in the way of power these days, especially when what you’re delivering are vanity domains with almost no traffic.

So I’ve got a single share (256MB memory) VPS from Gandi and over the last couple of days I’ve moved everything over, and I’m pleased at how easy it was.  The 1and1 VPS was running CentOS but I’ve gone for Debian with Gandi (I prefer Debian) so not everything could be simply copied over, but the content (static and mysql data) was easy enough to transfer, so now it’s just a case of making sure the VPS is secure and managed properly.

If you’re looking for some domains or a VPS, I really can’t recommend Gandi enough, my only gripe is there’s no easy way to pay monthly (no direct debit facility, so you have to top up a pre-pay account when you remember), but it’s a small issue when the service is so excellent.