Graphing INN2 stats through Munin

I run a news server (using INN2), and I graph the performance of the server on which it sits using Munin.  I used to use Cacti, but find Munin much, much easier to set up and get going, even if the interface isn’t quite as fancy.

Munin comes with a collection of regular plugins for graphing various linux services and system stats (such as Apache, MySQL, etc.) but it obviously can’t include everything.  I spent about 10 minutes seeing if anyone had written their own plugin for reporting on INN2 traffic stats, but couldn’t see anything.  So I rolled my own.  I tried this under Cacti and frankly, it was painful.  Not so much the script to gather the data, but getting the graphs actually up and running.

With Munin, it’s trivial.  You can read all about it here, which is where I worked out what I needed to do.

Disclaimer: My shell scripting is shockingly lazy, and my use of awk can be compared to someone playing an exquisite 400 year old violin with a hammer.  Sorry, but suck it up.

I use Munin on a Debian system installed, so the paths may vary depending on your setup.  You need to complete three steps.

1. Get your stats

Munin collects data every 300 seconds.  You need a script which can be run every 300 seconds to return data.  The output of your script should be,

datavariable.value actualvalue

where datavariable is the name you give your erm, data variable, value is the word value and actualvalue is the value you want to graph.  If you have several values, you just output them one line at a time,

datavariable.value actualvalue
datavariable_b.value actualvalue
datavariable_c.value actualvalue

Once you have a script which can output that, you’re most of the way there.

2. Describe your graph

What Munin actually graphs is based on what your script outputs when it is passed a command line parameter of config.  There are a bunch of values you should return which describe the graph, and then a bunch of values which describe the variables (i.e. the stuff above returned by your script), their format and how they should be put onto the graph.  Here’s a simple example,

[sourcecode language=”bash” gutter=”false”]
echo ‘graph_title CPU usage’
echo ‘graph_vlabel %’
echo ‘graph_scale no’
echo ‘graph_info This graph shows how CPU time is spent.’
echo ‘system.label system’
echo ‘system.draw AREA’
echo ‘system.min 0’
[/sourcecode]

That would tell Munin to create a graph with the relevant title, to label the vertical axis with a single %, add a little bit of text and graph a single variable called system.  The system variable is labelled on the graph as system, is an AREA plot and has a minimum value of 0.

There’s plenty more you can do, which you can read about on the Munin site, but that’s the basic stuff, and should be returned when your script is called with config on the command line.  (By the way, your script can be anything executable, perl, shell, etc.)

3. Tell Munin to use your script

You should place your script (executable) in /etc/munin/plugins or a link to it.  You should also add various config options to /etc/munin/plugin-conf.d/munin-node which control how your script behaves.  I’m not covering that in detail here but it’s fairly obvious.

You then restart the munin agent, and bingo your stuff turns up.  Like magic.

INN2 Graphs

So, if you want to graph INN2 data, set INN2 to output stats (inn.conf and innfeed.conf) every 300 seconds (or quicker).  Then use the following two scripts, and entries in the munin-node config file (as appropriate for your installation).

inn2_incoming

[sourcecode language=”bash”]
#!/bin/sh
#
# Plugin to monitor inn2 status file, and report incoming traffic
#
# Require read permissions on the appropriate status file, and INN2 to be configured
# to update the file at or quicker than every 5 minutes.
#
#
#
# Parameters:
#
#       config   (required)
#       autoconf (optional – used by munin-config)
#
# Magick markers (optional):
#%# family=auto
#%# capabilities=autoconf
#
# config example for /etc/munin/plugin-conf.d/munin-node
#[inn2_incoming]
#user news
#group news
#env.logfile /var/log/news/inn_status.html
#

LOG=${logfile:-/var/log/news/inn_status}
CATEGORY=${category:-inn2}

if [ "$1" = "autoconf" ]; then
if [ -r "$LOG" ]; then
echo yes
exit 0
else
echo no
exit 1
fi
fi

if [ "$1" = "config" ]; then

echo ‘graph_title INN2 incoming article stats’
echo ‘graph_args –base 1000 -l 0’
echo ‘graph_scale no’
echo ‘graph_vlabel article count’
echo ‘graph_category’ $CATEGORY
echo ‘graph_period minute’

echo ‘offered.label Total articles offered’
echo ‘offered.type DERIVE’
echo ‘offered.min 0’
echo ‘accepted.label Articles accepted’
echo ‘accepted.type DERIVE’
echo ‘accepted.min 0’
echo ‘refused.label Articles refused’
echo ‘refused.type DERIVE’
echo ‘refused.min 0’
echo ‘rejected.label Articles rejected’
echo ‘rejected.type DERIVE’
echo ‘rejected.min 0’
echo ‘dupe.label Duplicate articles’
echo ‘dupe.type DERIVE’
echo ‘dupe.min 0’

exit 0
fi

awk -v RS="" ‘/global/ {print $0}’ < $LOG | awk -v RS="" -F":" ‘{printf "offered.value %d\naccepted.value %d\nrefused.value %d\nrejected.value %d\ndupe.value %d\n",$5,$7,$10,$13,$16}’
[/sourcecode]

inn2_outgoing

[sourcecode language=”bash”]
#!/bin/sh
#
# Plugin to monitor inn2 status file, and report outgoing traffic
#
# Require read permissions on the appropriate status file, and INN2 to be configured
# to update the file at or quicker than every 5 minutes.
#
#
#
# Parameters:
#
# config (required)
# autoconf (optional – used by munin-config)
#
# Magick markers (optional):
#%# family=auto
#%# capabilities=autoconf
#
# config example for /etc/munin/plugin-conf.d/munin-node
#[inn2_incoming]
#user news
#group news
#env.logfile /var/log/news/innfeed.status
#

LOG=${logfile:-/var/log/news/innfeed.status}
CATEGORY=${category:-inn2}

if [ "$1" = "autoconf" ]; then
if [ -r "$LOG" ]; then
echo yes
exit 0
else
echo no
exit 1
fi
fi

if [ "$1" = "config" ]; then

echo ‘graph_title INN2 outgoing article stats’
echo ‘graph_args –base 1000 -l 0’
echo ‘graph_scale no’
echo ‘graph_vlabel article count’
echo ‘graph_category’ $CATEGORY
echo ‘graph_period minute’

echo ‘offered.label Total articles offered’
echo ‘offered.type DERIVE’
echo ‘offered.min 0’
echo ‘accepted.label Articles accepted’
echo ‘accepted.type DERIVE’
echo ‘accepted.min 0’
echo ‘refused.label Articles refused’
echo ‘refused.type DERIVE’
echo ‘refused.min 0’
echo ‘rejected.label Articles rejected’
echo ‘rejected.type DERIVE’
echo ‘rejected.min 0’
echo ‘missing.label Missing articles’
echo ‘missing.type DERIVE’
echo ‘missing.min 0’
echo ‘deferred.label Deferred articles’
echo ‘deferred.type DERIVE’
echo ‘deferred.min 0’

exit 0
fi

awk -v RS="" ‘/global \(process\)/ {print $0}’ < $LOG | awk -v RS="" -F":" ‘{printf "offered.value %d\naccepted.value %d\nrefused.value %d\nrejected.value %d\nmissing.value %d\ndeferred.value %d\n",$5,$7,$9,$11,$13,$15}’
[/sourcecode]

Obviously, depending on your version of INN2, you may have to tweak the awk entry to get the right values.

Lastly, put the following lines into /etc/munin/plugin-conf.d/munin-node

[sourcecode language=”plain” gutter=”false”]
[inn2_incoming]
user news
group news
env.logfile /var/log/news/inn_status.html

[inn2_outgoing]
user news
group news
env.logfile /var/log/news/innfeed.status
[/sourcecode]

And again, those values will need to change to match your distribution / INN2 setup.  Hope this helps someone.

Usenet / netnews

Usenet was one of the first things I did on the Internet, coming from a background of FidoNet / Bulletin Boards it was a natural progression, along with e-mail.  The web was still in it’s infancy, Gopher was still more useful, or e-mail responders, at the time.  IRC was huge and if I wasn’t reading Usenet I was probably on IRC (usually, both).

I spent a lot of time posting to Usenet and using software to run a news server locally (sort of a caching only server), I even wrote my own (newsmangler, although there appears to be a new tool out and about called that as well, resulting in various hits to this blog).

For whatever reason, my involvement in Usenet slowed and in fact, Usenet use for text-discussion slowed in general.  Binary use grew and is still huge, but I don’t really think it’s the right place for binaries.

Recently though my involvement has increased again, mainly in the uk.* hierarchy although I lurk a lot more than I used to and post a lot less.  I’ve been looking for an excuse to get another Virtual Private Server (this time with Gandi because their setup is so nice) and I’ve finally relented and put together an INN2 Usenet server ( http://matrix.darkstorm.co.uk/news/ ).  It’s been interesting, there’s a steep learning curve to getting it set up so that it behaves, and then getting peers (up to 3 now).  But it’s been fun.  I hope to keep developing the server and maybe end up offering access to other folk, if there are any others left who still read usenet that is 😉