Category Archives: Technology

Technology related things

Pointing and Shooting for 32 years (warning – photo heavy)

I had a camera when I was a kid, or maybe I borrowed my mum’s or sister’s camera, I’m not sure.  I know that it used 110 film though, because that I remember very well.    At some point, we changed to a 35mm instant camera, and I remember finding the film depressingly complex compared to the 110, and blew a few trying to wind it onto the spools and failing (before any of that stuff was automated).

I enjoyed taking photographs, and remember one school trip to Warkworth Castle, or maybe it was somewhere in York, when I took lots of photographs, mainly of ducks.  One teacher had words with me, about wasting film on ducks, but I quite liked the idea of taking some wild life shots.

Continue reading Pointing and Shooting for 32 years (warning – photo heavy)

Debian and KVM

I thought I’d spend a bit of time getting to know KVM on Debian.  How hard could it be, I thought?  I might fill a few hours, pass the time.

How wrong could I be!

It passed about 7 minutes.

apt-get install qemu-kvm libvirt-bin virtinst virt-manager virt-viewer

I bounced the box at this point, which may or may not have been necessary, then started up virt-manager, configured the VM, pointed it to the Debian net installation ISO, and off it went.

I was almost disappointed by the simplicity.  Guess I’ll have to find something else to play with.

Debian squeeze gdm / Xserver won’t start without a monitor

The little Atom based computer I was using as a Linux server in the house died.  Well technically, the fan died, which led to the rest of it dying.  It’s a proprietary case and motherboard,  so the fan isn’t something I can just pick up and replace (looks like a laptop fan, squeezed into the case), and I need something working faster than I could repair it.  I had a spare PC upstairs, which isn’t as quiet or as energy efficient as the Atom, but at least it works!  (This was after several days of Raspberry Pi’s failing and trashing SDHC cards, so I was already pretty pissed off with hardware, Linux and the whole building it yourself thing).

Anyway, as it happens, Debian reminded me how exceedingly trivial it is to build a server, and since I had full backups of the Atom PC it didn’t take long to get everything back up and running.  I was also reminded how slow the Atom chips can be, the P4 I’ve replaced it with is a world apart in terms of speed for whatever reason.

The Atom machine was running Ubuntu, but it used to frustrate me when it was quite the same as Debian, and I wanted to go back to a basic Debian build.  Also, since the P4 is more stock than the Atom I don’t need the bleeding edge drivers you get with Ubuntu.  All of this did leave me with one issue though.

When connected to a monitor, the Debian build works fine.  It works if you boot it with a monitor and then remove the monitor as well, but if you boot it without a monitor, it won’t start the X Server.  It tries about a million times [1] and then gives up.

I use Gnome under Debian, and because this machine is sitting physically in my house, I enable the autologon, and remote desktop control, so if I want, I can VNC in from my main machine.  I don’t usually need to do it, and I’m comfortable doing everything I want on that box from the command line, but every now and then it’s nice to use one or two GUI based apps.  Since Gnome supports this out of the box, I don’t feel the need to install VNC and start changing the config – it worked under Ubuntu, and it works under Debian if the monitor is there, I just needed it to work without the monitor.

I did a lot of reading around, there are plenty of suggestions about using VNC instead, some suggestions of modifying xorg.conf with some default display settings, and some other stuff.  I tried setting up an xorg.conf (new versions of X don’t use one by default, so you have to create one), but that didn’t seem to help.  More reading, and more playing around, and then I finally found the exact solution.

You can read the original page here.

Essentially, you need to add an entry to xorg.conf as I had been, but even then, X will probably refuse to start, because it detects modesetting drivers in the kernel and refuses to load the VESA driver.  Here’s the specific section from the error log if you get this,

(II) VESA: driver for VESA chipsets: vesa
(EE) VESA: Kernel modesetting driver in use, refusing to load
(WW) Falling back to old probe method for vesa
(EE) No devices detected.

So, as you can see from the document linked above, you need to disable the modesetting and then X will happily start.  This is the /etc/X11/xorg.conf config file I used (same as the original document),

Section "Device"
Identifier "VNC Device"
Driver "vesa"
EndSection

Section "Screen"
Identifier "VNC Screen"
Device "VNC Device"
Monitor "VNC Monitor"
SubSection "Display"
Modes "1280x1024"
EndSubSection
EndSection

Section "Monitor"
Identifier "VNC Monitor"
HorizSync 30-70
VertRefresh 50-75
EndSection

I also used both the i915 modeset change (because I had an i915 config file already), and the Nvidia one, since the machine has an Nvidia card in it.  A quick reboot, gdm and the X Server both started fine.  Very happy!

So the key isn’t just the xorg.conf above, which most people have posted about, you probably have to disable modesetting in the relevant graphics driver as well.

[1] okay, about half a million.

Problems with my Pi

I have a couple of Raspberry Pi’s (long story as to why I have two).  I spent a few hours one day a couple of weeks ago setting one of them up – pretty impressive, but I felt the overall performance was disappointing.  I moved the Pi between rooms in the house, did some more setup and then left it running for a few days.

Everything seemed okay.  Then one morning, it wasn’t responding.  I was running it headless and I could no longer SSH into it.  I had no choice but to power it off because it was no where near a composite or HDMI device.  Once powered off, it wouldn’t boot again at power on.  I carried it back through into the room with a TV, powered it up, and discovered the filesystem was corrupt.

No problem, pull out the SDHC card, whack it into a USB adapter, plug it into another Linux machine and fsck it.  That had a billion errors, I was impressed at quite how badly it had been corrupted.  I didn’t really think much more about it, I had a second SDHC card with raspbian already on it, and I didn’t really do much with the machines for a few weeks.

The day before yesterday, the Linux box I use to backup the other machines developed a fault, so I thought it would be the perfect time to replace it with one of the Pi’s.  Got it all set up on the shelf, hidden away, and ran it for a day.  Terrible disk IO performance, but otherwise okay.  I read around a bit and discovered that the SDHC card performance can be quite bad.  So I put a USB thumb drive in (16GB) with the intention of moving everything except /boot onto that to improve performance.

Except, when I reboot it – it wouldn’t boot.  Sure enough, corrupt filesystem.  I went back to check the original SDHC card and discovered not only was the filesystem corrupt, the card is damaged.  The second card doesn’t appear to be damaged but it would seem any power fluctuation in the Pi, or any power off without a shutdown, can trash the filesystem beyond recovery.

Now I’m concerned the Pi’s aren’t reliable enough to have sitting in the corner running 24×7.  A browse on the web shows I’m not alone either.  Quite disappointing.

Compiling analog 6.0 on Cygwin

Again, mostly for my own use later.  I had a need to run analog on my machine, and I didn’t want to download the Windows binary, because everything else I would be doing with the log files would be via Cygwin.  After a few unsuccessful attempts at compiling analog, I finally RTFM (read the flipping makefile) and made the following two changes.   In src/Makefile,

DEFS = -DHAVE_GD

LIBS = -lm -lz -ljpeg -lgd

That tells the analog makefile to use your pre-existing GD, JPEG and ZLIB libraries, rather than compiling the ones it comes with (it was those libraries which were giving me errors).  Once I’d done that, make clean and make worked fine and analog behaves as you would expect.

For reference, the errors I was getting before this change were,

libpng/pngwrite.o:pngwrite.c:(.text+0x1ec): undefined reference to `__imp__png_libpng_ver'
libpng/pngwrite.o:pngwrite.c:(.text+0x1f8): undefined reference to `__imp__png_libpng_ver'
libpng/pngwutil.o:pngwutil.c:(.text+0x45c): undefined reference to `__imp__png_IHDR'
libpng/pngwutil.o:pngwutil.c:(.text+0x6a5): undefined reference to `__imp__png_PLTE'
libpng/pngwutil.o:pngwutil.c:(.text+0x73d): undefined reference to `__imp__png_IDAT'
libpng/pngwutil.o:pngwutil.c:(.text+0x77e): undefined reference to `__imp__png_IEND'
/usr/lib/gcc/i686-pc-cygwin/4.5.3/../../../../i686-pc-cygwin/bin/ld: libpng/pngwutil.o: bad reloc address 0x12c in section `.rdata'
/usr/lib/gcc/i686-pc-cygwin/4.5.3/../../../../i686-pc-cygwin/bin/ld: final link failed: Invalid operation
collect2: ld returned 1 exit status
Makefile:76: recipe for target `analog' failed
make: *** [analog] Error 1

Compiling rrdtool on Cygwin

This post is half aide-mémoire and half public service announcement!  I use nmon at work to gather performance data, and I use customised version of nmon2web to graph it.  nmon2web relies on rrdtool, and I do a lot of my UNIX stuff these days using Cygwin.  Rather than use the Windows rrdtool binaries, I wanted to compile the latest version to run directly under Cygwin.

I’ve done this a few times now, with different Cygwin installs on both Windows XP and Windows 7, and every time I have to fight against various compilation issues.  I’ll cut to the chase, I hack at options and compiler settings and configure flags until it works, and I turn off most of the additional stuff to get it in.  But, it does work.

Get the Pre-requisites

All of the libraries and pre-requisites you need can be installed directly within Cygwin.  You’ll obviously need the regular development stuff (gcc, make, etc.) and you’ll also need the various libraries used by rrdtool.  Rather than worry about what you do and don’t need, I just whack on everything.  This is what I’ve got installed for each of the pre-reqs.

  • cairo (libcairo-devel, libcairo2)
  • glib (glib, glib-devel, glib2, glib2-devel, libglib1.2-devel, libglib1.2_0, libglib2.0-devel, libglib2.0_0)
  • libpng (libpng, libpng12, libpng14, libpng14-devel)
  • libxml2 (libxml2, libxml2-devel)
  • pango (libpango1.0-devel, libpango1.0_0, pango, pango-devel)
  • zlib (zlib, zlib-devel)
  • fontconfig (fontconfig, libfontconfig-devel, libfontconfig1)
  • freetype (libfreetype-devel, libfreetype6)
  • expat (expat, libexpat1, libexpat1-devel)

I’m certain that’s overkill, I had some of those installed already and installed a few extra libraries to get the compile working, but better safe than sorry!

configure options

The next step is working out what options to pass configure.  Some of these are required (on Windows 7 there are issues if you don’t use -no-undefined)

configure doesn’t seem to find the pango and cairo libraries under Cygwin unless I add these.

export CPPFLAGS="-I /usr/include/pango-1.0/pango/ -I /usr/include/cairo/cairo/"

and as I said, you need to prevent any undefined symbols in the libraries,

export LDFLAGS=-no-undefined

and then I basically turn off all the additional modules (perl, tcl, python, ruby) as well as mmap which doesn’t seem to work well under Cygwin anyway.  –prefix here is optional, it defaults to /opt, but I prefer everything under /usr/local.

./configure --disable-mmap --prefix=/usr/local/ --disable-tcl --disable-perl --disable-ruby --disable-python

Once that’s done, you can go for the compile.

Compiling

Run the usual make, you’ll get a whole bunch of warnings.  Such as,

warning: ‘optarg’ redeclared without dllimport attribute: previous dllimport ignored

No idea what they mean, but they don’t seem to break anything.

You’ll get a bunch of these,

*** Warning: linker path does not have real file for library -lstdc++.
*** I have the capability to make that library automatically link in when
*** you link to this library. But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libstdc++ and none of the candidates passed a file format test
*** using a file magic. Last file checked: /usr/lib/libpthread.a

because of the -no-undefined flag.  But again, doesn’t seem to break anything.

On Windows 7 (can’t remember if I got this on Windows XP) you’ll also get,

CCLD rrdupdate.exe
../libtool: line 8354: ./rrdupdate.exe: Permission denied
CC rrdcached-rrd_daemon.o

and you won’t be able to use rrdupdate.exe.  Running it gives the same permission denied error.  I’m not sure why yet, but I don’t use rrdupdate so it’s not been a big issue yet1.

Installing it all

Finally, after it all scrolls by, you can do a make install.

In /usr/local/bin you should end up with,

-rwxr-xr-x 1 User None 332K Jan 19 23:39 rrdcached.exe*
-rwxr-xr-x 1 User None 428K Jan 19 23:39 rrdcgi.exe*
-rwxr-xr-x 1 User None 641K Jan 19 23:39 rrdtool.exe*
-rwxr-xr-x 1 User None  14K Jan 19 23:39 rrdupdate.exe*

and rrdtool should work quite happily.

Finished!

So, know a better way?  Know why some bits still don’t work?  Know a sure fire way of fixing the warnings (if necessary), or getting rrdupdate to work, or being able to compile the additional modules?  Let me know!

  1. this only appears to be an issue with 64 bit Windows 7 []

NetFlix in the UK – going to be any good?

So, I know it’s early days, but NetFlix has just opened it’s doors in the UK and quite frankly, I thought they would have had a better line up to open with.

For the most part, the TV selection falls into four categories – old stuff of varying quality (Cracker, Fawlty Towers, 12 episodes of Men Behaving Badly), huge amounts of Kids TV and Animated stuff (Moschops, Thomas and Friends, X-Men), TV comedy shows like Saturday Night Live and a small, tiny amount of ‘new’ stuff (Dollhouse).

The movies don’t fare much better.  It’s not easy to list every movie, but here are the 2010-2011 movies on Netflix in various categories (this is every movie the interface returns, in that category for that year).

Action & Adventure 2010 & 2011

  • Blitz
  • Drive Angry
  • Delhi Belly
  • Thor: Tales of Asgard
  • Woochi
  • Shaolin
  • Red Hill
  • Fists of Rage
  • The Expendables
  • Little Big Soldier
  • Locked Down
  • Gotti the Mob Boss
  • Jackie Chan and the Kung Fu Kid
  • 71: Into the Fire
  • Nude Nuns With Big Guns
  • Rakht Charitra
  • Special Ops
  • Tees Maar Khan

Sci-Fi & Fantasy 2010 & 2011

  • Dead Space: Aftermath
  • Woochi
  • Area 51
  • Ticking Clock
  • The Disappearance of Haruhi Suzumiya
  • Hunter Prey

Thrillers 2010 & 2011

  • Blitz
  • Ticking Clock
  • Red Hill
  • Buried
  • My Soul to Take
  • The Disappearance of Haruhi Suzumiya
  • Winter in Wartime
  • Stone
  • Pimp
  • Raajneeti
  • House Under Siege

Maybe I’m missing something?  I mean, Blitz shows up on the main page which is a 2011 movie, but I can’t get it to show up on the basic lists, so perhaps I am missing something, but it still looks like a pretty lame opening gambit from NetFlix.   Unless they significantly improve the content very quickly and provide a more comprehensive way of listing movies (such as the simple LoveFilm A-Z view), I’ll be hard pushed to justify signing up.

Update: Aha, typically, after posting this I worked out what I was doing wrong.  You have to tell it to list things you’ve already seen in the summary lists.  I’ve added some items into the list, in italics, where they were missing first time.

Victory!

When we got our Windows 7 PC’s earlier in the year, I was really careful to take backups of absolutely everything before we wiped our old PC’s.  In fact, I ended up with about 3 backups of everything in several locations.  However, due to some issues with hardware, messing about, and frustration, I ended up losing my Picasa albums.  Not the ones on the web, but the definitions of albums on local disk.

I wasn’t too worried initially, I had all the photo’s.  Over time though I got more and more annoyed that there’s no easy way to re-link a web based album with Picasa on the PC.  No way to say – import this album and it’s settings, and relink to all the photos.  Very annoying.

Every time I start up Picasa I get a little twinge of annoyance.  Since I was painting some mini’s today and taking pictures, I’ve been starting Picasa a lot.

So I finally knuckled down, ransacked Google and my backups, and have restored all but one of the albums! Yay, success.  Picasa backups up the .pal files which represent the albums, and with some arcane copying too and fro you can convince Picasa to recreate them, although the behaviour seems inconsistent.  No idea why one of them didn’t work – but much easier to rebuild one album manually than 10, and now they’re all back fully synced and online.

Simple Debian Squeeze LAMP Config

Want the instant gratification solution?  Scroll down to “All In One” below.

Several of the Linux (Debian Squeeze) / Apache2 / MySQL / PHP (LAMP) configs I’ve seen on the ‘net include complexity you just don’t need (like suEXEC or suPHP).  If you’re setting up a basic server (physical, VPS, Xen, OpenVZ, whatever) and you’re going to be the only person running it, then setting up a LAMP environment is trivial.

This post assumes your server boots and you can log in and that it’s a basic Squeeze install with none of the relevant software pre-installed.  I hate sudo, so I’m going to assume you have either su’d to root (su – root) or that you’ve switched to root via sudo (sudo su – root).  If not, you’ll need to prefix all of these commands with sudo (after setting sudo up).

Although this little guide is for Debian, I think it’ll work unchanged on Ubuntu as well.  This guide also assumes that your server hosts a single website, so nothing fancy with any Apache2 VirtualHosts.

All of the follow steps can be combined, but I like to install by stages so that I can test each element on it’s own, and not get overwhelmed.

One of the big mistakes people new to Debian or Ubuntu do, is they think they have to manually edit config files to get things working.  The Debian packagers spend a lot of time making Debian packages work properly with each other with the absolute minimum manual effort.

Install Apache2

While Apache2 might be a bloated-warthog in the eyes of some system administrators, it’s ubiquitous, reliable, feature rich, well documented and well supported.  You can play with lighttpd, nginx and other options later, but this is LAMP, so let’s install the A.

The default Apache2 install on Debian will try and use the Worker MPM for Apache2.  We want the Prefork MPM when we use PHP5 later, so we’ll specify that straight away.

apt-get install apache2 apache2-mpm-prefork

And that’s it.  Assuming that runs successfully (errors are outside the scope of this walkthrough), you should be able to connect to the web server of your server on port 80 (iptables is also outside the scope of this, but a blank Debian install won’t have any firewalling in the way anyway).

If the install worked, connecting should return a page which says “It works!” and some information about this being the default web page.

Install PHP

Again, PHP gets bad press these days and maybe rightly so, but again it’s well used, well supported and well understood.  It’s probably easier to install it along with Apache2 in reality, because Debian does all the post config work at once then, but if you do it later, you are at least in control of the various stages.

We need to install two things, PHP5 and the Apache2 PHP module (on Debian, this package is libapache2-mod-php).

apt-get install php5 libapache2-mod-php5

and then restart Apache2 (I use the following command, any method that reloads the Apache2 config file works)

/etc/init.d/apache2 restart

That’s it.  You don’t need to edit anything, enable anything, configure anything.  Debian handles all that for you.  The default install of Apache2 creates a /var/www directory which it uses as the root for your web site.  If you go to that directory and create a file called test.php, and put this into it,

<?php phpinfo(); ?>

You can then test the PHP5 install has worked by connecting to your web server and requesting the file test.php.  If it asks you to save it, something went wrong (or you didn’t restart Apache2), otherwise you should get a full list of the PHP settings.

NB: Debian installs Suhosin by default.  This should be okay, but it can cause issues with some features of WordPress and phpMyAdmin.

Install MySQL

The MySQL install is the most complex, because you’ll have to create a password!  Otherwise, it’s as simple as the above steps.  As well as MySQL we need to remember to install the PHP5 MySQL module, otherwise we won’t be able to interact with the database.

apt-get install mysql-server php5-mysql

After some packages are downloaded and installed, you’ll be asked to set a password for the MySQL root user.  The dialog says this is not mandatory.  I think it should be.  You should absolutely set this password.  It does not need to match the Linux root user password, and as in all cases, it should actually differ significantly.

Restart Apache2 for good measure, and you’re done.  MySQL will be started, the relevant libraries are installed, and Apache2 / PHP5 / MySQL can all communicate as required.

By default, MySQL won’t be listening on any external interfaces, which is a good thing, so only your website can communicate with it.  Some guides recommend installing phpMyAdmin at this point, and you can if you want, although I prefer not to.

Permissions

In a default Debian install, Apache2 and PHP5 run as the www-data user.  By default, the permissions on /var/www are

drwxr-xr-x root root

That means the web server can’t create any files or directories in /var/www.  That’s a problem when installing things like WordPress which want to create their own config files or .htaccess files.  Because we’re not worried about multiple users on our server, or different customers, it’s safe to set the owner of /var/www to www-data:www-data, and do the same for all files in that directory.  This advice is only true for a server where you don’t mind all the websites running as the same user, but that’s the point of this example anyway!

All In One

The following command will install the whole thing in one go, and all you need to do is set the MySQL password.

apt-get install apache2 apache2-mpm-prefork php5 libapache2-mod-php5 mysql-server php5-mysql

Follow Up Steps

Later on you might want to install additional PHP modules (such as php5-curl, php5-gd, php5-mcrypt), and for sites which you expect to be busy on servers with not much memory you might want to look at using apache2-mpm-worker and FastCGI.

Exim4 (SMTP MTA) + Debian + Masquerading

I love Debian, and Exim4 seems to ‘just work’ for me most of the time, so I tend to use it for my MTA by preference.  Debconf handles the basic options for Exim4 pretty well, and usually I don’t need to mess with anything.

However, on one of my VPS’s I wanted to do what I used to refer to as masquerading.  I use the term to refer to having an SMTP masquerade as a different host on outbound e-mail addresses automatically.  So the server may be fred.example.net, but all outgoing mail comes from user@example.net.  It’s common if you want to handle the return mail via some other route – and for me I do.  My servers are in the darkstorm.co.uk domain, but I don’t want them handling mail for somehostname.darkstorm.co.uk, and I don’t want to have to configure every user with a different address, I just wanted a simple way to get Exim to masquerade.  Additionally, I only want external outbound mail re-writing, mail which is staying on the server should remain untouched.  This allows bob to mail fred on the server, and fred to reply without the mail suddenly going off the server, but if bob mails bill@example.org, then his address is re-written correctly.

I think I spent some time looking at this a couple of years ago, and had the same experience as recently – it’s a bit frustrating tracking down the best place to do it.  Firstly, Exim doesn’t have a masquerade option as such, and the manual doesn’t refer to masquerading in that way.  What it does have is an extensive rewriting section in the config file and support for doing that rewriting in various ways.

On top of this, the Debian configuration of Exim can be a little daunting at first, and how you achieve the configuration may depend on whether you’re using a split config or the combined config.

Anyway, enough rambling, you can get Exim to rewrite outgoing mail / masquerade by setting one macro.  This works on Debian 6 (Squeeze) with Exim4, but I assume it’ll work with Exim4 on any Debian installation.

Create (or edit)

/etc/exim4/exim4.conf.localmacros

Add the following line,

REMOTE_SMTP_HEADERS_REWRITE = *@hostname.example.net ${1}@example.net

Rebuild the Exim config (might not be essential but I do it every time anyway),

update-exim4.conf

and then recycle Exim (reload might work, but I tend to recycle stuff),

/etc/init.d/exim4 restart

That same macro is used for both the single monolithic config file, and the split config file.  It tells Exim that for remote SMTP only, it should rewrite any header that matches the left part of the line with the replacement on the right.  The ${1} on the right matches the * on the left (multiple *’s can be matched with ${1}, ${2}, etc.)

You can supply multiple rules by separating them with colons, such as this,

REMOTE_SMTP_HEADERS_REWRITE = *@hostname.example.net ${1}@example.net : *@hostname ${1}@example.net : *@localhost ${1}@example.net

There are more flags you can provide to the rewrite rules, and you can place rewrites in other locations, but the above will achieve the basic desire of ignoring locally delivered mail, but rewriting all headers on outbound e-mail which match.

Full details of Exim’s rewrite stuff is here.  Details about using Exim4 with Debian can be found in Debian’s Exim4 readme (which you can read online here).

I’m sure there are other ways of achieving this, there’s certainly an option in the debconf config (dc_hide_mailname) which seems hopeful, but it didn’t seem to do anything for me (maybe it only works when you’re using a smart relay?)  Either way, this option does what I wanted, and hey, this is UNIX, there’s always more than one way to skin a cat.

Edit: just had a look at the various Debian package files, and it looks like dc_hide_mailname only works if you’re using a smart relay option for Exim.  If you’re using the full internet host option from Debconf, it never asks you if you want to hide the mailname, and ignores that option when you rebuild the config files.