Playing Badminton through Clearpress

Yesterday I launched, finally, my badminton ladder web application for our sports and social club. Initially, I write something about 3 years ago, which used flat csv files and a cgi script for each page, which dealt with generating the html and processing results, and updating the ladder, and....

All not very practical, but at least showed what I could do at the time.

Earlier this year we had a change of hardware for our webservers, and we lost the functionality, due to the time lag of all the files being kept up to date as they changed on the multiple servers. A big problem.

So I decided it was time for a rewrite, which I did using Clearpress (http://clearpress.net/), trialling out git and github at the same time (I like this so much more than sourceforge! and svn).

I have blogged about Clearpress and its MVC framework before, so I won't spend time doing so, but with a little work, the setup used 5 tables in a database to produce a reliable system.

team - stores a team name, wins, losses and gives them a unique identifer
player - stores a player name, email and gives them a unique identifier
player_team - join table for a player to a team
ladder_type - our ladder has three sub ladders, to deal with new teams and those which haven't played for a long time
ladder - links team/position/ladder_type

The whole thing can be found on github

git://github.com/setitesuk/badminton-ladder.git

It is currently set up to deploy using a SQLLite database, but in production use, we are using a mysql database, and the schema is there. You just need to modify the config.ini file to use a mysql database, which is supported through clearpress.

So, if you are after a web app badminton ladder, then take a look. It is all available as Open Source (GNU Public Licence).

Next, to create a Tennis Competition app.

lc(x) - possible bad practice?

use strict; use warnings;

We all use the above, right? Well, certainly we should, and my team does;

Now this always causes a warning to occur when testing an undefined value (exception - if the test is for it to be undef).

my $arrayref = $self->method_returning_array_ref() || [];

Now, assuming that the method will always return an arrayref or undef, I will have an arrayref.

Now, it is common enough (returning stuff from an XML dom for example) that there is actually only 1 thing in the array, so I test against it

if ($arrayref->[0] eq 'yes') {
do something..
}

Now, if $arrayref->[0] is undef, this always throws an uninitialised variable warning. This normally leads me to change to

if ($arrayref->[0] && $arrayref->[0] eq 'yes') {
do something..
}

so that I don't spam up the logs with warnings.

However, we discovered today that by lowercasing the $arrayref->[0] variable, this will turn an undef into an empty string (for the conditional), therefore dispensing with any warnings.

if (lc$arrayref->[0] eq 'yes') {
do something..
}

Is this good or bad coding practice?

Reasons for it to be good
- You do not need an extra conditional just to dispense with the warning
- code less

Reasons for it to be bad
- It doesn't seem right
- The conditional is no longer testing against the pure result
- Are we getting rid of the warning for the wrong reason?
- Does it read correctly?

At this time, it is not something that the code police (perlcritic) seem to think is a bad practice, and certainly will make less code for us. It just seems like we are breaking an unwritten coding rule.

Stomping on the Rabbit, Pt2

So now I have this all installed, I have to start using it. Net::Stomp (see cpan.org) is the interface I would like to use, but the setup seems very much done for ActiveMQ (as some of the headers are 'activemq.xxxxx'.

This means that by default, I only seem to be able to send and receive messages to a queue that exists only whilst someone is listening.

The example scripts that come with the RabbitMQ-Stomp distribution only show this for PERL, however, the Ruby examples show much more (persistent queues, topics, etc).

Thankfully, I found the following post

http://tinyurl.com/pdznw9

which explains the headers for RabbitMQ to produce persistent queues and operating topics. For Net::Stomp, the key is to add these headers to the key-value pairs in your connection/send hash, and happily they go through. Here are some examples:

Persistent Queues:

Receiver -

my $stomp = Net::Stomp->new({hostname=>'localhost', port=>'61613'});
$stomp->connect({login=>'guest', passcode=>'guest'}) or croak $EVAL_ERROR;

$SIG{INT} = sub {
$stomp->unsubscribe({
destination => q(/queue/father/ted),
});
exit;
};

$stomp->subscribe({
destination => q(/queue/super/ted),
q{auto-delete} => q{false}, # setting these flags will make your queue remain whilst
q{durable} => q{true}, # the subscriber(s) go away, and pick up messages afterwards
ack => q(client),
});

while (1) {
my $frame = $stomp->receive_frame;
print $frame;
print $frame->body . "\n";
$stomp->ack({frame=>$frame});
last if $frame->body eq 'QUIT';
}

$stomp->disconnect;

Sender -

my $stomp = Net::Stomp->new({hostname=>'localhost', port=>'61613'});
$stomp->connect({login=>'guest', passcode=>'guest'}) or croak $EVAL_ERROR;
$stomp->send({destination => '/queue/super/ted',
bytes_message=>1,
body=>($ARGV[0] or "test\0message")});
$stomp->disconnect;

Topics:

Topics are handled differently in RabbitMQ to ActiveMQ. In ActiveMQ, they sit under a namespace /topic/xxx/yyy and a combination of the client-id, destination, exchange and routing key all make up the subscriber to the topic

In the case of RabbitMQ, it seems a bit more generic than that, but has some differences which make it (as far as I can see) non-persistent.

Receiver -

my $stomp = Net::Stomp->new({hostname=>'localhost', port=>'61613'});
$stomp->connect({login=>'guest', passcode=>'guest'}) or croak $EVAL_ERROR;

$SIG{INT} = sub {
exit;
};

$stomp->subscribe({
destination => q{bananaman}, # needs to be a unique client-id
exchange => q{amq.topic},
routing_key => q{bananas},
});

while (1) {
my $frame = $stomp->receive_frame;
print $frame;
print $frame->body . "\n";
last if $frame->body eq 'QUIT';
}

$stomp->disconnect;

This sets up a receiver, which if you use rabbitmqctl list_queues shows a queue bananaman which will receive messages (whilst he is listening) to the topic bananas

using the following gives an anonymous looking receiver queue

$stomp->subscribe({
id => q{banaman}, # needs to be a unique client-id
destination => q{},
exchange => q{amq.topic},
routing_key => q{bananas},
});

which you can unsubscribe from explicitly

$stomp->subscribe({
id => q{banaman},
});

However, it would appear that the unsubscription occurs anyway, so unfortunately, any posts to the topic bananas would be lost to bananaman whilst he is away.

Sender for topics -

my $stomp = Net::Stomp->new({hostname=>'localhost', port=>'61613'});
$stomp->connect({login=>'guest', passcode=>'guest'}) or croak $EVAL_ERROR;
$stomp->send({destination => q{bananas},
exchange => q{amq.topic},
body=>($ARGV[0] or "test\0message")});
$stomp->disconnect;

You will notice here that the destination for this message is the routing_key, and that the exchange flag is set.

The problem here is that essentially, each receiver to a particular topic (routing_key) forms a temporary queue to which the message is added, and then sent from, but the queue is just that, temporary and goes away when the receiver does, so that receiver will never get the message if they are not present when the message is sent to the message-queue.

I am very keen to hear from anyone who has found out how to work around this.

The other main problem that we have seen due to this is MQ agnosticism doesn't exist using Net::Stomp, as essentially the problem is that it is written with ActiveMQ in mind, and just feeds headers through. This means that there is no conversion of the headers between using ActiveMQ-STOMP, RabbitMQ-STOMP or any other message-queue-STOMP that is out there. This means that there is no way to easily hot-swap message-queues without code re-write. Again, I'd be happy to hear form anyone who has worked out how to get around this.

As for now, it looks as though it is going to be ActiveMQ, as this has been installed centrally for us, and appears to manage the persistence a bit better, along with headers which are documented in Net::Stomp.

Stomping on the Rabbit

we are trying to make a move to using message queues in my group to deal with pipelines and talking to other apps.

There should be some great advantages for us, and I am quite excited by this.

First thing, set up a STOMP message queue.

Now, I am currently writing a simple message_queue based on ClearPress, but this is not yet ready, so I have just downloaded and installed RabbitMQ.

RabbitMQ is written in Erlang, (which I have just set myself the challenge of learning from the Pragprog book Programming Erlang by Joe Armstrong). The first challenge was setting it up.

First: You need erlang installed. I had already done this, so wasn't a problem. Jsut make sure that erlc is in your path.

Now, I had dowloaded the latest release tarball, and expanded this, setting up in my sandbox area. However, this threw my a serious curveball with trying to install STOMP.

Thankfully, Google is my friend, and someone had the same problem, since the STOMP needs installing against the correct version number, so I hereby give the definitive installation guide to getting RabbitMQ up and running in a sandbox on MacOSX.

(I accept no responsibility for this not working on your machine!)

1) Install Mercurial (yet another distributed version control system, but the one which RabbitMQ is on).

2) (Thanks to everyone on this page http://tinyurl.com/pk7xfd)
hg clone http://hg.rabbitmq.com/rabbitmq-server
hg clone http://hg.rabbitmq.com/rabbitmq-codegen
hg clone http://hg.rabbitmq.com/rabbitmq-stomp
(cd rabbitmq-server; hg up rabbitmq_v1_5_4)
(cd rabbitmq-codegen; hg up rabbitmq_v1_5_4)
(cd rabbitmq-stomp; hg up rabbitmq_v1_5_3)

3) (At this point, you need to check your version of python and simplejson, which needs installing)

4) In the various MakeFiles, alter source roots to point to where you want the various db and log files to go, where your rabbit source root is, etc..

Eg.

I set up in my sandbox a folder 'rabbitmq', in which I put a logs dir, mnesia (db files) dir and rabbit-mnesia dir (into which I put the rabbitmq.conf file)

In 'rabbitmq-server/Makefile'

RABBITMQ_NODENAME=rabbit
RABBITMQ_SERVER_START_ARGS=/my/sandbox/rabbitmq/rabbitmq.conf
RABBITMQ_MNESIA_DIR=/my/sandbox/rabbitmq/$(RABBITMQ_NODENAME)-mnesia
RABBITMQ_LOG_BASE=/my/sandbox/rabbitmq/logs


4) make -C rabbitmq-server
make -C rabbitmq-stomp run

Hey presto, you now have a rabbitmq-stomp server up and running.

Now you have done this, you can try either the perl or ruby tests.

Hope this guide is of use to someone.

Cheers

Andy

Splitting a Project

As part of the Group I am in, we have had a project sanger-pipeline. Now, early on in the life of the group, it was quite small, and was essentially just a few wrappers around the Illumina Analysis pipeline, so that it hooked into our tracking system, and scripts to manage the movement of images off the individual Genome Analysers to our Farm.

However, as with most things, it got bigger...

and bigger...

and bigger until it was managing not only the wrappers (which were being maintained by someone who was in another group), and the movement of images, but processing other files, loading compressed versions of the images into other databases, deciding when it had everything it expected, what and when exactly to delete outdated files...and managing an apache webservice!

So, over the last 2 weeks, I have been splitting the project down. This was not a trivial thing to do, as many modules where used in multiple places, but split it I did to the following:

sanger-pipeline - this now only has wrapper scripts and modules relating to hooking the Illumina Analysis pipeline in to our system.

instrument_handling - this deals with scripts and modules relating to the smooth running and mirroring of data of the GA's, plus our controlcentre script, whose primary use is to run these as a special user (although it does do some other things).

data_handling - all these scripts and modules are responsible for managing data on the farm (once it has mirrored) including doing checks to ensure that data that needs long term storage are where they should be

sflogin_web_apps - the management of the apache server and the scripts that we run on the farm (instead of the usual webblades) to have access to files and directories located there.

At this time, the 4 projects are still linked, in so much as the top-level package namespace for the modules is srpipe:: (which means we need to be careful of ensuring that further down the line we don't accidently create two things which will conflict) and that many modules use modules from some of the other projects (code as DRY as possible).

However, these slight drawbacks are minor compared to the ability now to apply patches and new code to a project, without accidently deploying a broken development of something else (which was the tricky thing when the wrapper scripts were managed by someone separate to the group responsible for the (for example) mirroring of data).

So hopefully, now we have a much more stable codebase, and will be able to keep things running much more smoothly (or at least, develop much more fluidly).

The test coverage needs improvement, but this is something that we can further work on now. A fair amount of code had no tests at all past one which checked it compiled, so a revamp of the test suite was certainly in order, and with this, we can code directly in to ensure that the functionality doesn't change where code is reused in different projects.

Exciting times or screams ahead, who knows, but at least with hopefully more manageable project sizes, we should be able to aim for exciting...

Running Alone

John in my team at work has just pointed me at a great little CPAN module for ensuring that a script can only have one copy running at a time.

In our team, we have a lot of potentially slow running processes, which are often run as cron jobs. Now, ordinarily, you would pick cron start times
that ensure that you have finished any previous run of the script. But it is quite common for us to want to run as soon as possible, but not if the
previous run hasn't finished.

One method previously suggested was to touch a file in /tmp and then delete it when finished, wrapping the cron caller in a command not to run if
the touched file is present. However, what happens if /tmp is cleared out (which has happened to me!)

So, John found Sys::RunAlone. If the format of your script is to run a main method, and you put __END__ or __DATA__ at the bottom of the script,
then this can lock the text, and then knows no to exit if it finds a lock on the section. This means you could have your cronjob running every 30 minutes
and not worry if the script takes 5 minutes in a cycle, or 45 minutes.

A definite benefit.

A big thanks to Elizabeth Mattijsen for writing this very useful module.

(A quick example of this running is below. If you try to run the same script in another terminal before the first has finished it's sleep, you will get
the correct error).

#!/usr/bin/perl -wT
use strict;
use warnings;
use Sys::RunAlone;

main();
0;

sub main {
warn 'Hello ...';
go_to_sleep(10);
warn 'world!';
}

sub go_to_sleep {
my $sleepytime = shift;
sleep $sleepytime;
}

__END__

Blog entry

Data Formats

Check out this SlideShare Presentation:

Data Formats

View SlideShare presentation or Upload your own. (tags: csv tsv)

This presentation was given by me to foomongers to start off a bit of discussion on different data transfer formats.