Stomping on the Rabbit, Pt2

So now I have this all installed, I have to start using it. Net::Stomp (see cpan.org) is the interface I would like to use, but the setup seems very much done for ActiveMQ (as some of the headers are 'activemq.xxxxx'.

This means that by default, I only seem to be able to send and receive messages to a queue that exists only whilst someone is listening.

The example scripts that come with the RabbitMQ-Stomp distribution only show this for PERL, however, the Ruby examples show much more (persistent queues, topics, etc).

Thankfully, I found the following post

http://tinyurl.com/pdznw9

which explains the headers for RabbitMQ to produce persistent queues and operating topics. For Net::Stomp, the key is to add these headers to the key-value pairs in your connection/send hash, and happily they go through. Here are some examples:

Persistent Queues:

Receiver -

my $stomp = Net::Stomp->new({hostname=>'localhost', port=>'61613'});
$stomp->connect({login=>'guest', passcode=>'guest'}) or croak $EVAL_ERROR;

$SIG{INT} = sub {
$stomp->unsubscribe({
destination => q(/queue/father/ted),
});
exit;
};

$stomp->subscribe({
destination => q(/queue/super/ted),
q{auto-delete} => q{false}, # setting these flags will make your queue remain whilst
q{durable} => q{true}, # the subscriber(s) go away, and pick up messages afterwards
ack => q(client),
});

while (1) {
my $frame = $stomp->receive_frame;
print $frame;
print $frame->body . "\n";
$stomp->ack({frame=>$frame});
last if $frame->body eq 'QUIT';
}

$stomp->disconnect;

Sender -

my $stomp = Net::Stomp->new({hostname=>'localhost', port=>'61613'});
$stomp->connect({login=>'guest', passcode=>'guest'}) or croak $EVAL_ERROR;
$stomp->send({destination => '/queue/super/ted',
bytes_message=>1,
body=>($ARGV[0] or "test\0message")});
$stomp->disconnect;

Topics:

Topics are handled differently in RabbitMQ to ActiveMQ. In ActiveMQ, they sit under a namespace /topic/xxx/yyy and a combination of the client-id, destination, exchange and routing key all make up the subscriber to the topic

In the case of RabbitMQ, it seems a bit more generic than that, but has some differences which make it (as far as I can see) non-persistent.

Receiver -

my $stomp = Net::Stomp->new({hostname=>'localhost', port=>'61613'});
$stomp->connect({login=>'guest', passcode=>'guest'}) or croak $EVAL_ERROR;

$SIG{INT} = sub {
exit;
};

$stomp->subscribe({
destination => q{bananaman}, # needs to be a unique client-id
exchange => q{amq.topic},
routing_key => q{bananas},
});

while (1) {
my $frame = $stomp->receive_frame;
print $frame;
print $frame->body . "\n";
last if $frame->body eq 'QUIT';
}

$stomp->disconnect;

This sets up a receiver, which if you use rabbitmqctl list_queues shows a queue bananaman which will receive messages (whilst he is listening) to the topic bananas

using the following gives an anonymous looking receiver queue

$stomp->subscribe({
id => q{banaman}, # needs to be a unique client-id
destination => q{},
exchange => q{amq.topic},
routing_key => q{bananas},
});

which you can unsubscribe from explicitly

$stomp->subscribe({
id => q{banaman},
});

However, it would appear that the unsubscription occurs anyway, so unfortunately, any posts to the topic bananas would be lost to bananaman whilst he is away.

Sender for topics -

my $stomp = Net::Stomp->new({hostname=>'localhost', port=>'61613'});
$stomp->connect({login=>'guest', passcode=>'guest'}) or croak $EVAL_ERROR;
$stomp->send({destination => q{bananas},
exchange => q{amq.topic},
body=>($ARGV[0] or "test\0message")});
$stomp->disconnect;

You will notice here that the destination for this message is the routing_key, and that the exchange flag is set.

The problem here is that essentially, each receiver to a particular topic (routing_key) forms a temporary queue to which the message is added, and then sent from, but the queue is just that, temporary and goes away when the receiver does, so that receiver will never get the message if they are not present when the message is sent to the message-queue.

I am very keen to hear from anyone who has found out how to work around this.

The other main problem that we have seen due to this is MQ agnosticism doesn't exist using Net::Stomp, as essentially the problem is that it is written with ActiveMQ in mind, and just feeds headers through. This means that there is no conversion of the headers between using ActiveMQ-STOMP, RabbitMQ-STOMP or any other message-queue-STOMP that is out there. This means that there is no way to easily hot-swap message-queues without code re-write. Again, I'd be happy to hear form anyone who has worked out how to get around this.

As for now, it looks as though it is going to be ActiveMQ, as this has been installed centrally for us, and appears to manage the persistence a bit better, along with headers which are documented in Net::Stomp.

Stomping on the Rabbit

we are trying to make a move to using message queues in my group to deal with pipelines and talking to other apps.

There should be some great advantages for us, and I am quite excited by this.

First thing, set up a STOMP message queue.

Now, I am currently writing a simple message_queue based on ClearPress, but this is not yet ready, so I have just downloaded and installed RabbitMQ.

RabbitMQ is written in Erlang, (which I have just set myself the challenge of learning from the Pragprog book Programming Erlang by Joe Armstrong). The first challenge was setting it up.

First: You need erlang installed. I had already done this, so wasn't a problem. Jsut make sure that erlc is in your path.

Now, I had dowloaded the latest release tarball, and expanded this, setting up in my sandbox area. However, this threw my a serious curveball with trying to install STOMP.

Thankfully, Google is my friend, and someone had the same problem, since the STOMP needs installing against the correct version number, so I hereby give the definitive installation guide to getting RabbitMQ up and running in a sandbox on MacOSX.

(I accept no responsibility for this not working on your machine!)

1) Install Mercurial (yet another distributed version control system, but the one which RabbitMQ is on).

2) (Thanks to everyone on this page http://tinyurl.com/pk7xfd)
hg clone http://hg.rabbitmq.com/rabbitmq-server
hg clone http://hg.rabbitmq.com/rabbitmq-codegen
hg clone http://hg.rabbitmq.com/rabbitmq-stomp
(cd rabbitmq-server; hg up rabbitmq_v1_5_4)
(cd rabbitmq-codegen; hg up rabbitmq_v1_5_4)
(cd rabbitmq-stomp; hg up rabbitmq_v1_5_3)

3) (At this point, you need to check your version of python and simplejson, which needs installing)

4) In the various MakeFiles, alter source roots to point to where you want the various db and log files to go, where your rabbit source root is, etc..

Eg.

I set up in my sandbox a folder 'rabbitmq', in which I put a logs dir, mnesia (db files) dir and rabbit-mnesia dir (into which I put the rabbitmq.conf file)

In 'rabbitmq-server/Makefile'

RABBITMQ_NODENAME=rabbit
RABBITMQ_SERVER_START_ARGS=/my/sandbox/rabbitmq/rabbitmq.conf
RABBITMQ_MNESIA_DIR=/my/sandbox/rabbitmq/$(RABBITMQ_NODENAME)-mnesia
RABBITMQ_LOG_BASE=/my/sandbox/rabbitmq/logs


4) make -C rabbitmq-server
make -C rabbitmq-stomp run

Hey presto, you now have a rabbitmq-stomp server up and running.

Now you have done this, you can try either the perl or ruby tests.

Hope this guide is of use to someone.

Cheers

Andy

Splitting a Project

As part of the Group I am in, we have had a project sanger-pipeline. Now, early on in the life of the group, it was quite small, and was essentially just a few wrappers around the Illumina Analysis pipeline, so that it hooked into our tracking system, and scripts to manage the movement of images off the individual Genome Analysers to our Farm.

However, as with most things, it got bigger...

and bigger...

and bigger until it was managing not only the wrappers (which were being maintained by someone who was in another group), and the movement of images, but processing other files, loading compressed versions of the images into other databases, deciding when it had everything it expected, what and when exactly to delete outdated files...and managing an apache webservice!

So, over the last 2 weeks, I have been splitting the project down. This was not a trivial thing to do, as many modules where used in multiple places, but split it I did to the following:

sanger-pipeline - this now only has wrapper scripts and modules relating to hooking the Illumina Analysis pipeline in to our system.

instrument_handling - this deals with scripts and modules relating to the smooth running and mirroring of data of the GA's, plus our controlcentre script, whose primary use is to run these as a special user (although it does do some other things).

data_handling - all these scripts and modules are responsible for managing data on the farm (once it has mirrored) including doing checks to ensure that data that needs long term storage are where they should be

sflogin_web_apps - the management of the apache server and the scripts that we run on the farm (instead of the usual webblades) to have access to files and directories located there.

At this time, the 4 projects are still linked, in so much as the top-level package namespace for the modules is srpipe:: (which means we need to be careful of ensuring that further down the line we don't accidently create two things which will conflict) and that many modules use modules from some of the other projects (code as DRY as possible).

However, these slight drawbacks are minor compared to the ability now to apply patches and new code to a project, without accidently deploying a broken development of something else (which was the tricky thing when the wrapper scripts were managed by someone separate to the group responsible for the (for example) mirroring of data).

So hopefully, now we have a much more stable codebase, and will be able to keep things running much more smoothly (or at least, develop much more fluidly).

The test coverage needs improvement, but this is something that we can further work on now. A fair amount of code had no tests at all past one which checked it compiled, so a revamp of the test suite was certainly in order, and with this, we can code directly in to ensure that the functionality doesn't change where code is reused in different projects.

Exciting times or screams ahead, who knows, but at least with hopefully more manageable project sizes, we should be able to aim for exciting...

Running Alone

John in my team at work has just pointed me at a great little CPAN module for ensuring that a script can only have one copy running at a time.

In our team, we have a lot of potentially slow running processes, which are often run as cron jobs. Now, ordinarily, you would pick cron start times
that ensure that you have finished any previous run of the script. But it is quite common for us to want to run as soon as possible, but not if the
previous run hasn't finished.

One method previously suggested was to touch a file in /tmp and then delete it when finished, wrapping the cron caller in a command not to run if
the touched file is present. However, what happens if /tmp is cleared out (which has happened to me!)

So, John found Sys::RunAlone. If the format of your script is to run a main method, and you put __END__ or __DATA__ at the bottom of the script,
then this can lock the text, and then knows no to exit if it finds a lock on the section. This means you could have your cronjob running every 30 minutes
and not worry if the script takes 5 minutes in a cycle, or 45 minutes.

A definite benefit.

A big thanks to Elizabeth Mattijsen for writing this very useful module.

(A quick example of this running is below. If you try to run the same script in another terminal before the first has finished it's sleep, you will get
the correct error).

#!/usr/bin/perl -wT
use strict;
use warnings;
use Sys::RunAlone;

main();
0;

sub main {
warn 'Hello ...';
go_to_sleep(10);
warn 'world!';
}

sub go_to_sleep {
my $sleepytime = shift;
sleep $sleepytime;
}

__END__

Blog entry

Data Formats

Check out this SlideShare Presentation:

Data Formats

View SlideShare presentation or Upload your own. (tags: csv tsv)

This presentation was given by me to foomongers to start off a bit of discussion on different data transfer formats.

Disk is cheap?

I work at the Wellcome Trust Sanger Institute, where we do a lot of processing of raw sequence data, particularly from the Next Gen sequencing machines. Over any month period we can have 320Tb of data sitting around on on our sequencing farm. The run data then gets tarballed up and moved off into repositories.

Currently, Gouying and I have been working on a MySQL database to store the QC data from the runs, so that we can get rid of the Tarballs (each themselves around 60Gb), and allow easier access to the data. Note, this is not the DNA sequence, that is being stored elsewhere. The expected annual insert into this is likely to be around 50Tb at current Illumina machine levels (and we are planning to get more!).

Roger and others have been working on storing the images for the runs longer term. This has meant moving them into an Oracle database, which means that we should have them for perhaps a year, rather than around 3 weeks. This is around 100 (tiles) * 36 (cycles) * 2 (run pair) * 8 (lanes on a chip) images per run tiff images.

Speaking with Ed at work, we discussed what was mentioned at our last Town meeting. Ed used to work for IBM, and he talked me through a bit of how processors have developed and why they are now likely to be at their limit of capability, hence going to multiple core machines. He therefore raised a good question at the town meeting - is it cheaper to store the data from the runs long term, or just rerun the sequencing?

At the moment, it costs 2% of the cost of a run to keep the data on disk, so that answers the question. Or does it?

Torvus Linalds has been quoted as saying that disk is cheap, and yes, upgrading your desk/laptop to 4x the memory is pretty cheap. However, you still need to ensure you have space for it. And there is one thing that is certainly the case, space isn't cheap. All those disks have a physical footprint. (and we could very soon run out of space in the data centre)

They also have to be managed, and people aren't cheap. We have a very good systems team, that are on 24hr callout, but that costs money.

So, it is very much a trade off. The cost of resequencing is very expensive at the moment, and storage of data is cheap, but if the next gen sequencing costs come down, then it may become very much the case just to store a plate of DNA and resequence it every time you want data from it, rather than long term storage of 4Tb+ of data.

This may be a long way off, but if the $1000 genome becomes a reality, then I think that it may change. Is this a good thing? We shall see.

Ajax - Web 2.0 Primer course.

So last week I went on a four day course to find out more about using Ajax to make web2.0 sites. The course was actually 2x4 day courses attempted to be fitted in to 1x4 day course.

For me, the first course was pretty simple, it was supposedly the prerequisite course fro the Ajax primer. We covered the Javascript basics, but in order to move on, the course instructor assumed our knowledge of HTML and css. For me this was OK, but some people didn't seem too up with the whole css selector specs instead of using inline markup. However, this did 'get' us through the course in a day.

We then jumped to the Ajax course. As I have blogged before, earlier this year I bought Pragmatic Ajax: A web2.0 primer (www.pragprog.com). Most of the second course was actually summed up in this book (although I did learn more about some Best Practices). We went through from making an xhr object that is cross-browser compatible, to starting our own library, and then we did some examples using jQuery, Prototype and Scriptaculous.

The course was from Learning Tree, and the instructor was very knowledgable, showing us some other books that we might want to read, primarily Douglas Crockford - Javascript, The best bits (www.oreilly.com) and John Resig - Pro Javascript Techniques (apress).

The thing that interested me the most was probably something that had the least time spent on it. For my toy app, YADB, I wanted to be able to drag and drop cards from either a card list or inventory list into a deck container. We were shown how to do this using Scriptaculous (quite fortunate, considering YADB is Ruby on Rails).

Work wise, it was good to be shown exactly how to look at blackboxing functions, and how to pass JSON and objects around.

Two disappointing features of the course - example code snippets did not follow what 'Best Practices Guy' was saying, sometimes even in the same slide, and some of the exercises on the computer didn't match the code in the book.

Also, no fault of the instructor, but somewhere along the line, they hadn't explained to Learning Tree that most people on my site primarily use Perl,
so some of us were a bit stuck when it came to modifying some of the server side code, as we had little/no experience of the 3 they had chosen (jsp, php and .net)

Overall, a good course, that could have been better, but worth going on.

I must contact Learning Tree though about doing the 2 module tests for them though.

An aside to this, why is everything PHP. I had a couple of email links come though about web app jobs coming up, and everyone wants PHP. I could be a bit biased, but certainly a lot of people I know think that PHP is probably one of the worst technologies to come along for a while. Certainly one thing that has been blogged on Perl Buzz is that you can't bug report anything from an older version than the current release, which seems a bit pointless to me. However, I am not going to go into it now, I think I just clearly need to learn it. Better hit the bookshelves!