My first CPAN module

So I have pushed my first ever CPAN module, MooseX::Storage::File_or_DB

http://tinyurl.com/l3xfkc

I blogged about this as I started the module before

http://vampiresoftware.blogspot.com/2009/08/filesystemdatabase-or-both.html

and this weekend I was finally able to finish the first release.

As of the moment, you need to use this by extending it

package MyClass;
use Moose;
extends q{MooseX::Storage::File_or_DB};

But I hope for a future release just to be able to

use MooseX::Storage::File_or_DB;

This gives functionality to enable you to write to either a file a JSON string, or a database, and re-instantiate the object from either.

It makes heavy use of MooseX::Storage ( http://tinyurl.com/nujf4c ) - a big thanks to Tomas Doran for writing this - for inspecting the object, and providing the ability to write out to a file as a JSON string.

I hope that this will prove useful. Please do read the POD/CPAN page before use, and contact me about anything you feel relating to this - all constructive comments gratefully received.

Backending Catalyst

I am starting to look at Catalyst as a method to display some quality control results that are coming out of our analysis pipeline.
ClearPress is a nice MVC webapp builder, but is quite a light weight framework, and uses Class::Accessor for object base. We would like to move towards using Moose based objects and need a way to integrate these into Catalyst.

I am currently working my way through the latest Catalyst book (Diment & Trout), but before this arrived I found we had the following book on our Safari Subscription - Catalyst: Accelerating Perl Web Application Development: Design, develop, test and deploy applications with the open-source Catalyst MVC framework - Jonathan Rockway.

Now, note, I had been through the Tutorial on CPAN, but couldn't find on there anything about using a Filesystem as a source for the model (Did I miss something?), but this book luckily had a section on doing so.

Firstly, why have we QC data in a filesystem?

When we run the pipeline, this all happens on a staging area, which we write everything to, and then copy all our data into long term archival databases. The QC data is no exception, but we only want to archive the final agreed data. Bioinformaticians don't seem to ever be happy with a first pass that fails, if there is any chance it could be improved (i.e. a new test pipeline version, could rerunning jut squeeze 2% more...). As such we want to view the data in exactly the same way from the filesystem as from the database, because we don't want it stored until the last possible moment.

What have we done for this?

My team have been producing Moose objects which are:

1) Producing the data
2) Storing in JSON files (MooseX::Storage)
3) Reading in JSON files (MooseX::Storage) to re-instantiate the object
4) Saving to a Database (Fey)
5) Re-instantiating from a Database

I've been working with iterations of the objects, using the files, but want the objects to just sort it themselves - I shouldn't know where the data has come from, and these objects should be used in (in fact are being written for) other applications.

Catalyst very much guides you to using a Database, and seems to prefer using DBIx::Class for this, so I need a way of guiding the Model to provide the correct objects, which are not generated directly from Catalyst helpers.

What did I do?

So in the above book, I found the section 'Implementing a FileSystem model'. This shows us how to create a Backend, which takes us out of the ordinary Model style, and the call the the Model returns this Backend object instead. We then use this Backend object to contain the logic which can be used to obtain the objects from somewhere outside of the Catalyst application, de-coupling the data models from the app, and therefore increasing flexibility and maintainability. As I said, these objects are actually being written within another application project.

This has been an interesting venture, which has enabled me to write a web application which only concentrates on the logic for the view, and leave the data handling completely to someone else. We should be production ready with the application within the week, and displaying data for the users quickly and simply.

What the betting someone asks if we can regenerate the data for all previous runs? I won't be betting against it, that's for sure.

FileSystem/DataBase - or both?

I'm starting a new MooseX::Storage module called MooseX::Storage::File_or_DB.

The objective is that you can use the standard MooseX::Storage to serialize out the Moose
object as a JSON string to a file, and read it back again, but also save it out to
a Database so that it can be used in a usual database way (i.e. interrogate the db using
sql directly, so an attribute maps directly to a column).

There are a number of ORM or object db modules on CPAN (DBIx::Class, Fey, KiokuDB) but all seem a bit difficult to give the option to also obtain and save out the object to a filesystem as well, for more short term storage prior to archival to the database.

This is clearly something we need. MooseX::Storage is just the job for storing and retrieval from a filesystem, but linking both in one.

So, I'm setting to work on something that will do both. The current work in progress is on GitHub here

http://github.com/setitesuk/MooseX--Storage--File_or_DB

I would ask people to take a look and see what they think. The POD is currently where I want to end up, but the tests are working, and I think it is going in the right direction. Looking forward to more work on this to make it ready to submit to CPAN.

Am I too good

For our projects that are released through the Sanger Website as Open Source (most of our code is, but we don't have a specific release policy to putting it out there) we run David A. Wheelers sloccount to get a count of the lines of code.

I have just done release-3.0 of the pluggable pipeline system, so I thought it might be fun to get the stats of this.

Total Physical Source Lines of Code (SLOC) = 3,884

That's good, I've been working on this for 7 weeks, with other projects.

However, sloccount gives you further info:

Development Effort Estimate, Person-Years (Person-Months) = 0.83 (9.98)
(Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))

About 10 months development - I'd have been shot if what I have produced had taken that long :)

Schedule Estimate, Years (Months) = 0.50 (5.99)
(Basic COCOMO model, Months = 2.5 * (person-months**0.38))

6 months scheduling.

Estimated Average Number of Developers (Effort/Schedule) = 1.66

There's only me, and I've only been working for 7 weeks on this project!

Total Estimated Cost to Develop = $ 112,301
(average salary = $56,286/year, overhead = 2.40).

I need to ask for a raise! In fact, my boss covers his ears and refuses to listen when I mention these numbers.

Obviously, there has been discussion with other people about where the project is heading, but I think the Basic COCOMO model clearly doesn't quite cut the mustard with Agile Development practices. Or maybe I'm just too good.

Still, it is fun to watch my boss run screaming, refusing to listen when I quote the estimated cost to develop. At least I think I am worth at least what I am paid :)

'Non-'Unique indexing

We wanted to make a column in our MySQL database table (using InnoDB) nullable, but use the column as part of a composite key.

unique key C1,C2,C3,C4

C4 can be null

C1 C2 C3 C4

enter the following:

x y z a

goes into the table ok

enter those again - error that we break unique constraint.

This is as expected.

enter

x y z null

goes into table ok

enter those again - they also enter fine, and select * from table shows two separate row entries.

So, basically, you can have a nullable field in a composite unique index, but if the column is null, you loose the unique index checking.

I don't know about other DB's, but it is a shame that the null can't be part of the uniqueness of the index.

Musings on a Moose

come to sweden, see the majestic moose (paraphrased from Monty Python and the Holy Grail).

I am starting to look at Moose as an alternative to Class::Std and other modules for OO Perl. It has very good support from the community, and looks to be fast becoming the framework of choice.

I will start off by saying I like it. The setup of my::module is very easy, and reads very cleanly. I like the declaration of variable types on the accessors, and it obvious when an attribute is needed on new, (just set 'required' flag). I especially like the ability to make an attribute ro.

However, that leads me to a small bugbear I have. I have mentioned before about encapsulation, Class::Std objects are blessed scalars, and as such can't have keys. Your attributes are set up as keys on internal hashes, and as such can only be exposed via a method.

The Moose object created is a blessed hash, and, the attributes are stored in keys. This means that your user can still override the ro attribute by just using the key. Encapsulation is broken.

Someone in my office used a good simile here, which I think he attributed to Larry Wall, which is:

"Your neighbour stays out of your garden, because it is the right thing to do, not because you have a shotgun."

That might be the case, but I'd rather enforce the use of accessors from the start than run the risk of user's using the key. However, I plan to try out MooseX::InsideOut to see if this can enforce this once I'm more tightly used to the basics.

Another thing I find very good are the additional modifiers for the accessors. We have found that making use of predicates and lazy_build really improves the code layout and speed of object creation.

I have now been building the pluggable pipeline project using Moose, and it has been very quick to build the objects and code. I would say that I think once upto speed, and once I have tested out MooseX::InsideOut, I think that development time of code should reduce by about 10% and code maintainability should go up about 25%.

So a big thumbs up from myself, and my development team also. Whilst I think at the moment there would be no plans to convert our ClearPress based apps to Moose, I think I'm tempted to try to switch my Class::Std modules, and certainly all new projects will go that way.

Big Thanks to all the people who work on Moose. Is there a book in the works? I for one would get it, I'd even contribute if you'd like.

Pluggable Pipelines

My current main project is to replace the end of our analysis pipeline. It is currently all in one module, archive.pm. As with all things which are in one single file, this has been extended and changed around, and has become unmanageable.

We have also decided that we want to make it pluggable, so that we can easily add or remove parts depending on the project requirements.

For this I looked at creating a flag waver, whose job it is to take an array of function names, and launch each in turn, capturing any return values and submitting them as requirements to be fulfilled that may be needed for the next function.

The functions have no knowledge of anything except what requirements may come in, what information the object they control requires and how to capture and return further requirements for any processes further down the line.

Most importantly, the functions know nothing about any other function.

Each function loads an object, and gives it the parameters it needs. These objects handle doing any real work, which can be updating statuses, submitting jobs to LSF, manipulating files, obtaining data from databases/web services. All they have to return to the function which called them is something the next process might need to know for it's own ability to work. In our case an array of job ids from LSF submissions, to be used as job dependencies.

The structure is therefore a flagwaver, which calls the functions in a user specified order, which in turn call objects submitting jobs, returning dependencies for the next function.

What this means is that the flagwaver has very little responsibility itself. It relies on the user ensuring that any individual components will complete successfully (or at least error sensibly), and that the user has specified an order of components which will work (i.e. if 'B' depends upon an output of 'A', then they have put 'A' before 'B' in the array).

In our case, the flagwaver does have a little specific knowledge in that we have coded a few function which, if the order specified has them next to each other, they could be run at the same time, so we can parallelise as much as possible, but that is in a specific subclass of the pluggable base module, and can be overridden.

Pluggable Pipelines

View more presentations from setitesuk.