I'm starting a new MooseX::Storage module called MooseX::Storage::File_or_DB.
The objective is that you can use the standard MooseX::Storage to serialize out the Moose
object as a JSON string to a file, and read it back again, but also save it out to
a Database so that it can be used in a usual database way (i.e. interrogate the db using
sql directly, so an attribute maps directly to a column).
There are a number of ORM or object db modules on CPAN (DBIx::Class, Fey, KiokuDB) but all seem a bit difficult to give the option to also obtain and save out the object to a filesystem as well, for more short term storage prior to archival to the database.
This is clearly something we need. MooseX::Storage is just the job for storing and retrieval from a filesystem, but linking both in one.
So, I'm setting to work on something that will do both. The current work in progress is on GitHub here
http://github.com/setitesuk/MooseX--Storage--File_or_DB
I would ask people to take a look and see what they think. The POD is currently where I want to end up, but the tests are working, and I think it is going in the right direction. Looking forward to more work on this to make it ready to submit to CPAN.
Sunday, 30 August 2009
Friday, 28 August 2009
Am I too good
For our projects that are released through the Sanger Website as Open Source (most of our code is, but we don't have a specific release policy to putting it out there) we run David A. Wheelers sloccount to get a count of the lines of code.
I have just done release-3.0 of the pluggable pipeline system, so I thought it might be fun to get the stats of this.
Total Physical Source Lines of Code (SLOC) = 3,884
That's good, I've been working on this for 7 weeks, with other projects.
However, sloccount gives you further info:
Development Effort Estimate, Person-Years (Person-Months) = 0.83 (9.98)
(Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
About 10 months development - I'd have been shot if what I have produced had taken that long :)
Schedule Estimate, Years (Months) = 0.50 (5.99)
(Basic COCOMO model, Months = 2.5 * (person-months**0.38))
6 months scheduling.
Estimated Average Number of Developers (Effort/Schedule) = 1.66
There's only me, and I've only been working for 7 weeks on this project!
Total Estimated Cost to Develop = $ 112,301
(average salary = $56,286/year, overhead = 2.40).
I need to ask for a raise! In fact, my boss covers his ears and refuses to listen when I mention these numbers.
Obviously, there has been discussion with other people about where the project is heading, but I think the Basic COCOMO model clearly doesn't quite cut the mustard with Agile Development practices. Or maybe I'm just too good.
Still, it is fun to watch my boss run screaming, refusing to listen when I quote the estimated cost to develop. At least I think I am worth at least what I am paid :)
I have just done release-3.0 of the pluggable pipeline system, so I thought it might be fun to get the stats of this.
Total Physical Source Lines of Code (SLOC) = 3,884
That's good, I've been working on this for 7 weeks, with other projects.
However, sloccount gives you further info:
Development Effort Estimate, Person-Years (Person-Months) = 0.83 (9.98)
(Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
About 10 months development - I'd have been shot if what I have produced had taken that long :)
Schedule Estimate, Years (Months) = 0.50 (5.99)
(Basic COCOMO model, Months = 2.5 * (person-months**0.38))
6 months scheduling.
Estimated Average Number of Developers (Effort/Schedule) = 1.66
There's only me, and I've only been working for 7 weeks on this project!
Total Estimated Cost to Develop = $ 112,301
(average salary = $56,286/year, overhead = 2.40).
I need to ask for a raise! In fact, my boss covers his ears and refuses to listen when I mention these numbers.
Obviously, there has been discussion with other people about where the project is heading, but I think the Basic COCOMO model clearly doesn't quite cut the mustard with Agile Development practices. Or maybe I'm just too good.
Still, it is fun to watch my boss run screaming, refusing to listen when I quote the estimated cost to develop. At least I think I am worth at least what I am paid :)
'Non-'Unique indexing
We wanted to make a column in our MySQL database table (using InnoDB) nullable, but use the column as part of a composite key.
unique key C1,C2,C3,C4
C4 can be null
C1 C2 C3 C4
enter the following:
x y z a
goes into the table ok
enter those again - error that we break unique constraint.
This is as expected.
enter
x y z null
goes into table ok
enter those again - they also enter fine, and select * from table shows two separate row entries.
So, basically, you can have a nullable field in a composite unique index, but if the column is null, you loose the unique index checking.
I don't know about other DB's, but it is a shame that the null can't be part of the uniqueness of the index.
unique key C1,C2,C3,C4
C4 can be null
C1 C2 C3 C4
enter the following:
x y z a
goes into the table ok
enter those again - error that we break unique constraint.
This is as expected.
enter
x y z null
goes into table ok
enter those again - they also enter fine, and select * from table shows two separate row entries.
So, basically, you can have a nullable field in a composite unique index, but if the column is null, you loose the unique index checking.
I don't know about other DB's, but it is a shame that the null can't be part of the uniqueness of the index.
Wednesday, 12 August 2009
Musings on a Moose
come to sweden, see the majestic moose (paraphrased from Monty Python and the Holy Grail).
I am starting to look at Moose as an alternative to Class::Std and other modules for OO Perl. It has very good support from the community, and looks to be fast becoming the framework of choice.
I will start off by saying I like it. The setup of my::module is very easy, and reads very cleanly. I like the declaration of variable types on the accessors, and it obvious when an attribute is needed on new, (just set 'required' flag). I especially like the ability to make an attribute ro.
However, that leads me to a small bugbear I have. I have mentioned before about encapsulation, Class::Std objects are blessed scalars, and as such can't have keys. Your attributes are set up as keys on internal hashes, and as such can only be exposed via a method.
The Moose object created is a blessed hash, and, the attributes are stored in keys. This means that your user can still override the ro attribute by just using the key. Encapsulation is broken.
Someone in my office used a good simile here, which I think he attributed to Larry Wall, which is:
"Your neighbour stays out of your garden, because it is the right thing to do, not because you have a shotgun."
That might be the case, but I'd rather enforce the use of accessors from the start than run the risk of user's using the key. However, I plan to try out MooseX::InsideOut to see if this can enforce this once I'm more tightly used to the basics.
Another thing I find very good are the additional modifiers for the accessors. We have found that making use of predicates and lazy_build really improves the code layout and speed of object creation.
I have now been building the pluggable pipeline project using Moose, and it has been very quick to build the objects and code. I would say that I think once upto speed, and once I have tested out MooseX::InsideOut, I think that development time of code should reduce by about 10% and code maintainability should go up about 25%.
So a big thumbs up from myself, and my development team also. Whilst I think at the moment there would be no plans to convert our ClearPress based apps to Moose, I think I'm tempted to try to switch my Class::Std modules, and certainly all new projects will go that way.
Big Thanks to all the people who work on Moose. Is there a book in the works? I for one would get it, I'd even contribute if you'd like.
I am starting to look at Moose as an alternative to Class::Std and other modules for OO Perl. It has very good support from the community, and looks to be fast becoming the framework of choice.
I will start off by saying I like it. The setup of my::module is very easy, and reads very cleanly. I like the declaration of variable types on the accessors, and it obvious when an attribute is needed on new, (just set 'required' flag). I especially like the ability to make an attribute ro.
However, that leads me to a small bugbear I have. I have mentioned before about encapsulation, Class::Std objects are blessed scalars, and as such can't have keys. Your attributes are set up as keys on internal hashes, and as such can only be exposed via a method.
The Moose object created is a blessed hash, and, the attributes are stored in keys. This means that your user can still override the ro attribute by just using the key. Encapsulation is broken.
Someone in my office used a good simile here, which I think he attributed to Larry Wall, which is:
"Your neighbour stays out of your garden, because it is the right thing to do, not because you have a shotgun."
That might be the case, but I'd rather enforce the use of accessors from the start than run the risk of user's using the key. However, I plan to try out MooseX::InsideOut to see if this can enforce this once I'm more tightly used to the basics.
Another thing I find very good are the additional modifiers for the accessors. We have found that making use of predicates and lazy_build really improves the code layout and speed of object creation.
I have now been building the pluggable pipeline project using Moose, and it has been very quick to build the objects and code. I would say that I think once upto speed, and once I have tested out MooseX::InsideOut, I think that development time of code should reduce by about 10% and code maintainability should go up about 25%.
So a big thumbs up from myself, and my development team also. Whilst I think at the moment there would be no plans to convert our ClearPress based apps to Moose, I think I'm tempted to try to switch my Class::Std modules, and certainly all new projects will go that way.
Big Thanks to all the people who work on Moose. Is there a book in the works? I for one would get it, I'd even contribute if you'd like.
Friday, 31 July 2009
Pluggable Pipelines
My current main project is to replace the end of our analysis pipeline. It is currently all in one module, archive.pm. As with all things which are in one single file, this has been extended and changed around, and has become unmanageable.
We have also decided that we want to make it pluggable, so that we can easily add or remove parts depending on the project requirements.
For this I looked at creating a flag waver, whose job it is to take an array of function names, and launch each in turn, capturing any return values and submitting them as requirements to be fulfilled that may be needed for the next function.
The functions have no knowledge of anything except what requirements may come in, what information the object they control requires and how to capture and return further requirements for any processes further down the line.
Most importantly, the functions know nothing about any other function.
Each function loads an object, and gives it the parameters it needs. These objects handle doing any real work, which can be updating statuses, submitting jobs to LSF, manipulating files, obtaining data from databases/web services. All they have to return to the function which called them is something the next process might need to know for it's own ability to work. In our case an array of job ids from LSF submissions, to be used as job dependencies.
The structure is therefore a flagwaver, which calls the functions in a user specified order, which in turn call objects submitting jobs, returning dependencies for the next function.
What this means is that the flagwaver has very little responsibility itself. It relies on the user ensuring that any individual components will complete successfully (or at least error sensibly), and that the user has specified an order of components which will work (i.e. if 'B' depends upon an output of 'A', then they have put 'A' before 'B' in the array).
In our case, the flagwaver does have a little specific knowledge in that we have coded a few function which, if the order specified has them next to each other, they could be run at the same time, so we can parallelise as much as possible, but that is in a specific subclass of the pluggable base module, and can be overridden.
We have also decided that we want to make it pluggable, so that we can easily add or remove parts depending on the project requirements.
For this I looked at creating a flag waver, whose job it is to take an array of function names, and launch each in turn, capturing any return values and submitting them as requirements to be fulfilled that may be needed for the next function.
The functions have no knowledge of anything except what requirements may come in, what information the object they control requires and how to capture and return further requirements for any processes further down the line.
Most importantly, the functions know nothing about any other function.
Each function loads an object, and gives it the parameters it needs. These objects handle doing any real work, which can be updating statuses, submitting jobs to LSF, manipulating files, obtaining data from databases/web services. All they have to return to the function which called them is something the next process might need to know for it's own ability to work. In our case an array of job ids from LSF submissions, to be used as job dependencies.
The structure is therefore a flagwaver, which calls the functions in a user specified order, which in turn call objects submitting jobs, returning dependencies for the next function.
What this means is that the flagwaver has very little responsibility itself. It relies on the user ensuring that any individual components will complete successfully (or at least error sensibly), and that the user has specified an order of components which will work (i.e. if 'B' depends upon an output of 'A', then they have put 'A' before 'B' in the array).
In our case, the flagwaver does have a little specific knowledge in that we have coded a few function which, if the order specified has them next to each other, they could be run at the same time, so we can parallelise as much as possible, but that is in a specific subclass of the pluggable base module, and can be overridden.
Pluggable Pipelines
View more presentations from setitesuk.
Friday, 26 June 2009
Playing Badminton through Clearpress
Yesterday I launched, finally, my badminton ladder web application for our sports and social club. Initially, I write something about 3 years ago, which used flat csv files and a cgi script for each page, which dealt with generating the html and processing results, and updating the ladder, and....
All not very practical, but at least showed what I could do at the time.
Earlier this year we had a change of hardware for our webservers, and we lost the functionality, due to the time lag of all the files being kept up to date as they changed on the multiple servers. A big problem.
So I decided it was time for a rewrite, which I did using Clearpress (http://clearpress.net/), trialling out git and github at the same time (I like this so much more than sourceforge! and svn).
I have blogged about Clearpress and its MVC framework before, so I won't spend time doing so, but with a little work, the setup used 5 tables in a database to produce a reliable system.
team - stores a team name, wins, losses and gives them a unique identifer
player - stores a player name, email and gives them a unique identifier
player_team - join table for a player to a team
ladder_type - our ladder has three sub ladders, to deal with new teams and those which haven't played for a long time
ladder - links team/position/ladder_type
The whole thing can be found on github
git://github.com/setitesuk/badminton-ladder.git
It is currently set up to deploy using a SQLLite database, but in production use, we are using a mysql database, and the schema is there. You just need to modify the config.ini file to use a mysql database, which is supported through clearpress.
So, if you are after a web app badminton ladder, then take a look. It is all available as Open Source (GNU Public Licence).
Next, to create a Tennis Competition app.
All not very practical, but at least showed what I could do at the time.
Earlier this year we had a change of hardware for our webservers, and we lost the functionality, due to the time lag of all the files being kept up to date as they changed on the multiple servers. A big problem.
So I decided it was time for a rewrite, which I did using Clearpress (http://clearpress.net/), trialling out git and github at the same time (I like this so much more than sourceforge! and svn).
I have blogged about Clearpress and its MVC framework before, so I won't spend time doing so, but with a little work, the setup used 5 tables in a database to produce a reliable system.
team - stores a team name, wins, losses and gives them a unique identifer
player - stores a player name, email and gives them a unique identifier
player_team - join table for a player to a team
ladder_type - our ladder has three sub ladders, to deal with new teams and those which haven't played for a long time
ladder - links team/position/ladder_type
The whole thing can be found on github
git://github.com/setitesuk/badminton-ladder.git
It is currently set up to deploy using a SQLLite database, but in production use, we are using a mysql database, and the schema is there. You just need to modify the config.ini file to use a mysql database, which is supported through clearpress.
So, if you are after a web app badminton ladder, then take a look. It is all available as Open Source (GNU Public Licence).
Next, to create a Tennis Competition app.
Thursday, 11 June 2009
lc(x) - possible bad practice?
use strict; use warnings;
We all use the above, right? Well, certainly we should, and my team does;
Now this always causes a warning to occur when testing an undefined value (exception - if the test is for it to be undef).
my $arrayref = $self->method_returning_array_ref() || [];
Now, assuming that the method will always return an arrayref or undef, I will have an arrayref.
Now, it is common enough (returning stuff from an XML dom for example) that there is actually only 1 thing in the array, so I test against it
if ($arrayref->[0] eq 'yes') {
do something..
}
Now, if $arrayref->[0] is undef, this always throws an uninitialised variable warning. This normally leads me to change to
if ($arrayref->[0] && $arrayref->[0] eq 'yes') {
do something..
}
so that I don't spam up the logs with warnings.
However, we discovered today that by lowercasing the $arrayref->[0] variable, this will turn an undef into an empty string (for the conditional), therefore dispensing with any warnings.
if (lc$arrayref->[0] eq 'yes') {
do something..
}
Is this good or bad coding practice?
Reasons for it to be good
- You do not need an extra conditional just to dispense with the warning
- code less
Reasons for it to be bad
- It doesn't seem right
- The conditional is no longer testing against the pure result
- Are we getting rid of the warning for the wrong reason?
- Does it read correctly?
At this time, it is not something that the code police (perlcritic) seem to think is a bad practice, and certainly will make less code for us. It just seems like we are breaking an unwritten coding rule.
We all use the above, right? Well, certainly we should, and my team does;
Now this always causes a warning to occur when testing an undefined value (exception - if the test is for it to be undef).
my $arrayref = $self->method_returning_array_ref() || [];
Now, assuming that the method will always return an arrayref or undef, I will have an arrayref.
Now, it is common enough (returning stuff from an XML dom for example) that there is actually only 1 thing in the array, so I test against it
if ($arrayref->[0] eq 'yes') {
do something..
}
Now, if $arrayref->[0] is undef, this always throws an uninitialised variable warning. This normally leads me to change to
if ($arrayref->[0] && $arrayref->[0] eq 'yes') {
do something..
}
so that I don't spam up the logs with warnings.
However, we discovered today that by lowercasing the $arrayref->[0] variable, this will turn an undef into an empty string (for the conditional), therefore dispensing with any warnings.
if (lc$arrayref->[0] eq 'yes') {
do something..
}
Is this good or bad coding practice?
Reasons for it to be good
- You do not need an extra conditional just to dispense with the warning
- code less
Reasons for it to be bad
- It doesn't seem right
- The conditional is no longer testing against the pure result
- Are we getting rid of the warning for the wrong reason?
- Does it read correctly?
At this time, it is not something that the code police (perlcritic) seem to think is a bad practice, and certainly will make less code for us. It just seems like we are breaking an unwritten coding rule.
Subscribe to:
Posts (Atom)
