Vampire Software Development

TeX - an odd markup language

2012-01-13T02:19:00.000-08:00

Why do I say odd?

Over the last couple of iterations at $work, I have wanted to produce some automated PDF reports from out of our database, with the option to pull in some additional images should they be available.

Now, I'm using the perl ClearPress MVC framework for the application, and it is nice and friendly in that I can Template my TeX and push that through pdflatex to get the PDF.

No problems so far. The data is therefore obtained and pushed into the template in exactly the same way as with HTML.

A nice simple bit of TeX markup and I get a nicely formatted PDF report yes?

No.

Unlike scripting languages and HTML, it doesn't seem to be doing everything top to bottom. Some of my tables end up above Section headings, images too.

What?

Now, prior to this I have had no experience with TeX in any flavour before.
I start by looking into main uses of TeX. For those not in the know, it is a markup designed for Mathematical equation rendering, and is popular with Journal entries as with simple markup given with the paper, they can change the formatting to suit the Journal.

So I start looking around the Google querying how to do formatting. The answers to queries I have seem to always take me to forums, as opposed to any sort of manual. The first answer took me to the info in the paragraph above, basically saying that we want to do your formatting, don't do it yourself. And there was an interesting discussion in a forum with one person giving a way of telling your table/image 'Put yourself there!' and another asking 'Why on Earth would you want to limit your table/image to the one position'.

Also, another one was escaping special characters. Eventually, I came to the following link The Comprehensive LaTeX Symbol List - The UK TeX Archive (pdf download) but not after having been pushed around forum answers again.

It goes to show that forums are great places for information (although, sometimes the question to Google often brings up that in favour of the manual :) ).

But again, why do I say odd?

Well it seems that TeX doesn't do Markup in quite the logical way I would do Markup (which is the way HTML does it). HTML says start at the top, display the marked up code from top to bottom, unless the markup (combined with css/javascript) tells me to do it otherwise. TeX seems to say, Start at the top, but float everything to where I think it is best, unless the markup specifies something different. This seems to keep the order of the same types of items, but doesn't necessarily keep the items where written.

Also, special characters are not always represented with a textual markup version, or an actual escaped version. i.e.


  \ needs to be \textbackslash, not \\ (as that actually means newline)
  # needs to be \#, there isn't a \texthash, and \hash doesn't always work

I think that, had I known this at the start, I'd have then had a better idea of how to query Google for info. TeX is clearly a useful Markup, and by the looks of it, very extensible (I pulled in 'grffile' in order for it to deal with spaces in filenames and paths), but it isn't as quick to pick up as html.

Test::WWW::Selenium::Conversion::IDE

2011-12-03T00:48:00.000-08:00

At $work we have a LIMS system running as a web app. Nothing unusual there. However, we had a user experience a bug which had only be seen once before, and seemed impossible to recreate.

It seemed to come from asynchronous Ajax updates to one field on the user session.

My first thought, was how to recreate this bug which only comes up rarely. On my development version, I couldn't get it to happen. So, I decided to have a play with Selenium. After downloading the Firefox plugin, I recorded the actions that our user did, and ran through as fast as it would replay.

Success, the bug reappeared. If run at slow speed, the test ran through happily, but fast caused the issue with the Ajax.

Working on a solution then gave me a test case to work against, and I can happily report that the bug has gone away.

Now, as a reader of this blog, you know I am a Perl programmer, and, yes, this app is just that. A Clearpress app.

We have a good test suite exercising the model, view and api, but we don't have anything UX tests. So, can I run the Selenium IDE file (Selenese HTML) in the Perl test suite. CPAN has two modules providing drivers for the Selenium Server, but no converter from Selenese to Perl Test. That seemed to go some time ago.

So, time to actually do something about it. I chose to use the Test:: WWW::Selenium module over Selenium::Remote::Driver, as I preferred the API.

Now, I had two strategies here.

1) Fully convert the Selenese HTML file to a Perl Test file,
2) On the fly, convert and run the Selenese HTML file

I opted for 2, as I coul just hard code in a wrapper test a Selenese suite file, and then every time we record another user story, it will just get added to the suite, and run. Also, keeping the code DRY, as it will b handy to keep the Selenese HTML around to run individually in the IDE.

So, I started developing Test::WWW::Selenium::Conversion::IDE, to do just that. I released it yesterday to CPAN.

It should most definitely be considered ALPHA code. I am unlikely to change the function names, as $work will rely on it, but it is by no means feature complete. I welcome comments and patches if you need more features.

The way it works:

1) In a wrapper test file, create a Test::WWW::Selenium object with creds for your Selenium Server
2) Pass to the suite runner function this object and the filename of your suite file. You can also pass the location of the file, but it defaults to t/selenium_tests.
3) Rather than give a plan, use done_testing() at the end of your file, so the TAP plan can be accounted.

See POD for more options.

So, I push it out for the communities use.

Bye for now.

Deep Magic - recover a file

2011-09-02T01:49:00.000-07:00

We have hopefully managed to recover a file (or two) today from an existing process in which the files live in /tmp/, but if the process goes on too long, get deleted by the system


Find relevant Process ID
  >ps aux or top
  >pstree -p 

Work out files which have handles but are deleted
  >lsof -p 
  bwa     22039 srpipe    4r   REG      104,7  424718660  67171651 /tmp/iFCyWKyDjR/1.sai (deleted)
  bwa     22039 srpipe    5r   REG      104,7  401682372  67171652 /tmp/iFCyWKyDjR/2.sai (deleted)

Rebuild files
  >mkdir /tmp/iFCyWKyDjR
  >cat /proc/22039/fd/4 > /tmp/iFCyWKyDjR/1.sai
  >cat /proc/22039/fd/5 > /tmp/iFCyWKyDjR/2.sai

Check
  >ls -lh /tmp/iFCyWKyDjR
  total 789M
  -rw-rw-r-- 1 srpipe solexa 406M 2011-09-02 09:30 1.sai
  -rw-rw-r-- 1 srpipe solexa 384M 2011-09-02 09:30 2.sai

To remember

2011-06-14T07:13:00.000-07:00

my $last_base_index = ( length $fields[$SEQUENCE_INDEX] ) - 1;

The parenthesis are needed!

Bingo!

2011-06-03T03:50:00.000-07:00

Last year, my dad, as the Chairman of the local community association says that what might be a good fund raiser is a bingo evening.

Initially, my thought was, who are they going to get to run that then?

But he continued, and came up to the question of sourcing a bingo machine. This is were I came in and said,

'Oh, that's easy, I could write one in a couple of weekends on the computer, we jsut hook my laptop up to a big screen and that's that.'

Can you see where I'm going with this?

So, time passes, I had a quick stab last November, but stopped as Christmas was coming up, and I was writing a perl course for work...

Until March, when Dad turns round and says

'Are you going to be able to write the bingo machine in time for Annual fundraising week? What day of the week is best for you to run it?'

Eeep.

Still, never one to back down from a request to do something, I set to work. Went back to github, and decided to scrap most of what was there, start a new Catalyst project, and a couple of weekends later, I have a bingo caller, with big ball images, and a pretty simple interface.

It could still do with a pretty looking interface, but it does the job, and should run nicely next week - assuming I can sort out Moose

https://github.com/setitesuk/Bingo-Caller

Interesting bug

2011-03-16T03:17:00.000-07:00

I have just discovered an interesting bug, which took a lot of staring at to fathom out exactly what is wrong here:


  if ( $self->is_paired_read() ) {
    my $args = {
      dir => $arg_refs->{dir},
      position => $position,
      read => $self->general_values_conf()->{sanger_read_two},
      job_dependencies => $arg_refs->{job_dependencies},
      control => $is_control,
      is_spiked_phix => $is_spiked_phix,
    };

    if ( $self->is_indexed() ) {
      $args->{read} = $self->general_values_conf()->{illumina_read_two};
    }

    $bsub_command = $self->_calibration_table_bsub_command( {
      dir => $arg_refs->{dir},
      position => $position,
      read => $self->general_values_conf()->{sanger_read_two},
      job_dependencies => $arg_refs->{job_dependencies},
      control => $is_control,
      is_spiked_phix => $is_spiked_phix,
    } );

    if ( $self->verbose() ) {
      $self->log( $bsub_command );
    }

    push @{ $job_ids }, $self->submit_bsub_command($bsub_command);
  }

Some of you may have spotted it straight away, but others may not.

The problem comes from a bit of

1) copy and paste programming (there is a read 1 as well)
2) not paying attention

Had I not had the if ( $self->is_indexed ) block, then Perl::Critic would actually have saved me here, with a unused variable error, but because $args (lexically scoped) , is used in the block, we skip that.

Simply, when you go to the trouble of defining your args outside of the call to the method, then use the args, don't rewrite. (Bangs head against wall to try to remember the fact).

The bug in production is that we always get a calibration table name with read 'sanger_read_two', when it should be read 'illumina_read_two' on an indexed run.
(These are config variables, which actually equate to 2 and 3 respectively)

So a change to:


    $bsub_command = $self->_calibration_table_bsub_command( { $args } );

and all is fixed.

Perl Course help!

2011-02-07T08:21:00.000-08:00

I need some help as to whether or not to do include something in a beginners perl course I am writing to run at work.

Courses I went on and textbooks I have read (including the excellent Modern Perl) mention the idiom '0 but true', to have a true 0.

Everyone in my office when I asked either went 'what?', 'why?' or 'just something not needed and is confusing'.

Personally, I agree with the 'why?' and 'confusing', so I'm inclined to leave it out. However, what do the community at large think? One of my colleagues has been programming Perl for 15+years, and has never seen it. Is this generally the case?

Thanks to any responses in advance that support either side of the argument, particularly with good reasoning.

Just to confirm my 4 reasons for leaving it out:

1) Never seen it used
2) Too confusing to explain to people experienced in Perl, let alone Newbies
3) Too confusing to read in code
4) Won't be automatically assigned to a variable when answer is 0 (4-4), so better to use defined

Andy

Talk to Software East - Aglie Analysis Pipeline

2010-11-19T00:17:00.000-08:00

Last night I gave a talk to the monthly Software East meeting, held by Mark Dalgarno at RedGate Software.

Big Thanks to Mark for inviting me, and everyone who showed up for an interesting discussion on Agile techniques. Also, thanks to RedGate for hosting.

The slides have been uploaded to Slideshare:

Agile analysis development

View more presentations from setitesuk.

It was a great talk, and the first time I didn't get groans outside of work about having used Perl :)

Looking forward to the next Software East meeting in January, which will be Rachel Davies talking about Agile Retrospectives.
I saw Rachel give the second day's keynote at Agile Cambridge, and she was brilliant. She was also very nice to chat to at lunch that day. She has written the book Agile Coaching (Pragmatic Programmers) so for fear of overloading the session with people, I can thoroughly recommend going to this.

Web design Survey

2010-11-02T01:54:00.000-07:00

Day 2 of Agile Cambridge

2010-10-18T04:04:00.000-07:00

After far too little sleep, again I got the first bus out of Haverhill off to Cambridge, luckily not feeling bad after too much beer.

Arriving about 8.30ish, I chatted with a couple of people from the previous day, and we went through for the second Keynote, Building Trust in Agile Teams - Rachel Davies. Rachel is the co-author of Agile Coaching, and talked a lot about trust. What is trust, how do we trust people (both socially and in the work environment), and how do we build trust. Examples included borrowing £20 of a member of the audience, and doing the falling backwards into a group of peoples arms (which worked, and Rachel actually seemed a bit scared of the volunteers going through with). This was an excellent keynote. Rachel is a great speaker, and presented the material in a very easy to understand way.

After a coffee break, for me it was back into the Gilb Theatre for Scrum/XP Add-ons workshop, presented by Jon Mullen and Paul Fairless from BSkyB. The organisation of this was different, in that, with a bit of PowerPoint trickery, they were able to allow us to choose the order of the presentation topics. We did some voting (What we preferred, what our teams did agile), we choose the order we would put 'stories' based on complexity (I clearly thought that baking and decorating a cake for 20 was more complex than washing and waxing a car). The method of this though was (I thought) better than story poker). A story card is put on the table, then each person in turn either puts a new card either side (or between), or moves a card. This continues until an order is established of complexity. After this, the cards are then graded 1->8 for measure of complexity. They gave an excellent description of the way that they work (1 week sprints, pair programming, office layout), and a great little video (with sweets!) of a scrum (that had a number of things wrong with it). It all finished with an attempt to win a bottle of bubbly, in a game like they play on a Friday afternoon.

Next was lunch, and possibly the biggest thing ever to render me speechless. Rachel caught me and complimented my talk on the Pomodoro Technique. I was quite shocked, and as such completely failed to mention how much I had enjoyed her talk (Sorry Rachel). Here was someone who is experienced in presenting, complementing me on mine. After a little discussion about how I might start bringing Agile into my team here (including the idea of a taskboard, that needn't be on full display the whole time), she moved off and I had another Chat with Martin from Aptivate who was telling me about the work they had gone out to do in Zambia bringing women out there up to speed with IT, including the setting up of an Internet Cafe and training staff. It's great to hear about people doing this sort of work, and a shame that they possibly aren't getting as much funding this year to do it as fully as they have done in previous years.

After lunch, the final workshops session. I opted for Visual Management for Agile Teams - Xavier Quesada Allue. In this, we were tasked with creating a Taskboard which would be the focus of the scrum, and would keep track of stories, tasks, if we were on track, who was doing what and backlog.

Here is the one my team came up with

Although, as we went around the room, no two boards were the same. One team (featuring Conny) opted for a Kanban approach

But the coolest had their stories on hearts, which moved along a track. If they stopped, they picked up hairs (a rolling stone gathers no moss) or broke if they were blocked in any way.

This was a great way of getting on task with using a board, and showed that I think if the team is involved in the design, it probably adds more ownership and encourages it's use more.

Xavier also showed us some pictures of taskboards he had helped develop for teams. This was an excellent workshop, and well worth attending.

After the final coffee break of the conference (a chance to make sure people had my twitter alias - setitesuk), we headed back into the Gilb theatre for a panel Q&A session 'Creating a Development Process for your Team: What, How and Why'. The panel was led by Giovanni Asproni, who I met the previous day in Bob Marshalls workshop, and who previously worked at the EBI (so right next door to me!). Beforehand he admitted that he was happy to do lead the session, since he didn't need to do much talking (I must remember that trick next time).

The panel consisted of a number of the speakers from the 2 days (although I can only remember the names of Rachel, Allan Kelly and Willem). The main topic eventually was a debate around pair programming, although other aspects of Agile got mentioned (and even the Pomodoro Technique got dropped in as noting it had become a discussion point - yay me :) ).

I must say, that after two days I was starting to flag a little, so picked up the least amount from this session, but it was interesting nonetheless, and rounded off the two days in a great way.

Mark closed off the conference, and we all left (after grabbing more choccies from the RedGate guys!)

Overall, the conference for me was a great success. An opportunity to find out lots about Agile process. A steep learning curve in places, and just a great opportunity to meet others who are producing great software. My favourite workshop/talk has got to go to Gojko and David for 'The Specification Game', for the great way of teaching us all what we do wrong (it is certainly true that you learn more getting it wrong than right), and favourite snippet goes to James A. Whittaker for being so surprised that so many of us Brits still use a phone book.

Of all my sessions that I had my time again, I would have chosen not to go to, would be Code Debt. As I mentioned in my previous blog, not because it was lacking, but that it was the least relevant to Agile.

Thanks to Mark Dalgarno and his team for organising a great Conference, at a fantastic venue. Thanks to all the speakers that I saw, if I could present even half as well as you all, I'd be happy, and your material was in general excellent. Thanks to everyone I met for engaging discussions, and thanks to RedGate Software for the beers!

I look forward to next year.

Day 1 of Agile Cambridge 2010

2010-10-16T09:08:00.000-07:00

Agile Cambridge 2010

Day 1

With much excitement I left Haverhill on the first bus of the day in order to go to the conference. There was method to this madness, since the plan was drinks in the evening, and I wasn't going to pass that opportunity up.

I left with laptop, plan of the sessions I ideally wanted to achieve, and the bits for my lightning talk.

I arrived at Murray Edwards College around 8.30, and found those of my colleagues (Beth Jones and Conny Brunkvist) who were also attending. Also picking up a pack and some freebies from RedGate Software (choccies!)

At 9 we went through for the welcome talk (Mark Dalgarno - Software Acumen) and the first Keynote.

The Keynote was James A. Whittaker (Google) who delighted us all with his running of test at Google.
As well as unit tests and functional tests, they also have a suite of different Tours, which are designed to go off and fully test the whole applications under different situations, but which are named rather 'cleverly' (I would love to know what 'The Couch Potato Tour' is). He also showed us problems that had been sent in with Google Maps, including the best route to walk from Cambridge to Hull is taking the ferry via Holland.

He also seemed rather surprised that over half of the audience had actually used a phone book in the last 6 months. Clearly a difference between Americans and Brits.

After the first coffee break (good letting them be 30 mins, allows plenty of time to chat!), it was the first sessions with choices. I opted for the Workshop 'The Specification Game', since one thing I admit I am bad on is getting specifics up front, and in the end, this only proved it.

Gojko Adzic and David de Florinier introduced us to a 'simple task'. They were hiring (i.e. they were the customer - note that, I'll come back to it) us to do them a blackjack application. We were to go through the first iteration, and should produce them something which was playable. We needed a business/project owner (that was anyone who knew blackjack - me!) and at least one dev and 1 tester.

This was were we made our second mistake, (the first being that we completely forgot they were the customer). We went for having the dev and tester as separate people.

I spec'd out what I thought we could achieve in the first iteration, (not negotiating at all with the customer), and we went for it. I went for a playable version which wouldn't bother with betting, but would check blackjack, that the dealer would deal correctly, and would say if the player or the dealer won.

At the end of the time allowed, it is safe to say that we failed. Reasons:

1) Some of my specs weren't good enough
2) One of my specs wasn't dealt with
3) I spent too much time helping the tester, than guiding my two devs and the tester
4) I never consulted with the customer for any user acceptance tests
5) I never consulted with the user for exactly what they wanted in the first iteration, and negotiated that it was too much for my team

So, I accept the failure.

Initially afterwards we felt a bit aggrieved since the one thing that everyone in the room had failed to realise was that Gojko and David were our customers, and so we never felt we could consult with them, you notice that I mentioned that they did tell us this up front, so we all failed because we never got them to discuss with us the specifications. We chatted with Gojko nd David, and found that this was what they were trying to get across to us, and that this is the biggest problem with the specifications is that we forget to/or just don't get the customer involved in what to do that iteration. Also the other was that we never asked the customer for UATs, so when we finally got some, they inevitably failed. Also, no-one did the tests before the development. Whilst they said there must be at least one Dev and one Tester, they never said that they couldn't be the same person.

I think that this was an excellent session. One that I have certainly taken a lot away from.

After lunch, I went for the Hands-On session Code Debt (David Harvey and Peter Marks). This was a break away from Agile, but I was interested in seeing exactly what was meant by it (the term is bandied around a fair bit, but not with any real explanation). The best definition of Code Debt that was mentioned is 'It is code that you owe time to'.

We all have written terrible code (I am sure I still do). The purpose of the first exercise was to change/add to some javascript code to make a new set of tests of that code pass. We were given 10 minutes. Some in the group managed it, some didn't. What we were unaware of at the time was that one side of the group were given nice concise well written javascript, and the other just javascript that worked (it was all done via TDD), but had no best practice about it. As such it was essentially unmaintainable code, and us such, they weren't expected to be able to do the task.

And this seemed to be pretty much the point of the session. If the code just does the job, but hasn't been thought about (idiosyncrasies of the language, meaningful function/variable names, refactored) then there is some level of debt owed to the code to get it to that point.

It was a good workshop to have attended, but in this case, I think a little out of place at a conference about Agile, and possibly a bit too much time spent getting over the one point. Refactor and make sure that your naming means something. Still, well presented. (And I also discovered I was probably the only Perl programmer there - or at least willing to admit it).

Another coffee break, and the onto 'Understanding the Agile Mind: How Mindsets Transform as Organisations Rightshift Effectiveness' (Bob Marshall - Falling Blossoms twitter @flowchainsensei)

I had shared a few words with Bob at lunch (we seemed to have managed to be at all the same sessions so far), but hadn't actually twigged he was running this session.

The focus of this workshop was looking at how organisations worked at the effectiveness multipliers that give serious increases in production with lot less waste, but are so far above the norm, that the mindset seems 'Alien' to most. Working in small groups, we tried to come up with things that we thought companies that were at 0/1/2/3/4/5x effectiveness (1x being the norm) would be doing. Bob then mentioned
the 4 different types of organistional styles (Ad-Hoc, Analytic (sometimes called Mechanistic), Holistic/Synergistic and Chaordic) (I've just found a pdf paper by Bob about this)

I think that clearly this is a huge area to look into (I admit by this time of the day I was more thinking about the pub - sorry Bob), but looks incredibly interesting, and I shall be taking the time to read the above paper soon.

After another coffee break, we had the final session of the day, and a chance for me to Do My Stuff. So, obviously, I attended the Lightning talks (since I was presenting one). I should have gone first, but as with all things, the projection system decided to take that opportunity to have a nap, so I got a bit delayed.

The other talks were really interesting though. Nick ably stepped up to now go first, with some interesting insights into how agile and scrum very much mimics how humans have always done things, including that daily scrums are like sitting around the tribal campfire.

Bob then came up, with an explanation of how he came up with his twitter moniker. Tiss from RedGate (I think rather badgered into it by Helen) did some Improv very Whose Line is it Anyway. I then gave my talk on The Pomodoro Technique (see previous blog post), and then Alan Kelly came on to show how 'Doing it Right' against 'The Right (Business Aligned) Way to do it' affects the productivity, cost and sales. Took me right back to A-Level Economics, but a reminder of how things need levels of thought and balance.

After this, a move to the Castle Inn, for a well earned pint (or four) and food, all courtesy of those fine fellows at RedGate, who I spent most of the night chatting to, along with others who were there. (Sorry guys, I can't remember most of your names and it seems that there was no delegate list in the pack).

I left the pub (unfortunately rather abruptly - big apologies to those I was talking to at the time) in order to catch the bus back to Haverhill. I though some sleep might be useful before the following day.

Lightning Tomatoes

2010-10-15T02:34:00.000-07:00

I'm currently at the agile cambridge conference, having a great time finding more out about agile practices (what I'm doing wrong...and right).

In the lightning talk session I gave a quick 10 minute presentation of The Pomodoro Technique

Here are the slides:

Pomodoro lightning talk

View more presentations from setitesuk.

The talk generated a fair amount of discussion (and some people saying that they will give it a go). I just hope with most it wasn't just the beer talking (Thanks to Redgate Software for that).

The Tomato Cometh

2010-09-03T06:35:00.001-07:00

So I have now been doing the Pomodoro Technique for 3 weeks, and I have to say, that it is working very well.

I am even writing this blog within a Pomodoro - watch for live end/starts and interrupts/failures

I am averaging 8 completed Pomodoro a day. You can see my end of day report sheet here.

If you took a look, you'll see that the most I get is 11, and the least is 5. There is no direct correllation between failures and successes, but some things to note:

The Boss is away: Less completed and more interruptions/failures as I am being (Pomodoro ends) requested in his place. (finished sentence and save)

Quick 5 minute break and I'm back (start)

Here is a one day todo when my boss was off

Discussion with others: This is difficult, especially when I'm the one being asked, to keep to within 25 minute slots, so an hour can get lost outside of my daily tasks.

So, how is my day worked.

First hour - no Pomodori

8 -> 9am
get in, check emails and relevant blogs/websites.
write out plan for today. If there is nothing already on my radar, then set aside a Pomodoro to go through emails/RT to get new things onto my radar, and replan the day

9->9.05 - 5 minute break, ready for first Pomodoro to start.
From now on, I work in Pomodori, 25 minutes and then a 5 minute break.
(Just had a failure - interruption where I needed to discuss something with someone for > 30s - start new Pomodoro)

If I get an interruption of approx 30s (very quick question from my team members) then this only counts (to me) as a brief interruption, and I don't fail the Pomodoro. If it is longer than this, then the Pomodoro fails.

Now, I am expected to act on emails if they are urgent, so at the end of any Pomodoro or interruption, I check my inbox, and if necessary, replan my day for the next Pomodoro (or x) to act on an urgent request, else they get deleted or put to act later.

In my break, I typically get a drink, look out of the window, go for a quick stretch of my legs.

This tends to go until the end of the Pomodoro that occurs after 4pm. At this point, I don't start another one, but do the daily tidy up (although not of my desk :) ) and fill in my report chart, final check of emails, and twitter out my successful Pomodoro for the day, plus the last song I was listening to.

Interesting things I have noted over the last 3 weeks:

1) Time goes quickly

2) Breaking jobs up into smaller tasks really helps

3) I can't always predict the number of Pomodori a task will take, but I'm getting slightly better

4) Failing at 20 minutes can make it seem like a task has taken less Pomodori than it actually did

5) I can get more done this way (one set of tasks took about 1/2 day less)

6) A five minute break away increases the likelihood of the Eureka solution to the problem you spent the last 15 minutes looking at

7) Time really does go very fast

So now what?

I am going to continue with this, and I am planning a talk on it now. I think the technique really works for me, and I can track what I have done so much easier than just closing RT tickets/completing features in Pivotal Tracker.

I strongly recommend this to anyone, and if you need any more suggestion to give it a go, here is a paraphrased Tweet I read a couple of weeks ago:

The PomodoroTechnique quite literally saved my arse - Software developers at increased risk of Hemorrhoids.

So get up and move every 25 minutes - you'll be better for it.

(9 minutes left of Pomodoro, time for review of the post)

The two Pomodori this was written in

- first 15 minutes of P1 - taking pictures of my sheets and storing on Flickr
- review - fix all cases of lower case pomodoro(i) to Pomodoro(i)
- review - deciding that I would add this link to yesterdays To Do Today sheet

- review - find out how to embed the photos instead of links

MooseX::AttributeCloner v0.21

2010-08-25T06:36:00.000-07:00

I have just completed version 0.21 of MooseX::AttributeCloner.

The only change here is that I have forced the command options to come back in a sorted order. This was discovered when we found that in some tests, on my developers MacBook, we get a different order of production commands than on the Linux boxes. As such, this is causing tests to fail. A fix of this to return in a fixed order should solve this problem.

It has been submitted to PAUSE (I hope, first time I have tried uploading from a tag in github) but you can also find it on my github account

github.com/setitesuk/MooseX--AttributeCloner

3 Days of Tomatos

2010-08-16T08:27:00.000-07:00

I've been umming and aahing over looking at the Pomodoro Technique as a way of personally increasing my capacity to focus on the task at hand, having heard bits about it on Twitter feeds.

Finally, I bit the bullet, and spent some money on the book 'Pomodoro Technique Illustrated' by Staffan Noteberg (Pragmatic Bookshelf) and read it.

So, all I need is a timer, pen, paper and that's it. Well, that I can achieve.

I had a go the first day after reading the book, and thought I had done reasonably well, but then it all went out the window.

So, two weeks later, (last Thursday to be exact) I decide to try again. In earnest.

What a success!

Although I have made a few errors in judging how many pomodoro certain tasks will take (which actually shows that some jobs needed cutting into more smaller tasks) I have felt more focused throughout the day and felt at the end, I have achieved what I needed to.

I have averaged 10 pomodoro over the last 3 working days, and feel in that this is a realistic target. Tomorrow, plan for 10 - lets see if I can keep at this.

Anyone who is reading this, and finds themselves procrastinating, then I recommend this at the moment. A 5 minute break every 25 minutes really helps, and it is very easy to concentrate on task for just 25 minutes at a time.

The biggest downside I find - my alarm goes off, and the song on my iTunes hasn't finished.

A word to the Mooses out there (Miice?)

2010-07-29T07:29:00.000-07:00

Just found today with upgrade to latest Perl::Critic

Subroutines::ProhibitUnusedPrivateSubroutines

this throws a problem with all the many

_build_

as it thinks they are unused private subroutines, not seeing them elsewhere in the code

To fix (thanks to the CPAN documentation for helping me get to this) add the following in your (.)perlcriticrc

[Subroutines::ProhibitUnusedPrivateSubroutines]
private_name_regex = _(?!build_)\w+

This will pattern match and allow anything beginning _build_

Cheers

Difficult to track bug

2010-07-22T00:18:00.000-07:00

Here was a difficult bug to spot (using 5.8.8 and 5.10.1, not tried on 5.12):


my $output_path = $self->output_path();
my $bam_filename = $self->bam_filename_root();

if ( some condition ) {

  $output_path .= q{lane} . $self->position_decode_string() . q{/}.
  $bam_filename .=  $self->position_decode_string();

}

This was caused by a bit of lack of due care and attention after a bit of copy and paste refactoring.

Unfortunately, the code is perfectly legit, and parses as though the .= after $bam_filename is just a ., without also doing the concat to $bam_filename.

Bit of a pain in the backside to find this one.

A small journey in Benchmarking

2010-07-12T08:10:00.000-07:00

20ish lines of verbose code involving hashes, arrays and grouping, in comparison to a magic piece of regex which does in 3 lines the same thing.


use Benchmark q{:all};

my @lsf_indices = ( 1000,1001,1002,1003,1006,3000,3300,3301,3302,3303,3304,3305,3306,3998,3999,4000,4001,4002);

my %methods = (

  regex => sub {
    my $array_string = join q{,}, @lsf_indices;
    $array_string =~ s/\b(\d+)(,((??{$+ + 1}))\b)+/$1-$+/g;
    $array_string = q{[} . $array_string . q{]};
  },
  verbose => sub {
      my ( $previous, $current_working_index );
      my %consecutive;

       foreach my $index ( @lsf_indices ) {

         if ( $previous && ( $index == $previous + 1 ) ) {
           push @{ $consecutive{ $current_working_index } }, $index;
           $previous = $index;
         } else {
           $previous = $index;
           $current_working_index = $index;
           push @{ $consecutive{ $current_working_index } }, $index;
         }

       }

       my @array;
       foreach my $index ( sort { $a <=> $b } keys %consecutive ) {

         if ( scalar @{ $consecutive{$index} } == 1 ) {

           push @array, qq{$consecutive{$index}->[0]};

         } else {

           my $last  = pop   @{ $consecutive{$index} };
           my $first = shift @{ $consecutive{$index} };
           push @array, $first . q{-} . $last;

         }

       }

       my $array_string = q{[} . ( join q{,}, @array ) . q{]};
     },

);

cmpthese(  30_000, \%methods);
timethese( 30_000, \%methods);

Result


home$ ./benchmark.pl 
           Rate   regex verbose
regex    6048/s      --    -72%
verbose 21898/s    262%      --
Benchmark: timing 30000 iterations of regex, verbose...
     regex:  5 wallclock secs ( 4.96 usr +  0.01 sys =  4.97 CPU) @ 6036.22/s (n=30000)
   verbose:  1 wallclock secs ( 1.37 usr +  0.00 sys =  1.37 CPU) @ 21897.81/s (n=30000)

Verbose lines of code is around 5 times faster. Happy :) Code I can read, and speed benefits as well.

Admittedly, a bit of an exercise, since this isn't really a bottleneck. ;)

File::Spit

2010-06-24T06:09:00.001-07:00

John in my office a couple of days ago said we need the function 'spit' for writing to files much like we have 'slurp' in Perl6 (or by using one of my 3 fave cpan module Perl6::Slurp)

Since I have just had a couple of hours, I have just done this. File::Spit exports automatically the symbol 'spit'.

This method takes a file_path, $data and an optional delimiter flag.

From the POD:

spit:

will croak if once it has a file_path and a possible delimiter, it finds no data (note, this is different from an empty string or 0)

delimiter - if the delimiter matches against /\A[\s:=,.\/;]+\z/xms, it will be used, else it will just append to the file
perl false values (undef, q{} or 0) will be classified as though there is no delimiter given

will croak if it fails to write, with value of $EVAL_ERROR

Write to file, killing any previous versions of file (note, no warnings)


  eval {
    spit ( $FilePath, $data );
  } or do {
    your error handling here...
  };

Append to file, creating if needed, but no delimiter


  eval {
    spit ( $FilePath, $string, 1 ); # note, it is just a true value, any perl false values will use write and kill previous file
  } or do {
    your error handling here...
  };

Append as above, but with delimiter


  eval {
    spit ( $FilePath, $string, qq{\n\n} );
  } or do {
    your error handling here...
  };

On success, this returns a true value (1)

This is probably completely unnecessary, or already exists on CPAN. If it doesn't, I'm happy to push to it, but for now, you can get it from github

http://github.com/setitesuk/File--Spit/tree/v0.1

This in theory could work with any type of data, but the tests only check strings.

I'm off on holiday now.

Happy Coding

Andy

Is there anything wrong with this?

2010-06-16T01:21:00.000-07:00

So, we all think reusing other code and not reinventing the wheel is generally a good thing.

And using Moose is generally good.

So, I want to test if something is an object. I am using Moose.

1) Create a large hash to compare keys against, and then do


my $not_object_ref = {
  HASH => 1, ARRAY => 1, GLOB => 1,...
};

my $is_object;
if ( my $ref = ref $object ) {
  if ( ! $not_object_ref->{$ref} ) {
    $is_object++;
  }
}

# stuff that uses boolean value of $is_object

2) Use a Moose Attribute and eval


has q{_i_am_an_object} => (
  isa => q{Object},
  is => q{rw},
);

sub _is_object {
  my ( $self, $object ) = @_;
  my $is_object = 0;
  eval {
    $self->_i_am_an_object( $object ); # test if this is an object
    $is_object++;
  } or do {}; # I like PBP and perl critic :)
  return $is_object;
}

my $is_object = $self->_is_object( $object );
# stuff that uses boolean value of $is_object

I don't know if this is a pure abuse of the Moose Attribute system, or if there is a much neater way of doing it, but certainly it has merit over the whole keeping track of what refs are not objects. And certainly, if you want to check against the object type, you change the isa to the class name, and give it a better method name.

It's probably more of an abuse of eval :)

Andy

When not to shift...

2010-06-10T04:39:00.000-07:00

I'm not a big fan of 'shift @array'. And I have just sealed why I think that is.

Anyone who has looked at my code will see that my methods always use array assignments

sub my_method {
  my ( $self, $arg_refs ) = @_;
}

Even if it is just $self, rather than

sub my_method {
  my $self = shift;
}

Today, I further found another bug whilst testing. I'm trying out Test::MockObject, and know that I want to return an arrayref, with 1 element in the array.

The code loops through 8 times, and I expect a result from this array each time, since the webservices I am mocking would provide this.

However, in my tests, I get a result of 1 true, rather than 8.

Why is this?

I check the arrayrefs are the same with some debug (they are). I check the array length each time through. 1st time, 1. All other 7, 0.

The reason is that instead of just assigning

my $ref_seq = $arrayref->[0];

I have shifted

my $ref_seq = shift @{ $arrayref };

So of course, I have within the code removed the element off the array, rather than leave it there, and since $arrayref goes out of scope in the code as soon as $ref_seq is assigned, either works.

This is one time when I would expect the 'real world' to have still been fine, but my 'test environment' can't cope.

Obviously, there are times when shift is most appropriate, i.e. you deliberately need to truncate out the first element because you must not allow that to get somewhere else as the array carries on, but that niggling little voice that keeps telling me never to shift has finally had a reason to give me.

Still, liking Test::MockObject :)

It's been a while

2010-04-21T08:19:00.000-07:00

It has been a while since I last blogged. Unfortunately, family health problems are still not in a resolved state, so it probably will be so again.

However, last week, I went on Dave Cross's advanced perl course/seminar. In fact, most of my team did.

Dave knew his stuff (and apologised that he was talking about the 'new' 5.10 when 5.12 was released less than 36 hours earlier).

The course was interesting, however, due to the limitations of time (1 day isn't long enough, and I do like Lab-time) we didn't cover as much about Moose/DBIx::Class/Catalyst than I would have liked.

However, he did cover Test::Builder, which I hadn't really seen before, and how to create your own bespoke tests which will fit in with the plan.

And so I just have.

I have released Test::Structures::Data today. At the moment it only exposes one method

is_value_found_in_hash_values

Now, this is not to detract from Test::Deeply (which I like a lot!) or Test::Data::Hash, however, I could not find any methods which go to see if a value is present in a data structure, and this was particularly what I was after.

It is on CPAN

http://tinyurl.com/y5owods

and my github repository is

http://github.com/setitesuk/Test--Data--Structures

I plan to add some other methods shortly. These are meant to be simple things though. For example, you may have a hash, and a value. You don't know which key the value should be for, or even care which key it belongs to. The above test just sees if it can find it in the hash values.

Hoping this might be of use to someone (anyone?)

Andy

Retrieving the $schema object from the resultset object

2010-04-08T04:05:00.000-07:00

More for my own personal reference

Within a resultset (row) object

my $schema = $self->result_source->schema();

Example reasoning

User wants to be able to obtain (via helper method) all the public groups s/he doesn't belong to

sub public_usergroups {
my ($self) = @_;

my $schema = $self->result_source->schema();

my @public_usergroups = $schema->resultset('Usergroup')->search({
id_usergroup => {
'NOT IN' => $self->user2usergroups->get_column('id_usergroup')->as_query,
},
is_public => 1,
});

return @public_usergroups;
}

The returned set are then also Usergroup objects, so they have the methods relating to those.

BarCamb3

2010-03-19T03:17:00.001-07:00

Ive just been one of the lucky 20 to get a shiny ticket to BarCamb3 in the first batch of tickets released. WooHoo! Just hope now that my wife is better so that I can go.

More tickets are to be released next week, and the following week.

I went to the first 2 BarCambs, which where hosted by Matt Wood at the Wellcome Trust Sanger Institute. They were very interesting events, and I spoke at the 2nd one with a presentation called 'It's Too Much Information For Me', which came to me when I was 5 minutes away from the event in the car, thinking about a Duran Duran song and even the group hadn't been played on the radio.

That's the great thing about BarCamps. They are unconferences, so pretty much anything goes. You turn up on the day, with or without anything to talk about. Mill around having coffee, and put your name up to speak with a topic, or don't, or join some others.

The website for BarCamb is

http://barcamb.ltheobald.co.uk/

and the sponsors this year are Red Gate Software, Paypal and Taylor Vinters Solicitors.

I don't yet know what to talk about/discuss yet. Perhaps it will again come to me in the car. Also, I don't know how 2 days will be filled up, but whatever happens, it is bound to be interesting, and I'm hoping Simon turns up with Mbed again. (I want to play this year please!)

Andy

Artificial Intelligence vs the human brain

2010-03-11T05:07:00.000-08:00

Here is a plea to all people working on Artificial Intelligence.

Please think about what you are doing, and try to avoid loose wiring.

My wife has suffered from Post-Natal depression ever since the birth of our son 4 and half years ago. Yesterday, I took her into hospital again to ensure that she is safe (she is plagued by voices at the moment) whilst the new medication ramps up to a level which is therapeutic for her.

This, as you may expect, is quite distressing for all of us. We have support systems in place though, and my work are being frankly fantastic about it all.

It gets me thinking though. I can program computers to get them to behave in particular ways, but how do you rewrite the programming in a brain.

I recently read 'Pragmatic Thinking and Learning - Refactor your Wetware' by Andy Hunt (www.pragprog.com).

This is great for yourself, or even trying to suggest to others. I have found some easy to apply tips and some which I need to get round to trying, but one thing it shows is that we have set ways of doing things. Our Logic/Linear-mode (L-mode) tends to be dominant, and seems to have the most influence, unless, as the book suggests, we deliberately try to get information from our Rich-mode (R-mode).

The key thing with this though is that we control the flow of information, we determine if we will do it. We seem to have a controlling Sensible-mode (S-mode).

And this is what I like about Computer Programming. A computer really only sources from L-mode, and it allows the L-mode to control it. So I can write a program that is logical, instructs the computer, and it doesn't get anything else influencing it (well, assuming I have taken care of external running factors such as OS, file locations...)

My wife though, is not controlling her own thoughts. What is, we don't know, but not her. Her S-mode seems to lose it's control. Whether the thoughts are L-mode or R-mode based (traditionally, it would look like those things are "Right-brain" thought processes, although when she explains her thoughts, you could argue the Logic-mode is having a say).

Luckily, this time, my wife's S-mode seems to be working enough to stop her finishing the act, but she is getting somewhere close to it (popping enough pills out of a blister pack, but at the last minute, throwing them in the bin instead of taking them).

So where am I going with this.

My opening request is to think about AI, and not introduce loose wiring. Truly Artificial Intelligence should be able to think both in L-mode and have R-mode, but with an overriding S-mode to control all the thoughts.

I like programming because what I produce only has to act logically. I am in turmoil because my wife has some 'loose wires' which are not allowing her to act completely rationally, and I can't fix the bug in the program to correct it.

If AI is a true goal of the computer technology industry, then lets hope that the coding which goes into it won't allow for the loose wires which screw up the controller, or else we might just have a lot of depressive machines which need careful looking after whilst we feed them lots of pill programs to try to make them better.

Here's hoping that the pills my wife is now taking soon start to sort out her programming. I'm glad I never became a psychiatrist.

crontab or daemon

2010-03-04T07:45:00.000-08:00

So here is an interesting choice I need to make.

I have just rewritten a bit of code to email interested parties when a run with their data on it reaches 2 points. One when it reaches run complete (i.e. the instrument has done all it's processing) and then again when the data has been post-processed and qc'd and deposited in the central archive space for them to obtain.

I'm quite pleased with the code. It is more robust than the previous hack which we had never intended to be all encompassing, and actually mails the parties that should be interested (rather than some 'user' which may or may not be the right person).

It is, of course, also written using Moose.

However, now I have to decide, which do I choose, a cronjob, or a daemon process.

Cronjob:

Pros - very quick. Just decide how often to launch it, and run the script.
Cons - need to remember which node the cronjob is running, need to do something with the outputs (logs, etc), need to ensure that jobs don't relaunch on top of each other

Daemon:

Pros - Use a monitor to keep us informed it is still running, cyclical so won't launch over each other, write to a log file easy
Cons - Need to write a daemon controller script

I'm sure that there are others, I'm mostly babbling and writing this down as I think. Certainly, for the first release of this, I will start it as a cronjob, but down the line, I think I will move this to a Daemon, once the script has been in a production environment for a while. (i.e. we know it is working correctly!)

Readonly::Scalar, $VERSION and Module::Build/Install

2010-02-25T06:38:00.000-08:00

As I blogged before, I have just rebuilt my development area. Within this, I installed the latest version of Module::Build (0.3603).

Now, two things have happened since this. The first, I mentioned in my post on the latest release of MooseX::AttributeCloner, about the fact that 'passthrough' is being deprecated as an option for dist producing a Makefile.PL.

I won't go into that one now, but the second is potentially more disastrous.

Before I go any further, I would like to make it clear that this is not an attempt to slag off the people who write/maintain Module::Build. The tool is extremely useful, and I got a very helpful response from David Golden about my RT ticket. I would like to say Thankyou for producing the tool. The below is possibly more our abuse of it than their failings with it.

We use Test::Perl::Critic to monitor our coding standards. (OK, I accept that it is a set of guidelines, and is in no way compulsory, but it is a start.) Perlcritic wants $VERSION to be a constant, and to do this within the code, we use (as for all constants)

Readonly::Scalar our $VERSION => do { my ($r) = q$LastChangedRevision: 8362 $ =~ /(\d+)/mxs; $r; };

(Again, yes, taking the version number from a version control system is supposedly 'not good', but it works for us, and others I know)

When I run perl Makefile.PL, I get the following:
(note, I use the passthrough or small Makefile.PL and perl Makefile.PL to run Build.PL)

Creating new 'MYMETA.yml' with configuration results
Error evaling version line 'BEGIN { q# Hide from _packages_inside()
#; package Module::Build::ModuleInfo::_version::p56;
use Module::Build::Version;
no strict;

local $VERSION;
$VERSION=undef;
$vsub = sub {
Readonly::Scalar our $VERSION => do { my ($r) = q$LastChangedRevision: 8362 $ =~ /(\d+)/mxs; $r; };;
$VERSION
};
}' in /this/package/lib/module.pm: syntax error at (eval 92) line 9, near "Readonly::Scalar our "
BEGIN not safe after errors--compilation aborted at (eval 92) line 11, line 36.

failed to build version sub for /this/package/lib/module.pm at /Users/ajb/dev/perl/5.10.1/lib/5.10.1/Module/Build/ModuleInfo.pm line 332, line 36.

WARNING: Possible missing or corrupt 'MANIFEST' file.
Nothing to enter for 'provides' field in metafile.
Creating new 'Build' script for 'my_project' version '8391.'

OK, in this case, it's a warning, but the Build file is created, and I can do what I need to.

However, updating the Build.PL 'requires' to include some other package libs, the problem becomes more serious.

Error evaling version line 'BEGIN { q# Hide from _packages_inside()
#; package Module::Build::ModuleInfo::_version::p5;
use Module::Build::Version;
no strict;

local $VERSION;
$VERSION=undef;
$vsub = sub {
Readonly::Scalar our $VERSION => do { my ($r) = q$LastChangedRevision: 8212 $ =~ /(\d+)/mxs; $r; };;
$VERSION
};
}' in /another/package/lib/module.pm: syntax error at (eval 28) line 9, near "Readonly::Scalar our "
BEGIN not safe after errors--compilation aborted at (eval 28) line 11, line 19.

failed to build version sub for /another/package/lib/module.pm at /Users/ajb/dev/perl/5.10.1/lib/5.10.1/Module/Build/ModuleInfo.pm line 332, line 19.
Couldn't run Build.PL: No such file or directory at /Users/ajb/dev/perl/5.10.1/lib/5.10.1/Module/Build/Compat.pm line 335.

Here, because it can't identify the version of the 'external' module, it has croaked out.

I submitted a bug report, and David Golden (Big thanks to him for responding) suggested that before the line could perhaps be made

use Readonly; Readonly::Scalar our $VERSION => do { my ($r) = q$LastChangedRevision: 8212 $ =~ /(\d+)/mxs; $r; };

since the problem is in the eval block.

This is fine to do for internal modules, but a problem for anything we install centrally from CPAN (maintenance of code, root access, etc).

I tried using Module::Install as an alternative. This carps errors in both cases, but in both cases doesn't cause the Makefile not to be created.

This post therefore comes down to 2 things.

1) Information to anyone who really cares or reads this blog.
2) Are we the only people who use Readonly::Scalar to declare 'our $VERSION'? (in which case, MooseX::AttributeCloner is possibly the only module on CPAN which does this)

Thoughts and comments welcome, although I would appreciate people not calling us Idiots (or stronger) for using Readonly::Scalar to 'constant'ify the variable (or at least not without good reason). We have looked at use version;, but it doesn't seem to be right with using a version control value.

Again, thanks to the authors/maintainers of Module::Build and Module::Install for these wonderful tools.

MooseX::AttributeCloner v0.2

2010-02-19T00:35:00.000-08:00

Yesterday I released v0.2 of MooseX::AttributeCloner. This is just a bugfix release, thanks to those lovely people over at CPANTs.

The problem was that I missed a file in my MANIFEST, so when I built my distribution package, it left it out. Upshot - tests failed.

This has now been fixed, however, I since discovered a deprecation on Module::Build, which I have fixed.

I had initially set up my Build.PL file to use

  create_makefile_pl => 'passthrough',

which generated a Makefile.PL, which loaded Module::Build if not installed.

However, this feature is deprecated, and may be removed, since newer versions of CPAN.pm/CPANPLUS and 5.10.1 accept the 'configure_requires' option. So, I have converted to using

  create_makefile_pl => 'small',
  configure_requires => { 'Module::Build' => 0.3603 }

in Build.PL. This is the new way to do it. It is mentioned in the POD and README.

The new version can be found on CPAN here

http://tinyurl.com/ylztfvz

Cheers

Andy

Generated code from DBIx::Class and Test::Perl::Critic

2010-02-17T03:59:00.000-08:00

We have hit an interesting thing in our ever changing code base.

We want to start managing the database schema and use the power of DBIx::Class to keep the code and database in sync by auto generation.

However, we also want to keep to coding standards by running Test::Perl::Critic at maximum severity over the code base.

Problem - the auto generated code fails tests, but if we modify above the comment line

#DO NOT MODIFY ANYTHING ABOVE THIS LINE

we lose the ability to run regeneration of the code if we make any database changes.

One solution is to say, don't run Perl::Critic over the resultset files. However, this means we need to separate out any manually written code, which should be here (running Catalyst, etc), or we never Test::Perl::Critic our own code. It also means additional maintenance to our auto test each time we add more files.

Another is never to be able to auto-regenerate and manually keep the database and code in sync.

So, has anyone come across any solutions to this problem? All comments gratefully received.

BTW - we understand that

1) TIMTOADY - some people write good maintainable code which doesn't follow Perl::Critic standards.
2) It would be very difficult for maintainers of code which generates code to constantly keep up with the latest Perl::Critic.

New release of MooseX::AttributeCloner

2010-02-16T00:39:00.001-08:00

I released a new version of MooseX:AttributeCloner yesterday to CPAN.

Here are the changes:

1) BugFix - CPANTs put in a bug report that MooseX::Getopt was not in the dependencies list in the Build.PL module

2) You can now do

my $NewObject = new::object->new_with_cloned_attributes($CurrentObject);

instead of only

my $NewObject = $CurrentObject->new_with_cloned_attributes(q{new::object);

However, to do this, both objects need to use the MooseX:AttributeCloner role. This is on my TODO list that $CurrentObject would only need to be a Moose object, and not have to utilise the MooseX:AttributeCloner role.

It's out there now. Any feedback appreciated.

Andy

It's amazing - how many more CPAN dependencies

2010-02-16T00:29:00.000-08:00

It's true, if you need something done, someone may well have done it before and released a CPAN module. However, there are some that I think might have not been necessary.

After the initial install of my dev area (see last post), I checkout of svn and git my projects again, and started work. However, over the last week, I found a number of other lacking CPAN modules, which I then needed to download.

LibXSLT:

Download Latest version from ftp://xmlsoft.org/libxslt/

./configure --prefix=$HOME/dev
make
make install

Further CPAN modules:

i Cache::Memcached
i XML::Generator
i File::Type
i Math::Round
i Data::ICal
i Date::ICal
i Statistics::Lite
i MIME::Parser
i Perl6::Slurp
i Sys::Filesystem::MountPoint
i XSLT::Cache
i Net::Stomp
i Net::Stomp::Receipt
i GD::Graph::bars3d
i IO::Prompt
i Parallel::ForkManager
i HTML::Tidy
i SQL::Translator
i http://search.cpan.org/CPAN/authors/id/D/DR/DROLSKY/List-AllUtils-0.02.tar.gz (dodgy md5, so use full url)
i Fey::ORM
i Fey
i Fey::DBIManager
i Fey::Loader

I'm not really sure why we particularly needed List-AllUtils, since List-Utils and List-MoreUtils are already being used, but someone has done so. Also, I am not sure why this (and List-MoreUtils before it) had dodgy md5sums. (I should probably put in an RT ticket).

Anyway, just an update.

Rebuilding my development area

2010-02-08T00:51:00.001-08:00

I thought it was about time to get around to rebuilding my development area, for a few reasons:

1) Housekeeping - My dev area was getting a lot of junk floating around, and rather than just go through and delete, I thought it better to restart

2) About time I upgraded to perl-5.10.1

3) I need to start up a VM soon, and thought it a good opportunity to set make notes about what was needed

4) cpan/cpanplus wasn't working for me

5) I've never had GD working properly, and I could be about to lose my desktop at work

So plenty of reasons. I also wanted to try to structure how I setup various apps, so that it should (in theory) be easier to upgrade an of them. Getting further on, I think this might not be so worthwhile, but at least it's a try.

Here is 'What I Have Done' so far

INITIAL SETUP:

In $HOME

mkdir dev
cd dev

This gives me a base dev directory to use

PERL:

mkdir perl

Download the version of perl you want to install

mkdir perl/ (i.e. mkdir perl/5.10.1)

Unarchive the download and go into the directory for created from unarchiving and do the following

./Configure -des -Dprefix=$HOME/dev/perl/
make
make test (go away and make a cup of tea)
make install

This will now give you in $HOME/dev/perl/ the bin/, lib/ and man/ directories

Once you have done this, symlink this version to $HOME/dev/perl/current, and add $HOME/dev/perl/current/bin to $PATH

This should make your default perl $HOME/dev/perl/current/bin/perl

Since you have done this, if you now want to download and try another perl, then you can do the same, and just switch the current softlink

LIBGD:

You need to download
freetype-2.3.11.tar.gz
jpegsrc.v8.tar.gz
libpng-1.2.23.tar.gz
zlib-1.2.3.tar.gz
gd-2.0.35.tar.gz

unpack and install

freetype

cd freetype-2.3.11
./configure --prefix=$HOME/dev
make install
cd ..

jpeg-8
(need to look at)
cd jpeg-8
./configure --prefix=$HOME/dev --enable-shared --enable-static
make
make install
cd ..

zlib-1.2.3

cd zlib-1.2.3
./configure --prefix=$HOME/dev
make
make install
cd ..

[edit] libpng-1.2.x

cd libpng-1.2.x
CFLAGS="-I$HOME/dev/include" LDFLAGS="-L$HOME/dev/lib" ./configure --prefix=$HOME/dev
make
make install
cd ..

[edit] gd-2.0.35

cd gd-2.0.35
CFLAGS="-I$HOME/dev/include" LDFLAGS="-L$HOME/dev/lib" ./configure --prefix=$HOME/dev --with-png=$HOME/dev --with-freetype=$HOME/dev --with-jpeg=$HOME/dev
make INCLUDEDIRS="-I. -I$HOME/dev" LIBDIRS="-L$HOME/dev" LIBS="-lgd -lpng -lz -lm" CFLAGS="-O -DHAVE_LIBPNG"
make install
cd..

For these, I chose not to create individual versions of them. You will also note that libpng is 1.2 and jpeg is V8 but says need to look at, since this doesn't seem to work with this version of gd. However, since I mostly create png images, I'm not too concerned at this time. Must sort it though eventually.

Graphviz:

This is needed for installation of some CPAN modules
http://www.graphviz.org

Follow instructions on how to install, using $HOME/dev as the prefix

Again, no version specific route taken

SLEEPYCAT libdb-4:

Download from Oracle

unarchive latest version and install

cd build_unix/
../dist/configure --prefix=$HOME/dev
make
make install

Again, no version specific route, and needed for some CPAN modules

CPAN modules:

cpanp is the recommended method to download and install modules from cpan

type

cpanp

and the interactive shell will be launched

If this is the first time, then enter the following

s conf prereqs 1; s save

This will save some of the hassle of needing to confirm installation of required modules

These are chosen because I need to set up a webserver, I work in a Bio place, and some are personal choice. Obviously, if you need others, or not some of these, then pick and choose. They are also loaded in this order for convenience and dependencies.

To install a cpan module, just type

i

i Bundle::LWP
i LWP::Parallel::UserAgent
i YAML::Tiny
i Module::Build
i Module::PortablePath
i Task::Moose (select all the optional loads)
i IO::Stringy
i Calendar::Simple
i List::MoreUtils (This had a checksum error, so manually downloaded)
i DateTime
i DateTime::Format::ICal
i iCal::Parser
i Digest::SHA1
i Class::Std
i Crypt::CBC
i Crypt::Blowfish
i MIME::Lite
i DBI
i DBD::mysql # force install if you've no test database available, also requires the mysql client development headers - mysql_config needs to be in your $PATH (probably ~/dev/bin).
i DBD::SQLite
i Tie::IxHash
i XML::XPathEngine
i XML::Parser (again, I got a dodgy md5)
i XML::XPath
i HTML::TreeBuilder
i XML::SAX
i XML::Simple
i XML::Handler::YAWriter
i XML::Filter::BufferText
i MLDBM
i Jcode
i Spreadsheet::WriteExcel
i Unicode::Map
i Apache::DBI
i Readonly (another dodgy md5)
i XML::FeedLite (causes lots of prereqs to be installed - would suggest a cup of tea if you have selected auto download of prereqs)
i Chart::OFC
i YAML
i Digest::SHA
i Ace # force install if fails to make as it may have problems connecting to Ace database during tests
i Bio::ASN1::EntrezGene # force install if fails as it looks as though for tests it needs a non-existent CPAN module
i Bundle::BioPerl
i GD (Why has this failed tests?)
i B/BI/BIRNEY/bioperl-1.4.tar.gz # Requires sleepycat libdb-4 to pass tests
i Bio::Das
i Bio::Das::Lite
i App::Ack

manually download and install DB_File - as you need to Change config.in to point at dev/lib and dev/include
manually download and install BerkeleyDB - as you need to Change config.in as above.

APACHE and MOD-PERL:

Apache = httpd-2.2.14;

http://httpd.apache.org/download.cgi

in dev, mkdir -p apache/2.2.14
cd apache
ln -s 2.2.14/ current

This gives space to install this version of apache into, and a softlink to the version we want to use (similar to perl above)

export LD_LIBRARY_PATH=$HOME/dev/lib
./configure --prefix=$HOME/dev/apache/2.2.14 LDFLAGS="-L/$HOME/dev/lib"

add $HOME/dev/apache/current/bin to $PATH

mod_perl 2.0:

http://perl.apache.org/download/index.html

Get latest version of 2.0
$HOME/dev/bin/perl Makefile.PL
# follow instructions, e.g. apxs is at $HOME/dev/apache/current/bin/apxs

make
make install

Change/create the $HOME/dev/apache/current/conf/httpd.conf and $HOME/dev/apache/current/conf/perlconfig.ini as you need to.

Catalyst:

Now the biggie. Catalyst has lots of dependencies. It will take some time, plus it is interactive.
Just install everything - except the extra DBD supports.
You can do them in your own time, but they may make the Install fall over now, which you don't want.

cpanp
i Task::Catalyst

If you have got through this, then congrats.

I have also downloaded into my dev area subversion and git, and have tried to do ImageMagick (although this is erroring that my C compiler won't compile executables, even though it has done svn and git).

subversion:

Retrieve the latest version and dependency from http://subversion.apache.org/source-code.html
unpack both, the dependency folder should end up in the same directory, and will then be installed with svn

mkdir -p $HOME/dev/subversion/
cd $HOME/dev/subversion
ln -s current

cd into unpacked folder

./configure --prefix=$HOME/dev/subversion/ --eprefix=$HOME/dev/subversion/
make
make install

add $HOME/dev/subversion/current/bin to your $PATH

This will enable you to have/try multiple versions of svn in the same way as Perl and Apache above

git:

Retrieve the latest version from http://git-scm.com/
unpack

mkdir -p $HOME/dev/git/
cd $HOME/dev/git
ln -s current

make configure
./configure --prefix=$HOME/dev/git/
make
make install (had to do as root)

ImageMagick:

problem with my gcc version at this time

mkdir -p $HOME/dev/imageMagick/
cd $HOME/dev/imageMagick
ln -s current

./configure PREFIX=/Users/ajb/dev/imageMagick/6.5.9 EXEC-PREFIX=/Users/ajb/dev/imageMagick/6.5.9 LIBS=-l/Users/ajb/dev/lib --enable-shared --disable-static

After this, you should have a nice fairly 'clean' version of a dev area. If you want ot install other stuff, then I would recommend the method suggested for versioning the download you have. (I can also recommend the MOCA installation idea for mysql, although I choose to not have that in my dev area).

Once inside this, I then create folders for my projects, using svn or git to version control within those folders, just adding the directories to my path as I need to.

Note: I am using MAC OSX Leopard. At times for the make install, I have needed to sudo make install. I accept no liability for anything that happens should you follow these instructions on any system, but hope that they might be useful for anyone who would like to set up a dev/test area, but are not sure how to go about it.

Cheers

Call a Spade a SPADE!

2010-01-29T06:55:00.000-08:00

AAARRGGGHHHH! Once again, I have found myself trawling around some code (ours) which sits on top of someone elses API representation because someone just can't call a spade a spade.

Has the world gone mad over the last year? What is the benefit of making people trawl through some huge long chain of calls just to find out that your asset is (or is not) what you thought it was.

OK, so what is the issue?

Imagine you have Food that you want to describe. Now, each and every item of food is an asset to you.

On top of that, some of those have child 'assets'. So I want to some Weetabix and Banana for Breakfast.

I look in my 'cupboard' (an asset I have) for the Weetabix. What I find is a whole store cupboard full of further 'assets'. So, I check each asset to see if it is Weetabix. I find an asset which is a 'cereal box', containing 24 'assets'. I check its contents, expecting to find Weetabix.

But wait. It also contains 'assets'. Eh? I interrogate the first of these assets, and find that it is in fact a Weetabix biscuit, but what did I miss? Why not have a subgroup or collection name to the cereal box which describes the assets it contains.

It goes further. Someone has taken my fruit bowl away, and instead replaced that with an 'asset'. In this case it tells me it's a bowl, but not what of? Again, I find myself searching down to discover where my Banana is.

And don't get me started on the milk...

I work in a fairly good object oriented world. I accept that objects can have different (and sometimes multiple) class inheritances, (Weetabix is a 'Cereal', which can also be 'Breakfast', 'Food') but why try to force someone to go down from the top - I have an Asset, it is food, that is also Cereal, that I might eat for Breakfast, it contains further assets, which are CornFlakes. That's not the box containing Weetabix then, Great, lets try the next one.
(Of course, finding CornFlakes might make me suddenly change my mind, but my mind 'asset' has had Weetabix coded into it, so I will only accept that today).

There is a reason that Food comes in well labelled cartons (usually). It is so that you don't need to open that carton to find out what is in it (I would imagine Supermarkets not being too happy about that!). I'm all for class inheritance (or Role inheritance in the fantastic case of Moose), but I should be able to get my Weetabix, and then ask questions (should I want to) like: Are you a cereal, breakfast, (an Asset?)?

(It is true that Supermarkets tend to put food into groups, in aisles, but they are usually well labelled as to what you will find in the aisle, you don't need to go down each one).

Rant over. Now, how should I describe Chocolate? Asset, Food, Meal, Essential...

Fighting with a LightSaber

2010-01-25T00:55:00.000-08:00

Today's coding could be fun, as soon as the Ibuprofen wears off.

Yesterday I hit 35, and my son, already a huge star wars fanatic at 4, gave me LightSaber Duels for the Wii, and a Lightsaber attachment for the WiiMote.

Fantastic. This was what I wanted the Wii for. It took my wife to want it for WiiFit to actually have a valid reason to get it, but I can finally be a Jedi Knight.

Or so I thought. I hadn't quite accounted for my fitness levels. I play Badminton at least once a week, and have been doing some exercise, but after 2x45 minutes of Duelling, my Bicep is starting to seriously complain.

So, what has this told me about todays work?

1) Write as little code as possible. Always the mantra of a coder, since you can't have bugs in code that was never written.

2) Try to take those microbreaks that H&S are going on about.

3) Don't sit there wishing to play your latest computer game, when you should be coding. You need to take a break from that!

Some inane ramblings from me today. If you have got this far, then don't forget to raise a glass of whisky to Robert Burns today, and eat plenty of Haggis and Orkney Clapshot.

From Clearpress to Catalyst

2010-01-18T00:28:00.001-08:00

We have had a minor problem with our tracking system which has made it difficult to give out completely to the outside world. It is due to the deployment onto our webserver setup.

In order to be able to 'give away' responsibility for running servers, and some design aspects, we have tied it to the our works internal modules for locating lib files (WorkPaths) and content/page styling and authentication (WorkWeb).

Unfortunately, because of this, without a lot of work, we are rather stuck with it's limitations, and so there is 'work to be done' if outside people want to use it.

So, now, I am taking a break from working just on the pipeline, and attempting to break it into a standalone application, using Catalyst. Sorry Roger, ClearPress is great, but we need something with a bigger support network.

Now, ClearPress works. I still am a fan of, it it ain't broken, don't fix it. And Catalyst has many methods of the MODEL being used, so for this reason, I have worked on the following strategy to move it.

1) Leave the models alone.

ClearPress already has a good way of talking to the database, the models already have most of the supporting code in them, so why change this. Catalyst can use other models as a back end, so I am going to leave these as they are for now. This does mean that to operate, you need two frameworks installed from CPAN, but ClearPress is pretty lightweight in it's dependencies, so I don't feel that this is an issue for now.

2) Keep the templates.

The preferred templating system for both ClearPress and Catalyst is Template Toolkit (much to my new bosses wish that Mason might be better - i.e. using PERL, but I have wittered on about this in the past, so won't here). This is great, because all I have to do is move the templates to the Catalyst structure, with a small amount of recoding.

Here, I have also gone for the approach of putting the files into directories named like their parent Controller/View, so the directories are easier to navigate.

3) Views/Controllers.

This is a semantic thing. In ClearPress, they are called Views, in Catalyst - Controllers. This is the main part of recoding, because the methods are often autogenerated for you in ClearPress, or at least must follow a strict pattern. Catalyst operates differently here (Chained dispatch, different location to find form params, etc), so this is the slow part.

However, again, here is an opportunity not to be missed. We got a little out of the Fat Model, Thin Controller/View philosophy in the ClearPress app (not a failing of ClearPress, it is very easy to do the same in Catalyst/Rails or probably any other MVC framework, it was pure laziness on our part as developers). So, I have taken it upon myself, to move code to the relevant Model where necessary.

Some URLs have changed, and this is again a clear difference between the two. Particularly trying to make the URLs RESTful, but I am managing it.

4) Styling and JavaScript

Most of of our styling uses css, so I am taking the opportunity to create a css sheet with everything that is needed, and hopefully not any more. Our internal web styling is changing, and also, for external release to other users, they need a css sheet which will work for them.

A similar thign with JavaScript. However, a big change is the fact that the core web here is moving from Prototype/Scriptaculous to jQuery. I have already seen some instances that are a problem, since we have coded against P/S and there are conflict with jQuery. So again, I am trying to 'shield' the app against conflict here.

So far, this move is going well, and pretty rapidly. The CSS/JavaScript is pretty much complete. The templates take minutes to alter. So, I can concentrate on the Controllers, and refactoring some code to the Model, where it belongs. Ultimately, the intention is to move the Models as well. We have already run DBIx:Class over the tracking database, so it shouldn't take much to 'fix' them with the code needed, but small steps, and they ain't broken.

The only thing I have left off here is authentication. The new boss said he'll deal with that. Currently, I am permanently signed in as an admin, so we shall see how that goes, when I have migrated all that I can.

Speaking of which, better get to it!

I Don't Play WoW

2010-01-11T00:33:00.000-08:00

Over the last few weeks, I have noticed a surprising large number of phishing emails in my inbox. All allegedly coming from blizzard.com relating to my World of Warcraft account.

They have improved as well.

1) The first ones where every 3-4 days, telling me that I appear to be trying to sell my WoW account. This is against the rules, and I am in danger of having the account shutdown as per guidelines if this can be proven.

- I find this difficult to believe for 2 reasons:
i) I don't have WoW account or even own/play a copy of the game
ii) If I did, I doubt (from past gaming experience) I'd ever have one worthy enough to actually sell.

2) Then came the ones with all the misspellings that you 'would never notice'.

- I wouldn't follow these links because:
i) See 1i
ii) I have a pretty good grasp of the English Language, and I can spell (I only ever failed 1 spelling test at school, and the word did have more than 3 syllables)

3) Then a ramp up of number - 2 a day, and the misspellings disappeared.

- I wouldn't follow because:
i) See 1i
ii) I am not as thick as two short planks

Now, Blizzard, if you are trying to contact me, then please feel free to cancel my account. My email address is already being fraudulently used, and so I am happy for you to ban it (please note, the email address used is different to this one).

If, however, guys who are phishing for the account details - please give up. I am getting rather tired of picking apart your emails for potential problems. You need to realise that, if this is to work, you need to be a little bit more subtle about your approach than you are. People are waking up to the fact that these emails are simply not real - the Ebay/Paypal ones have stopped, just give it a rest. Why don't you take up real 'Fishing'? Perhaps you could make money selling whatever you catch, surplus to what you eat. Start a business, and become a useful member of whichever society you live in.

For anyone who does read this blog, and thinks that I should be playing WoW, especially as a Perl Software Developer/Techie/Geek/Nerd, please don't spam/flame me. I chose the life I live, and I am happy with Lego StarWars and other simple Wii games.

Rant over.

I managed it.

2010-01-08T08:13:00.000-08:00

So, finishing from the last post, I had to do another bit of file manipulation, both internally and readname, but I got the files back to how they should be. Now, hopefully, we have a pipeline that does what it is supposed to again.

No-one would ever break it again, no would they?

:)

Finally, I get somewhere, I hope

2010-01-07T08:11:00.000-08:00

We can't expect things to stay static, but you would hope that when they don't, you find out with time to make changes, and that they tell you!

However, this doesn't always happen, and sometimes the changes are quite major. As any regular readers know, I work at a high throughput sequencing facility, with a major analysis pipeline running all the time. We need to be adaptable, and have become more so, but something changed which hit us hard.

All of a sudden, something that previously expected 3 inputs, suddenly wanted 2. Also, those two inputs are 1 and 3. And the description needed the middle part stripped out. And the files needed changing both name and internally. And ...

You get the picture.

The problem occurs because we are expecting certain outputs, in a certain way to produce the final files that our researchers are expecting. This was no longer going to be what we would get. AAHHH!

So, a think about it, a look around the code producing the files, and we come up with a strategy.

1) Rename the 3 files such that the middle file only psuedo exists to the next part of the pipeline.

2) Run the next part of the pipeline with only those two files.

3) Rename the files back to the original expected names, moving in the middle file again.

4) Keep our fingers crossed :)

So, first things first, rename the files. Now at this point, we think there are only 3 input files to 'Stage 2' as I'll call it. By stripping out the middle descriptor, and renaming, we run the pipeline.

Error...

The error, 'Stage 2' believes there to be a middle descriptor still. It turns out two files are created before 'Stage 1', in prep for 'Stage 2', still with the middle descriptor in it. 'Stage 2' then reads this file, and submits the middle descriptor to the scripts it kicks off. Pants.

So, change these created files. Unfortunately, this can't be done from the config file we create, so after creating the other files, go in and physically change them

Result! But then...

We thought it wanted 3 files, reduced to 2. Actually, it wants 6 reduced to 4. Another set of 3 files, are required.

So, rename those and Result.

I have now successfully run 'Stage 2'.

The next step is to get the stuff back to how we expect for the production of our own output files. Wish me luck, I'm going in.

Pragmatic Thinking and Learning

2009-12-29T00:40:00.000-08:00

I have just finished reading Pragmatic Thinking and Learning - Refactor your Wetware by Andy Hunt. This is a very interesting book, which is well worth a read by anyone who is interested in:

a) Different ways of learning new skills
b) The way the mind works
c) The whole left side/right side debate
d) You just fancy a change from an ordinary software development textbook.

As I was reading it, a few things came to light that I did 'right' and 'wrong' last year.

Learning Moose and Erlang.

Last year I learnt Moose, and didn't learn Erlang. I set out to do both, but didn't manage it. Why? I am capable. I managed to learn to use oo perl, ruby and rails in 3 months, why not Moose and Erlang.

Simple, I had no opportunity to play with Erlang, and I had no defined targets (I want to write in Erlang using concurrency by ).

However, with Moose, it was completely different. I wanted to write a pluggable pipeline system using a modern oo perl system which others would be able to use easily. I had time to play as I learnt the basics, and then start building the structure of the framework as I learnt more.

Allowing time and scribbling.

I started to turn away from my screen and think a bit more. I also scribbled more. I created more scrap paper this year than the previous. I roughly mapped out where the code went. This worked. Thinking back to when I set up the QC application and database, I just got on and did it. That took a lot longer to achieve than it should (I got a stern warning from my bosses boss about that!) whereas the pipeline was in theory tougher, but took less than a quarter of the time in the end to get to it's initial start.

Both of these positives are described in the book as using R-mode to lead L-mode. Since I have seen some of this in action last year, I am going to try to use that in guiding me forward next year. I really need to learn some C. If there is likely to be space, I would like to learn Erlang still. However, the key is to be able to set a reason for doing so. I will.

So, first off C. Erlang isn't required for my job at this time, so I'll leave it for now.

My target for C. Finish going through the C book I have and then take a C script that my pluggable pipeline sets off, and try to work out how it all works, and see if I can refactor it. In the process of this, I should also try to write a small program to sort a user inputted list. This shoudl probably be my first mini-target once completing the book.

There are lots of other things in this book to follow. I think I probably need to reread it. I'll not write about more here now (I have a 4 year old desperate to assemble a Lego Star Wars toy with me) but I shall try to keep up to date my blogging more, and detail when it is something in from this book which has helped guide me towards it.

Another year passes...

2009-12-18T23:06:00.000-08:00

So, another work year has passed. With it including my 10th anniversary at The Sanger Institute, and 2 years as officially a software developer, and a change of team leader, its been a pretty full year.

So, what have I achieved:

Well, first off, I decided to split up a project which helped us a lot, as we have been able to be more agile in changes and bug fixes when dealing with parts of our pipelines and code.

Following the change of leader in our team, we started to spend some time looking at new technologies. I started to take a look message queues. See Stomping on the Rabbit and Pt2.
This was my first time at actually properly failing something. It felt good. I got lots of help from monadic and their team down in London relating to RabbitMQ, and we tried ActiveMQ as well. But the reason we failed it: after attempting a comprehensive test suite to test cases where servers go down, we decided that (at this time) it wasn't stable enough for passing around the code as reliably as we hoped. I spent about 3 weeks looking at this, but we believe that is time well spent. (Another team started to employ it directly, but continually had problems with their poller of a queue.)

I also spent some time at this time taking a look at Erlang, but since there didn't seem to be much opportunity to apply this at work, I stopped for the time being.

I then deployed the badminton ladder I was working on for the Sports and Social Club. See Playing Badminton through Clearpress. This has gone down well, although I do need to make some improvements to the interface.

We had been deliberating about making the analysis pipeline pluggable, so I took this task on. I also used it as a chance to take a good look at Moose as an O-O framework. I have pretty much spent the rest of the year doing this.

This had had me really excited, and probably the thing which has kept me most interested in my job for the last 6 months, and also one thing which proved I didn't waste my time splitting up the project earlier in the year.

After much discussion of whether to relaunch scripts, or load everything to LSF in one go, we went for something that does as much as it can, then submits jobs sequentially to LSF, but with code being as decoupled as possible. You can see in the powerpoint presentation Pluggable Pipelines the style we took. With the power of the Moose behind it utilising Roles and even developing my own CPAN extensions MooseX::File_or_DB::Storage and MooseX::AttributeCloner.

With this pluggable pipeline setup, of course it is now multiple times easier to add new stuff, particularly QC testing. So, we need a way to display the results. The new boss is keen on Catalyst, so I decided now is the time to look at this. Whilst this project has moved onto another team member, I gave it a bit of a start, initially to see how it would deal with getting stuff from a file server rather than a database 'Backending Catalyst'. A couple of book sources later and I get a working app up. In fact, really, the biggest challenge was tring to apply some css that I stole from two Rails apps.

I pass my 10 year anniversary without a hitch, it seems. It is funny to find that I have worked somewhere for 10 years. Although saying that, I have done a number of different roles here, from washing glassware to sequencing dna, to instituting the widespread use of Transposon Insertion Libraries in finishing, to Quality Control, to Software Development.

And then the slope to Christmas. The pipeline systems aim to be flexible has now been pushed to the limit, with virtually unreported changes to the 3rd party pipeline which it has had to keep up with - and whilst the code unavoidably has to change, we have spent most time discussing the planned changes to cope, and testing it, rather than coding.

I've also had to come to the decision that I think Class::Std is well and truly dead. RIP. You helped me become a better, O-O programmer, and you will be remembered with a fondness, but Moose has just dealt you too many blows.

Overall, it has been quite a productive agile year. I'm taking a look at the C language and trying to lobby to get a course run at work on Moose and Catalyst. Hopefully, these will be big things for me next year. I look forward to it.

Merry Christmas, and a Happy New Year to all.

Speedy does it.

2009-12-12T00:38:00.000-08:00

In order to extend the flexibility to the pipeline, I developed MooseX::AttributeCloner (see CPAN) in order to pass around variables set on the original command line.

However, this has led to the need to redevelop some further loading scripts that had been originally written with Class::Std and Class::Accessor. Switching them to Moose has had a two-fold effect that I have been rather happy with.

1) Less code. Since the Roles that I had written for the pipeline (but made sure I left as potentially 'common code'), consuming these ditched a substantial amount of code. Woohoo!

2) Faster. The loading of the data runs exceptionally fast now. The dropping of the need to generate extra objects (since code was refactored into consumed roles) has increased the speed of data lookups. I haven't truly benchmarked the loading of the data, but something that was taking a couple of minutes to run through completely, now takes mere seconds. In fact, the logs don't show any change in the timestamp used in each of the print statements.

The longer of the two refactored scripts had a reduction of nearly half the code. The shorter about 45%. The slow parts (conversion to xml and database loading) of the long script needs some refactoring of how it loads. (drop xml mid stage) but now it seems that some improvements have certainly been achieved with the switch to Moose.

Another factor which has also been able to speed this up is the fact that, if the pipeline has told the scripts lots of information (such as file system paths, filenames, etc) then the code doesn't need to work it out again. A definite advantage of the combination of ustilising MooseX::Getopt and MooseX::AttributeCloner.

Always change the branch!

2009-12-05T00:42:00.000-08:00

Again, this one of those blogs which have shown some numptiness by me.

Why did some things break, when they were working yesterday.

We have a lot of file passing to do, and unfortunately, a third party analysis pipeline starts changing file locations and filenames, such that we have to remain agile to cope with the third party changes.

So, we discover this happening and quickly fix our pipeline to cope with a filename change. Deploy and go, everything is working as expected.

A new release of the pipeline occurs from the development branch into trunk, and then deploy. But suddenly, the files aren't being found as expected. What has happened?

Quite simply, I didn't make a corresponding change in the development branch for the original bugfix. 4 hours of developer time spent trying to track this down for something that I broke my own rule.

Always change the branch/backport trunk to ensure that the bug is fixed everywhere.

Sometimes, just trying to be too agile, sorry read trying to fix a bug fast, just causes the bug to get reimplemented further down the line. Take your time Brown! A little caution and doing it right will make it better.

Schoolboy error teaches a lesson.

2009-11-26T02:12:00.000-08:00

This is not a slander of Perl::Critic, but more a need to ensure that I remember the following.

Things break when there are no tests!

perl critic threw a paddy over

print STDERR q{Hello World!} or carp;

Because STDERR is not preceded by a *

So in my program I preceded to be a good little programmer, and put a * in front of all my STDERRs. (Before people ask, this is in a help statement to be outputted to the user if they didn't provide a run id, not really Hello World!, and was actually a quickie script to fix an external bug initially).

In PBP, it mentions the good practice of surrounding *STDERR with braces, but in my case, Perl::Critic didn't throw for this.

Now, here is my problem (my fault). I didn't have a test which at least compiled this script.

Can you guess what happens next? It goes into production, is launched and fails to compile, since

print *STDERR q{Hello World!} or carp;

isn't allowed. Here it is in a one liner:

>perl -e 'print *STDERR qq{hello world};'
syntax error at -e line 1, near "*STDERR qq{hello world}"
Execution of -e aborted due to compilation errors.

AAAGGHHH! It is a basic schoolboy error, I'm sure, but certainly teaches a lesson after around an hour of digging through logs to try to work out exactly why it worked last week, and not this.

All braced and ready to fly.

Cloning via the command line

2009-11-25T05:15:00.000-08:00

I have update MooseX::AttributeCloner today on CPAN - v0.17

It had a few revisions over the last few days. Some due to the CPAN testers firing me back errors in my tests, some due to development requirements.

The last new thing about it though is the method 'attributes_as_command_options'.

This method goes through all of your built non-private attributes which have an init_arg, and builds a string of them for using in a command line.

The default is to render them as follows

--Boolean --attr1 val1 --hash_attr key1=val1 --hash_attr key2=val2

but you can request that the values be quoted, an = sign be placed between arg and val, and that only a single dash is used.

You can also exclude some attributes by supplying an arrayref of 'init_arg's to be excluded.

For more info, see the CPAN page

or once installed, perldoc MooseX::AttributeCloner

Unlike the new_from_cloned_attributes, you cannot supply additional key/value pairs to be added, but then, if you are generating a command line you can always add these yourself.

I hope that this proves of use to some people. It is certainly helping to reduce code and increase the flexibility of the pluggable pipeline system I have been developing.

Refactoring into Roles

2009-11-17T00:06:00.000-08:00

So, the pipeline moves along, and then the bossman says:

'I want to be able to run it outside of the normal pipeline locations - how do I do it?'

After a discussion as to why you want to do this (since the normal person responsible for testing out new Illumina pipelines often does it in the usual place, moving softlinks as necessary), it is that he wanted to run some stuff to dump the output into his home directory.

OK, if that is what you want.

'Oh yes, also, can we make it more generic. A lot of things we have done imply the setup we have here (most of which is generated by the Illumina pipeline) and I would like it to move away from that'.

This is a major refactor then. Separate Business logic from Generic logic, and allow everything to be user defined, should the user want to.

Looking at it, it seems an obvious thing really, but at the time I have to say, I panicked a bit. The pipeline has been written to be pluggable, so new 'components' can be switched in or out quickly, but I hadn't really planned it to be used outside of the current setup. The principle is easy enough to apply elsewhere, and should not take long to set up, but the components were specialised to apply to what they represent.

So, the first thing I did - go and get a drink. I don't drink Tea or Coffee, but popping off for a break seemed like the right thing to do. This break lasted a while, as I thought about strategies to take. Should I go for all new objects, passing them around. How about pushing them from the command line? MooseX::Getopt seemed to give an option to put any attributes onto the command line, but what about subsequent component objects from the pipeline?

In the end, I wondered about using Moose Roles. The Manual seemed to suggest that this could be a good way forward. A 'class' that doesn't need to be instantiated or extended, but instead is consumed to become part of the class that you want. So, everything could have all the same attributes, in exactly the same way.

Now, I have to say, I could see an immediate danger here. If everything can be given to everything, then what makes anything different to be a component, or the pipeline 'flag waver'. So, I needed to be sensible, in that the pipelines need to match the components that are going to be launched from them, but not others.

First job: Sort out the pipelines. This was quite simple, and needed doing anyway, since the flag waver had a component launcher for every possible component, but most components are only used for 1 of 3 pipelines, so separate out the 3 pipeline components into subclasses. This had the added advantage of naming the 3 pipeline flag wavers as well.

OK, so what's the next thing.

Go through all components looking for common attributes and methods, and labelling them as generic, or specific. Once I had done this, I created some roles which I refactored these into. Sometimes leaving them in the class if they were unnecessary to be available in multiple classes, otherwise putting them in a role which described if they were generic/business logic and which described the type of feature they give.

After that, I just needed to apply roles to classes which meant that the flag waver had the same attributes as any component classes it would launch. This enables me to have the attribute value in the flag waver, and pass it to anything launched from it.

This left a headache in that new instantiation would need me to loop through all attributes and pass them through. So, I put together another role to do this (see http://vampiresoftware.blogspot.com/2009/11/moosexattributecloner.html for details on this).

So, now I'm left with the final problem, how to allow the user to run a pipeline with the options they want. A quick solution to this was already being used by another of my team, MooseX::Getopt. This is a great Role to apply to a class, which then enables any attribute to instantly become a command line option. Since the the Roles created above give attributes to a class, then they become command line options. Hurrah, problem solved.

So, after I initially thought that Roles would not really be worth it much, considering how much subclassing I have normally done and have been used to, now I'm convinced that this is a very useful technology.

Result of all this refactoring - over 900 lines of code lost. The value according to sloccount if $35,000 less. The code is now more maintainable, and more useful for users who want to define their own parameters.

Sorry Class::Std, you have just had another nail forcefully bashed into your coffin.

MooseX::AttributeCloner

2009-11-08T14:05:00.000-08:00

I have just released a MooseX::AttributeCloner to the CPAN. The purpose of this role is to go pass all built attribute values from one class object to a newly instantiated object.

The reason for the creation of this role is for our pluggable pipeline system, where we want to be able to pass any parameters from the command line to any of the objects which are responsible for submitting jobs.

The Role inspects all the attributes the class has via meta, and checks that the attribute is defined. If it is, it passes the value into a hash, which it then uses in the object instantiation of the new object.

my $object_with_attributes = $self->new_with_cloned_attributes('Package::Name');

This will take $self's built attributes, and uses them in a hash_ref to create a new object instance of 'Package::Name'.

It does check the init_arg of the attribute in $self, and uses that as the key for the value. If you have a Ref in the attribute, then it will be the same Ref in the new object. (So, in this case, it isn't strictly cloned, please don't be pedantic).

If you wish to pass through additional information, or use a different value for an attribute that you already have a value for in the current object, then a hash_ref can also be provided with key/value pairs so that these can be used/added as as well

my $object_with_attributes = $self->new_with_cloned_attributes('Package::Name',{
attr1 => q{val1},...
});

There are also two additional methods provided by this role, 'attributes_as_json' and attributes_as_escaped_json'. With these methods, you can get a JSON string with the values of all built attributes, which you could then use elsewhere, such as just building a data hash or pushing into a commandline. More info can be found in the Documentation, but it is worth noting, objects are not stringified, throughout the whole data structure. In the case where this would be in an array, a null will be found instead.

I hope that this will be found to be useful. Please give me any feedback for this that you wish.

Extending a Role attribute

2009-11-06T03:27:00.000-08:00

Not so much a blog post, more a blog question.

When you are 'extend'ing a Moose base class, Attributes can be extended

package Super::Hero;
use Moose;

has q{super_abilitity} => (isa => q{Str}, is => q{rw});

...

package My::Hero;
use Moose;
extends qw{Super::Hero};

has q{+super_abilitity} => (required => 1);

...

However, in this case, I would like Super::Hero to be a Role to be consumed by some class.

package Super::Hero;
use Moose::Role;

has q{super_abilitity} => (isa => q{Str}, is => q{rw});

...

However, the following doesn't work:

package My::Hero;
use Moose;

with qw{Super::Hero};

has q{+super_abilitity} => (required => 1);

...

The required 'extension' is just ignored. I have to actually declare the whole attribute again.

package My::Hero;
use Moose;

with qw{Super::Hero};

has q{super_abilitity} => (isa => q{Str}, is => q{ro}, required => 1);

...

I can't find anything in CPAN Documentation to confirm or deny that this the deliberate design. Does anyone have any suggestions as to any solutions to this?

Any help appreciated.

10 years at The Sanger - Have I finally achieved my career goal

2009-10-31T00:58:00.000-07:00

Friday the 30th was my last working day of year 10 at the Sanger Institute. I have worked in 4 teams whilst there, plus I had been there for 2.5 months before officially starting temping in the glasswash team. Also, in January, I hit the momentous age of 35. (Which, according to bible references to 3score and 10 being the life of a man, makes me middle aged).

So I wonder if I have finally reached what I wanted my career to really be. Over the next month, NaNoWriMo (http://www.nanowrimo.org/) is on and I have decided that I might just spend the next month putting to paper my life so far. I don't claim it will be great (I'm fairly average in most things), but I think it will be good to recap and look back, coming from a middle class background, who wanted to be a bricklayer when I played with Lego aged 5 and a computer programmer aged 13 (but couldn't find anything at school to help him), to being a teacher, scientist on Human Genome Project and a Perl developer.

Who knows, maybe it will interest people, maybe it won't. But a life recap can't be bad.
And maybe it will just make me think, life really isn't too bad, you know.

Course on Moose and Catalyst

2009-10-25T00:44:00.000-07:00

I work at the Wellcome Trust Sanger Institute. In my team, New Pipeline Development, we have identified a need for some training in Moose and Catalyst. If anyone has any course recommendations, I'd be interested in hearing about them to propose to my team and training co-ordinator.

The course can be aimed at people who are experienced Perl developers, but only have a small amount of experience of Moose and Catalyst (as anyone who reads my blog will probably identify).

Replies as Comments, or emails will be appreciated. Many Thanks in advance.

Does anyone else do this?

2009-10-21T06:20:00.000-07:00

I've been away on holiday, and I'm just going back over some comments left on my blog, and got this one?

"Is that a new fashion to use q{string} instead of 'string'?"

My response:

"Just a habit I have gotten into since using perl::critic and PBP.

As '' and ' ' are not allowed by 'the rulez' and you should use q{} and q{ }, (and their double quote equivalents}, I have the coding habit of just using then straight off."

Anyone else doing this as a matter of course, or is it just me? I notice going through a number of recent books, ' and " are used in most of the code examples.

How to put your perl ironman pic on blogspot

2009-10-12T00:09:00.000-07:00

Go to your dashboard, and under the chosen blog, select layout

Under Add and arrange page elements, select one of the two add a gadget depending if you want it on your sidebar, or at the bottom.

Select the html/javascript gadget.

Give it a title.

In content type a html img tag containing the following

http://ironman.enlightenedperl.org/munger/mybadge/male/setitesuk.png

replacing setitesuk with your username or nickname (male can be switched to female).

click save and hey presto, it will now appear the next time you load your blog.

Note: I am sure I am duplicating this post from somewhere. Partly, this is for my own info, partly to help anyone who also can't find the details easily (apart from mst's blog about the url for the image.)

Is it easier to be specific?

2009-10-11T00:32:00.000-07:00

I'm re-assessing some code, trying to refactor it into a re-usable role. Simple, you might think? But is it?

It is very easy to write your code to perform a task on a specific item, or in a specific way. From variable names which mean something tangible, to methods designed to act on a pathway which is unique to your production setup. But, how do you make it more usable?

1 - Variable-Method names

I was taught to make my variable/method names mean something. This makes the code more readable.

$donut = q{jam donut};
...
eat($donut);

instead of

$d = q{jam donut};
...
eat($d);

This is a trivial example, but the principle is there.

However, you (read I) can take it too far. One such point is in directories. For the analysis pipeline, we end up with a directory, after a step called GERALD, called GERALD-date.

In the code, we put this into $gerald_dir. Sounds reasonable. Everywhere I read $gerald_dir, I know exactly what it represents.

However, here is the problem. What happens when the step and directory are renamed Harold. Whilst the principle is the same, and the same files are there, suddenly the variable name is wrong. Just grepping the filesystem won't find something like Gerald. At this point you are probably screaming at me, give it a semantic name, and document what it represents.

Exactly, but that is easy with internal local variable names, not so with public exposed method names. Suddenly I need the role to include a deprecation cycle for the replacement method names, aaahhh!

2 - Application/Locally specific

Anyone should argue that locally specific logic should exist as far up as possible, leaving it out of the generic process end logic as much as possible. No arguements there.

But how do you determine which is app specific, and which is generic.

Obviously, naming conventions are app specific. Or are they? Many things might need to know how to construct a filename in a particular way.

Ok, then how about directory structure? Again, you may find many apps wanting to access the Recalibrated data dir. They all need to know how to do get there.

This is, as you can see, quite a grey area. One where, as I am finding, I think the refactor into a generic role still needs to be a little more specific than might be first thought. You can't get to a directory without at least some knowledge of where it is likely to be. You can't open a file without at least some knowledge of how it is named.

Determine a row in a database - Some way of constructing a query to get it...

Paul Weller said "No one ever said it was gonna be easy", and I wouldnt want it any other way, but remember, if you want your code to be reusable, try to make names semantic, but not specific, and document what they represent.

Also, ensure you document why you process with particular assumptions. At least then, no-one can say you didn't tell them.

That is my plan at least (where I can) from now on.

Just a bit of info

2009-10-10T01:09:00.000-07:00

Whilst I don't have the code here to show this, an interesting thing that we found this week regarding a Moose attribute.

has q{attr} => (isa => q{Str}, is => q{rw});

This makes $class->attr() both reader and writer.

Now, it is mentioned in the manual that if you specify an attribute option, it will override the default by Moose, but what I was interested in was the error message.

has q{attr} => (isa => q{Str}, is => q{rw}, writer => q{_set_attr});

The error message you get here, if you try to use

$class->attr($some_string);

Is that you are trying to modify a read_only attribute. I was expecting it to have something different to this, since 'is => q{rw}'.

Using 'is => q{ro}' still allows you to use your private writer to set the value.

Personally, I always use 'is => q{ro}' unless I want the attr name to be the writer, but this is mildly interesting that setting a writer effectively overwrites the declaration of rw to be ro. Does this therefore make it more or less confusing to the user?

Looking at it from two different perspectives:

Public attribute setting:

Someone inspecting the code for this attribute would see rw and this would tell them, hey you can set this. Hopefully they would look for a writer option before diving in to use the attr name itself. But I can see the use of declaring rw in order to give some hint.

Private attribute setting:

Someone inspecting the code would see rw, and may assume that this means they should be able to set this attribute. However, the writer should only be used internally to the class, therefore giving a hint that it can, and perhaps should, be overwritten if needed. Of course, as with any situation, let the buyer beware, this functionality may change.
However, declaring ro, you are dropping the hint that outside of the class, if you didn't set it on construction, you really shouldn't be thinking of doing so.

I'll leave it up to someone else to determine a best practice. It was an interesting thing to discover. I shall stick with declaring 'is => q{ro}, writer => q{_set_attr}' since most of the time, it is private to the class to set anyway.

The Definitive Guide to Catalyst

2009-10-05T01:43:00.000-07:00

I have just finished reading The Definitive Guide to Catalyst - Diment and Trout (Apress). As Catalyst Newbie, all I can say is buy and read this book.

It has lots of helpful advice and guidance, from creating your first application to extending Catalyst, and lots in between.

It also takes a good look at DBIx::Class, Moose and the Reaction framework, plus a good explanation of the MVC pattern.

I think this book is going to prove a valuable addition to our teams bookshelf, and be referenced for some time to come.

Pluggable pipelines - the return

2009-10-05T00:27:00.000-07:00

On Friday, we deployed v5.0 of the new pluggable style analysis pipeline. This now includes a pluggable analysis pipeline, the archival file creation, and qc pipeline, and archival pipline.

With this new setup, we can now add additional steps (or remove them) easily in a matter of a few hours (usually actually minutes, but then we need those tests).

If you look back to my previous post, the style is there, and we have found it really works well. It is keeping the idea of individual application code separate from the pipeline code which runs it, and even loosely couples individual apps so they run more smoothly.

An additional bonus which has occured, without even trying, is that we have a significant improvement in run time. The reason is that, since we have queued all as separate jobs, instead of as batched makefiles, LSF runs them as it finds space. This means it doesn't need to find, or earmark, 8processors with max memory allowance, but just one or two with smaller memory, so they can be assigned more efficiently. This passes stuff through faster (assuming dependency trees pan out correctly), and so users can get the data quicker. Fantastic, and even better since we didn't plan for it.

No time to stop yet though, we have lots more features to add, but since we have managed to move successfully to this more agile structure, I don't see them taking long to deploy.

Here's to agile structures and development.

Role reversal

2009-10-03T01:28:00.000-07:00

So, in our further investigations into using Moose, I have started to look at using Roles to produce reusable code instead of base classes.

You can find out in the Moose::Manual pages on CPAN about setting them up, but what we had found initially confusing was how much we should make a role do. So our initial investigations just led us to produce ordinary Moose objects to do the functions, but not import them into the consuming objects.

So, we move further on, and find ourselves repeating code as we don't want to import some objects which mostly do other things, so what is the solution.

I'm now taking another look at Roles, to see if that might be the solution, but with another key factor. Keep the role doing as specific a thing as possible.

An example of this is that we often need an object to have methods to expose a run id, a short run name and a run_folder. We do have an object which can provide this, along with other functions, but it is not really reusable in some situations. So I have just done a very small role run::short_info, which can then be used to import just these methods in.

The result, a small amount of (what should be) very reusable code over most, if not all, of our frameworks.

The next one I am doing is run::path_info, which will hopefully give just the next level of features that many need, but not all.

This feels very much the right way to go now, and I hope that we will have a good amount of reusable code which will increase flexibility, and keep the code low maintenance. (Well, let's keep our fingers crossed).

Just now to hope that someone fixes the requires attribute/method in Moose:Role soon.

system($cmd) vs qx/$cmd/ in tests

2009-09-29T03:41:00.000-07:00

I have just found an interesting thing in my test suite.

The other day, I had some silent failures since I was capturing the output, but not checking the error code, of some commands.

my $output = `/some/command/which/normally/works -params`;

Since the output was unimportant, but the command working was, I thought I'd switch as follows

my $rc = system(q{/some/command/which/normally/works -params});
if ($rc != 0) {
croak q{Meaningfull message};
}

All seems fine, until I run the test suite.
t/20-mytests.t ..
1..9
ok 1
ok 2
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
00000000ok 9
Failed 1/9 subtests

Test Summary Report
-------------------
t/20-mytests.t (Wstat: 0 Tests: 8 Failed: 0)
Parse errors: Bad plan. You planned 9 tests but ran 8.

So what is wrong here.

I don't know the ins and outs, but after a bit of debugging, it is that the system command under the test framework causes the TAP not to count the test, regardless of it passing or not.

Solution to this:

Go back to

my $output = `/some/command/which/normally/works -params`;
if ($CHILD_ERROR != 0) {
croak q{Meaningfull message};
}

Everything is now fine again.

Sometimes there just isn't enough

2009-09-26T00:24:00.000-07:00

I'm starting to get confused. (OK, starting is wrong). What is the problem with more error/debug than less.

I mean, sure, your average user is not going to have the foggiest idea what your error information is, and so a nice helpful message is good. But what about in a system,(which has many many many tests) where the average time to run in production takes 1000 times longer than any of your tests. And you get for Joe User

HI,

I received this error message:

An error occured in processing your request. Please let us know at line 360 in something.pl

Could you do something about this. I had been running this process for 10 hours.

Cheers

Joe

This is great, except, I know absolutely nothing further. Joe, understandably, reported the error he got (all of it!) and so thinks he has been helpful, but when you look at line 360, it's a croak of an eval block, where all your code is in multiple modules running inside it. AAAHHHH!

From looking around (and I may only have looked at a relatively small subset) people seem to think that letting the user see all of the error stack is a bad thing. Why? Speaking with some users, they don't understand the output, and would like an additional friendly message, but want to be helpful. They just don't know what to say.

My response to get info from Joe above would be:

On which node did you run this?
What parameters did you set/include?
At what time did you start and stop this process?
Did you have an error log?

To which Joe replies:

Parameters: a,b,c
I submitted it to the farm from work_cluster2
My bsub command: ....
Error log: dev/null (because, that is what he was instructed)

Great, no further output. I can't reproduce, I can't write a test, I can't track down the problem without running the whole thing myself.

So what is the potential solution:

1) Don't be afraid to show your user what looks like the internal workings of the script. Give them the whole stack output. With a friendly message too, of course.
2) Training to ensure that they write an error log (dev/null is not a log, but users don't know that often).
3) Training to ensure that they email you/bug report through rt the whole stack. If they know it will help you solve their problem, they ought to be happy. (Certainly users I spoke to said they would be happy to provide 200 lines of 'garbage output' if it meant that I could solve a problem even 25% faster.

I don't know who suggested it, or who started the culture. It might not be great for commercially sold apps, but certainly where you have internal or free (in it's many forms) software, then surely you shouldn't be worried about 'what the user might find out', because, I would be 95% certain, they don't actually care!

So big it up for confess and cluck, or even occasional print 'debug info trace points' as they ought to be the way of the future, in dealing with those bugs in a timely fashion.

readdir in the wrong order

2009-09-18T23:55:00.000-07:00

This week we found an interesting bug (which seems quite obvious really, but still threw a spanner in the works) which was that using the readdir function doesn't equate to ls'ing a directory, i.e. you can't be sure that the files will be in alphanumeric order.

So, as you can probably work out, I had a method using this function to obtain a list of files in a directory, and then was passing the list to another program.

This program, although we make a promise that if two(or three if multpilexed) files for a lane generated with this program everything would be in the correct order, doesn't actually do any internal ordering of the list of files passed to it (where was that documented?) and so just readdir alone, passing the list of files, meant that they weren't guaranteed to be in the correct order needed. AAHHHH!

So, a quick sort to the list, and everything is now fine, but that was a bit of a surprise. So was spending 2 days trying to reorder the files that had been created wrongly, although I now have a script that can do this should it happen again, which on the farm only takes about 10 minutes.

So, as I said, an interesting bug. I will just have to remember that if I do anything with a list from readdir in future, run a sort on it afterwards, just in case.

My first CPAN module - update

2009-09-12T23:41:00.000-07:00

I have been politely requested to rename my first module to MooseX::File_or_DB::Storage, so I have done so and it can now be found here:

http://tinyurl.com/po5voa

It is exactly the same as v0.2, just a different package name. MooseX::Storage::File_or_DB is scheduled for deletion off CPAN on Wed 16th, so I urge anyone to update now.

Cheers

Andy

My first CPAN module

2009-09-07T05:33:00.000-07:00

So I have pushed my first ever CPAN module, MooseX::Storage::File_or_DB

http://tinyurl.com/l3xfkc

I blogged about this as I started the module before

http://vampiresoftware.blogspot.com/2009/08/filesystemdatabase-or-both.html

and this weekend I was finally able to finish the first release.

As of the moment, you need to use this by extending it

package MyClass;
use Moose;
extends q{MooseX::Storage::File_or_DB};

But I hope for a future release just to be able to

use MooseX::Storage::File_or_DB;

This gives functionality to enable you to write to either a file a JSON string, or a database, and re-instantiate the object from either.

It makes heavy use of MooseX::Storage ( http://tinyurl.com/nujf4c ) - a big thanks to Tomas Doran for writing this - for inspecting the object, and providing the ability to write out to a file as a JSON string.

I hope that this will prove useful. Please do read the POD/CPAN page before use, and contact me about anything you feel relating to this - all constructive comments gratefully received.

Backending Catalyst

2009-09-06T02:08:00.000-07:00

I am starting to look at Catalyst as a method to display some quality control results that are coming out of our analysis pipeline.
ClearPress is a nice MVC webapp builder, but is quite a light weight framework, and uses Class::Accessor for object base. We would like to move towards using Moose based objects and need a way to integrate these into Catalyst.

I am currently working my way through the latest Catalyst book (Diment & Trout), but before this arrived I found we had the following book on our Safari Subscription - Catalyst: Accelerating Perl Web Application Development: Design, develop, test and deploy applications with the open-source Catalyst MVC framework - Jonathan Rockway.

Now, note, I had been through the Tutorial on CPAN, but couldn't find on there anything about using a Filesystem as a source for the model (Did I miss something?), but this book luckily had a section on doing so.

Firstly, why have we QC data in a filesystem?

When we run the pipeline, this all happens on a staging area, which we write everything to, and then copy all our data into long term archival databases. The QC data is no exception, but we only want to archive the final agreed data. Bioinformaticians don't seem to ever be happy with a first pass that fails, if there is any chance it could be improved (i.e. a new test pipeline version, could rerunning jut squeeze 2% more...). As such we want to view the data in exactly the same way from the filesystem as from the database, because we don't want it stored until the last possible moment.

What have we done for this?

My team have been producing Moose objects which are:

1) Producing the data
2) Storing in JSON files (MooseX::Storage)
3) Reading in JSON files (MooseX::Storage) to re-instantiate the object
4) Saving to a Database (Fey)
5) Re-instantiating from a Database

I've been working with iterations of the objects, using the files, but want the objects to just sort it themselves - I shouldn't know where the data has come from, and these objects should be used in (in fact are being written for) other applications.

Catalyst very much guides you to using a Database, and seems to prefer using DBIx::Class for this, so I need a way of guiding the Model to provide the correct objects, which are not generated directly from Catalyst helpers.

What did I do?

So in the above book, I found the section 'Implementing a FileSystem model'. This shows us how to create a Backend, which takes us out of the ordinary Model style, and the call the the Model returns this Backend object instead. We then use this Backend object to contain the logic which can be used to obtain the objects from somewhere outside of the Catalyst application, de-coupling the data models from the app, and therefore increasing flexibility and maintainability. As I said, these objects are actually being written within another application project.

This has been an interesting venture, which has enabled me to write a web application which only concentrates on the logic for the view, and leave the data handling completely to someone else. We should be production ready with the application within the week, and displaying data for the users quickly and simply.

What the betting someone asks if we can regenerate the data for all previous runs? I won't be betting against it, that's for sure.

FileSystem/DataBase - or both?

2009-08-30T09:42:00.001-07:00

I'm starting a new MooseX::Storage module called MooseX::Storage::File_or_DB.

The objective is that you can use the standard MooseX::Storage to serialize out the Moose
object as a JSON string to a file, and read it back again, but also save it out to
a Database so that it can be used in a usual database way (i.e. interrogate the db using
sql directly, so an attribute maps directly to a column).

There are a number of ORM or object db modules on CPAN (DBIx::Class, Fey, KiokuDB) but all seem a bit difficult to give the option to also obtain and save out the object to a filesystem as well, for more short term storage prior to archival to the database.

This is clearly something we need. MooseX::Storage is just the job for storing and retrieval from a filesystem, but linking both in one.

So, I'm setting to work on something that will do both. The current work in progress is on GitHub here

http://github.com/setitesuk/MooseX--Storage--File_or_DB

I would ask people to take a look and see what they think. The POD is currently where I want to end up, but the tests are working, and I think it is going in the right direction. Looking forward to more work on this to make it ready to submit to CPAN.

Am I too good

2009-08-28T04:06:00.000-07:00

For our projects that are released through the Sanger Website as Open Source (most of our code is, but we don't have a specific release policy to putting it out there) we run David A. Wheelers sloccount to get a count of the lines of code.

I have just done release-3.0 of the pluggable pipeline system, so I thought it might be fun to get the stats of this.

Total Physical Source Lines of Code (SLOC) = 3,884

That's good, I've been working on this for 7 weeks, with other projects.

However, sloccount gives you further info:

Development Effort Estimate, Person-Years (Person-Months) = 0.83 (9.98)
(Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))

About 10 months development - I'd have been shot if what I have produced had taken that long :)

Schedule Estimate, Years (Months) = 0.50 (5.99)
(Basic COCOMO model, Months = 2.5 * (person-months**0.38))

6 months scheduling.

Estimated Average Number of Developers (Effort/Schedule) = 1.66

There's only me, and I've only been working for 7 weeks on this project!

Total Estimated Cost to Develop = $ 112,301
(average salary = $56,286/year, overhead = 2.40).

I need to ask for a raise! In fact, my boss covers his ears and refuses to listen when I mention these numbers.

Obviously, there has been discussion with other people about where the project is heading, but I think the Basic COCOMO model clearly doesn't quite cut the mustard with Agile Development practices. Or maybe I'm just too good.

Still, it is fun to watch my boss run screaming, refusing to listen when I quote the estimated cost to develop. At least I think I am worth at least what I am paid :)

'Non-'Unique indexing

2009-08-28T03:54:00.000-07:00

We wanted to make a column in our MySQL database table (using InnoDB) nullable, but use the column as part of a composite key.

unique key C1,C2,C3,C4

C4 can be null

C1 C2 C3 C4

enter the following:

x y z a

goes into the table ok

enter those again - error that we break unique constraint.

This is as expected.

enter

x y z null

goes into table ok

enter those again - they also enter fine, and select * from table shows two separate row entries.

So, basically, you can have a nullable field in a composite unique index, but if the column is null, you loose the unique index checking.

I don't know about other DB's, but it is a shame that the null can't be part of the uniqueness of the index.

Musings on a Moose

2009-08-12T08:55:00.000-07:00

come to sweden, see the majestic moose (paraphrased from Monty Python and the Holy Grail).

I am starting to look at Moose as an alternative to Class::Std and other modules for OO Perl. It has very good support from the community, and looks to be fast becoming the framework of choice.

I will start off by saying I like it. The setup of my::module is very easy, and reads very cleanly. I like the declaration of variable types on the accessors, and it obvious when an attribute is needed on new, (just set 'required' flag). I especially like the ability to make an attribute ro.

However, that leads me to a small bugbear I have. I have mentioned before about encapsulation, Class::Std objects are blessed scalars, and as such can't have keys. Your attributes are set up as keys on internal hashes, and as such can only be exposed via a method.

The Moose object created is a blessed hash, and, the attributes are stored in keys. This means that your user can still override the ro attribute by just using the key. Encapsulation is broken.

Someone in my office used a good simile here, which I think he attributed to Larry Wall, which is:

"Your neighbour stays out of your garden, because it is the right thing to do, not because you have a shotgun."

That might be the case, but I'd rather enforce the use of accessors from the start than run the risk of user's using the key. However, I plan to try out MooseX::InsideOut to see if this can enforce this once I'm more tightly used to the basics.

Another thing I find very good are the additional modifiers for the accessors. We have found that making use of predicates and lazy_build really improves the code layout and speed of object creation.

I have now been building the pluggable pipeline project using Moose, and it has been very quick to build the objects and code. I would say that I think once upto speed, and once I have tested out MooseX::InsideOut, I think that development time of code should reduce by about 10% and code maintainability should go up about 25%.

So a big thumbs up from myself, and my development team also. Whilst I think at the moment there would be no plans to convert our ClearPress based apps to Moose, I think I'm tempted to try to switch my Class::Std modules, and certainly all new projects will go that way.

Big Thanks to all the people who work on Moose. Is there a book in the works? I for one would get it, I'd even contribute if you'd like.

Pluggable Pipelines

2009-07-31T03:49:00.001-07:00

My current main project is to replace the end of our analysis pipeline. It is currently all in one module, archive.pm. As with all things which are in one single file, this has been extended and changed around, and has become unmanageable.

We have also decided that we want to make it pluggable, so that we can easily add or remove parts depending on the project requirements.

For this I looked at creating a flag waver, whose job it is to take an array of function names, and launch each in turn, capturing any return values and submitting them as requirements to be fulfilled that may be needed for the next function.

The functions have no knowledge of anything except what requirements may come in, what information the object they control requires and how to capture and return further requirements for any processes further down the line.

Most importantly, the functions know nothing about any other function.

Each function loads an object, and gives it the parameters it needs. These objects handle doing any real work, which can be updating statuses, submitting jobs to LSF, manipulating files, obtaining data from databases/web services. All they have to return to the function which called them is something the next process might need to know for it's own ability to work. In our case an array of job ids from LSF submissions, to be used as job dependencies.

The structure is therefore a flagwaver, which calls the functions in a user specified order, which in turn call objects submitting jobs, returning dependencies for the next function.

What this means is that the flagwaver has very little responsibility itself. It relies on the user ensuring that any individual components will complete successfully (or at least error sensibly), and that the user has specified an order of components which will work (i.e. if 'B' depends upon an output of 'A', then they have put 'A' before 'B' in the array).

In our case, the flagwaver does have a little specific knowledge in that we have coded a few function which, if the order specified has them next to each other, they could be run at the same time, so we can parallelise as much as possible, but that is in a specific subclass of the pluggable base module, and can be overridden.

Pluggable Pipelines

View more presentations from setitesuk.

Playing Badminton through Clearpress

2009-06-26T00:07:00.000-07:00

Yesterday I launched, finally, my badminton ladder web application for our sports and social club. Initially, I write something about 3 years ago, which used flat csv files and a cgi script for each page, which dealt with generating the html and processing results, and updating the ladder, and....

All not very practical, but at least showed what I could do at the time.

Earlier this year we had a change of hardware for our webservers, and we lost the functionality, due to the time lag of all the files being kept up to date as they changed on the multiple servers. A big problem.

So I decided it was time for a rewrite, which I did using Clearpress (http://clearpress.net/), trialling out git and github at the same time (I like this so much more than sourceforge! and svn).

I have blogged about Clearpress and its MVC framework before, so I won't spend time doing so, but with a little work, the setup used 5 tables in a database to produce a reliable system.

team - stores a team name, wins, losses and gives them a unique identifer
player - stores a player name, email and gives them a unique identifier
player_team - join table for a player to a team
ladder_type - our ladder has three sub ladders, to deal with new teams and those which haven't played for a long time
ladder - links team/position/ladder_type

The whole thing can be found on github

git://github.com/setitesuk/badminton-ladder.git

It is currently set up to deploy using a SQLLite database, but in production use, we are using a mysql database, and the schema is there. You just need to modify the config.ini file to use a mysql database, which is supported through clearpress.

So, if you are after a web app badminton ladder, then take a look. It is all available as Open Source (GNU Public Licence).

Next, to create a Tennis Competition app.

lc(x) - possible bad practice?

2009-06-11T05:05:00.000-07:00

use strict; use warnings;

We all use the above, right? Well, certainly we should, and my team does;

Now this always causes a warning to occur when testing an undefined value (exception - if the test is for it to be undef).

my $arrayref = $self->method_returning_array_ref() || [];

Now, assuming that the method will always return an arrayref or undef, I will have an arrayref.

Now, it is common enough (returning stuff from an XML dom for example) that there is actually only 1 thing in the array, so I test against it

if ($arrayref->[0] eq 'yes') {
do something..
}

Now, if $arrayref->[0] is undef, this always throws an uninitialised variable warning. This normally leads me to change to

if ($arrayref->[0] && $arrayref->[0] eq 'yes') {
do something..
}

so that I don't spam up the logs with warnings.

However, we discovered today that by lowercasing the $arrayref->[0] variable, this will turn an undef into an empty string (for the conditional), therefore dispensing with any warnings.

if (lc$arrayref->[0] eq 'yes') {
do something..
}

Is this good or bad coding practice?

Reasons for it to be good
- You do not need an extra conditional just to dispense with the warning
- code less

Reasons for it to be bad
- It doesn't seem right
- The conditional is no longer testing against the pure result
- Are we getting rid of the warning for the wrong reason?
- Does it read correctly?

At this time, it is not something that the code police (perlcritic) seem to think is a bad practice, and certainly will make less code for us. It just seems like we are breaking an unwritten coding rule.

Stomping on the Rabbit, Pt2

2009-05-20T02:53:00.001-07:00

So now I have this all installed, I have to start using it. Net::Stomp (see cpan.org) is the interface I would like to use, but the setup seems very much done for ActiveMQ (as some of the headers are 'activemq.xxxxx'.

This means that by default, I only seem to be able to send and receive messages to a queue that exists only whilst someone is listening.

The example scripts that come with the RabbitMQ-Stomp distribution only show this for PERL, however, the Ruby examples show much more (persistent queues, topics, etc).

Thankfully, I found the following post

http://tinyurl.com/pdznw9

which explains the headers for RabbitMQ to produce persistent queues and operating topics. For Net::Stomp, the key is to add these headers to the key-value pairs in your connection/send hash, and happily they go through. Here are some examples:

Persistent Queues:

Receiver -

my $stomp = Net::Stomp->new({hostname=>'localhost', port=>'61613'});
$stomp->connect({login=>'guest', passcode=>'guest'}) or croak $EVAL_ERROR;

$SIG{INT} = sub {
$stomp->unsubscribe({
destination => q(/queue/father/ted),
});
exit;
};

$stomp->subscribe({
destination => q(/queue/super/ted),
q{auto-delete} => q{false}, # setting these flags will make your queue remain whilst
q{durable} => q{true}, # the subscriber(s) go away, and pick up messages afterwards
ack => q(client),
});

while (1) {
my $frame = $stomp->receive_frame;
print $frame;
print $frame->body . "\n";
$stomp->ack({frame=>$frame});
last if $frame->body eq 'QUIT';
}

$stomp->disconnect;

Sender -

my $stomp = Net::Stomp->new({hostname=>'localhost', port=>'61613'});
$stomp->connect({login=>'guest', passcode=>'guest'}) or croak $EVAL_ERROR;
$stomp->send({destination => '/queue/super/ted',
bytes_message=>1,
body=>($ARGV[0] or "test\0message")});
$stomp->disconnect;

Topics:

Topics are handled differently in RabbitMQ to ActiveMQ. In ActiveMQ, they sit under a namespace /topic/xxx/yyy and a combination of the client-id, destination, exchange and routing key all make up the subscriber to the topic

In the case of RabbitMQ, it seems a bit more generic than that, but has some differences which make it (as far as I can see) non-persistent.

Receiver -

my $stomp = Net::Stomp->new({hostname=>'localhost', port=>'61613'});
$stomp->connect({login=>'guest', passcode=>'guest'}) or croak $EVAL_ERROR;

$SIG{INT} = sub {
exit;
};

$stomp->subscribe({
destination => q{bananaman}, # needs to be a unique client-id
exchange => q{amq.topic},
routing_key => q{bananas},
});

while (1) {
my $frame = $stomp->receive_frame;
print $frame;
print $frame->body . "\n";
last if $frame->body eq 'QUIT';
}

$stomp->disconnect;

This sets up a receiver, which if you use rabbitmqctl list_queues shows a queue bananaman which will receive messages (whilst he is listening) to the topic bananas

using the following gives an anonymous looking receiver queue

$stomp->subscribe({
id => q{banaman}, # needs to be a unique client-id
destination => q{},
exchange => q{amq.topic},
routing_key => q{bananas},
});

which you can unsubscribe from explicitly

$stomp->subscribe({
id => q{banaman},
});

However, it would appear that the unsubscription occurs anyway, so unfortunately, any posts to the topic bananas would be lost to bananaman whilst he is away.

Sender for topics -

my $stomp = Net::Stomp->new({hostname=>'localhost', port=>'61613'});
$stomp->connect({login=>'guest', passcode=>'guest'}) or croak $EVAL_ERROR;
$stomp->send({destination => q{bananas},
exchange => q{amq.topic},
body=>($ARGV[0] or "test\0message")});
$stomp->disconnect;

You will notice here that the destination for this message is the routing_key, and that the exchange flag is set.

The problem here is that essentially, each receiver to a particular topic (routing_key) forms a temporary queue to which the message is added, and then sent from, but the queue is just that, temporary and goes away when the receiver does, so that receiver will never get the message if they are not present when the message is sent to the message-queue.

I am very keen to hear from anyone who has found out how to work around this.

The other main problem that we have seen due to this is MQ agnosticism doesn't exist using Net::Stomp, as essentially the problem is that it is written with ActiveMQ in mind, and just feeds headers through. This means that there is no conversion of the headers between using ActiveMQ-STOMP, RabbitMQ-STOMP or any other message-queue-STOMP that is out there. This means that there is no way to easily hot-swap message-queues without code re-write. Again, I'd be happy to hear form anyone who has worked out how to get around this.

As for now, it looks as though it is going to be ActiveMQ, as this has been installed centrally for us, and appears to manage the persistence a bit better, along with headers which are documented in Net::Stomp.

Stomping on the Rabbit

2009-05-15T07:30:00.000-07:00

we are trying to make a move to using message queues in my group to deal with pipelines and talking to other apps.

There should be some great advantages for us, and I am quite excited by this.

First thing, set up a STOMP message queue.

Now, I am currently writing a simple message_queue based on ClearPress, but this is not yet ready, so I have just downloaded and installed RabbitMQ.

RabbitMQ is written in Erlang, (which I have just set myself the challenge of learning from the Pragprog book Programming Erlang by Joe Armstrong). The first challenge was setting it up.

First: You need erlang installed. I had already done this, so wasn't a problem. Jsut make sure that erlc is in your path.

Now, I had dowloaded the latest release tarball, and expanded this, setting up in my sandbox area. However, this threw my a serious curveball with trying to install STOMP.

Thankfully, Google is my friend, and someone had the same problem, since the STOMP needs installing against the correct version number, so I hereby give the definitive installation guide to getting RabbitMQ up and running in a sandbox on MacOSX.

(I accept no responsibility for this not working on your machine!)

1) Install Mercurial (yet another distributed version control system, but the one which RabbitMQ is on).

2) (Thanks to everyone on this page http://tinyurl.com/pk7xfd)
hg clone http://hg.rabbitmq.com/rabbitmq-server
hg clone http://hg.rabbitmq.com/rabbitmq-codegen
hg clone http://hg.rabbitmq.com/rabbitmq-stomp
(cd rabbitmq-server; hg up rabbitmq_v1_5_4)
(cd rabbitmq-codegen; hg up rabbitmq_v1_5_4)
(cd rabbitmq-stomp; hg up rabbitmq_v1_5_3)

3) (At this point, you need to check your version of python and simplejson, which needs installing)

4) In the various MakeFiles, alter source roots to point to where you want the various db and log files to go, where your rabbit source root is, etc..

Eg.

I set up in my sandbox a folder 'rabbitmq', in which I put a logs dir, mnesia (db files) dir and rabbit-mnesia dir (into which I put the rabbitmq.conf file)

In 'rabbitmq-server/Makefile'

RABBITMQ_NODENAME=rabbit
RABBITMQ_SERVER_START_ARGS=/my/sandbox/rabbitmq/rabbitmq.conf
RABBITMQ_MNESIA_DIR=/my/sandbox/rabbitmq/$(RABBITMQ_NODENAME)-mnesia
RABBITMQ_LOG_BASE=/my/sandbox/rabbitmq/logs

4) make -C rabbitmq-server
make -C rabbitmq-stomp run

Hey presto, you now have a rabbitmq-stomp server up and running.

Now you have done this, you can try either the perl or ruby tests.

Hope this guide is of use to someone.

Cheers

Andy

Splitting a Project

2009-03-27T05:45:00.000-07:00

As part of the Group I am in, we have had a project sanger-pipeline. Now, early on in the life of the group, it was quite small, and was essentially just a few wrappers around the Illumina Analysis pipeline, so that it hooked into our tracking system, and scripts to manage the movement of images off the individual Genome Analysers to our Farm.

However, as with most things, it got bigger...

and bigger...

and bigger until it was managing not only the wrappers (which were being maintained by someone who was in another group), and the movement of images, but processing other files, loading compressed versions of the images into other databases, deciding when it had everything it expected, what and when exactly to delete outdated files...and managing an apache webservice!

So, over the last 2 weeks, I have been splitting the project down. This was not a trivial thing to do, as many modules where used in multiple places, but split it I did to the following:

sanger-pipeline - this now only has wrapper scripts and modules relating to hooking the Illumina Analysis pipeline in to our system.

instrument_handling - this deals with scripts and modules relating to the smooth running and mirroring of data of the GA's, plus our controlcentre script, whose primary use is to run these as a special user (although it does do some other things).

data_handling - all these scripts and modules are responsible for managing data on the farm (once it has mirrored) including doing checks to ensure that data that needs long term storage are where they should be

sflogin_web_apps - the management of the apache server and the scripts that we run on the farm (instead of the usual webblades) to have access to files and directories located there.

At this time, the 4 projects are still linked, in so much as the top-level package namespace for the modules is srpipe:: (which means we need to be careful of ensuring that further down the line we don't accidently create two things which will conflict) and that many modules use modules from some of the other projects (code as DRY as possible).

However, these slight drawbacks are minor compared to the ability now to apply patches and new code to a project, without accidently deploying a broken development of something else (which was the tricky thing when the wrapper scripts were managed by someone separate to the group responsible for the (for example) mirroring of data).

So hopefully, now we have a much more stable codebase, and will be able to keep things running much more smoothly (or at least, develop much more fluidly).

The test coverage needs improvement, but this is something that we can further work on now. A fair amount of code had no tests at all past one which checked it compiled, so a revamp of the test suite was certainly in order, and with this, we can code directly in to ensure that the functionality doesn't change where code is reused in different projects.

Exciting times or screams ahead, who knows, but at least with hopefully more manageable project sizes, we should be able to aim for exciting...

Running Alone

2008-12-18T08:58:00.000-08:00

John in my team at work has just pointed me at a great little CPAN module for ensuring that a script can only have one copy running at a time.

In our team, we have a lot of potentially slow running processes, which are often run as cron jobs. Now, ordinarily, you would pick cron start times
that ensure that you have finished any previous run of the script. But it is quite common for us to want to run as soon as possible, but not if the
previous run hasn't finished.

One method previously suggested was to touch a file in /tmp and then delete it when finished, wrapping the cron caller in a command not to run if
the touched file is present. However, what happens if /tmp is cleared out (which has happened to me!)

So, John found Sys::RunAlone. If the format of your script is to run a main method, and you put __END__ or __DATA__ at the bottom of the script,
then this can lock the text, and then knows no to exit if it finds a lock on the section. This means you could have your cronjob running every 30 minutes
and not worry if the script takes 5 minutes in a cycle, or 45 minutes.

A definite benefit.

A big thanks to Elizabeth Mattijsen for writing this very useful module.

(A quick example of this running is below. If you try to run the same script in another terminal before the first has finished it's sleep, you will get
the correct error).

#!/usr/bin/perl -wT
use strict;
use warnings;
use Sys::RunAlone;

main();
0;

sub main {
warn 'Hello ...';
go_to_sleep(10);
warn 'world!';
}

sub go_to_sleep {
my $sleepytime = shift;
sleep $sleepytime;
}

__END__

Blog entry

Data Formats

2008-12-18T08:17:00.001-08:00

Check out this SlideShare Presentation:

Data Formats

View SlideShare presentation or Upload your own. (tags: csv tsv)

This presentation was given by me to foomongers to start off a bit of discussion on different data transfer formats.

Disk is cheap?

2008-10-18T00:35:00.000-07:00

I work at the Wellcome Trust Sanger Institute, where we do a lot of processing of raw sequence data, particularly from the Next Gen sequencing machines. Over any month period we can have 320Tb of data sitting around on on our sequencing farm. The run data then gets tarballed up and moved off into repositories.

Currently, Gouying and I have been working on a MySQL database to store the QC data from the runs, so that we can get rid of the Tarballs (each themselves around 60Gb), and allow easier access to the data. Note, this is not the DNA sequence, that is being stored elsewhere. The expected annual insert into this is likely to be around 50Tb at current Illumina machine levels (and we are planning to get more!).

Roger and others have been working on storing the images for the runs longer term. This has meant moving them into an Oracle database, which means that we should have them for perhaps a year, rather than around 3 weeks. This is around 100 (tiles) * 36 (cycles) * 2 (run pair) * 8 (lanes on a chip) images per run tiff images.

Speaking with Ed at work, we discussed what was mentioned at our last Town meeting. Ed used to work for IBM, and he talked me through a bit of how processors have developed and why they are now likely to be at their limit of capability, hence going to multiple core machines. He therefore raised a good question at the town meeting - is it cheaper to store the data from the runs long term, or just rerun the sequencing?

At the moment, it costs 2% of the cost of a run to keep the data on disk, so that answers the question. Or does it?

Torvus Linalds has been quoted as saying that disk is cheap, and yes, upgrading your desk/laptop to 4x the memory is pretty cheap. However, you still need to ensure you have space for it. And there is one thing that is certainly the case, space isn't cheap. All those disks have a physical footprint. (and we could very soon run out of space in the data centre)

They also have to be managed, and people aren't cheap. We have a very good systems team, that are on 24hr callout, but that costs money.

So, it is very much a trade off. The cost of resequencing is very expensive at the moment, and storage of data is cheap, but if the next gen sequencing costs come down, then it may become very much the case just to store a plate of DNA and resequence it every time you want data from it, rather than long term storage of 4Tb+ of data.

This may be a long way off, but if the $1000 genome becomes a reality, then I think that it may change. Is this a good thing? We shall see.

Ajax - Web 2.0 Primer course.

2008-10-08T00:24:00.000-07:00

So last week I went on a four day course to find out more about using Ajax to make web2.0 sites. The course was actually 2x4 day courses attempted to be fitted in to 1x4 day course.

For me, the first course was pretty simple, it was supposedly the prerequisite course fro the Ajax primer. We covered the Javascript basics, but in order to move on, the course instructor assumed our knowledge of HTML and css. For me this was OK, but some people didn't seem too up with the whole css selector specs instead of using inline markup. However, this did 'get' us through the course in a day.

We then jumped to the Ajax course. As I have blogged before, earlier this year I bought Pragmatic Ajax: A web2.0 primer (www.pragprog.com). Most of the second course was actually summed up in this book (although I did learn more about some Best Practices). We went through from making an xhr object that is cross-browser compatible, to starting our own library, and then we did some examples using jQuery, Prototype and Scriptaculous.

The course was from Learning Tree, and the instructor was very knowledgable, showing us some other books that we might want to read, primarily Douglas Crockford - Javascript, The best bits (www.oreilly.com) and John Resig - Pro Javascript Techniques (apress).

The thing that interested me the most was probably something that had the least time spent on it. For my toy app, YADB, I wanted to be able to drag and drop cards from either a card list or inventory list into a deck container. We were shown how to do this using Scriptaculous (quite fortunate, considering YADB is Ruby on Rails).

Work wise, it was good to be shown exactly how to look at blackboxing functions, and how to pass JSON and objects around.

Two disappointing features of the course - example code snippets did not follow what 'Best Practices Guy' was saying, sometimes even in the same slide, and some of the exercises on the computer didn't match the code in the book.

Also, no fault of the instructor, but somewhere along the line, they hadn't explained to Learning Tree that most people on my site primarily use Perl,
so some of us were a bit stuck when it came to modifying some of the server side code, as we had little/no experience of the 3 they had chosen (jsp, php and .net)

Overall, a good course, that could have been better, but worth going on.

I must contact Learning Tree though about doing the 2 module tests for them though.

An aside to this, why is everything PHP. I had a couple of email links come though about web app jobs coming up, and everyone wants PHP. I could be a bit biased, but certainly a lot of people I know think that PHP is probably one of the worst technologies to come along for a while. Certainly one thing that has been blogged on Perl Buzz is that you can't bug report anything from an older version than the current release, which seems a bit pointless to me. However, I am not going to go into it now, I think I just clearly need to learn it. Better hit the bookshelves!

#barcamb

2008-08-03T00:08:00.000-07:00

So on Friday I attended the second barcamb, at the Wellcome Trust Genome Campus. For those not in the know, a barcamp is an unconference. People turn up with stuff to talk about, and a plan for the day is organised over coffee and biscuits at the start of the day.

It was very good interesting day. Many of the faces we saw last year came again, and gave either updates, or new talks, and we had soem new people as well, including a chap who was using the internet to help the Neighbourhood Watch in his village.

Simon Ford brought MBED with him again, showing some more of the exciting stuff he had been doing with it (including something at a 24hr hackathon, which uses a social idea to move packages to their destination).

Unfortunately, I can't remember many peoples names, but is was great to talk with so many of you, and I was rather surprised that my quick unplanned 10minuter on 'It's Too Much Information for ME!' seemed to generate a lot of little discussions with me. It's nice to see that either other people have had the same problem, or that people realise that you end up needing to program for other peoples lack of foresight/rushing development code into production. My talk followed quite nicely from Nava Whiteford's talk on the Swift Analysis pipeline, and Matthew Astleys ad-hoc talk/discussion on Panic Driven Development.

I have some photo's which I have uploaded to Facebook. Please feel free to have a look and tag yourself (or anyone else you know) in any of them.

http://www.facebook.com/photo.php?pid=1183440&l=5b93d&id=665450745

Cheers

Andy

Perl::Critic - new release

2008-07-18T04:17:00.000-07:00

So a new release of Perl::Critic was released, and all I want to say is what a faff.

Some key new features:

1) You must check the return value of all eval statement - don't rely on $EVAL_ERROR/$@

Now, this is a good thing(tm) but it does have some pitfalls, such as where you might be evalling a transactional commit.

eval {
$transaction_state and $dbh->commit();
} or do {
...some 'croak' code
};

The problem here is that the eval isn't a croak if $transaction_state is 0 (as it could be inside a much larger transaction), but the return code would be 0, therefore firing off your 'croak' code.

So, to get round this, you need to return a true value after the statement;

eval {
$transaction_state and $dbh->commit();
1;
} or do {
...some 'croak' code
};

A bit off a faff, but it is better code for it.

2) Declaration of all numbers other than 0,1 and 2

So, as it could be difficult to understand what a number represents when reading the code, you now need to use Readonly to declare any numbers with a named variable at the top of your code.

So now instead of

$percentage = $x * 100/$y;

You need

use Readonly;
Readonly our $PERCENTAGE_MAKER => 100;
...
$percentage = $x * $PERCENTAGE_MAKER/$y;
...

This 'might' make sense to odd numbers floating around, but it also applies to indices on arrays. So if you want the 5th element on an array intstead of requesting

$wanted = $array[4];

You now need

use Readonly;
Readonly our $FIFTH_ARRAY_ELEMENT => 4;
...
$wanted = $array[$FIFTH_ARRAY_ELEMENT];
...

Now, I admit it is often bad to pick out many individual elements from an array by specific number, but it will seriously clutter code where this may be necessary. I admit, for a couple of lines in my code, I have now used ##no critic at the end.

I also wonder why specifically 0,1 and 2 get let off. If the problem is that you don't know if 60 means

i) minutes in an hour
ii) seconds in a minute
iii) degrees in the angle of an equilateral triangle

then 0 could mean off, false, nothing, 0
1 could mean on, true, positive, 1
2 could mean wheels on a bicycle, eyes on a face, hands, 2

Just some examples where I imagine it is because they are the most heavily used for certain features, it would almost impossible to change them (especially 1 as the return value of a perl module)

3) Declaring a variable, which you never (appear to) use

This is annoying if, like me, you use Class::Std to create inside out objects, as you need to declare a hash for your attributes, but this has is never referred to again in the code.

Now, whilst I understand that if you don't use a variable, don't declare it, in this case you are using it, just via the Class::Std methods of the accessors created. However, much like the declaration of numbers, it isn't looking at the context in which you use the declared variable. Again, in this case I have had to wrap these with a ## no critic {} ## use critic in order to not have it fail.

--

So as I said at the beginning, all I want to say is what a faff and the reason is as follows:

We are using Perl::Critic extensively here in New Pipeline Development, and it does force our hand to a Good Coding Practice, but I can't help but wonder here if a couple of these new standards are just a little too overzealous, and causing some things to be overcritised (i.e. a faff).

As with many things, it will take time to get used to programming in advance of critic'ing the code, but I think some of these new features need a little tweaking.

Class::Std or Blessed Hash

2008-06-20T05:39:00.001-07:00

Objects, Objects, Objects

Everything is objects these days, well, certainly in the world of agile, well structured, extensible, easy to maintain BioInformatics software.

Even Perl6 is aiming to be OO. Probably because of the fact that so many of the modules on CPAN at least expose an OO layer, if not are only OO.

When I started programming PERL, I was writing straight forward top to bottom scripts.

I then moved on to using and producing code in modules, but just exporting the subroutines into the script that used it, for simple code reuse.

Last summer, I got finally taught with hands on development of exactly how OO works and is used. I got a bit confused, but at least I had none of the confusion of

$him = Person->new({args});
$her = $him->new({args});

Which implies a relationship which 'is not there'.

Last summer I discovered Class:Std, which I think is probably my favourite CPAN module of all time. Why?

Well this is the thing. PERL is not an OO language, and it isn't slower because of it. I also learnt Ruby on Rails (as I mentioned in a previous post) and Ruby is slower because everything is an object. Something that clearly sets the two languaged apart.

Now, that isn't the thing that bugs me about OO. In fact, I have learned to embrace PERL OO, and enjoy programming in it. But what does bug me, is that the vast majority of PERL OO breaks encapsulation because all most objects are are HASHes. You have a new constructor, which blesses the package name around a HASH reference. So, when all is said and done, whilst good packages have constructors written to expose the stored data within the object via method calls, you can just access a lot of it via a key.

$him->eye_colour() is equivalent to $him->{eye_colour}

and this encourages lazy programming, because the other advantage is that you can just say 'I need to store some data, what should I do with it, as Person doesn't have an address accessor'

Now, presumably Person does have something that links it to Address. Perhaps Address and Person both have an id_person accessor. But you can cheat. If you want to grab address now, and cache it for later, just do

$person->{address} = $address->house_and_street();

The you can drop the address object, and person now knows exactly where they live.

However, this is dangerous, because

1) Have you deleted something specifically stored in key address
2) What if they move whilst person object is still in memory. You have two places to correct the data.

Why, I hear you cry - I won't do that with my program. No, but someone else will (or you will forget).

Solution use Class::Std;

Class:Std enforces encapsulation. You still get a blessed package, but this time it is a SCALAR, which can't have keys.

You then in each package declare what accessors you want the object to have, and as such enforce people to only use those accessors. You don't have to worry about AUTOLOAD in the history of used modules, as Class::Std handles creating you accessors. You don't even need a new constructor, although you can add a BUILD method which will operate at construction.

So in my example

package Person;

use Class::Std;

{
my %eye_colour_of :ATTR( 'init_arg' => eye_colour, :get<eye_colour>, :set<eye_colour>);
}
1;

my $him = Person->new({eye_colour => 'blue'});

Job done. Less code for initial construction than blessing via new, and you cannot be tempted to throw the address onto the person when it is being used, as

print $him = Person=SCALAR(0x9f2c68)

So, unless you specify in the code (documented and tested, of course) that you want an accessor which allows this object to store the address, it can't be done, and your later code is more robust for it.

Now, where am I going with all of this?

Well, I use Clearpress to form a base for my PERL web apps in my current role. It is a good solid platform which I have mentioned before, and I am very happy to work within it. However, I am writing an API to use the services it provides. Clearpress doesn't use Class::Std. My API does. This is no problem as they talk via LWP::UserAgent requests, but it is quite confusing as the live in the same project in subversion. And my big thing is that I am programming both at the same time. This is bad news, as I have been trying to use features of one type of Object with the other. It hasn't really made a significant difference, as the package name reminds me which I should be using, but is is wierd getting the error when you try to cheat, and use a key to cache some info in the Class::Std object, as it is a scalar.

So, from this, I am going to finish the project in the way I have started it, but I think from now on there is one golden rule:

Use only one type of object, and just ensure you enforce encapsulation by the way you program - don't get lazy.

Now, to convince my boss to refactor Clearpress into Class::Std...

To scroll within or out

2008-05-07T17:52:00.000-07:00

So an interesting thing arose out of my work recently, both NPG and a personal project, which is how to scroll tables.

In NPG, we have many tables of data, from run information, to search results, to instrument information. These are generated from templates and render to the browser.

In V:YaDB, I have the same issue, as well in excess of 2000 cards is quite a bit to scroll through, but again it is templated.

Following good practice and webstandards, we are all using

<table>
<thead>
.
.
</thead>
<tfoot>
.
.
</tfoot>
<tbody>
.
.
.
.
</tbody>
</table>

or at least we should be. Most modern browsers insert this anyway, but putting it in gives you some additional css tags, and ensures you and future developers of your project know what is going where (and for anyone not familiar with this, <tfoot> should be before <tbody>, although it is optional to have a footer to your table).

Anyway, with this in mind, it should be easy to just put the following css in to make the body part of your table scroll, therefore leaving the head fixed, to keep column header visible

tbody {height:300px;overflow:auto;overflow-x:hidden;}

However, this only works in the firefox browser. What is going on there? But it is true. I spent some time this weekend on IE7 and Safari and found this to be true.

So, back to square one. Searching google found a lot of real hacks, from serious amounts of Javascript, to running different css files dependent on browser (including separate versions for IE6 and IE7). Madness.

One idea I came across that I liked the most though was the idea of two table renders. Whilst this means it is only really suitable for quick to render tables, this is something that is possible.

Now, there are two options here.

1) Using declared fixed width columns, produce one table which only has the thead part of your table. Then immediately beneath it produce the table with the data in. This one could be a bigger table because it only has to render data once, but you have no flexibility should you need to add a new column of data.

2) In a fixed height div, render two copies of the table using absolute positioning over each other. Then wrap each in it's own div labelled with an id. Using using z-align:1; for the one you want to scroll, and z-align:2; for the head, and fix the height of the head div to that only the head row is shown, and set overflow:hidden; Set the height of the scroll table to the height of the outer div, and set overflow:auto; overflow-x:hidden;

(with 2, obviously, you could also fix the width, and for both show the overflow-x)

Also with both, you need to declare a spacer column, which will then ensure room for your scrollbar.

I personally prefer 2, which I managed to create some css to produce a nice effect with the fact that with 2 tables, I was able to manipulate the header style without needing to worry about if it affected the rest of the data, and also revealed the bottom border only to give a ruled effect.

The downside to 2 is that anyone with css turned off will end up viewing two copies of your table, but hey, no-one should be turning off css or javascript in their browsers, and if you know someone who does, 'send the boyz round to ave a wurd'.

I think that I am going to expand scrumptious to have javascript and css effects, and this will be the first css effect in it. I'll let you know when the sourceforge svn trunk is updated.

However, start lobbying your local MP today to get the simplest option put into your favourite browser, or if your fave is already firefox/iceweasel, then at least IE and Safari.

Please note: I have nothing against Opera, Camino or any other browsers out there, I just don't use them on a regular basis.

History Meme

2008-04-30T01:24:00.000-07:00

So I got tagged to do this by my boss Roger, and a lot of my work colleagues are doing it. So here is the result from my macbook:

history|awk '{print $2}'|sort|uniq -c|sort -rn|head

167 prove
62 cd
55 svn
45 cover
42 ls
32 rake
28 mate
11 make
8 ./bin/apachectl
7 sudo

So I am testing quite a lot on my machine (prove and cover) (always trying for test driven development is the reason for this). cd is obvious. svn is vitally important.

I am surprised rake turns up more than make though

More service soon.

Andy

Javascript for all

2008-04-07T04:51:00.000-07:00

So on Friday, as a gentle way of trying to get back into work mode after the Rails course had finished, I started
by trying to refactor out a lot of javascript from the templates.

I have just bought 'Pragmatic Ajax (A Web 2.0 Primer)' from The Pragmatic Programmers. It is a very interesting read, and
inspired me to 'get the code out of the view'

It's true to say, that we have been quite lax in simply putting <script> tags in with fairly specialised javascript
functions, which don't really need any variables passed to them (as the function ends up with the paths and div ids hard=coded).

Well, I managed to refactor out most of the functions that we had written, and with a few additions to variables being passed to them, managed to reduce the number of some function (or make them more genericised for future reference). I even discovered a slight problem with my scrumptious.js which I need to tweak and document.

The great thing is that we have now reduced the code in the views. This makes the views easier to read and keep upto date.

There are still a few functions which I should be able to refactor, but I need to find out a couple of extra things first.

I've only got through the first 3 chapters of 'Pragmatic Ajax' so far, but it has explained a bit that so far I hadn't known from just my experience learning some Ajax through RoRails. Chapter 1 explains about what Ajax is, Chapter 2 shows you how to develop Ajaxian Maps (a google maps clone). Then it has started to go into the Nitty Gritty details of Ajax and Client-side Javascript.

However, so far the javascript examples have all been written in the html head, rather than in a separate .js file. I imagine (hope) that this will change in a best practice suggestion. I'm also hoping it will show a bit on testing javascript, which so far is something that I haven't done.

My experience of programming books has led me to find the Pragmatic Programmers books are a great way of finding the information in an easy to read style. So far, Pragmatic Ajax is a good book and hasn't let me down in it's style and (most importantly) content.

Advancing with Rails course - Day 4 pt 3

2008-04-03T06:59:00.001-07:00

So, the final bit of the course has been looking at integration testing

objective is to go through the processes to a conclusion, i.e.

login >>
attempt to bid on an auction you started >>
fail

login >>
bid on auction as highest bidder >>
fail

login >>
bid on auction >>
pass

and so on

can cross controllers which is why next level up from functional tests. May be more than one/two asserts as
they are linked and this is useful to ensure it doesn't bother trying tests it can't even get to

very good place to often refactor heavily

integration testing routes

def test_show_route
assert_recognizes({:controller => :auctions,
:action => :show,
:id => "1"}, auction_path(1)) <==named routes
end
def test_generate_route
assert_generates("/auctions/3", :controller => :auctions,
:action => :show,
:id => "3")
end

also can assert_routes

you could get back responses to look at when rjs is rendered

irb>> app.get("/auctions/destroy/1")
>> 200
irb>> app.response.body
(output of rjs file)

rcov tool

coverage of code with tests

ZenTest

runs in another terminal window, and everytime you make and save a change to a file, it works out the tests which are affected and reruns those tests.

Advancing with Rails course - Day 4 pt 2

2008-04-03T03:46:00.001-07:00

Plugins

How to install and generate a plugin

guest095:yadb ajb$ script/plugin install http://repo.pragprog.com/svn/Public/plugins/annotate_models
+ ./ChangeLog
+ ./README
+ ./lib/annotate_models.rb
+ ./tasks/annotate_models_tasks.rake
guest095:yadb ajb$ ls vendor/plugins/
annotate_models/ nested_has_many_through/
guest095:yadb ajb$ ls vendor/plugins/annotate_models/
ChangeLog README lib/ tasks/
guest095:yadb ajb$ ls vendor/plugins/annotate_models/tasks/annotate_models_tasks.rake
vendor/plugins/annotate_models/tasks/annotate_models_tasks.rake
guest095:yadb ajb$

adds schema info to the top of your model files

guest095:yadb ajb$ rake annotate_models
(in /Users/ajb/dev/vtes/yadb)
Annotating Card
Annotating CardDiscipline
Annotating Clan
Unable to annotate Clan: Could not find table 'clans'
Annotating CostType
Annotating Deck
Annotating Discipline
Annotating Minion

card.rb:

# == Schema Information
# Schema version: 6
#
# Table name: cards
#
# id :integer not null, primary key
# name :string(255)
# text :string(255)
# requirements :string(255)
# cost :integer
# cost_type_id :integer
# minion_id :integer
# deck_id :integer
# discipline_id :integer
#

class Card < ActiveRecord::Base
belongs_to :deck
belongs_to :cost_type
belongs_to :minion
has_many :card_disciplines
has_many :disciplines, :through => :card_disciplines
belongs_to :vampire, :class_name => "Minion", :conditions => "minion.name = 'vampire'"
belongs_to :werewolf, :class_name => "Minion", :conditions => "minion.name = 'werewolf'"

validates_presence_of :name
validates_uniqueness_of :name
validates_presence_of :deck_id

end

writing a plugin

guest095:yadb ajb$ script/generate plugin nice_error_fields
create vendor/plugins/nice_error_fields/lib
create vendor/plugins/nice_error_fields/tasks
create vendor/plugins/nice_error_fields/test
create vendor/plugins/nice_error_fields/README
create vendor/plugins/nice_error_fields/MIT-LICENSE
create vendor/plugins/nice_error_fields/Rakefile
create vendor/plugins/nice_error_fields/init.rb
create vendor/plugins/nice_error_fields/install.rb
create vendor/plugins/nice_error_fields/uninstall.rb
create vendor/plugins/nice_error_fields/lib/nice_error_fields.rb
create vendor/plugins/nice_error_fields/tasks/nice_error_fields_tasks.rake
create vendor/plugins/nice_error_fields/test/nice_error_fields_test.rb

guest095:yadb ajb$ cd vendor/plugins/nice_error_fields/
guest095:nice_error_fields ajb$ ls
MIT-LICENSE Rakefile install.rb tasks uninstall.rb
README init.rb lib test

install.rb needs the require 'nice_error_fields.rb'
and the lib directory is added to the path

the init.rb files are all read (so no need to create an initializer) and all the lib paths are added

Advancing with Rails course - Day 4 pt 1

2008-04-03T03:10:00.000-07:00

Forms first off, a bit of idioms created based on RESTful:

forms if you where to put in def new to render edit, when rendered it knows to post to create rather than edit,
and know is calling edit to put to update

AciveRecord can query objects via the new_record? method (@thing.new_record?), which means that it can determine whether it is a new object and so RESTfully can determine whether to create or update

David doesn't like this as it has the feeling that it is not very elegant. It is over economising. Use a partial if there is >serious overlap, and then have different templates, which can have extra bits.

(This was originally a plugin simply_helpful, now core and only works with REST)

Subclass

subdirectories

You can

script/generate controller admin/things

in routes

map.namespace :admin do |n|
n.resources :things
end

this gives a set of restful routes like new_admin_thing
with a
parallel tree of views app/views/admin/things/

Then onto some more Ruby

Procs and Callbacks

Ruby: proc (function) objects

anonymous functions which are themselves objects (and can bepassed around, etc)

Proc can remember a local variable within itself, rather than it going out of scope and being garbage collected

guest095:yadb ajb$ irb
>> Proc.new {}
=> #<Proc:0x00000000@(irb):1>
>> lambda {}
=> #<Proc:0x00000000@(irb):2>
>> proc {}
=> #<Proc:0x00000000@(irb):3>

lambda and proc are synonyms, but different from Proc.new

code blocks can be captured within a def method

p = lambda {|x| puts x * 10}
array = [1,2,3,4,5,]
array.each(&p)

def convert(n, &block) <-- special & argument syntax captures code block as a Proc object
end

puts convert (10) { |x| x * 30 }

alternative

def convert(n)
if block_given?
yield(n)
else
n * 2
end
end

good unless you need to objectify the method for some reason

it is a closure on the variables that exist around where it is created
>> y= 1; [1,2,3].each {|x| puts x * 10; puts y; y += 1 }
10
1
20
2
30
3
=> [1, 2, 3]
>> puts y
4
=> nil

but leaves alone ones created after its creation

>> class Counter
>> def self.create(n=0, inc=1)
>> return Proc.new{ n += inc; n - inc }
>> end
>> end
=> nil
>> c = Counter.create
=> #<Proc:0x0059f330@(irb):3>
>> puts c.call
0
=> nil
>> puts c.call
1
=> nil
>> n = 222
=> 222
>> c = Counter.create
=> #<Proc:0x0059f330@(irb):3>
>> puts c.call
0
=> nil
>> puts c.call
1
=> nil
>> c = Counter.create(5,5)
=> #<Proc:0x0059f330@(irb):3>
>> puts c.call
5
=> nil
>> puts c.call
10
=> nil
>> puts n
222
=> nil

you can write code blocks with {} or do - end, but they are not interchangeable

>> puts [1,2,3].map {|x| x * 10 }
10
20
30
=> nil
>> puts [1,2,3].map do |x| x * 10 end
1
2
3
=> nil

you can write a method to warn about the presence of a block (or not), but not every ruby method will do this

>> def m; raise "A block!" if block_given?; end
=> nil
>> m
=> nil
>> m {}
RuntimeError: A block!
from (irb):21:in `m'
from (irb):23

>> m do |x| x*10 end
RuntimeError: A block!
from (irb):27:in `m'
from (irb):28

>> def m; raise "A block!" if !block_given?; end
=> nil
>> m {}
=> nil
>> m
RuntimeError: A block!
from (irb):24:in `m'
from (irb):26

Built in callbacks
method_missing
Modules: - included
Classes: - inherited

>> module M
>> def talk; puts "Hi!"; end
>> end
=> nil
>> M.new
NoMethodError: undefined method `new' for M:Module
from (irb):4

you can't instantiate a module, but they are good for mixins

you can include Classes in Modules

>> module Violin
>> class String; end
>> end
=> nil

modules can give their methods to a class, via include

?> class Person
>> include M
>> end
=> Person
>> andy = Person.new
=> #<Person:0x59b71c>
>> andy.talk
Hi!
=> nil

Classes - inherited

>> class Furniture
>> def self.inherited(c)
>> puts "#{self} has been inherited by class #{c}"
>> end
>> end
=> nil
>>
?> class Chair < Furniture
>> end
Furniture has been inherited by class Chair
=> nil

extend

?> module N
>> def walk; puts "I am walking!"; end
>> end
=> nil
>> Person.ancestors
=> [Person, M, Object, Kernel]
>> andy.walk
NoMethodError: undefined method `walk' for #<Person:0x59b71c>
from (irb):29
>> andy.extend(N)
=> #<Person:0x59b71c>
>> andy.walk
I am walking!
=> nil

use extend via a module if you want to replace core methods as this is a low impact way

>> module M;def shout;puts "HI!!!!";end;end
=> nil
>> class C;end
=> nil
>> C.extend(M)
=> C
>> C.shout
HI!!!!
=> nil

Classes are objects. It extends the singleton methods on the object

equivalent to

class << C; include M; end

extend gives class methods, include are instance methods, all though David wasn't really sure what that ultimately would mean

Advancing with Rails course - Day 3 - pt3

2008-04-02T09:23:00.001-07:00

Final blog for day 3 (I'll probably end up summarising a lot of this is a presentation to foomongers, so I'll post the slides if I do. However, for now:

We started to look at REST.

Just a few notes- not in any structure. A good resource to read is Roy T. Fielding's dissertation (http://www.ics.uci.edu/~fielding/)

Representational State Transfer

request sequence (slide 198)

by default

link_to => GET
form_for => POST

link_to with movie_path => GET
link_to using edit_movie_path => GET

as browsers don't issue PUT and DELETE requests, rails cheats with:

to get PUT:
form_for with :url => movie_path(@movie) and :html => (:method => "put")
you get => method="post", intput type="hidden" name="_method" value="put"

to get DELETE
link_to wih item_path(@item), :method => "delete"
you get => DELETE (but wrapped in a form so that spiders don't follow it)

REST and CRUD have no actual real connection, but when brought into rails, then they meet up,
because it has been orchestrated

And then finally during the day onto some Ajax stuff

Ajax:

Ajax request happens overall with in the cycle of a request

link_to_remote "Click me", :url => {....}
-> goes to server, does stuff C/A -> sends back to client

typically, not a whole cycle, as only sends back a snippet/fragment which you want to drop into your document

link_to_remote "Click me", :url => {:action => click_me}, :update => "mydiv"

so drops it into a div entitled mydiv

def click_me
render :partial => "clicked"
end

renders the partial and drops it in

link_to_remote "Click me", :url => {:action => click_me}, :div => "mydiv"

as no update

def click_me
@div = params[:div]
@del = Auction.delete(params[:id])
end

hands off to click_me.rjs

if @del.zero?
page.alert("Destroy operation failed")
else
page.visual_effect :shrink, @div, :duration =>1
page.delay(2) do
page.alert("That auction is history!")
end
end

page sends back the javascript to do what you want

form_remote_for :item, :url => {:action => :update}, :update => "mydiv"

flash is normally waiting for the next request cycle

flash.now[:notice] will have it come through to the ajx response there and then, rather than wait for the page refresh

raise request.xhr? method which means is this an ajax request, which means that you can fork on the request.

name.html.erb will take preference over name.rjs, so you either shouldn't have them with the same name,
or render rjs directly in the controller

render :update do |page|
page.alert("Hello")
end

which would bypass that issue

you can put rjs directly in the view to get javascript directly in the page

xhr requests are post by default, so if you are requesting a view only allowed by get, put or delete, then you need to explicity state this in the link_to_remote tag

Advancing with Rails course - Day 3 - pt2

2008-04-02T04:05:00.000-07:00

Rails Routing

routing system does 2 things

1) recognises and inteprets URLs
2) generating URLs and path strings in your views

recognition
- goals to determine controller and action
store other values in params hash

generation
- arguments to link_to, form_for, etc
- for redirection in controllers
anywhere you need a path or URL in your actual code

top-level route

map.root :controller => "cards"

link_to "top", root_path

map.connect ':controller/:action/:id'

shows what you want your URL to look like

routing system only makes sense to itself -> only the routing system can know what the URL means if it generated it.

routing system doesn't know how to resolve for another application

model doesn't know the controller which is manipulating it.

to inject css into a link_to

<%= link_to "log in", {:controller => 'this', :action => 'that'}, :class => "blah" %>

hard coding

map.connect 'help',
:controller => "leagues",
;action => "assist"

this will resolve http://www.web.site.com/help to the specific path leagues/assist
and will only resolve /help as that

map.connect 'help/:controller', :action => "assist"

resolves controller in http://www.web.site.com/help/wibble to the path wibble/assist

extra wildcards can also be added and matched positionally, which are then stored in params
:id is special case though, which allows it to be nil, whereas anything else needs to specified,
or will find no route (unless you use globbing)

you can constrain the wildcards with pattern matching, etc and have many of these, but orderis important
(higher up the routes.rb file, will hit first)

You may need to catch a routing error.

current and default values

missing components get defaults

:controller and :action from current ones
:id, :topic, etc from params hash

default gets turned off after the first one you specify

named routes:

map.<name> "name",
:controller => :x,
:action => :y

then allows
redirect_to name_url
link_to "text" name_path

map.vampires 'vampire/:name/:capacity',
:controller => :cards,
:action => :vampires

<%= link_to "#{vampire} with capacity #{capacity}",
vampires_path :name => vampire, :capacity => capacity %>

<%= link_to "#{vampire} with capacity #{capacity}",
vampires_path (vampire, capacity) %>
(walks through the list into the url space)

Named routes and CRUD

these can be expressive and encourage good action names

cluster thinking around the operation (borrow book is create loan, renew is update, return is destroy/delete, view all loans is read)

Advancing with Rails course - Day 3 - pt1

2008-04-02T04:01:00.000-07:00

protect_from_forgery -> ensures that posts are only acted on if from your application

session hash is now stored as a cookie on the client side
typical to put in a user_id

before_filter s : anything is true unless false and will stop processing if it gets false,
so explicitly return true

attr_accessible and attr_protected

in model

attr_protected :admin, :hashed_password
black list of stuff which can't be changed

attr_accessible :price, :size
white list of stuff which can be changed

however, if not in attr_accessible and this is declared, can't be updated

needs maintenance by hand, but pretty cheap

untrusted_input

item = Item.find(:first, :conditions => ["material = ?", untrusted_input])

Don't know how the database does boolean
User.find(:first, :conditions => ["admin = ?", true])

for doing in
User.find(:first, :conditions => ["email in (?)", ["a@c", "b@d]])

User.find(:first, :conditions => ["created_at < ?", Time.now])

method h

<%=h item.name %>

when there is the chance that it could be html which could get injected, which then escapes the html and so doesn't drag in a <script> or <img>

Advancing with Rails course - Day 2

2008-04-02T00:46:00.000-07:00

April 1st

So, a lot to blog about from day 2. Don't feel you need to read it all, and some may hardly be a revelation to most, but part of the purpose of this blog is also for my own record of what occurred and what I learnt, so you'll have to bear with me.

We did a lot of working within our own apps today, doing little bits of lecture, and then seeing if it can be applied.
I seem to have picked a good little app to work on, as it does forms, and needs associations, which was some of what we worked on today. (Looking through my notes again, please note that some of these are my interpretations of the discussions an what notes I grabbed, so may not be the whole truth, or even everything David said)

The first lecture bit started on the "Request Cycle".

tThis was hardly a revelation to myself, but was nice to see it formalised. Basically, all rails apps work on the idea of a request cycle.

A user clicks somewhere (to launch the application, a hyperlink, sends a form...)
Mongrel sets off a dispatcher, which goes to routing system to load share (if your server is set up this way)
then calls controller/action/params[x]

i.e. items/show/1
-> contr/act/params[:id]

It then takes items and adds up to make it items_controller.rb and looks for ItemsController class.

This is now just all in the hands of the ruby interpreter, the server is just waiting for a response, which it then sends back to the user.

The server then cares nothing about what happens until it receives another request from the user. Quite obvious, but it explains why new instances of the objects are created each time, whereas within a gui app, the program loads once, and then there are very few times that a new instance of an object would get created. However, the application expects further user input (even if it is just a kill command), whereas the Rails app doesn't. Even if you request a form, the rails app doesn't actually expect you to fill it in and send it back.

So what is happening during the ruby interpreter phase.

firstly it creates an instance or object of the controller

controller = ItemsController.new

and then looks for an action on that instance of the given name

controller.show

passing in the params[:id] if given

now by default active record objects know that they should render a view with the same name, but you can use the render method to render another template. They also know how to render themselves as xml by default.

render :xml => @item

or you could use a redirect, but this starts a new request cycle (sends a 302 to the user/client, which the goes back to mongrel), which therefore needs to reassign and item id (and any other params that might need to pass through)

(update doesn't typically have an update.html.erb, as you usually want a redirect or to render the form again)

render uses the changed values, whereas redirect sets a new instance, which then enables the wrong values to propagate through to the rendered template

before_filter command

before_filter :set_item,
:only => ['edit', 'udpate', 'show']
this would cause set_item action to run before the stated actions

This is good, because it then enforces use of the correct variable use, and then, should you end up with a blank action (ie: def show;end) because the set_item does everything, then it will jump through and render the show.html.erb file, without even needing def show;end to be there at all

Now, instance variables are how the controller talks to the views, but in this case the instance variable does not belong purely to self, as

self in controller is a controller object
self in view is an ActionView::Base object

when rails hands off a variable to the view, it walks through the array and sets its type to the ActionView::Base,
so they don't technically share them.

--

Before lunch we looked at non-default associations. This got very interesting for my app YADB, we found a plugin which allowed nested has many through associations.

belongs_to :auction (gives methods from Auction)
belongs_to :bidder, :class_name => "User" (enables User to be used as a bidder, but no bidder object)

self.bidder = User.new(x)

has_many :bids, :foreign_key => "bidder_id" (sets the use of a different foreign key as otherwise would look for bids_id)
has_many :auctions_held, :class_name => "Auction", :foreign_key => "seller_id"
(combination of the previous two)
has_many :auctions_bid_on (no class), :through => :bids,
:source => :auction, (however you got there, I want to know via bid.auction)
:uniq => true (creates many to many relationship, but with useful join table with it's own model)

define singleton methods on the association

has_many :bids do
def average_interval
i = 0.0
inject {|a,b| i += b.created_at = a.created_at; b }
i / (size-1)/ 1.day
end
end

A bit on inject:

it is a method on enumerable

[1,2,3,4].inject do |a,b|
a+b
end

loop

1st time 1 is a, 2 is b
next time a is result of code block, b is the next element

1: a = 1, b = 2
2: a = 3, b = 3
3: a = 6, b = 4

also used with hashes (both are equivalent)

[1,2,3,4,5].inject({}) do |hash,e| hash[e] = e * 10; hash; end
[1,2,3,4,5].inject({}) do |hash,e| hash.update(e => e * 10); end

I had a problem about trying to link through 2 tables, with the following idea

class Disciplines < ActiveRecord::Base
has_many :card_disciplines
has_many :cards, :through => :card_disciplines
has_many :vampires,
:through => :cards,
:source => :minion, :conditions => "minions.name = 'vampire'"
end

Now this isn't in 2.0.2, but there had been a lot of discussion about it. A patch has been submitted and the writers have created the following plugin

nested_has_many_through.rb
https://svn.colivre.coop.br/svn/noosfero/trunk/vendor/plugins/nested_has_many_through/

which works. David mentioned that the discussion about this has reached a point where he thought it had been incorporated, and may be in edge rails, but clearly didn't make the cut to 2.0.2

I hope that it makes it in, as when I first attempted YADB with 1.2, I was looking for some links like this. There is some discussion about how this hits the database, but it keeps the Rails pragma, instead of writing a SQL statement myself, and still only generated 1 single SQL statement. What more do you want?

--

We then looked at errors and validation.

ActiveRecord examines the object to decide if it is valid. If it fails, then it never even attempts to save the information in the database.

validates_size_of :name, minimum => 5
validates_size_of :name, maximum => 50
- > must be on separate lines

You can create your own validations by using validate

def validate
errors.add("name", "That's impossible") if name =~ /\d/
end

All ActiveRecord objects have and errors attribute. If the errors attribute is empty, then the object is valid

You can trap the errors if the database has the constraints on it that disallow data to be be saved, but do this via the controller, as you are expecting a big error, rather than a protection of the data integrity

--

This then lead onto forms and processing, starting with a bit on the difference between each and map

each vs map

each loops over 1 element at a time, and return value is the object

x = [1,2,3,4]
y = x.each {|e| puts e*10}
y.equal?(x) => true

map returns a mapping (an accumulator of the results of the block)

newx = x.map {|e| e* 10 }
newx => [10,20,30,40]

What goes on when you display and process forms

<% form_tag :action => 'update', :id => @item.id do %>
<p> name: <%= text_field 'item', 'name' %></p>
<p> year: <%= text_field 'item', 'year' %></p>
prepopulates these on form generation

then, for each first argument

params[:item]

and then

params[:item][:name]
params[:item][:year]

fields_for will allow you to override the default selected using form_for

<% form_for 'auction' :url => {:action => create} do |f| %>

<p>Title: <%= f.text_field "title" %></p>
<% fields_for 'item' do |fi| %>
<p>Description: <%= fi.text_field 'description' %>
<% end %>
<%= submit_tag %>
<% end %>

gives

params[:auction][:title]
params[:item][:description]

errors which occur get wrapped with a <div class="fieldWithErrors"></div> which causes the box to jump down beneath the title.

This means you could style it

You can change it.

It calls a proc, which is a function and you can replace it.

Lazy (in environment.rb, should be an initializer)

ActionView::Base.field_error_proc = Proc.new {|a.b|
"<p>Andy's placeholder</p>"
}

This is an executable object that I can call again and again when I want it.

Look on google with 'field_error_proc'

--

The final lecture session was our daily dose of ruby. We were looking at the singleton method. This was interesting to find out, as we looked the singleton class, and a bit on how an object looks to find if it has a method.

The method lookup path

1) The object's singleton class
- modules mixed into singleton class
2) The object's class
- modules mixed into object's class
3) The object's superclass
- modules mixed into object's superclass

repeat 3 as needed until
-Object
a) kernel

To open the singleton class definition for self

class << self

this is a very frequent idiom for class methods, and means if you have lots of

def self.method
...
end

then you can save the 'self.' by

class << self
def method
end
def another_singleton_method
end
end

To demonstrate this, I'll just dump here the irb that I did whilst trying this, and that will be all from day 2. Looking forward to day 3.

guest095:~ ajb$ irb
>> class Person; attr_accessor :name; end
=> nil
>> andy = Person.new
=> #<Person:0x5a2abc>
>> class << andy
>> def talk
>> puts "Hi"
>> end
>> end
=> nil
>> andy.talk
Hi
=> nil
>> aclass = Person
=> Person
>> aclass.methods
=> ["to_yaml_style", "inspect", "private_class_method", "const_missing", "clone", "method", "public_methods", "public_instance_methods", "yaml_as", "instance_variable_defined?", "method_defined?", "superclass", "equal?", "freeze", "included_modules", "const_get", "to_yaml_properties", "methods", "respond_to?", "module_eval", "class_variables", "dup", "instance_variables", "protected_instance_methods", "to_yaml", "__id__", "public_method_defined?", "eql?", "object_id", "require", "const_set", "id", "send", "singleton_methods", "taguri", "class_eval", "taint", "require_gem", "instance_variable_get", "frozen?", "yaml_tag_class_name", "taguri=", "include?", "private_instance_methods", "instance_of?", "__send__", "private_method_defined?", "to_a", "name", "yaml_tag_read_class", "autoload", "type", "new", "<", "instance_eval", "gem", "protected_methods", "<=>", "display", "==", ">", "===", "instance_method", "instance_variable_set", "extend", "kind_of?", "protected_method_defined?", "const_defined?", ">=", "ancestors", "to_s", "<=", "public_class_method", "allocate", "class", "hash", "private_methods", "=~", "tainted?", "instance_methods", "class_variable_defined?", "untaint", "nil?", "constants", "is_a?", "yaml_tag_subclasses?", "autoload?"]
>> dc = class << andy; self; end
=> #<Class:#<Person:0x5a2abc>>
>> dc
=> #<Class:#<Person:0x5a2abc>>
>> dc.instance_methods.sort
=> ["==", "===", "=~", "__id__", "__send__", "class", "clone", "display", "dup", "eql?", "equal?", "extend", "freeze", "frozen?", "gem", "hash", "id", "inspect", "instance_eval", "instance_of?", "instance_variable_defined?", "instance_variable_get", "instance_variable_set", "instance_variables", "is_a?", "kind_of?", "method", "methods", "name", "name=", "nil?", "object_id", "private_methods", "protected_methods", "public_methods", "require", "require_gem", "respond_to?", "send", "singleton_methods", "taguri", "taguri=", "taint", "tainted?", "talk", "to_a", "to_s", "to_yaml", "to_yaml_properties", "to_yaml_style", "type", "untaint"]
>> dc.instance_methods(false)
=> ["name", "talk", "name="]

Advancing with Rails course - Day 1

2008-04-01T01:12:00.000-07:00

Mar 31 was the first of our 4 day Ruby on Rails course at work. It is being presented by David Black, author of Ruby for Rails, and Director of Ruby Power and Light LLC.

We have essentially managed to catch him between the European Ruby conference and Scotland on Rails.

The day was very much an introductory day, with a few little tidbits. We set up our own little Sandbox project to work within. I have opted to start with something I have been looking to do for sometime which is a VTES CCG deck-building program. (VTES is a vampire based collectable card game which is where the Vampire Software comes from. It will be something I'll get into sourceforge soon!)

David also explained a bit more about Ruby objects and rails functional testing. The former was a good tidy up of what I already knew/had worked out. The latter was a good formalisation of what they are trying to achieve, as whilst I had done some functional testing in rails before, most was concentrated on the unit testing. (A note here. functional testing can mean different things in different frameworks. Some prefer to think of functional testing in Rails as just an extension of Unit testing specifically for controllers, as in other frameworks, functional tests are what Rails calls Integration tests).

At this point, David also gave us a reason for NOT using scaffold generation to generate model backed controllers and views. Personally, I like this approach (I think that code which writes code is a very good thing) but he says that as you get further into rails, you end up removing more from scaffold generated code, than you would write that could have been auto-generated, and as such prefers to suggest that you should learn writing from scratch. With my limited experience, I'll take a rain-check on making a confirmed decision, but I will (for now), stick with auto-generation when it comes to a model backed controllers.

One thing here though that I did discover is in Rails 2.0.2, if you have generated your model first, and then run script/generate scaffold , it stops after it finds a migration with the same name as it wants to generate. You actually need to do the following

1) script/generate scaffold --skip-migration
2) not do script/generate model first
3) rename the migration file and class within it before running

This was annoying, as before when I did this, it used to ask if I wanted to replace it, and it generates everything up to the point where it barfs, but nothing after it. Oh well, I suppose it is something I can get used to. Just think I'll need to read up on the new stuff for 2.0.2 sooner than I expected. :)

We also touched upon the new Initializers, which look very useful for keeping track of requirements. I hope we might look at those again.

The final thing to talk about here was how to implement Composite Primary Keys. Firstly - why? ActiveRecord is never going to support them.

Secondly - However, we have a big legacy database using them extensively, so David showed us how to implement Dr. Nic's composite_primary_keys gem (requiring it via an Initializer!). I won't explain it all here, but the scratchpad attempt I made of it worked well. (to install it though - sudo gem install composite_primary_keys)

And finally, speaking of Legacy databases. In Rails 'type' is a reserved column type for when using Single Table Inheritance, and ActiveRecord expects to use this as a class name. We have fallen foul of this with our legacy database, but fall foul of it no more. In the model put:

set_inheritance_column :nothing

and hey presto! you can use the 'type' column in your legacy schema as planned.

That will do for now. Let's see what the next day brings.

Composite Primary Keys - Yuuurrghhh!

2008-03-19T08:14:00.000-07:00

Well, nearly.

You have seen me write how I think that Composite Primary Keys are above all else a BAD THING(tm).

Well, I think they still are, as Primary Keys.

However, there is no reason not to have a unique combination of keys in a table. This I have never had an issue with.

So why?

Well, look at our tagging example. We don't want to assign a tag to an entity_type twice in our frequency table, at this point we just want to increment the frequency count. Here we hit a snag.
When we send info to our model, we don't know the id_tag_frequency, only the id_tag and the id_entity_type. This isn't a unique primary key.

However, not to worry. We just need a constraint to ensure that the combination here is unique (tags will be assigned to multiple entity_types, and entity_types will have multiple tags).

If we do this, then you can do a query lookup to return just one row.

So why not make this the Primary Key?

Well, if you do this, then you could lose out on simple coding like this

if (unique a + b) and not primary key { fetch primary key from database or create new entry }

but, more importantly, if later on you need to reference the table row in another table (which is the benefit of relational databases, isn't it?) you only need the single primary key put in the new table, instead of all the parts which make up the composite key. This maintains a DRY principle within the database, because, should you need to change part of the composite uniqueness in table 1, you don't need to do it in table 2 as well.

So, never worry about needing a unique combination of fields, that is great, but always have a unique single field primary key on the row. This should help future proof your database when you need new tables, and will make it much easier to create new programs, as you have now given people the option of searching via the composite unique key, or the single primary key.

An important aside to this. If you use an iscurrent boolean for your rows. It can be difficult to make this part of the unique composite key (because only 1 will be current), but do ensure that your code makes only 1 current for the entity it relates to. This will ensure that anyone querying the database will always be able to find the current one. I have had this problem working with a legacy database before, due to the fact that someone had put the constraint:

entity + status + date = unique composite primary key

because date (especially if stored as datetime) will generally give you a guaranteed unique combination, new rows were getting stored as iscurrent = 1, but all other rows for that entity and status were not getting set to iscurrent = 0. Whilst you could always use the following sql

select entity, status, date from table order by date

and select the one with the most recent date, you lose the benefit of just asking

select entity, status, date from table where iscurrent = 1

if that is all you want. And again, this future proofs development of new applications, as if the table is very large (and in bioinformatics data storage, they often get very exceptionally large) then developers will need to rely on the field names to determine how to use the table.

So, if you have an iscurrent, keep it up to date, otherwise, DON'T USE IT.

Rant over. Normal service resumes whenever we can determine what is normal anyway.

Define Existence

2008-03-18T05:28:00.000-07:00

Not an existential question this one, but an interesting 'bug'.

I have never used if (exists $var->{key}) {}

I don't know why, but I just have never needed to. In fact, I can only just barely remember being told about it on a course.

I have always used if (defined $var->{key}) {}

What is the difference.

To quote the Camel

A variable can only be defined if it exists, however, the reverse is not necessarily true.

So, if during an initialisation step, you do

$model->{primary_key} = [result of some sql query for primary key]

but there is no primary key result as the sql statement returned no results, then

(exists $model->{primary_key}) == true

however

(defined $model->{primary_key}) == false

as $model->{primary_key} == NULL

Just something I found whilst trying to bug-fix why my save kept trying to go to update (and then croaking because of, you guessed it, no primary key) when it should have been going to create.

I suppose I should have know this, but when I have always, in the past, used defined to tell me if it both exists and is defined, then it's an easy thing to miss.

Andy

Are Clouds Real?

2008-03-17T07:28:00.000-07:00

This is a bit of a summary of what we decided in the production of our database tables for tags.

Very simply, we wanted to be able to tag different entities with tags. We also want to be able
to track the tags used for an entity type (frequency, use for that entity type), so that we
could lookup that, and offer suggestions based on the usage already of that tag - i.e. a cloud
of tags for that entity_type (see scrumptious tagging for some application of that).

So, the first is simply to have a table of tags

id_tag | tag
-------------------

then we have a table of runs

id_run | other info about run
_________________________________

and then a join table

id_tag_run | id_run | id_tag
_________________________________

OK, so that is quite easy. But, we don't have any opportunity to look at any information on this,
particularly our 'clouds' of tags.

We looked at the possibility of a cloud table

id_cloud | ids_of_tags_in_cloud

But how do we know the entity (remember, we want to tag many different things)

In this case, we have an entity_type table

id_entity_type | description | iscurrent
_______________________________________________

From our models, we can get them to work out their entity_type, as our models are named
xxx::model:: where entity type is the name of the table. This is just a simple
method. So, we make sure the description matches. This gives us an id_entity_type.

id_cloud | ids_of_tags_in_cloud | id_entity_type
____________________________________________________

However, now we nothing of the frequencies the tags are used for an entity. We also need to
keep appending the tag id's to a field, which could make it difficult to represent each
individual tag in the model.

At this point, our discussion came about to what clouds actually are. (at this point the
science teacher in me said a collection of water vapour). We wondered about abstracting the
cloud, and try to collect the frequencies of a tag against the entity type

So, out with the cloud table (for now?) and in with the following

id_tag_frequency | id_entity_type | id_tag | frequency
_________________________________________________________

This is looking good. We can store the frequencies that a tag has been used on an entity,
and we also get an easy lookup for all tags that have been saved against an entity, which means
that we can abstract our cloud with an sql statement like this

cloud = select e.description as entity_type, t.tag as tag, frequency
from entity_type e, tag_frequency tf, tag t
where e.description = ?
and e.id_entity_type = tf.id_entity_type
and tf.id_tag = t.id_tag;

So, do we need any more information. It is useful to know who first saved a tag for an entity, and a date.
We can add this to our tag_ join table

id_tag_run | id_run | id_tag | id_user | date
_______________________________________________________

So what we have

tag table:
id_tag | tag

's table:
id_ | info about entity

tag_ tables: (one for each entity)
id_tag_ | id_ | id_tag | id_user | date

entity_type dictionary table:
id_entity_type | description | iscurrent(optional)

tag_frequency:
id_tag_frequency | id_entity_type | id_tag | frequency

and clouds are purely abstract entities created from the data in the tables. We just need to use to code
to (sql or within your program if another language) to calculate the frequency.

So, in the Clearpress MVC framework (or in Rails) we create a model for each of tag* tables with the accessors
for the table and hey presto, a setup for tagging any entities in our database. You could even tag a tag :)

How many tests does it take to change a lightbulb? - the film

2008-03-13T09:10:00.000-07:00

The film of my talk is now available - you can see it here:

http://www.foomongers.org.uk/videos/andybrown-testing.html

Cheers

Andy

How many tests does it take to change a lightbulb?

2008-03-12T09:33:00.001-07:00

I gave this talk at lunchtime today during our Wednesday FooMongers meeting to probably the largest number of people who have attended. It was well received.

It focused on why we should test, and how to go about testing, mostly test-driven development wise.

Rob is putting a video up on YouTube tonight of the talk, so I'll post the link tomorrow.

Andy

From: setitesuk, 8 minutes ago

| View | Upload your own

Slides for a presentation on testing given to foomongers at the EBI/Sanger Insts 2008/03/12

SlideShare Link

Scrumptious Tagging

2008-03-11T03:42:00.000-07:00

So, we needed to look at, as well as writing full annotations for runs, the ability to tag runs (and other entities) with keywords. Being as we are writing a web-based application, this is very web2.0.

For examples of tagging, look at de.licio.us, facebook or flickr.

We wanted a style very similar to de.licio.us. However, with the range of autocomplete/autofill javascript out there in the open source community, I couldn't find anything that mimics the way de.licio.us works.

In a nutshell:

Display tags already given to a run (I'll use this, but replace with entity).
Click to Add tags
Change display to an entry field which has the tags in it, a cloud of all tags which have already been assigned to runs, which you can click to add or remove from the entry field, and they highlight if already in the entry field,
and the clever bit - if the user starts typing into the field, it comes up with suggestions in another revealed field of tags from the cloud, sorted by the frequency the tags have been used, showing the ten most common for the current suggestions, changing as the user types more letters.

(You may have noticed here that we are trying to get the users to essentially try to use the same tags again and again, rather than stick in a hyphen, etc)

My boss already had a two function javascript to toggle the tags in and out from the cloud, and highlight them (with some css) if they are already in the text field, but I needed to do the rest. This I managed to some up in 4 more functions.

With my bosses permission, I have included his 2 functions (crediting him of course) and I have put the javascript on sourceforge (http://sourceforge.net/projects/scrumptious/). You can just checkout the current svn trunk to obtain the code.

Mocking out to retrieve an email

2008-02-25T05:52:00.000-08:00

So, we are using MIME::Lite to generate emails from our scripts/modules. The problem I came across in
testing was:

How do I test the $msg->send() to see that the email I have generated is what was expected?

In Ruby on Rails, the test framework automatically allows for email to be appended to an array, which
you can then test against.

I want something like this in PERL!

Well, it turns out that I can.

After the module using MIME::Lite has been loaded, and my object created, do the following before running a method calling $msg->send(); ($model is an object created from xx.pm containing the method you want to call $msg->send() from)

my $sub = sub {
my $msg = shift;
push @{$model->{emails}}, $msg->as_string;
return;
};
MIME::Lite->send('sub', $sub);

Now, if you run the method containing $msg->send(), you can then access an array, where each element contain the MIME::Lite output of the email generated (and no emails would have been sent, thus not spamming people at random:).

If you model method only creates a MIME::Lite object, and you are testing what a script might do with it, then an alternative $sub is

my $sub = sub {
my $msg = shift;
return $msg->as_string();
};
MIME::Lite->send('sub', $sub);

Then yous can happily do:

my $email = $msg->send();

and test $email to your hearts content, including sending it through MIME::Parser to obtain the headers, attachments, body etc.

I hope this may be of help for someone.

Andy

Contacting Authors and creating SQL

2008-02-08T04:39:00.001-08:00

So yesterday I was having a problem. I wanted to send an email using MIME::Lite from a PERL
script,
but such that auto responders wouldn't send back.

I trawled through the RFC to see what things they suggested. It is a lot of boring documentation, but I found 3 useful things

1) Subjects should have autoreply in them
2) The auto-submitted header should be set
3) Auto responders should ignore anything that comes with a Precendence: list header

The first two I could programmatically cope with (if (x) { ignore in some way }).

The third needed me to change my outgoing posts. So, off to the documentation for MIME::Lite to find how to set the Precendence header.

Unfortunately, I couldn't find this, and just using MIME::Lite->new(Precendence => 'list'); wasn't working. Give up I did not! I mailed the very nice man who wrote MIME::Lite as to whether a) it did it, or b) if it would be put in a future release.

Bingo, he mailed me back. Simply do MIME::Lite->new('Precendence:' => 'list'); Hey presto,it worked - brilliant. (My biggest thanks to eryq for his quick informative response!)

In fact, he mentioned you can set any of the non-standard MIME::Lite setting headers in this way. A very useful thing to know indeed.

So, one quick change and a test later, and we have this in svn!

On to the next issue - generating advanced queries in SQL programmatically.

So, I have been set for this sprint to write an advanced query page. I have taken this slowly and steadily, as I want to get it right! The first task is just to get it self generating some multi-table queries, that also use join tables, performing a simple one statement select.

Easy or hard? That is the question.

The way I have looked at it is that I, the wonderful software guru, know the database tables. After all, why wouldn't I? In a search model, I create 4 new methods.

1) advanced_search - this is to generate the SQL, perform it and return the results
2) search_for - this returns a hash of all the required fields and corresponding tables based on what the user has requested.
3) search_conditions - this returns a hash of what fields are being selected on, and corresponding tables for those. Here, I can write some additional extra WHERE statements to group together more than one field lookup if needed (which for loader was)
4) table_links - this returns a hash of all tables that have foreign keys, and the table for that foreign key (all keys have the format "id_").

With this and some looped code, I have managed to generate some rather complex single SELECT statements in the format of

SELECT DISTINCT tablea.id_tablea, tablea.name, tableb.id_tableb, tablec.comments
FROM tablea, tableb, tablec, tabled
WHERE tabela.position IN (x,y,z)
AND tablea.id_tabled = tabled.id_tabled
AND tabled.id_tablec = tablec.id_tablec
AND tablec.comment LIKE '%good%'
AND tablec.id_tableb = tableb.id_tableb

I have chosen to put DISTINCT in as a default, as sometimes, when x,y,z are all chosen, it can return multiple rows that are the same.

This seems to be going quite fast, however my ever faithful friend Test::Perl::Critic says that method advanced_search has a high complexity score, so it definitely needs a bit of refactoring.

However, it seems to be going fairly well. So, refactoring can wait until next week.

By for now

It's been a (rather busy) while

2008-02-05T08:39:00.000-08:00

So, a little while since I last posted. However, much has been done.

We had another release last week, some of which has needed some tweaking since for the next release. GD doesn't help make pages go quickly! However, a bit of ajaxing later and we have a page which loads quickly, but allows the graphs to be opened on the page afterwards.

Nice!

Also, I have been experimenting with creating a script ot parse a MIME format email, to then store the body as an annotation and get the id from the subject.

Fairly easy. I am using MIME::Lite and MIME::Parser to do the hard work, and the api for the application to contact the main models, which in turn handle saving the annotation. You got to love well constructed and documented APIs.

I have written extensive tests over the code for the main part. I wrote most within a module, so that this could be tested easily and reused, with a small script to handle receiving the email, and if anything croaks, doing something with the croak. (An email back, unless it is an Out of Office reply - why do people need to use them? I think we should all start immediately spamming anyone who uses them so an end such that perhaps they won't in future*).

Anyway, I've done a fair bit over the past couple of weeks towards new features, or bettering existing ones. The learning curve keeps going up. I am now also looking at trying to dynamically create SQL in order to be able to do advanced searching. The problem, however, is how to store the foreign key pattern from across the database. I need to look into if the models can provide the information they possess, or if I need to store it within a hash (as current).

Must dash for now. More soon.

*This is not a serious suggestion. I do not condone spamming in any way. A polite request should be used instead.

Adding a graph

2008-01-23T00:58:00.000-08:00

So, Hands up all those that have 'use(d) GD;'?

Now that includes me!

So, I needed to show a table of numbers in a bar chart format. Quite simply, I had an arrayref of hash refs, each hash ref using keys date and percentage.

First off, the GD plot needs it as an arrayref containing 2 arraysrefs, the first of the x-axis (in this case the date) and the second of the percentages. ($cmap is an object of in-house stuff to work out appropriate colours to use).

Easy

my $plot = [[],[]];
foreach my $href (@{$arrayref}) {
push $plot->[0], $href->{date};
push $plot->[1], $href->{percentage};
}

my $graph = GD::Graph::bars->new(1000,400);
$graph->set(
'dclrs' => [
map {
GD::Graph::colour::add_colour(q(#).$cmap->hex_by_name($_));
} (qw(red purple orange blue green yellow magenta cyan))],
'fgclr' => 'black',
'boxclr' => 'white',
'accentclr' => 'black',
'shadowclr' => 'black',
'y_long_ticks' => 1,
'x_label' => 'Date',
'y_label' => 'Percentage Activity'
);
return $graph->plot($plot)->png();

See, I said it was easy. :)

So, of course now I have said that, everyone wants their data in a graph format. Luckily, GD provides lots of graph formats. I just wish that we could submit to them as array of hrefs, rather than array of arrays, which I personlly think is

1) messier
2) relies on the end user to know the order the arrays should be in
3) Sends no additional information for legends (should this be needed), whereas the key can be the legend.

Oh well. I shouldn't complain really. I didn't have to write it. Don't you just love the CPAN!

(GD and it's suite of components by Lincoln D. Stein and searchable on CPAN. There are lots of additional stuff to link into Template::Toolkit, etc written by others. Have a look!)

FooMongers today. Don't yet know what we'll discuss.

Andy