Talk to Software East - Aglie Analysis Pipeline

Last night I gave a talk to the monthly Software East meeting, held by Mark Dalgarno at RedGate Software.

Big Thanks to Mark for inviting me, and everyone who showed up for an interesting discussion on Agile techniques. Also, thanks to RedGate for hosting.

The slides have been uploaded to Slideshare:

Agile analysis development

View more presentations from setitesuk.

It was a great talk, and the first time I didn't get groans outside of work about having used Perl :)

Looking forward to the next Software East meeting in January, which will be Rachel Davies talking about Agile Retrospectives.
I saw Rachel give the second day's keynote at Agile Cambridge, and she was brilliant. She was also very nice to chat to at lunch that day. She has written the book Agile Coaching (Pragmatic Programmers) so for fear of overloading the session with people, I can thoroughly recommend going to this.

Day 2 of Agile Cambridge

After far too little sleep, again I got the first bus out of Haverhill off to Cambridge, luckily not feeling bad after too much beer.

Arriving about 8.30ish, I chatted with a couple of people from the previous day, and we went through for the second Keynote, Building Trust in Agile Teams - Rachel Davies. Rachel is the co-author of Agile Coaching, and talked a lot about trust. What is trust, how do we trust people (both socially and in the work environment), and how do we build trust. Examples included borrowing £20 of a member of the audience, and doing the falling backwards into a group of peoples arms (which worked, and Rachel actually seemed a bit scared of the volunteers going through with). This was an excellent keynote. Rachel is a great speaker, and presented the material in a very easy to understand way.

After a coffee break, for me it was back into the Gilb Theatre for Scrum/XP Add-ons workshop, presented by Jon Mullen and Paul Fairless from BSkyB. The organisation of this was different, in that, with a bit of PowerPoint trickery, they were able to allow us to choose the order of the presentation topics. We did some voting (What we preferred, what our teams did agile), we choose the order we would put 'stories' based on complexity (I clearly thought that baking and decorating a cake for 20 was more complex than washing and waxing a car). The method of this though was (I thought) better than story poker). A story card is put on the table, then each person in turn either puts a new card either side (or between), or moves a card. This continues until an order is established of complexity. After this, the cards are then graded 1->8 for measure of complexity. They gave an excellent description of the way that they work (1 week sprints, pair programming, office layout), and a great little video (with sweets!) of a scrum (that had a number of things wrong with it). It all finished with an attempt to win a bottle of bubbly, in a game like they play on a Friday afternoon.

Next was lunch, and possibly the biggest thing ever to render me speechless. Rachel caught me and complimented my talk on the Pomodoro Technique. I was quite shocked, and as such completely failed to mention how much I had enjoyed her talk (Sorry Rachel). Here was someone who is experienced in presenting, complementing me on mine. After a little discussion about how I might start bringing Agile into my team here (including the idea of a taskboard, that needn't be on full display the whole time), she moved off and I had another Chat with Martin from Aptivate who was telling me about the work they had gone out to do in Zambia bringing women out there up to speed with IT, including the setting up of an Internet Cafe and training staff. It's great to hear about people doing this sort of work, and a shame that they possibly aren't getting as much funding this year to do it as fully as they have done in previous years.

After lunch, the final workshops session. I opted for Visual Management for Agile Teams - Xavier Quesada Allue. In this, we were tasked with creating a Taskboard which would be the focus of the scrum, and would keep track of stories, tasks, if we were on track, who was doing what and backlog.

Here is the one my team came up with



Although, as we went around the room, no two boards were the same. One team (featuring Conny) opted for a Kanban approach



But the coolest had their stories on hearts, which moved along a track. If they stopped, they picked up hairs (a rolling stone gathers no moss) or broke if they were blocked in any way.



This was a great way of getting on task with using a board, and showed that I think if the team is involved in the design, it probably adds more ownership and encourages it's use more.

Xavier also showed us some pictures of taskboards he had helped develop for teams. This was an excellent workshop, and well worth attending.


After the final coffee break of the conference (a chance to make sure people had my twitter alias - setitesuk), we headed back into the Gilb theatre for a panel Q&A session 'Creating a Development Process for your Team: What, How and Why'. The panel was led by Giovanni Asproni, who I met the previous day in Bob Marshalls workshop, and who previously worked at the EBI (so right next door to me!). Beforehand he admitted that he was happy to do lead the session, since he didn't need to do much talking (I must remember that trick next time).

The panel consisted of a number of the speakers from the 2 days (although I can only remember the names of Rachel, Allan Kelly and Willem). The main topic eventually was a debate around pair programming, although other aspects of Agile got mentioned (and even the Pomodoro Technique got dropped in as noting it had become a discussion point - yay me :) ).

I must say, that after two days I was starting to flag a little, so picked up the least amount from this session, but it was interesting nonetheless, and rounded off the two days in a great way.

Mark closed off the conference, and we all left (after grabbing more choccies from the RedGate guys!)

Overall, the conference for me was a great success. An opportunity to find out lots about Agile process. A steep learning curve in places, and just a great opportunity to meet others who are producing great software. My favourite workshop/talk has got to go to Gojko and David for 'The Specification Game', for the great way of teaching us all what we do wrong (it is certainly true that you learn more getting it wrong than right), and favourite snippet goes to James A. Whittaker for being so surprised that so many of us Brits still use a phone book.

Of all my sessions that I had my time again, I would have chosen not to go to, would be Code Debt. As I mentioned in my previous blog, not because it was lacking, but that it was the least relevant to Agile.

Thanks to Mark Dalgarno and his team for organising a great Conference, at a fantastic venue. Thanks to all the speakers that I saw, if I could present even half as well as you all, I'd be happy, and your material was in general excellent. Thanks to everyone I met for engaging discussions, and thanks to RedGate Software for the beers!

I look forward to next year.

Day 1 of Agile Cambridge 2010

Agile Cambridge 2010

Day 1

With much excitement I left Haverhill on the first bus of the day in order to go to the conference. There was method to this madness, since the plan was drinks in the evening, and I wasn't going to pass that opportunity up.

I left with laptop, plan of the sessions I ideally wanted to achieve, and the bits for my lightning talk.

I arrived at Murray Edwards College around 8.30, and found those of my colleagues (Beth Jones and Conny Brunkvist) who were also attending. Also picking up a pack and some freebies from RedGate Software (choccies!)

At 9 we went through for the welcome talk (Mark Dalgarno - Software Acumen) and the first Keynote.

The Keynote was James A. Whittaker (Google) who delighted us all with his running of test at Google.
As well as unit tests and functional tests, they also have a suite of different Tours, which are designed to go off and fully test the whole applications under different situations, but which are named rather 'cleverly' (I would love to know what 'The Couch Potato Tour' is). He also showed us problems that had been sent in with Google Maps, including the best route to walk from Cambridge to Hull is taking the ferry via Holland.

He also seemed rather surprised that over half of the audience had actually used a phone book in the last 6 months. Clearly a difference between Americans and Brits.

After the first coffee break (good letting them be 30 mins, allows plenty of time to chat!), it was the first sessions with choices. I opted for the Workshop 'The Specification Game', since one thing I admit I am bad on is getting specifics up front, and in the end, this only proved it.

Gojko Adzic and David de Florinier introduced us to a 'simple task'. They were hiring (i.e. they were the customer - note that, I'll come back to it) us to do them a blackjack application. We were to go through the first iteration, and should produce them something which was playable. We needed a business/project owner (that was anyone who knew blackjack - me!) and at least one dev and 1 tester.

This was were we made our second mistake, (the first being that we completely forgot they were the customer). We went for having the dev and tester as separate people.

I spec'd out what I thought we could achieve in the first iteration, (not negotiating at all with the customer), and we went for it. I went for a playable version which wouldn't bother with betting, but would check blackjack, that the dealer would deal correctly, and would say if the player or the dealer won.

At the end of the time allowed, it is safe to say that we failed. Reasons:

1) Some of my specs weren't good enough
2) One of my specs wasn't dealt with
3) I spent too much time helping the tester, than guiding my two devs and the tester
4) I never consulted with the customer for any user acceptance tests
5) I never consulted with the user for exactly what they wanted in the first iteration, and negotiated that it was too much for my team

So, I accept the failure.

Initially afterwards we felt a bit aggrieved since the one thing that everyone in the room had failed to realise was that Gojko and David were our customers, and so we never felt we could consult with them, you notice that I mentioned that they did tell us this up front, so we all failed because we never got them to discuss with us the specifications. We chatted with Gojko nd David, and found that this was what they were trying to get across to us, and that this is the biggest problem with the specifications is that we forget to/or just don't get the customer involved in what to do that iteration. Also the other was that we never asked the customer for UATs, so when we finally got some, they inevitably failed. Also, no-one did the tests before the development. Whilst they said there must be at least one Dev and one Tester, they never said that they couldn't be the same person.

I think that this was an excellent session. One that I have certainly taken a lot away from.

After lunch, I went for the Hands-On session Code Debt (David Harvey and Peter Marks). This was a break away from Agile, but I was interested in seeing exactly what was meant by it (the term is bandied around a fair bit, but not with any real explanation). The best definition of Code Debt that was mentioned is 'It is code that you owe time to'.

We all have written terrible code (I am sure I still do). The purpose of the first exercise was to change/add to some javascript code to make a new set of tests of that code pass. We were given 10 minutes. Some in the group managed it, some didn't. What we were unaware of at the time was that one side of the group were given nice concise well written javascript, and the other just javascript that worked (it was all done via TDD), but had no best practice about it. As such it was essentially unmaintainable code, and us such, they weren't expected to be able to do the task.

And this seemed to be pretty much the point of the session. If the code just does the job, but hasn't been thought about (idiosyncrasies of the language, meaningful function/variable names, refactored) then there is some level of debt owed to the code to get it to that point.

It was a good workshop to have attended, but in this case, I think a little out of place at a conference about Agile, and possibly a bit too much time spent getting over the one point. Refactor and make sure that your naming means something. Still, well presented. (And I also discovered I was probably the only Perl programmer there - or at least willing to admit it).

Another coffee break, and the onto 'Understanding the Agile Mind: How Mindsets Transform as Organisations Rightshift Effectiveness' (Bob Marshall - Falling Blossoms twitter @flowchainsensei)

I had shared a few words with Bob at lunch (we seemed to have managed to be at all the same sessions so far), but hadn't actually twigged he was running this session.

The focus of this workshop was looking at how organisations worked at the effectiveness multipliers that give serious increases in production with lot less waste, but are so far above the norm, that the mindset seems 'Alien' to most. Working in small groups, we tried to come up with things that we thought companies that were at 0/1/2/3/4/5x effectiveness (1x being the norm) would be doing. Bob then mentioned
the 4 different types of organistional styles (Ad-Hoc, Analytic (sometimes called Mechanistic), Holistic/Synergistic and Chaordic) (I've just found a pdf paper by Bob about this)

I think that clearly this is a huge area to look into (I admit by this time of the day I was more thinking about the pub - sorry Bob), but looks incredibly interesting, and I shall be taking the time to read the above paper soon.

After another coffee break, we had the final session of the day, and a chance for me to Do My Stuff. So, obviously, I attended the Lightning talks (since I was presenting one). I should have gone first, but as with all things, the projection system decided to take that opportunity to have a nap, so I got a bit delayed.

The other talks were really interesting though. Nick ably stepped up to now go first, with some interesting insights into how agile and scrum very much mimics how humans have always done things, including that daily scrums are like sitting around the tribal campfire.

Bob then came up, with an explanation of how he came up with his twitter moniker. Tiss from RedGate (I think rather badgered into it by Helen) did some Improv very Whose Line is it Anyway. I then gave my talk on The Pomodoro Technique (see previous blog post), and then Alan Kelly came on to show how 'Doing it Right' against 'The Right (Business Aligned) Way to do it' affects the productivity, cost and sales. Took me right back to A-Level Economics, but a reminder of how things need levels of thought and balance.

After this, a move to the Castle Inn, for a well earned pint (or four) and food, all courtesy of those fine fellows at RedGate, who I spent most of the night chatting to, along with others who were there. (Sorry guys, I can't remember most of your names and it seems that there was no delegate list in the pack).

I left the pub (unfortunately rather abruptly - big apologies to those I was talking to at the time) in order to catch the bus back to Haverhill. I though some sleep might be useful before the following day.

3 Days of Tomatos

I've been umming and aahing over looking at the Pomodoro Technique as a way of personally increasing my capacity to focus on the task at hand, having heard bits about it on Twitter feeds.

Finally, I bit the bullet, and spent some money on the book 'Pomodoro Technique Illustrated' by Staffan Noteberg (Pragmatic Bookshelf) and read it.

So, all I need is a timer, pen, paper and that's it. Well, that I can achieve.

I had a go the first day after reading the book, and thought I had done reasonably well, but then it all went out the window.

So, two weeks later, (last Thursday to be exact) I decide to try again. In earnest.

What a success!

Although I have made a few errors in judging how many pomodoro certain tasks will take (which actually shows that some jobs needed cutting into more smaller tasks) I have felt more focused throughout the day and felt at the end, I have achieved what I needed to.

I have averaged 10 pomodoro over the last 3 working days, and feel in that this is a realistic target. Tomorrow, plan for 10 - lets see if I can keep at this.

Anyone who is reading this, and finds themselves procrastinating, then I recommend this at the moment. A 5 minute break every 25 minutes really helps, and it is very easy to concentrate on task for just 25 minutes at a time.

The biggest downside I find - my alarm goes off, and the song on my iTunes hasn't finished.

Another year passes...

So, another work year has passed. With it including my 10th anniversary at The Sanger Institute, and 2 years as officially a software developer, and a change of team leader, its been a pretty full year.

So, what have I achieved:

Well, first off, I decided to split up a project which helped us a lot, as we have been able to be more agile in changes and bug fixes when dealing with parts of our pipelines and code.

Following the change of leader in our team, we started to spend some time looking at new technologies. I started to take a look message queues. See Stomping on the Rabbit and Pt2.
This was my first time at actually properly failing something. It felt good. I got lots of help from monadic and their team down in London relating to RabbitMQ, and we tried ActiveMQ as well. But the reason we failed it: after attempting a comprehensive test suite to test cases where servers go down, we decided that (at this time) it wasn't stable enough for passing around the code as reliably as we hoped. I spent about 3 weeks looking at this, but we believe that is time well spent. (Another team started to employ it directly, but continually had problems with their poller of a queue.)

I also spent some time at this time taking a look at Erlang, but since there didn't seem to be much opportunity to apply this at work, I stopped for the time being.

I then deployed the badminton ladder I was working on for the Sports and Social Club. See Playing Badminton through Clearpress. This has gone down well, although I do need to make some improvements to the interface.

We had been deliberating about making the analysis pipeline pluggable, so I took this task on. I also used it as a chance to take a good look at Moose as an O-O framework. I have pretty much spent the rest of the year doing this.

This had had me really excited, and probably the thing which has kept me most interested in my job for the last 6 months, and also one thing which proved I didn't waste my time splitting up the project earlier in the year.

After much discussion of whether to relaunch scripts, or load everything to LSF in one go, we went for something that does as much as it can, then submits jobs sequentially to LSF, but with code being as decoupled as possible. You can see in the powerpoint presentation Pluggable Pipelines the style we took. With the power of the Moose behind it utilising Roles and even developing my own CPAN extensions MooseX::File_or_DB::Storage and MooseX::AttributeCloner.

With this pluggable pipeline setup, of course it is now multiple times easier to add new stuff, particularly QC testing. So, we need a way to display the results. The new boss is keen on Catalyst, so I decided now is the time to look at this. Whilst this project has moved onto another team member, I gave it a bit of a start, initially to see how it would deal with getting stuff from a file server rather than a database 'Backending Catalyst'. A couple of book sources later and I get a working app up. In fact, really, the biggest challenge was tring to apply some css that I stole from two Rails apps.

I pass my 10 year anniversary without a hitch, it seems. It is funny to find that I have worked somewhere for 10 years. Although saying that, I have done a number of different roles here, from washing glassware to sequencing dna, to instituting the widespread use of Transposon Insertion Libraries in finishing, to Quality Control, to Software Development.

And then the slope to Christmas. The pipeline systems aim to be flexible has now been pushed to the limit, with virtually unreported changes to the 3rd party pipeline which it has had to keep up with - and whilst the code unavoidably has to change, we have spent most time discussing the planned changes to cope, and testing it, rather than coding.

I've also had to come to the decision that I think Class::Std is well and truly dead. RIP. You helped me become a better, O-O programmer, and you will be remembered with a fondness, but Moose has just dealt you too many blows.

Overall, it has been quite a productive agile year. I'm taking a look at the C language and trying to lobby to get a course run at work on Moose and Catalyst. Hopefully, these will be big things for me next year. I look forward to it.

Merry Christmas, and a Happy New Year to all.

Always change the branch!

Again, this one of those blogs which have shown some numptiness by me.

Why did some things break, when they were working yesterday.

We have a lot of file passing to do, and unfortunately, a third party analysis pipeline starts changing file locations and filenames, such that we have to remain agile to cope with the third party changes.

So, we discover this happening and quickly fix our pipeline to cope with a filename change. Deploy and go, everything is working as expected.

A new release of the pipeline occurs from the development branch into trunk, and then deploy. But suddenly, the files aren't being found as expected. What has happened?

Quite simply, I didn't make a corresponding change in the development branch for the original bugfix. 4 hours of developer time spent trying to track this down for something that I broke my own rule.

Always change the branch/backport trunk to ensure that the bug is fixed everywhere.

Sometimes, just trying to be too agile, sorry read trying to fix a bug fast, just causes the bug to get reimplemented further down the line. Take your time Brown! A little caution and doing it right will make it better.

Pluggable pipelines - the return

On Friday, we deployed v5.0 of the new pluggable style analysis pipeline. This now includes a pluggable analysis pipeline, the archival file creation, and qc pipeline, and archival pipline.

With this new setup, we can now add additional steps (or remove them) easily in a matter of a few hours (usually actually minutes, but then we need those tests).

If you look back to my previous post, the style is there, and we have found it really works well. It is keeping the idea of individual application code separate from the pipeline code which runs it, and even loosely couples individual apps so they run more smoothly.

An additional bonus which has occured, without even trying, is that we have a significant improvement in run time. The reason is that, since we have queued all as separate jobs, instead of as batched makefiles, LSF runs them as it finds space. This means it doesn't need to find, or earmark, 8processors with max memory allowance, but just one or two with smaller memory, so they can be assigned more efficiently. This passes stuff through faster (assuming dependency trees pan out correctly), and so users can get the data quicker. Fantastic, and even better since we didn't plan for it.

No time to stop yet though, we have lots more features to add, but since we have managed to move successfully to this more agile structure, I don't see them taking long to deploy.

Here's to agile structures and development.