On Friday, we deployed v5.0 of the new pluggable style analysis pipeline. This now includes a pluggable analysis pipeline, the archival file creation, and qc pipeline, and archival pipline.
With this new setup, we can now add additional steps (or remove them) easily in a matter of a few hours (usually actually minutes, but then we need those tests).
If you look back to my previous post, the style is there, and we have found it really works well. It is keeping the idea of individual application code separate from the pipeline code which runs it, and even loosely couples individual apps so they run more smoothly.
An additional bonus which has occured, without even trying, is that we have a significant improvement in run time. The reason is that, since we have queued all as separate jobs, instead of as batched makefiles, LSF runs them as it finds space. This means it doesn't need to find, or earmark, 8processors with max memory allowance, but just one or two with smaller memory, so they can be assigned more efficiently. This passes stuff through faster (assuming dependency trees pan out correctly), and so users can get the data quicker. Fantastic, and even better since we didn't plan for it.
No time to stop yet though, we have lots more features to add, but since we have managed to move successfully to this more agile structure, I don't see them taking long to deploy.
Here's to agile structures and development.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment