Bingo Node

Much has happened since I last posted here. At $work, a shift has occurred in programming responsibilities and whilst I stilll have lead on the internal LIMs, much of the last few months has been working with Nodejs to produce an analysis framework.

Node uses the v8 engine to run javascript server side, and it's asynchronous nature, and easy use of sockets, makes it perfect to produce an analysis framework for small packet based analysis.

Take info, wrap in JSON packet, pass through socket to new process, process asynchronously enhances packet and pushes through new socket to next new process...

This has been a steep learning curve for me, as I have been primarily a Perl developer with some JS on the client side. Writing purely asynchronous code is very much a change of mind set.

And it works both ways, I was recently helping a colleague with a Perl script which actually would have benefited from being Asynchronous, and I spent a day trying to do that in Perl. (Mind you, I lost count of the amount of times I tried to do synchronous stuff in Node :) Long live the promise module)

Also, we discovered the node-webkit library. This was just fantastic, as we need to be able to deploy a user interface on multiple platforms, and this allows just that, but with everything in done in Node. A standalone application is produced, compiled for Mac, Linux and Windows (one for each).

Now, as we all know, mastership of a skill takes 10,000 hours, and that ain't only gonna happen at work. So I needed a pet project to play with.

Bring in a rewrite of my bingo caller. I have full love for Perl, and the Catalyst/Moose frameworks, but the bingo-caller was a bit slow and clunky. I needed a server and SQL-Lite running in the background. It really needed a rewrite.

I also was thinking of setting myself up as a freelance Bingo caller, as I now have a couple of regular gigs, but the Catalyst app version just won't cut the mustard any more;

So, in a weekend, with a substantially reduced codebase, I got a bingo caller running in Node with node-webkit. It compiles, is now a double click app (my first ever!), and is much much faster. Even putting in the traditional bingo calls hasn't worried the memory usage. There are clear advantages to writing amd running it all in javascript. So, to all involved in NodeJS, and it's many libraries (especially node-webkit), I salute you all.

The code is hosted on GitHub. Versions 0.1/0.2 are under the MIT License, v1.0 is copyright me. I would ask for you to ask permission if you wish to use the caller, but feel free to peruse the code and take hints from it.

github.com/setitesuk/node_webkit_bingo_caller

On a comparison note, I now have 2 real tools for server side programming now, and whilst Perl was my hammer, I now don't need to look at everything as a nail. NodeJS is a great screwdriver, and I can certainly start to identify screws.

If you are interested in hiring a Bingo Caller

https://www.facebook.com/setitesukbingo
twitter.com/setitesukbingo
setitesukbingo@gmail.com

Stumped by range

Today at $work we can across the following in someone's perl script

  if (/\Q$start/x../\Q$end/x) {print q{match};}

This rather stumped all of us from $boss down. Between us, we have over 25 years worth of perl experience, and no-one had seen this before. What is the range operator doing between 2 regexes?

It turns out, that we can use this to loop over (for example) lines in a file, and when $start is seen, start printing out until $end is seen. Then reset and finish looping.

A google search brought up the following page

http://perlmeme.org/faqs/file_io/middle_of_a_file.html

Interestingly (although it is probably the right way to do it), it only keeps a track of whether you have seen $start once, and $end once (before a 'reset') so if $start is there twice before $end, it only finds the first $end, and stops 'matching'. ie:

 foo       no_match
 $start    match
 foo       match
 foo       match
 $start    match
 foo       match
 $end      match
 foo       no_match
 $end      no_match
 $start    match
 foo       match
 foo       match
 $end      match
 foo       no_match



Whilst we worked this out (quicker with a hint from a comment), I think we universally decided that it wasn't particularly readable. However, I think my first ever production script probably would have been a bit shorter had I used it back then :)

I'll leave the merits of this down to the reader. Personally, I don't know how useful I'd find it, but it might help someone out there. Hello to you if you are that person!

(Hopefully, I'll remember this post next time I see this!)

A very handsontable(.js)

At $work, everyone loves spreadsheets (well, almost everyone) and they like to use them at any and all opportunities they can.

This has meant the ability to upload and download data from the LIMs as xls(x), including downloadable template files.

However, people don't seem to like data entry (well, ok, who does - my first non-supermarket job I spent most of my first afternoon, past shift end, data entering), no matter how you construct the web form to do it.

User: Can I enter my data directly into LIMs, modifying many rows of data at a time, in a style like Excel, because I know Excel?

Developer(AKA me): Yes, you can download the data as xls, change it and batch upload the changed sheet. Expected time delay of less than 5 minutes, (generally less than 30s).

User: But I don't like the idea of a time lag between my upload and seeing the data.

Developer: So what would you like?

User: Excel table entry, but within the web application.

Developer: (silent _scream - AAAHHHHH!) I'll see what I can do, but it will take a bit of time.

User: It is quite urgent that the interface is changed.

Developer: (thinks - funny that, user needs something doing urgently) I can give you a ball park estimate of a possibility it can be done in 2 weeks, with a rough usable prototype system in a further 2 weeks. (Starts searching Google)

User: Thankyou (I do on the whole have nice users)

So, I need a Tabular style of entry, for rows of data editing, that can save, and do much as Excel does.

Well, I could do an html table (yes, this thought did originally cross my mind, I'm not proud), with each box being a field in a form. However, it is ugly and you can't do cut and paste, value dragging...

So, I looked around for alternatives using JavaScript libraries (preferably jQuery) as I really didn't want to roll my own.

Luckily, after a quick search, I found it handsontable.js which is jQuery based, and the examples on the website did very much like I wanted it to.

So, after pointing User to the website, I got a good thumbs up, and started to put together a prototype app with some real data to test it out.

The great thing about handsontable.js is that the API is very compact. You only have to provide a set of data, and a few other params (helpfully named) to get a working table up, and it provides the functions to dumpout the data to be directly sent via AJAX.

So, I was able to get the working prototype out within those first 2 weeks easily. Once I had had theat tested by User, it was quite simple to import it into the LIMs, so within the allotted time I had the feature working.

I had some interesting Bugs to then deal with. Perhaps Bugs is too strong a word. The first was that a button to remove table entry (hide) and then bring it back didn't revert unsaved changes. Luckily, handsontables loadData function saw to helping me there. Really simple. Then column sizes and scrollability/change col width. Both again very simple to implement (although there does seem to be a finite size to a column, which I unfortunately hit)

The website has all the features you need, including the css files, and has some of the best online tutorial/examples I've seen in a very long time.

So, big up thanks to Marcin Warpechowski for providing a brilliant solution. I can thoroughly recommend this as a solution for table style entry in a web interface.

Bookmarklets and closing a window with js

I want to allow users to update one application (LIMS) with basic information whilst the product generating the output was still running.

Prior to this, the finished output is seen by a daemon, which uses an api and creates the information into the application, and users then update.

The problem is that the users are using the product's own software, and we don't want to modify that to use the api to post, so what do we do?

Solution: create a bookmarklet

I hadn't come across bookmarklets before this week, as I just haven't needed them. But it is simply a bit of javascript that you drop into you bookmarks toolbar, and it deals with loading a page from a website, maybe scraping from the one you are on (see delicious.com) and giving a pop-up window, or overlays the page with additional features that you might want, such as ability to draw on the page.

So, this enables me to load into a popup, a cutdown form from the external application, screen scraping for a filename, and then closing once saved.

Here is an example of a bookmarklet (drop it into a href attribute on a link, and you can then drop it into bookmarks easily) NB: I am using Template Toolkit to fill in SCRIPT_NAME/class/method

  javascript:(function()%7Bf='[% SCRIPT_NAME %]/[% class %]/[% method %]
  ?nodecor=1&filename='+document.getElementById('filename').val+'&';
  a=function()%7Bif(!window.open(f+'noui=1&jump=doclose','',
  'location=1,links=0,scrollbars=1,toolbar=0,width=1100,height=600'))
  location.href=f+'jump=yes'%7D;if(/Firefox
  /.test(navigator.userAgent))%7BsetTimeout(a,0)%7Delse%7Ba()%7D%7D)()

To close the window once it had successfully saved out the data was interesting. Javascript doesn't have an actual sleep function, and many many people were commenting 'Why would you want one?'

Well folks, the reason is that sometimes, sleeping is convenient. I want people to have the chance to be told their data is saved, but close the window down and get it out of their face, to save them having to do it.

It turns out in this case that I could just use the setTimeout function, as below (jQuery is utilised)

  $(document).ready( function () {
    setTimeout( function () { window.close() }, 5000 ); } );

So, now I have a working support for the users, who can better utilise the time that they are waiting for the product to produce it's output, and save data more upfront.

TeX - an odd markup language

Why do I say odd?

Over the last couple of iterations at $work, I have wanted to produce some automated PDF reports from out of our database, with the option to pull in some additional images should they be available.

Now, I'm using the perl ClearPress MVC framework for the application, and it is nice and friendly in that I can Template my TeX and push that through pdflatex to get the PDF.

No problems so far. The data is therefore obtained and pushed into the template in exactly the same way as with HTML.

A nice simple bit of TeX markup and I get a nicely formatted PDF report yes?

No.

Unlike scripting languages and HTML, it doesn't seem to be doing everything top to bottom. Some of my tables end up above Section headings, images too.

What?

Now, prior to this I have had no experience with TeX in any flavour before.
I start by looking into main uses of TeX. For those not in the know, it is a markup designed for Mathematical equation rendering, and is popular with Journal entries as with simple markup given with the paper, they can change the formatting to suit the Journal.

So I start looking around the Google querying how to do formatting. The answers to queries I have seem to always take me to forums, as opposed to any sort of manual. The first answer took me to the info in the paragraph above, basically saying that we want to do your formatting, don't do it yourself. And there was an interesting discussion in a forum with one person giving a way of telling your table/image 'Put yourself there!' and another asking 'Why on Earth would you want to limit your table/image to the one position'.

Also, another one was escaping special characters. Eventually, I came to the following link The Comprehensive LaTeX Symbol List - The UK TeX Archive (pdf download) but not after having been pushed around forum answers again.

It goes to show that forums are great places for information (although, sometimes the question to Google often brings up that in favour of the manual :) ).

But again, why do I say odd?

Well it seems that TeX doesn't do Markup in quite the logical way I would do Markup (which is the way HTML does it). HTML says start at the top, display the marked up code from top to bottom, unless the markup (combined with css/javascript) tells me to do it otherwise. TeX seems to say, Start at the top, but float everything to where I think it is best, unless the markup specifies something different. This seems to keep the order of the same types of items, but doesn't necessarily keep the items where written.

Also, special characters are not always represented with a textual markup version, or an actual escaped version. i.e.

  \ needs to be \textbackslash, not \\ (as that actually means newline)
  # needs to be \#, there isn't a \texthash, and \hash doesn't always work

I think that, had I known this at the start, I'd have then had a better idea of how to query Google for info. TeX is clearly a useful Markup, and by the looks of it, very extensible (I pulled in 'grffile' in order for it to deal with spaces in filenames and paths), but it isn't as quick to pick up as html.

Test::WWW::Selenium::Conversion::IDE

At $work we have a LIMS system running as a web app. Nothing unusual there. However, we had a user experience a bug which had only be seen once before, and seemed impossible to recreate.

It seemed to come from asynchronous Ajax updates to one field on the user session.

My first thought, was how to recreate this bug which only comes up rarely. On my development version, I couldn't get it to happen. So, I decided to have a play with Selenium. After downloading the Firefox plugin, I recorded the actions that our user did, and ran through as fast as it would replay.

Success, the bug reappeared. If run at slow speed, the test ran through happily, but fast caused the issue with the Ajax.

Working on a solution then gave me a test case to work against, and I can happily report that the bug has gone away.

Now, as a reader of this blog, you know I am a Perl programmer, and, yes, this app is just that. A Clearpress app.

We have a good test suite exercising the model, view and api, but we don't have anything UX tests. So, can I run the Selenium IDE file (Selenese HTML) in the Perl test suite. CPAN has two modules providing drivers for the Selenium Server, but no converter from Selenese to Perl Test. That seemed to go some time ago.

So, time to actually do something about it. I chose to use the Test:: WWW::Selenium module over Selenium::Remote::Driver, as I preferred the API.

Now, I had two strategies here.

1) Fully convert the Selenese HTML file to a Perl Test file,
2) On the fly, convert and run the Selenese HTML file

I opted for 2, as I coul just hard code in a wrapper test a Selenese suite file, and then every time we record another user story, it will just get added to the suite, and run. Also, keeping the code DRY, as it will b handy to keep the Selenese HTML around to run individually in the IDE.

So, I started developing Test::WWW::Selenium::Conversion::IDE, to do just that. I released it yesterday to CPAN.

It should most definitely be considered ALPHA code. I am unlikely to change the function names, as $work will rely on it, but it is by no means feature complete. I welcome comments and patches if you need more features.

The way it works:

1) In a wrapper test file, create a Test::WWW::Selenium object with creds for your Selenium Server
2) Pass to the suite runner function this object and the filename of your suite file. You can also pass the location of the file, but it defaults to t/selenium_tests.
3) Rather than give a plan, use done_testing() at the end of your file, so the TAP plan can be accounted.

See POD for more options.

So, I push it out for the communities use.

Bye for now.

Deep Magic - recover a file

We have hopefully managed to recover a file (or two) today from an existing process in which the files live in /tmp/, but if the process goes on too long, get deleted by the system

Find relevant Process ID
  >ps aux or top
  >pstree -p 

Work out files which have handles but are deleted
  >lsof -p 
  bwa     22039 srpipe    4r   REG      104,7  424718660  67171651 /tmp/iFCyWKyDjR/1.sai (deleted)
  bwa     22039 srpipe    5r   REG      104,7  401682372  67171652 /tmp/iFCyWKyDjR/2.sai (deleted)

Rebuild files
  >mkdir /tmp/iFCyWKyDjR
  >cat /proc/22039/fd/4 > /tmp/iFCyWKyDjR/1.sai
  >cat /proc/22039/fd/5 > /tmp/iFCyWKyDjR/2.sai

Check
  >ls -lh /tmp/iFCyWKyDjR
  total 789M
  -rw-rw-r-- 1 srpipe solexa 406M 2011-09-02 09:30 1.sai
  -rw-rw-r-- 1 srpipe solexa 384M 2011-09-02 09:30 2.sai