Reviving this blog

Alas, it’s been nearly 3 years since I updated this blog!

During this time a lot has changed in my life, from my personal life to my professional life.  Here’s just a few of the major changes:

  1. I now work in one of the world’s top investment bank as an IT security software engineer.
  2. I haven’t touched Python in ages and have converted to using Perl
  3. I have gotten really really good at AngularJS.
  4. Lastly, very recently, I received my GISF Security Fundamentals Certification.  I know it’s not a big deal, but it’s my very first certification and it’s a huge deal to me.
  5. I have a son and he’s 1 year and 5 months now!  My biggest love in the whole wide world.

Points 1-4 are actually related to my current job at the investment bank.  I never envisioned myself to work in the financial industry doing IT.  I always thought it was selling out since that’s exactly what most of my friends in Singapore do after graduating from college.

Strangely, I was always under the impression that banking IT is all about maintaining gigantic legacy systems.  Okay, maybe certain banks are still like that, however, the one I’m working for is pretty fantastic in this area.  We are working on pretty cutting edge stuff:  Hadoop, Spark, ElasticSearch, Hive, AngularJS to mention a few.  There’s also a bunch of really cool tools developed in-house that mimics tools like Heroku, Docker, Chef, etc.  I’m very impressed to say the least.

My team uses Perl for all their backend and scripting needs.  To “play nice” with my team, I had to pick up Perl as well.  I have to say that I was really hating it for the first 1.5 years of using this language.  I really hate the idea of “multiple ways of doing one thing”.  Then there’s also the special variables that Perl syntax is riddled with, like:  $_, @_, $!, $#, %+.  All that is still ok for me.  The thing I really dislike about Perl is the way their documentation is styled.  It’s near impossible to find out the return value/type without reading the entire contents of the documentation.  Having said all of this, I have recently started to appreciate this language because of its brevity and the amount of code to get up and running.  I recently had to write a RESTful API, and chose to use Mojolicious web framework to do the job.  I was initially thinking of using the familiar Python Flask, however I found that the amount of code to write was just too much to get going.  Highly recommend Mojolicious!  It’s a pretty awesome web framework!

I also started teaching AngularJS at my work place.  It started out as an interest group where we meet bi-weekly to tackle some of the harder topics of AngularJS.  The training department took notice of this and have requested me to teach 2016 Intro to AngularJS course.  That should be lots of fun and am really looking forward to this.

Lastly and most importantly, my wife gave birth to a beautiful little boy on August 14, 2014.  His name is Huang Kaitong, Scott and he is now 1 year 5 months old.  I’m mighty proud of him and he is the love of my life!  Fatherhood has changed me greatly and I feel that everything I do has a purpose all of a sudden.

Tests

Nosetests: Simple How To

Introduction to Nose

I’m 2 weeks new to a mobile gaming company.  Reading their vast library of code and can’t seem to figure out how everything is interconnected.  I decided to write some unit tests with the help of Nose Testing Framework for me to understand how things work.  I’m very new to testing and thought of applying TDD practices to my future code development.

What makes this so awesome?

I really like the Nose Testing Framework because it makes testing so much more easier.  Some highlights from their docs:

  • If it looks like a test, it’s a test.  Basically Nose will match file names and modules with testMatch regular expression that searches for the word “test”.  If a class is a subclass of unittest.TestCase AND is also in a module that looks like a test, it will be a test aswell.  How awesome is that?
  • Packages are always inspected, but they are only collected if they look like tests. This means that you can include your tests inside of your packages (somepackage/tests) and nose will collect the tests without running package code inappropriately.
  • If an object defines a __test__ attribute that does not evaluate to True, that object will not be collected, nor will any objects it contains.

The above are some of the things I love about Nose.

 How to begin?

Usually I start with a setup.py file.  This small script provides a lot of powerful features like console scripts as well as ease of testing your project.

We add a tests_require option to specify what packages are required to be installed when you perform the following command:

python setup.py test

Another important option to add is test_suite.  We use nose.collector so that we are able to issue the above command.  Further details can be found here.

My first test

  • This is a test of a python wrapper I wrote for redis.
  • I included all of nose.tools functions so that we can write python’s unittest module functions lower-cased and with underscores instead of the typical Java-like camelCased functions.
    • eg.  assertIsInstance would be written as assert_is_instance
  • RedisTestCase inherits unittest.TestCase.  Whether this is a Nose test or a unittest test, Nose is able to “sniff” (pun intended) out all tests
  • In my setUp fixture, I setup an instance of my redis connection as well as populate it with the sorted set called “abc” and a set called “tomongo”
    • subsequently, when the unit test runs, the sequence of events will be as such:
First test
----------
setUp()
test_connection_instance()
tearDown()
 
Second test
-----------
setUp()
test_ping_function_exists()
tearDown()
...
    • As you can see, tearDown is called for every single test, whether it passed or failed.  This ensures that each test is run exactly under the same circumstances as described in the setUp fixture.
  • Lastly, I setup a test suite to consolidate all TestCases within this module.  I can optionally include/exclude certain TestCases from a TestSuite.  It is a collection of TestCases or TestStuites.

Running the test

The above test maybe named something like test_redis.py.  It could also be in a directory named “tests” under multiple levels of directories.  Nose will find it all!  To run tests, go back to the project’s root directory where your setup.py file lives.  Run the following line:

python setup.py test

This should run all your tests within your project and make you a happy man/girl.

 Trouble “sniffing”

The only trouble I encountered during this experience was of something that was related to the multiprocessing module.  An example is shown below:

The solution to this is to import the multiprocessing module at the start of your setup.py file (you can check the above setup.py code).  That’s it and everything will run fine.

Bells & whistles

Nose comes with some very interesting plugins that you can use to enhance your unit testing experience.  Among these I have used the doctests plugin for running doctests as well as the testid plugin for numbered test output.  A full comprehensive list can be found here.

Summary

Unit testing for the first time has been an enjoyable experience and very very fruitful.  Since then, I’ve been writing tests for all of my code and ensuring that all of them pass before I push code to repositories.  It gives you a piece of mind and provides others with proof saying that your code actually works and it’s not my fault that your thing ain’t working.  If you haven’t tried writing tests in python, I STRONGLY recommend that you do; it would be an unforgivable sin not to.

Gitlab 1 Gitorious 0

I was recently tasked to setup a git server in my new company.  I’ve used Gitorious in the past and thought of setting that up.  However, my colleague informed me of this new tool called GitLab.  I was impressed immediately by the striking resemblance to Github.

The installation process was very detailed and was quite a breeze to setup.  After 20 minutes, Gitlab was finally setup.  Initially we wanted to setup Redmine for issue tracking but Gitlab covered that!

Another great tool completely FOC.  Any donate button on the site would be great.

Encrypting web passwords with scrypt

I’m working on a fresh project, re-designing it from scratch.  The first page I had to make was the Login page.  I’ve always wondered how passwords are stored.  In the past, before I read up online, I thought the plain text passwords were stored in the database and everytime someone enters their password, it will just match against the database.  I was so terribly wrong.

What really goes on is that, there’s usually this concept of a salt.  A salt is a random string generated whenever a user creates an account.  The salt is then appended to the end/beginning of the user’s chosen password.  The result is then passed through a encryption algorithm.  The output together with the original salt is stored in the database.  Now….when the user authenticates by logging into your web application, the salt is obtained from the database and appended to the entered password.  The output from the original encryption is also retrieved and decrypted and the result is matched with the appended result.

Because the unencrypted password is never stored in the database, the system admins of the web application can never know your true password.  This is why whenever you click on “Forgot your password?”, they always reset your password and email you the new one and ask you to change it.

In my web application, I researched a couple of encryption algorithms out there and the easiest one to use right out of the box seemed to be scrypt.  It had a really good python wrapper around it.  The article by Alecco Locco was really useful to get me up and running.  I highly recommend this because not only was it easy to implement, it was supposedly one of the best in terms of security out there.

Feel free to let me know how it turned out for you.

Meddling with urllib2

It’s been a full year since I last updated my blog, this is my new year’s resolution (to update it more often) in practice!

I’ve used parts of urllib2 in the past to do some simple authentication to my own web application to run some scripts.  However, I never really studied the module in detail.  I’ve also seen many instances where httplib was used.  After spending 2-3 hours reading the python documentation, I learnt that httplib is rarely used as it’s more of a low-level interface to HTTP.  Sure it has lots of features, but it’s a bit verbose for my taste.

After reading urllib2 documentation, I decided to convert the CouchDB Wrapper class provided in the Getting started with Python CouchDB Wiki to using urllib2 instead of httplib as a form of exercise.

The main functions to change would be the connect and all of the get/put/delete/post functions.  Let’s have a look at the original get function:

    def get(self, uri):
        c = self.connect()
        headers = {"Accept": "application/json"}
        c.request("GET", uri, None, headers)
        return c.getresponse()

self.connect returns a HTTPConnection object. It sets the headers and method calls getresponse() to obtain the output from the uri provided.  I wanted to use urllib2.Request object to set the headers and any relevant data.  First I had to modify the connect function to return a Request object.  The modified connect function is shown below:

   def connect(self, uri):
        '''Returns the urllib2.Request object to connect to'''
        full_url = ''.join(['http://', self.host, ':', self.port, uri])
        return urllib2.Request(full_url)

Now that we have a Request object, I modified the get function like so:

    def get(self, uri):
        r = self.connect(uri)
        r.add_header('Accept', 'application/json')
        c = urllib2.urlopen(r)
        return c.read()

The Request instance here is r. I add a header using the add_header() function which accepts two strings. Then open the url with urllib2’s might urlopen function which in this case accepts the Request instance and return the contents. That’s easy right?

One thing to take note is that, urllib2 doesn’t seem to support PUT or DELETE requests which are required for the put() and delete() functions. I read one article that creates a quick hack to do just that.  The rest of the functions are found below:

   def get(self, uri):
        r = self.connect(uri)
        r.add_header('Accept', 'application/json')
        c = urllib2.urlopen(r)
        return c.read()
 
    def post(self, uri, body):
        r = self.connect(uri)
        r.add_header('Content-type', 'application/json')
        r.add_data(body)
        c = urllib2.urlopen(r)
        return c.read()
 
    def put(self, uri, body):
        r = self.connect(uri)
        if len(body) > 0:
            r.add_header('Content-type', 'application/json')
            r.get_method = lambda: "PUT"
        else:
            r.get_method = lambda: "PUT"
        r.add_data(body)
        c = urllib2.urlopen(r)
        return c.read()
 
    def delete(self, uri):
        r = self.connect(uri)
        r.get_method = lambda: 'DELETE'
        c = urllib2.urlopen(r)
        return c.read()

You would also have to make a fix to the original prettyPrint function provided. Just change s.read() to just s.

I hope this helps any new Python developers out there. Let me know what you think.

R is freaking amazing!

Over the past weekend, I installed a bunch of development related packages and languages on my Mac. One of the many interesting stuff I stumbled upon was the mighty “R Project for Statistical Computing”.

R has a very simple syntax that can be easily picked up by any novice programmer.  While I was fiddling around with R and it’s awesome plotting libraries, I thought back of the days in university when I was learning Information Coding, dealing with topics like correlation, hypothesis testing and pdfs.

I found this particular tutorial to be extremely useful in understanding the basics:  http://www.cyclismo.org/tutorial/R/

I strongly urge everyone to give it a try.  I was going to do a small write up on Rpy2, which is Python’s integration with R, but my Python configuration on my Mac got screwed up by Mac OS X Lion installation.

Adding TinyMCE into a jQuery UI Dialog

Recently, at work, I’ve been trying to integrate TinyMCE to a textarea within a jQuery UI Dialog. This is actually an email dialog where users can fill in the To, Cc, Subject and Email body (this is the textarea) fields to email specific people. Going to the TinyMCE website, you’ll realize that they provide a plain javascript version and a jQuery plug-in version.  I’ve tried both and found that the jQuery one definitely makes things easier.

Here’s my tinyMCE initialization code that I’ve wrapped into a function:

function addTinyMCE() {
    jQuery('#email_body').tinymce({
        script_url: '/js/tiny_mce_3.2.7_jquery/jscripts/tiny_mce/tiny_mce.js',
        width: "550px",
        height: "290px",
        mode: "textarea",
        // General options
        theme : "simple",
    });
}

However, when implementing the TinyMCE editor to the textarea within the jQuery UI Dialog,  I ran into a bunch of problems:

  1. The first time you open the dialog, everything is fine and dandy
    • However, the second time you try to open the dialog, the dialog won’t show
    • Or, the second time you open the dialog, the TinyMCE editor doesn’t render, leaving a plain textarea
    • Or, the second time you open the dialog, the TinyMCE editor comes out blank.
  2. The Cancel button on my dialog fails on the second opening of the dialog OR the ‘x’ at the top right corner fails to work properly.

With my Firebug open, I tried to debug this issue.  I found out that somehow my original textarea’s inline css was set to “display:none” on opening the dialog the second time.  I thought I would be smart by resetting that value on open of the dialog; this didn’t work.  There was also a span introduced by TinyMCE directly beneath the textarea.  Removing it first didn’t work either.

After 5 days of debugging, I found a blog that illustrates a similar issue.  The link is here.

Before I go on…this is what my jQuery UI Dialog initialization looked like:

$('#email_body').dialog({
    autoOpen: true,
    modal: true,
    title: "Email Dialog",
    open: addTinyMCE,
    close: function() {
        $(this).dialog('destroy');
    },
    buttons:  {
        'Send': function() {...},
        'Cancel': function() {
            $(this).dialog('destroy');
        }
    }
});

SOLUTION:

Turns out that what I’ve been doing was trying to instantiate the TinyMCE editor on a hidden textarea hence, causing me hours of hair pulling.  Everytime the dialog is open, the TinyMCE needs to be re-initialized.  To prevent the textarea from being hidden before re-initialization, you need to make one small but VERY IMPORTANT change to the initialization code.

Change:

mode: "textarea"

to:

mode: "none"

This sets up TinyMCE without converting any fields to the editor. With this in place and since you ‘hooked’ the tinyMCE initialization to the textarea element with id=”email_body”, the text area is initialized properly to the contents of the textarea.  This solves problem 1 as stated above.

Next, you want to remove the TinyMCE instance whenever the dialog is closed either by clicking on the ‘x’ or by pressing the Cancel button.  Since the tinymce javascript file was imported when you use the tinyMCE jquery plugin, you have full access to the TinyMCE api (I didn’t know this initially).  To remove the instance, I created a function as such:

function removeTinyMCE () {
    tinyMCE.execCommand('mceFocus', false, 'email_body');
    tinyMCE.execCommand('mceRemoveControl', false, 'email_body');
}

Now you just have to call this just before you close or destroy the jQuery UI Dialog. This solves problem number 2.

I hope this clears up some doubts and solves some of your problems.  This worked for me, but if you know of a better way, please share it in the comments below this post. My final jQuery UI Dialog code is below:

$('#email_body').dialog({
    autoOpen: true,
    modal: true,
    title: "Email Dialog",
    open: addTinyMCE,
    close: function() {
        removeTinyMCE();
        $(this).dialog('destroy');
    },
    buttons:  {
        'Send': function() {...},
        'Cancel': function() {
            removeTinyMCE();
            $(this).dialog('destroy');
        }
    }
});

Sorting Python complex python lists

Today I required to sort a python list of tuples.  The normal way to sort a list would be:

list_of_courses = [(crse.id, crse.name, crse.start_time, crse.end_time) for crse in course]
sorted_list_of_courses = sorted(list_of_courses)

This didn’t work for me as I wanted the the tuples in the list to be sorted based on the start time of a course. One solution I found here was:

sorted_list_of_courses = sorted(list_of_courses, key=lambda cs: cs[2])

After reading further I realized that a more efficient way of doing the above was to import the itemgetter function from the operator module and use the function to reference the sort key of interest instead of using confusing lambdas:

from operator import itemgetter
sorted_list_of_courses = sorted(list_of_courses, key=itemgetter(2), reverse=True)

Adding a reverse keyword argument to the sorted function allows the list to be reversed hence sorting in descending order.

I found this to be really useful so I decided to log this down in my blog. Hope this helps.

Adding more test data in a rails migration

I was following the tutorial in “Agile Web Development with Rails (3rd edition)” and wanted to experiment with the examples in Chapter 6.  I wanted to add more test data into an already existing migration, so I thought I could just append the following:

1
2
3
4
5
Product.create(:title => 'Javascript: The Good Parts', 
    :description =>
    %{<p>If you really want a good explanation of JavaScript’s components this is the book to get.</p>},
    :image_url => '/images/javascript.jpg',
    :price => 47.99)

and save the migration file and run:

rake db:migrate

Sadly I was wrong. What was required was to either:

  1. Do a rake db:rollback first, make the necessary changes and run rake db:migrate again  OR
  2. Create a new migration by running ruby script/generate migration AddXXXtoYYY and run the necessary changes. (Preferred way)
Option 2 is the preferred way as stated in “1.3 Changing Migrations” here.