Hacking Python

I was asked to help solve a problem today, involving checking the voter registration status of a database of voters against a web-form. This is not a totally straight-forward scripting issue as the web-form contains a randomly generated key and a number of other values that need to be included in the query. So, any program written would need to have a bit of smarts.

It was designed that way, don’t ask me why.

So, I thought to myself, this would be a good opportunity to learn some Python! Honestly, I have never written any Python before and I thought that since Python was a great scripting language to hack stuff together quickly, I should try to see if I could do this in Python. To make it a little challenging, I gave myself two hours to finish it (the length of free time I had while sitting at a local pasar malam this evening).

So, I started by looking up the official Python examples on how to form a HTTP query – straightforward and easy with Python. Then, I did some text processing on the return results to grep the various random keys. Next, I followed the official examples on how to perform a HTTP post. Finally, I used the official examples on how to process a CSV file and perform the checkup under a sub-routine inside a loop.

It took me just an hour to finish the script. I think that this is more of a reflection on the ease of use of Python, rather than my skills. I have just proven to myself that it is possible for me to pick up a new language and accomplish something useful with it in under an hour. Amazing.

Anyway, the code isn’t totally pretty but it is available at github.

Sassy Sass

I had a public holiday today and I ended up spending it on hacking some code. I decided to do a WordPress Theme for my company blog instead of lazing around. It was a fun thing to do on a lazy holiday – not very healthy but quite fun. In order to spice things up, I decided to take this opportunity to learn some Syntactically Awesome Style Sheet (SASS).

That is how I do every project – I make sure that there is something new to learn.

According to their website, “Sass makes CSS fun again. Sass is an extension of CSS3, adding nested rules, variables, mixins, selector inheritance, and more. It’s translated to well-formatted, standard CSS using the command line tool or a web-framework plugin.” Essentially, it is a meta-language that can be transformed into legal CSS using an external tool. On top of all the advantages stated, it can be used to automate the generation of complicated style-sheets.

One other advantage that it can deliver is eliminate the need for multi-file CSS that is currently used for organising CSS. SASS is like a programming language in many ways and can include stuff from other files, perform variable substitution and even evaluate some basic expressions and built-in functions. With all that, it can still generate a single compressed CSS file for the browser, which will reduce the number of loads needed.

Anyway, it was a fun project. I will convert the other subdomains on my company site later.

OpenSC and MyKAD

I recently bought myself a smart-card reader in order to fool around with the MyKAD (Malaysian National Identity Card). The main reason for this is to figure out how to read the information contained within the MyKAD. I went online to find myself an OpenSC compatible reader and got myself one for just under RM 23. Next, I found some information about MyKAD APDU on a Lowyat forum along with the associated data file offsets.

The result is the following shell script that dumps all the MyKAD information onto the screen. OpenSC automatically detects the MyKAD as an EMV compatible card.




opensc-tool -v -s $AP_JPN \
-s $DF_PAGE1 -s $PR_PAGE1 -s $SZ_PAGE1 \
-s $DF_PAGE4 -s $PR_PAGE4 -s $SZ_PAGE4 2>/dev/null

All it took was 8 lines of shell script and it would dump all the MyKAD information onto the screen, raw. All that is needed now is some extra processing magic to slice up the data into its constituent parts. Imagine, all this without a single like of C/C++ code and under RM25!

CakePHP and OpenID

I got the chance to integrate OpenID authentication into a CakePHP application that I am working on. I tried the OpenID component at first but kept running into troubles with the return_url not matching what was expected. After getting no where for a couple of hours, I found a simpler library to use – the LightOpenID library.

All I had to do was to load it using App::import('Vendor','openid'); and that was it.

The library is pretty straight forward to use and even the examples given were more than enough to get it working. However, I ran into another problem very quickly. Some OpenID worked while others did not. After some investigation, it turned out that it was due to the use of mod_rewrite that caused a url to have two “?” in it. To fix this problem, I forced the return_url to have a “?” in it that made the OpenID provider use a “&” instead.

That was it!

So, the return_url was something like:

End of File

I’ve found that many novice programmers are confused when it comes to reading files. I had to educate an apprentice today on this subject and I thought that I would put down a short summary of my lesson today.

To most people, there are many different types of files out there – videos, images, music, text and what nots.

All these do not mean anything to programmers. A programmer is only concerned with two types of files – text and binary. A text file is delimited by a carriage-return on Unices and a carriage-return line-feed on Windows. A binary file has no delimiter. Therefore, a text file is read in line by line while a binary file is read in blob by blob. In addition, it is possible to terminate a text file with an end-of-file marker (^Z) but that would not be possible in a binary file.

So, when trying to read a text file, it is possible to read in line by line until an EOF is encountered. However, there are a few different ways to read a binary file.

It is possible to keep reading the file in blobs until the number of bytes actually read is less than the size of a blob. This indicates that the end of file has been reached. This is the preferred method of reading binary files. It is also possible to detect the size of a file and keep track of it with a counter until it is decremented to zero. This requires careful management of the size counter and limits the size of the file to the maximum value for an integer.

Software Engineering

A colleague of mine sent me an article by Tom DeMarco, one of the pioneers of structured analysis and also a strong believer in software engineering processes. Previously advocating metrics, metrics and more metrics in the past, he has come to realise that software engineering is rather a misnomer. In one sense it is engineering but in another sense, it is not. After decades of invaluable experience, he has come to one conclusion:

For the past 40 years, for example, we’ve tortured ourselves over our inability to finish a software project on time and on budget. But as I hinted earlier, this never should have been the supreme goal. The more important goal is transformation, creating software that changes the world or that transforms a company or how it does business.

Since I have also been writing code for a couple of decades since my humble beginnings with LOGO and BASIC, I have to say that I am of the opinion that it is very difficult to control the process of software creation – and it is a creative process, no doubt about it. However, as an engineer, I do believe that metrics is useful but only after the fact. What I mean to say is that metrics are only useful in documenting failures.

It is silly to try to control software creation during its inception and conception. You just need to hire the best people, give them the best tools and then hope for the best. Project managers who try to micro-manage the project will invariably fail because of the nature of the metrics used – they try to attribute success to certain values. Unfortunately, the success of a software rarely depends on the number of lines of code maintained, nor does it depend on the number of faults found.

After a project has come to a finish – and when I say finished, I mean that the people involved have come to a unanimous decision that they are happy with the state of the product at the time – that is when software metrics can be used to measure certain things. For example, it could be useful to measure individual contributions to the code base and identify good managers. It could also possibly measure the number of significant changes made to base code.

Anyway, I think that the video below is as good a metric as any for measuring software quality. I like the fact that the contributors seem to come in waves. I wonder if it correlates with real-world events in any way.