Sijin Joseph's blog

What will be the next generation internet application platform?

Sijin Joseph — Tue, 24 Jun 2008 14:45:00 GMT

A few years ago I was a firm believer in the Rich Connected Client application model, which was based on running applications installed locally on the users desktop. From the time of the Ajaxian explosion, the quality and quantity of Ajax based web applications has continued to increase, applications like FaceBook have introduced new paradigms whereas apps like Live Maps have made existing apps much more convenient and accessible. Today you have to really argue hard to even consider a desktop based application for anything that is non-computation intensive (Even this category is questionable now, for e.g. a few years back movie editing web apps would have been out of the question). So what is it that makes the web such a successful application platform

Uniform and simple model (Web Browser, urls, can click when hand is visible) - Once a user learns the basics of working with a web application that knowledge can be easily applied to other applications.
Client platform independence - The decoupling of the server and client with an agreed contract (HTML+CSS+JS) means that the traditional problems of targetting various platforms with different APIs is no longer existent on the client side.
Machine independence - The user is no longer restricted to the machine on which the application was installed. This also results in a much simpler deployment model.
Data independence - The user's data is now available on the network which means that not only can the user run the application from anywhere but can also access his data from anywhere.

Now what would the next generation internet application platform look like? I think that in addition to the above characteristics, the next generation of platforms would involve the following.

Full use of computing resources available locally - Having a powerful CPU and GPU seems like such a waste when all your applications have to be funnelled through the browser. So the next generation platform would allow access to the computing power available locally.
Better integration with the local resources - This is sort of related to the point above, but would allow internet applications to access local disks, settings, registry etc.
Better security model - Of course all this has already been attempted with ActiveX and XPCOM, but the security models there have been weak and non-intutive to users, a better solution is needed.

So it looks like the direction being taken by Microsoft Silverlight and Adobe AIR are steps in the right direction to building the next generation internet application platform. However Microsoft has a great oppurtunity here push the envelope with Silverlight and introduce new standards for desktop integration of internet applications, their extensive user base means that any API created by them has a very good chance of being successful and catching on with the other players in this space.

Subversion as a deployment tool

Sijin Joseph — Mon, 05 May 2008 15:05:00 GMT

I was thinking on the way to work today that subversion would be a great tool to overcome some of the difficulties associated with frequent deployments to the web serevers. Here's how I see it working

Create a production/live build folder in your source tree and add it to the repository.
Modify our build system to create the live builds in this folder and commit to the repository.
On the live server the site is deployed as a checkout of the live build folder.
Once the build passes unit tests and QA all we need to deploy is to update the working copy on the live server. The big advantage here is that rollbacks etc. are automatically handled because we can always roll back to a previous version. Also you get a nice history of all the updates to the live server.

Programmer Competency Matrix

Sijin Joseph — Wed, 30 Apr 2008 16:49:00 GMT

Having worked with programmers with an extreme variance in skills, I sometimes get the feeling that there is an big lack of good programmers but when I thought about it a little more I realized that it's not very clear cut, some of the programmers have strong areas and if you confine the tasks into their strong areas then they tend to deliver well. So I started thinking about all the lines on which we can evaluate a programmer, here's what I have so far...

Programmer Competency Matrix (the table is too big to fit on this blog post and needs a whole page of it's own)

After having spent a whole afternoon on this I realize that even this is not comprehensive, this matrix is more biased towards non-visual programmers, so a big majority of web devs will not be able to relate well to this matrix, but I am tired and will come back to this at a later time.

An alternative model of computation for concurrency

Sijin Joseph — Sat, 12 Apr 2008 12:11:00 GMT

I recently came across an old article that I had written for my company newsletter, it's always fun to discover old stuff that you've written and see how much your perception has changed since then. Copied verbatim below, this was a print article which is why the links are not hyperlinked.

Concurrency is one of the hot subjects in computer science today. This has partly to do with the fact that processors (1) are reaching their physical limits and thus we need to start looking at new avenues of achieving performance. Herb Sutter the renowned author has written an excellent article (2) named “The Free Lunch Is Over - A Fundamental Turn Toward Concurrency in Software” which captures very beautifully why concurrency is going to be important in the years ahead.

Let us look at a few issues surrounding concurrency today and also look at an alternative model of computation that solves these problems.

The most prevalent model of computation in both hardware and software today is the Von Neumann architecture (3) which is an implementation of the Turing (4) machine which in turn is based on the work of Alan Turing(5). Programs written in this model have a shared memory area also known as the store which can be to store data. The store is divided into chunks called cells and named cells are called variables.

A function written in the Neumann model can make use of the store to aid in its computation, thus any time a function uses data other than those provided as parameters to the function, it is in fact using the store.

The alternative model of computation we'll look at is called the functional model. The functional model is based on the work of Alonzo Church (6) called Lambda Calculus(7), who came up with this model at approximately the same time as Alan Turing. The good thing though is that both models of computation are equivalent i.e. any computation that can be expressed in one model can also be expressed in the other.

The functional model is based on mathematics, in this model there is no store and all computation is done by evaluating mathematical functions. A mathematical function is one in which the value of a function is totally dependent on the values of the parameters and not on any external state. Also the term variable in this model refers to a named value and not to a named cell.

Consider a simple function
f(x) = g(x) + h(x)

The function f takes x as a parameters and returns the result of evaluating function g with parameter x and adding it to the result of evaluating the function h with parameter x.

In Neumann style programs it is possible that f will return a different output for the same value of x, where as in functional style programs it is guaranteed that no matter how many times f is called with a particular value of x, the output will always be the same. This is because functional programs do their computations without the use of any external state whereas the computations in Neumann style programs are affected by the state of the store.

Let us understand this with a concrete example
int y = 2;
foo(x)
{
    return x *y;
}

bar(x)
{
    int y = 2;
    return x * y;
}

The function foo uses a variable y from the free store to do its computation, thus if some other function were to alter the value of the cell y, foo would start returning different results.

On the other hand bar does not use any external state and for any value of x it will always return twice of x.

It is possible to write functional programs in Neumann languages by totally avoiding the use of store and modeling the entire computation using only functions that use local variables. On the other hand functional languages do not have the concept of a shared store so writing Neumann style programs in them is not possible.

Now let's take a look at how these computation models interact with concurrency. There are two ways parallelize programs, implicitly and explicitly.

In implicit parallelization, the compiler does the hard work of looking at the program and deciding what instruction sequences can be parallelized. This is the most commonly used method of parallelization and in fact most of the programmers are not even aware that the compiler does this under the hood.

Neumann style programs are hard to optimize for parallelization since functions can be implicitly dependent on each other via the store. For e.g.
baz(x)
{
    foo(x);
    return bar(x);
}

int y = 0;
foo(x)
{
    y = 2 * x;
}

bar(x)
{
    return y * 6;
}
In this case foo and bar need to execute sequentially because they make use of the cell 'y' from the store. This is an extremely simple example but hidden dependencies like this can get incredibly complicated to figure out. Thus the compiler has to be conservative about what it can parallelize and only parallelize code that it absolutely knows is safe to do so which unfortunately is not a lot.

On the other hand since functions written in a functional language are not dependent on external state they can be very easily parallelized by evaluating different functions in parallel.
For e.g.
baz(x)
{
    foo(x);
    bar(x)
    combo(foo(x));
}
Since foo, bar and combo are purely mathematical functions, the evaluation of baz can be parallelized by evaluating foo and bar in parallel and then evaluating combo, since combo depends on the value from foo.

Now let’s turn our attention to explicit parallelism, where the programmer explicitly programs parallelism. This is done by creating parallel execution sequences called threads that are scheduled by the operating system to execute on the available processors.

In the Neumann style the problem that arises with explicit parallelism is that since all threads share the same common store they need to be careful not to overwrite the data of other threads. The canonical way to handle this has been to but checks around code that accesses shared data to ensure that only a single thread has access to shared data at any point of time. This blocking behavior in turns leads to issues like deadlock where all threads keep waiting for access to a shared location on the store. Further the complexity of this solution increases exponentially with the number of threads, number of shared data areas and the number of places where the shared data is accessed.

A better solution to this problem is actually doing the lock on the memory itself rather than on the code that accesses the memory. Transactional memory systems (8) have started making their way into mainstream computing but they're still a long way off from being an ideal solution.

In 1978 C.A.R Hoare(9) (best known for inventing QuickSort) wrote a paper titled Communicating Sequential Processes (10), this classic paper introduces a new way to tackle the problem of explicit parallelism, which is in a way similar to the model used by functional programming languages. In this model concurrent processes share state information by passing messages to each other rather than sharing a common store. These messages can be passed asynchronously or synchronously. Thus the whole problem of preventing access to shared data is made non-existent.

There is one language today that combines aspects of functional programming and message passing concurrency to come up as an ideal distributed computing platform. That language is Erlang (11) from Ericsson laboratories. This language and platform came about as the result of research in 1980s to find out which features of computer languages were suited for programming telecommunication systems. In addition to concurrency, this language has some other cool features like error recovery and hot updates that enable any system built using Erlang to keep on running. Joe Armstrong, the brains behind Erlang has written a very good article (12) on all the fuss about Erlang and I would encourage anyone who is interested to take a look at it.

I had originally intended for this article to be an introduction to Erlang but ran out of my 1000 word limit even before I could get past the introduction, thus I decided to pick up the one big feature of Erlang which is concurrency and focus on that. In terms of support for reliability and concurrency, Erlang is unrivaled and I believe that in the years ahead some aspects of Erlang or Erlang itself will become mainstream the way has Ruby has.

References
1.    www.cise.ufl.edu/research/revcomp/physlim/PhysLim-CiSE/PhysLim-CiSE-5.doc - Physical limits of computing.
2.    http://www.gotw.ca/publications/concurrency-ddj.htm - The free lunch is over.
3.    http://en.wikipedia.org/wiki/Von_Neumann_architecture - The Von Neumann architecture.
4.    http://en.wikipedia.org/wiki/Turing_machine - Turing machine.
5.    http://en.wikipedia.org/wiki/Alan_Turing - Alan Turing
6.    http://en.wikipedia.org/wiki/Alonzo_Church - Alonzo Church
7.    http://en.wikipedia.org/wiki/Lambda_calculus - Lambda calculus
8.    http://en.wikipedia.org/wiki/Transactional_memory - Software Transactional Memory
9.    http://en.wikipedia.org/wiki/C._A._R._Hoare - C.A.R. Hoare
10.    http://en.wikipedia.org/wiki/Communicating_sequential_processes - Communicating Sequential Processes
11.    http://www.erlang.org/ - Erlang
12.    http://www.pragmaticprogrammer.com/articles/erlang.html - More about Erlang
13.    http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=10142 - Concepts, Techniques, and Models of Computer Programming – This is an amazing book that talks about the various models of computation.

First alternative to google that I actually liked

Sijin Joseph — Fri, 28 Mar 2008 11:20:00 GMT

SearchMe - Still in beta and I was able to get an account easily. Check out the screenshot of their search results

Not only are the results categorized but I loved the visual search results that don't require me to open the page in another window to further filter interesting results.

How to leak memory in .Net - Statics and Event Handlers

Sijin Joseph — Fri, 03 Aug 2007 06:43:00 GMT

For the past few days I’ve been investigating some memory leak issues in our desktop application. The problem started showing up when we saw that opening new documents and then closing them didn’t have any negative impact on the memory usage. Initial tests using vadump and process explorer confirmed that there was an issue and so we the developers started looking into it.

Initially it looked like the problem was that certain event handlers were causing references to closed documents to hang on a couple of ones that I remember are Application.Idle and SystemEvents.PowerModeChanged. Next there were some references via event handlers that were being held by Singleton objects and some service objects and those were easily handled as well.

After this we could see that the references were still hanging around, btw we were using a combination of ANTS profiler, WinDBG (dumpheap + gcroot) and the VS.Net debugger all this while to investigate the issue. After fixing the obvious issues we struggled to find the root cause of the memory leaks. Then I looked around for alternate profilers and came across .Net Memory Profiler from SciTec, using this gave a much clearer picture into the issue, you see the new profiler gave you allocation stacks for all references and the ability to reflect over the instance fields, using this I started seeing that two third-party components that we were using were causing the issue.

Basically both third-party components, one a very well known UI toolkit and another one that provides skinning support to controls were storing references to controls in static hashtables. In one toolkit a bug in the code caused the reference from the hashtable to remain even after it was not required and in the second case I think those guys just didn’t know how to remove an entry from a hashtable, they were simply setting the value of the key to be null causing a reference to the key to be held by the static hashtable.

We now need to hack around these issues either by using Reflection or getting an updated build from the vendors.

Some of the lessons I’ve learnt from this

Always, Always have source code for any third-party component that you are using in your application. For any non-trivial usage you’ll always end up fixing bugs in the component.
When putting any data in static fields, double check to make sure that it’s really required and keep in mind the memory impact of the decision also provide a clean API to clean up the static data.
If your’re hooking onto an event then make sure to unhook when it’s no longer required.

.Net framework hotfix wreaks havoc

Sijin Joseph — Wed, 18 Jul 2007 12:46:00 GMT

Last week all of us were baffled when suddenly one part of our application that uploads files to a FTP server stopped working. The strange thing was that the same build has been working without any issues for the past one week. We looked at everything that could have gone wrong, server, configuration, code but everything was setup fine and hadn't been changed. Also interestingly it stopped working for everyone except the developer who was responsible for the feature.

The first thing we did was to enable detailed logging to see what was happening, the logs showed two problems

We were incorrectly formatting the path of file to upload
The .Net framework code was changing folders after login to the root folder of the ftp where it didn't have permissions to upload the file

Further investigation showed that the second issue was not coming on the developer's machine. Most puzzling indeed.

Then I remembered that last week .Net had issued a critical hotfix for .Net 2.0, could this be the issue. We verified that the developer didn't have the hotfix and all machines which were failing did have, Strike 1! Next we uninstalled the hotfix from one of the machines and the FTP uploads started working, Strike 2!! Finally we fixed the incorrect formatting of the ftp url and the issue got resolved on all machines with or without the hotfix, Strike 3! Issue resolved!

The problem was that the hotfix changed the implementation of the FTP code inside the .Net framework so that it behaved differently when passed an incorrectly formatted url.

This was the first time I saw a working app fail because of the way an incorrect argument was handled by a newer version of the framework. It was a good learning experience though :) Also this strengthens my belief in asserting all assumptions in code because if we had asserted that the url was infact of the format that we were expecting, this issue would never have happened in the first place.

In defense of hacking

Sijin Joseph — Mon, 09 Jul 2007 05:44:00 GMT

I read a very interesting essay today - Hacknot - To Those About to Hack
that talks about why planning upfront always pays in the long run. There is a very nice story that illustrates the value of planning upfront.

I think that when people write essays like this they tend to provide an analogy that suits the point that they’re trying to make, for e.g. in this case Pro BDUF and Agile bashing.

There are a couple of reasons why the analogy is not quite relevant in this case. Firstly software is not like chopping wood, it’s not like construction infact any comparison that tries to compare software development with any kind of physical object creation is flawed. Physical objects have limitations with respect to the time and effort required to shape them or construct them. The values of these physical constants are irrelevant when it comes to software and in some cases the physical limitations do not exist at all.

Secondly the requirements in almost every software product that I’ve worked on always change after the initial code has been implemented, first because the customer/user is himself not very clear on what is required. It’s quite difficult to describe a large state machine for a CS grad let alone a layman. Also usability design itself is an iterative process for for a product with a UI the requirement churn rate is absurdly high when compared to any physical engineering activity.

So given that requirements are bound to change doesn’t it make sense to practice the one thing that you know for sure is going to happen i.e. change. No amount of planning is going to prepare you for change, you need to practice for change day in and day out by following the training regime of agile methods. TDD, pair programming, daily meetings, refactoring, rejection of BDUF etc. these things prepare the programmer for the inevitable.

I can imagine a version of the story that favors the Agile camp in which the carpenter in the middle of the day decides that he does not need a big log of wood at all! But the fact that wood chopping is a physical activity again prevents me from going ahead on that analogy.

What they don't teach you in CS class

Sijin Joseph — Sat, 26 May 2007 10:58:00 GMT

Software Engineering!!!

A scientist builds in order to learn; an engineer learns in order to build. - Fred Brooks in the Mythical Man-Month.

Following up on my post about the need for a CS degree for programmers, I had started writing this post on how software engineering requires a different set of skills than what is required for a computer scientist. But then I saw that most of what I wanted to say had already been very well captured by a lot of other very famous people, so instead of reiterating, I’ll be posting links to some good reads on this topic.

But before that here is a quick summary of what I think are the most important skills for a programmer, which has somethings in common with those required by a computer scientist but also some that are not.

Given a system, have a very good understanding of it’s rules. The systems that a programmer typically works with are the language, the OS, the implementation platform(Java, .Net, Python etc.) and libraries. This knowledge is essential when writing code as well as when debugging issues. Most good programmers have encyclopedic knowledge of the systems that they’re working with, one of the best examples that I can think of is Raymond Chen.
Be able to come up with efficient ways to get a particular task done using the rules of the system. I think this is something that you’re born and although this can potentially be learnt, I think the best programmers have an innate talent for this aspect of programming. Some common techniques for solving problems are taught in CS class, but the ones most used in reality are mostly based on common sense. One of the most excellent books that I’ve read on abstract problem solving is “How to solve it: Modern Heuristics” by Zbigniew Michalewicz and David B. Fogel
Be able to express their thoughts in a manner that can be easily understood by other programmers. This aspect is something that can only be learnt by experience. This is one area that is very important yet gets very little weightage in CS class. I’ve seen some extremely unreadable code, that when deciphered showed extraordinary problem solving ability. For examples, browse some of the solutions submitted by top rankers at TopCoder.com. One of the best books on this aspect and my recommendation as a first book for any programmer is Code Complete by Steve McConnell
Be a good problem solver, this includes having related abilities like systematic elimination of possibilities to reach a solution, hypothesis testing to narrow down causes etc. This again is something that you’re born with and can potentially be learnt to some level. Best book on this aspect that I’ve read is Debugging Applications by John Robbins of NuMega, although this book is windows specific, some of the chapters that deal with debugging strategies and techniques to prevent bugs are invaluable.
Use your experience to prevent mistakes. This is another area about which very little is written but you can easily make out professional code by the way in which bugs are fixed. Newbies tend to fix the bug at the point of its happening and that’s it, a professional on the other hand thinks about what caused this kind of error to get introduced in the first place and then puts in checks to ensure that similar kinds of issues don’t enter into code and if they do then get flushed out immediately. Also when faced with similar types of problems, good programmers are able to look at the meta problem and come up with reusable solutions for them.
Experience, nothing can compare to having written and maintained 1000000+ lines of code.

And here are the links on the CS vs SE question…

Programmer's tools

Sijin Joseph — Mon, 21 May 2007 14:30:00 GMT

Here are some of the tools that I currently have installed on my dev machine. Most of them are freeware or have free versions available.

IDEs

Visual Studio 2003 and 2005 - The essential IDE for any professional windows developer, I only have C# and C++ installed btw to keep resource usage to a minimum. The Visual Studio series are some of the best IDEs I have used, everything can be easily customized and the whole product has great automation and command line support, for e.g. instead of searching for a file to open in the Solution Explorer, you can simply open the command window and say “open <filename>”, you even get autocomplete help for all commands and filenames. If you don’t have Visual Studio 2005 professional, you can get the freely available Visual Studio Express editions which are good enough for most purposes.

DevMetrics - Allows you to generate project statistics like LoC, Cyclomatic Complexity etc.
GhostDoc - Fantastic add-in that generates comments.
TestDriven.Net - Great support for Test Driven Development without ever leaving the IDE.

Eclipse 3.2 - Great IDE for Java development. One of the things I disliked about it is the number of configuration options you have to make to customize a basic install. An alternative that has better defaults out of the box is NetBeans 5.5 but I found that to be a bigger resource hog than Eclipse and finally uninstalled it.

Aptana - This is the new kid on the block, targeted at web developers, this IDE provides great support for HTML, CSS and JS editing and debugging. The latest milestone 8 release incorporates a Javascript debugger that is integrated with firefox and also works with FireBug. Another awesome feature is the content-assist(intellisense) for javascript files. In fact the content-assist is so cool that for core js methods it actually shows the DOM level compatibility and supported browsers as well.

Editors

Notepad++ - I like Notepad++ for primarily for its visual appeal and keyboard shortcuts that are more or less like VS.Net, which I find quite comfortable to work with. Also this is the one editor I use most frequently for editing plain text. This rocks as a Notepad replacement.
PSPad - This is a great programmer oriented editor, it has tons of functions that you can use to make editing code easy, like code explorer, hash generator, lorem ipsum generator, autocomplete, char codes, inbuilt hex editor, file comparer, macros and tons more. Also PSPad has excellent multifile and project support.
Komodo Edit 4.0 - Stripped down version of Komodo IDE, but still has the most useful functions of editing and intellisense for dynamic languages like Perl, PHP, Python, Ruby and Tcl; plus has support for browser-side code including JavaScript, CSS, HTML and XML. I found the intellisense for JS, CSS and HTML to be lacking when compared to Aptana.

Debugging

Microsoft Debugging Tools for Windows - Includes WinDbg and the console debuggers, useful for some low level windows debugging.
OllyDbg - I use both OllyDbg and WinDbg , OllyDbg has a more intuitive UI, but WinDbg is more powerful in the commands that it provides, not to mention that for .Net debugging there is no other alternative than WinDbg

System monitoring

SysInternals tools - One of the first tools I install on a new system, awesome set of utilities from Mark Russinovich and Bryce Cogswell, Mark is one of the authors of the best book on windows internals, aptly named Windows Internals. Process Explorer is an essential replacement for Windows Task Manager, Autoruns to get rid of those pesky startup items and a whole host of other things that delay startup, and process monitor to monitor registry and filesystem activity of a process are the top 3 tools from this suite that I use.
Microsoft Network Monitor 3.0 - Sort of like ethereal but with Microsoft branding, great for examining low-level network traffic to debug communication issues.
Fiddler 2 - Awesome tool from Microsoft to intercept HTTP traffic.

.Net

Reflector - Absolutely essential for a professional .Net developer, decompiles .Net assemblies and has a host of wonderful plugins.
Snippet Compiler - A nifty little tool to quickly test out code snippets, without creating a full fledged VS.Net project.
Regulator - Essential companion to any sort of regular expression work, initially written by Roy Osherove.

Image Editing

Paint.Net - I use this as my standard image editor, Paint.Net replaces GIMP which I used to use a couple of years back.
IrfanView with plugins - Best tool for image viewing, has tons of plugins and the base application allows gives you the most information possible about an image and has excellent support for the most basic of image manipulation functions.

Browsers and Add-ons

Firefox - Standard browser.

FireBug - One of the best ff add-ons for web developers. Supports DOM and CSS manipulation on the fly which really cuts down on the development time.
Javascript Debugger - Great for debugging javascript
Web Developer toolbar - Another useful web developer centric add-on

IE 7.0 - Can’t help it.

IE Developer toolbar - MS’s answer to Firebug, more limited in functionality though.
Tamper IE - For intercepting HTTP GET and POST requests. Great for security testing.

Opera - For testing, love the speed dial.

Utilities

SlickRun - Extensively used to launch most tasks that I do on a daily basis. This is a small window that floats on your desktop, which you can use to launch applications and commands. Very lightweight and stable.
Launchy - A recent addition, but this has now become my fav and most used util, this is like spotlight on the Mac OSX if you’ve used it, basically allows you to quickly launch applications, documents and files. I’ve never had to open up my programs menu since I’ve started using this. A must for every power user.
WinMerge - For resolving conflicts and for checking file differences.
Stickies - A free desktop post-it like notes tool. Recently released but quite good. I was using TurboNotes free edition before switching to this.
Clean Sources Plus - I use this on almost a daily basis to clean out my vs.net code tree with all unnecessary files and folders like .suo, .user, bin, obj etc.
Foxit PDF reader 2.0 - This is seriously fast, makes you wonder what the Adobe people are upto.
xPlorer2 Lite - An excellent replacement for the standard windows explorer, has tab support, multi-panes and tons of useful utilities.
WinDirStat - For finding out where your hard disk space is going.
ClipX - This is another one of those small utilities that make life so much easier. This basically works like an extended clipboard and remembers everything you ever copied to the clipboard, including images. highly customizable and works with all applications, simply press Ctrl+Shift+V(default) and up pops the last 25 items you had copied.
Startup Delayer - It took me a while to understand the concept of this, but it really works. Basically it allows you to time when each of your startup applications will launch after the machine starts up, also it shows a small progress bar at the top, which allows you to pause or stop the startup of the tasks if needed.
Daemon Tools - For loading all those ISOs into a virtual CD-ROM drive. Has support for SafeDisk and other copy protection mechanisms if you’re into gaming.
Microsoft Virtual PC 2007 - I use this to have a clean WinXP system, where I can test out various things.
VMWare Server - A truly professional Virtual machine application. I found this to be less resource intensive than VPC and also the guest image has better performance when run on VMWare as compared to VPC.
CamStudio - For making flash demo movies of your application.
WinRar - My preferred archiving tool.
7-Zip - Another archiving tool, I use this mainly for commandline scripting of archiving actions.
CCleaner - Cleans out crap from your system, like temp files, invalid registry entries, browser caches etc.
FolderShare - For sharing files between you own PCs and keeping them in sync.
KeePass - Allows you to keep all your secrets safe, I use this to store all my online passwords. Has great support for USB based operations as well so that you can carry all your important passwords on a USB.
Windows Install Cleanup - For cleaning up those corrupted installations. If you ever run into windows installer errors because of a corrupted or unfinished installation give this tool from Microsoft a try.
IcoFX - Icon editor.
WinSnap - Screenshot taker.
Gadwin PrintScreen - Another screenshot taker. Predecessor to WinSnap.
Safe Add-Remove programs - A really fast alternative to the windows add/remove programs. The only downside is that you need to be careful with Microsoft product installs, since they tend to be spread out across multiple entries when seen through this tool.
w.Bloggar - What I am using to write this post.
TortoiseSVN and TortoiseCVS - For accessing source of the various internal and open source code repositories.

Misc

NSIS - For whipping up a quick windows installer.
Atmosphere Lite - Listen to natural sounds and relaxing waves, great for concentration while working.
Foobar 2000- Lightweight and excellent music player from one of the developers on Winamp. Great for listening to music while working.

Make sure you know what you're fixing

Sijin Joseph — Fri, 18 May 2007 16:13:00 GMT

I’ve been working on this bug for the past one week, basically a call to the GDI+ APIs MeasureString and DrawString was failing with a very useful exception “A generic GDI+ error has occured” , my initial hypotehesis was that the problem was coming because of the length of the string that we were trying to measure, at that time around 100000+ characters. In retrospect this now seems like such a foolish thing to hypotheize.

So anyways the bug kept bouncing between me and the tester who did a great job of coming up with new scenarios to cause the crash and we came up with all sorts of complicated rules about how we should trim the length of the string etc. Then today the tester got a crash scenario which was dependent on the number of newlines in the string, this really got me thinking and I realized that my original guess about the length of the string being a problem could not be an issue because so many of the controls display text in excess of 100000 characters without any issue and all of them internally use DrawString.

So then I started testing out my guess that it was the newlines in the string that was causing the issue and this did in fact turn out to be the root cause. The GDI+ MeasureString API makes my 2GHz laptop behave line a Pentium Pro when asked to measure a string containing ~7000 newlines and at 8000+ it starts failing with the generic GDI+ error.

I think I learned an important lesson today that spending some time thinking about the root causes of issues and doing a simple analysis can go a long way in making sure that a bug that is once fixed does not pop up again in some other form.

A new kind of adsense scam?

Sijin Joseph — Wed, 16 May 2007 08:55:00 GMT

I was reading this article today A Few Secrets You Need to Know about Client-Side Scripting which I picked up from joel.reddit.com

The odd thing is that the site is not a blog, its just one page that someone has setup on tripod, I then remembered reading similarly formatted sites yesterday and for sure they were there in the hot list of joel sub-reddit, check them out

http://zisweenews.tripod.com/things-to-consider-before-redesigning-or-redeveloping-a-website.html
http://kjctyynews.tripod.com/the-web-effect-the-characteristics-of-a-web-website.html

These sites only have this one page and even the usernames seem like fakes. So it seems that someone is setting up these one off pages and then getting them to the top of the ranking in reddit, virtually guaranteeing traffic. The only reason I can think of for someone to do this would be to make money from google adsense.

Also this could be a test-bed before attempting to try the same on Digg.

Is a CS degree required to make a good programmer?

Sijin Joseph — Wed, 16 May 2007 02:54:00 GMT

I’ve been thinking about this for some time now and wanted to jot down my thoughts, but this post Hackers and Fighters by Mark Traver captures beautifully my feelings on this subject.

There are a few things which are different in Indian universities which I’ll talk about in another post.

Is Ruby "The One"?

Sijin Joseph — Mon, 14 May 2007 10:11:00 GMT

I got back to application development using Java after spending the last 4 years developing applications using .Net and C++ and things have really changed!!! I have spent the last one month trying to catch up on all the new stuff in the Java world. Updating myself with the latest changes to the syntax only took one day though and I was surprised to see that in the latest incarnation Java 5 and .Net are like twins. If you don’t believe me then read this.

So after updating myself with the platform changes I started going through the technology stack, it’s been a month and I think I have only scratched the surface of whats out there and I’ve been only looking at enterprise application development technologies, it actually took me a week to review all the web application development options! So far here’s what I’ve looked at, as in, read through the reference manuals, installed and played around a bit, Hibernate, JPA, JDBC, Spring, Struts, EJB, Velocity, Tapestry, WebWorks, Cocoon, Log4j, JBoss, Tomcat, JNDI, JMS, NetBeans, Eclipse, Jetty, Glassfish.

Today while I was trying to read the official docs for Java EE, I spent half an hour on the Sun site trying to decide what I wanted to download, did I want Java SE 6 or Java EE 5 Well after finally finding what I wanted, I went for a coffee break and for some reason the dialogue between Neo and the Architect from “Matrix Reloaded” kept popping up in my mind. I think the connection I was making is that the Java eco-system has become extremely complex, not unlike the stage C++ had reached before the advent of Java, at that time Java was “Neo”.

So who’s the new “Neo”? I really can’t predict, but it’ll be simple to use, have less configuration, it just works defaults, easy to get started, eventually have good performance and most importantly have a passionate user community, that will keep improving on it and ultimately cause it to become the next Java . Looking at todays options, I think “Ruby” seems to meet this criteria the most. You never know 5 years down the line we could all be thinking about how it took weeks to setup a simple application using the Java platform and how going through all those xml files and annotations gave you a headache.

I’ve included the entire dialogue between Neo and the Architect below, substitute the Matrix for Java and Neo for Ruby and njoy!

My favourites, keeping in mind the above context

The problem is choice - Hell Yeah!!!!
Denial is the most predictable of all human responses. - Which is what a lot of people who read this will be feeling
As you are undoubtedly gathering, the anomaly’s systemic - C++ and now Java

The Architect - Hello, Neo.

Neo - Who are you?

The Architect - I am the Architect. I created the matrix. I’ve been waiting for you. You have many questions, and although the process has altered your consciousness, you remain irrevocably human. Ergo, some of my answers you will understand, and some of them you will not. Concordantly, while your first question may be the most pertinent, you may or may not realize it is also irrelevant.

Neo - Why am I here?

The Architect - Your life is the sum of a remainder of an unbalanced equation inherent to the programming of the matrix. You are the eventuality of an anomaly, which despite my sincerest efforts I have been unable to eliminate from what is otherwise a harmony of mathematical precision. While it remains a burden to sedulously avoid it, it is not unexpected, and thus not beyond a measure of control. Which has led you, inexorably, here.

Neo - You haven’t answered my question.

The Architect - Quite right. Interesting. That was quicker than the others.

*The responses of the other Ones appear on the monitors: “Others? What others? How many? Answer me!”*

The Architect - The matrix is older than you know. I prefer counting from the emergence of one integral anomaly to the emergence of the next, in which case this is the sixth version.

*Again, the responses of the other Ones appear on the monitors: “Five versions? Three? I’ve been lied too. This is bullshit.”*

Neo: There are only two possible explanations: either no one told me, or no one knows.

The Architect - Precisely. As you are undoubtedly gathering, the anomaly’s systemic, creating fluctuations in even the most simplistic equations.

*Once again, the responses of the other Ones appear on the monitors: “You can’t control me! F*ck you! I’m going to kill you! You can’t make me do anything!*

Neo - Choice. The problem is choice.

*The scene cuts to Trinity fighting an agent, and then back to the Architect’s room*

The Architect - The first matrix I designed was quite naturally perfect, it was a work of art, flawless, sublime. A triumph equaled only by its monumental failure. The inevitability of its doom is as apparent to me now as a consequence of the imperfection inherent in every human being, thus I redesigned it based on your history to more accurately reflect the varying grotesqueries of your nature. However, I was again frustrated by failure. I have since come to understand that the answer eluded me because it required a lesser mind, or perhaps a mind less bound by the parameters of perfection. Thus, the answer was stumbled upon by another, an intuitive program, initially created to investigate certain aspects of the human psyche. If I am the father of the matrix, she would undoubtedly be its mother.

Neo - The Oracle.

The Architect - Please. As I was saying, she stumbled upon a solution whereby nearly 99.9% of all test subjects accepted the program, as long as they were given a choice, even if they were only aware of the choice at a near unconscious level. While this answer functioned, it was obviously fundamentally flawed, thus creating the otherwise contradictory systemic anomaly, that if left unchecked might threaten the system itself. Ergo, those that refused the program, while a minority, if unchecked, would constitute an escalating probability of disaster.

Neo - This is about Zion.

The Architect - You are here because Zion is about to be destroyed. Its every living inhabitant terminated, its entire existence eradicated.

Neo - Bullshit.

*The responses of the other Ones appear on the monitors: “Bullshit!”*

The Architect - Denial is the most predictable of all human responses. But, rest assured, this will be the sixth time we have destroyed it, and we have become exceedingly efficient at it.

*Scene cuts to Trinity fighting an agent, and then back to the Architects room.*

The Architect - The function of the One is now to return to the source, allowing a temporary dissemination of the code you carry, reinserting the prime program. After which you will be required to select from the matrix 23 individuals, 16 female, 7 male, to rebuild Zion. Failure to comply with this process will result in a cataclysmic system crash killing everyone connected to the matrix, which coupled with the extermination of Zion will ultimately result in the extinction of the entire human race.

Neo - You won’t let it happen, you can’t. You need human beings to survive.

The Architect - There are levels of survival we are prepared to accept. However, the relevant issue is whether or not you are ready to accept the responsibility for the death of every human being in this world.

*The Architect presses a button on a pen that he is holding, and images of people from all over the matrix appear on the monitors*

The Architect - It is interesting reading your reactions. Your five predecessors were by design based on a similar predication, a contingent affirmation that was meant to create a profound attachment to the rest of your species, facilitating the function of the one. While the others experienced this in a very general way, your experience is far more specific. Vis-a-vis, love.

*Images of Trinity fighting the agent from Neo’s dream appear on the monitors*

Neo - Trinity.

The Architect - Apropos, she entered the matrix to save your life at the cost of her own.

Neo - No!

The Architect - Which brings us at last to the moment of truth, wherein the fundamental flaw is ultimately expressed, and the anomaly revealed as both beginning, and end. There are two doors. The door to your right leads to the source, and the salvation of Zion. The door to the left leads back to the matrix, to her, and to the end of your species. As you adequately put, the problem is choice. But we already know what you’re going to do, don’t we? Already I can see the chain reaction, the chemical precursors that signal the onset of emotion, designed specifically to overwhelm logic, and reason. An emotion that is already blinding you from the simple, and obvious truth: she is going to die, and there is nothing that you can do to stop it.

*Neo walks to the door on his left*

The Architect - Humph. Hope, it is the quintessential human delusion, simultaneously the source of your greatest strength, and your greatest weakness.

Neo - If I were you, I would hope that we don’t meet again.

The Architect - We won’t.

Notes on the software build process

Sijin Joseph — Sun, 06 May 2007 15:45:00 GMT

The build is the pulse of any software development activity and a good build system facilitates quality software development. My current thoughts on a build process are as follows

Build Machine

The build machine is a dedicated physical or virtual machine whose sole purpose is to build your product. It should not be used for development or QA activities. Here are some points to keep in mind when setting up the build machine.

Use a physical or virtual build machine depending on how often you plan to build and whether the build is initiated manually or automatically. A virtual machine has the advantage that it can do the build, copy the files to a file server and then it can be reset to it's initial clean state. Also a virtual machine does not require an expensive machine that will be idle most of the time. On the other hand if the build is initiated automatically or happens multiple times during the day, it is best to use a dedicated physical machine.

Another possibility is to use one Virtual Server that hosts build machines for multiple projects, this way the physical hardware is used optimally.

Keep the number of softwares installed on the build machine to a minimum.
Should have ample storage space, if the builds will also be archived on the build machine itself.
Build machine should be fast to keep build times low. This is especially important in C++ like projects where typical build times can be 30+ minutes. Also having a fast build process means that the time from a checkin to a build pass/fail result is minimized thus ensuring that time is not wasted waiting for the build results.
Have some sort of remote desktop capability installed on the machine. This proves to be invaluable when you need to do emergency builds from home at night.

Process

The build process is more than just a compilation of the code that is done on a developer's workstation, a good build process is able to generate the final set of artifacts that is sent to the user in one step. For e.g. a build process for a desktop application would build the installer, documentation, licenses etc. for the entire product, in case of a web application the build will compile the code files, supporting assets and deploy to a test or staging server. A good build system should be configurable and able to build multiple editions of the product.

There are a lot of options available for build tools, ranging from the simple make files and batch files to continuous integration systems like CruiseControl, in between are the new age build systems like Ant, NAnt, MSBuild etc.

Here are some tasks that need to be done when creating a build system. Not all of them will be relevant to all projects and there will be some steps that are specific to individual projects. Also the specifics on how to achieve the tasks will be different based on the build tool you use.

Clean up/Create the folders where the project output will be generated.
Get the latest sources from source control.
Update the build numbers in code and documentation.
Tag the code in source control.
Build the code.
Run automated unit tests.
Run automated functional tests.
Run automated regression tests.
Build the documentation.
Generate release notes
Build the installers.
Copy the artifacts to a folder for archiving or to the machine that is used for archiving the builds. Make sure that the folder are named appropriately.
Deploy the product to a test server.
Send out build status email.
Clean up and temp files and folders.

The following sections talk in detail about some of the tasks mentioned above

Build numbers

The build number should be of the format major.minor.patch.build (Note that this is slightly different from the .Net recommendations)

major and minor revision numbers are generally decided by the product marketing. The patch and build numbers are generally used by the engineering team to identify the product.

The build number should be a monotonically increasing sequence across your mainline code and is independent of the major and minor version numbers. For a branch the build number should remain constant and the patch version should increment with each labeled build, the labeled builds on a branch would usually correspond to updates to already released products, which is why the major minor and build numbers remain the same.

For e.g.

Mainline - 1.0.0.100, 1.0.0.101 ... 1.5.0.125 ... 2.0.0.180

In the above case the build number starts off with 100 and each subsequent labeled build of the mainline has the next higher build number, this remains true even as the major and minor version numbers progress to 1.5 and 2.0

Branch - 1.5.0.125, 1.5.1.125, 1.5.2.125

In the above case we create a branch for the 1.5.0.125 build which is a release branch. Any subsequent bug fixes/minor changes to this branch get labeled with increasing patch numbers while the major, minor and build numbers remain constant, this makes it clear that the builds are just patch updates to the release.

There are many alternatives to the above scheme for build numbering, one of them is major.minor.build.patch, the problem with this one is that the user facing version number of the product major,minor and patch are not contiguous, in case of major.minor.patch.build, we can simply drop off the last part of the version string and get the user facing version string major.minor.patch

Tagging code

If a labeled build is being made as opposed to a test build, then the code should be tagged in source control. This usual convention is to just tag the code as Build_ for e.g. Tags/Build_1.5.0.125 When a build is being released to the customer, then in addition to the tag, a branch should also be created for the build on which any subsequent bug fixes and minor changes can be made without affecting the mainline. for e.g. Branches/Release_1.5.0.125

Build the code

When building the code, consider building both the Debug and Release versions of the code, the Debug versions are useful when you have to debug a particular version of the product at a later stage. Although the code can always be retrieved from source control, it can be time consuming in case the code base size is big.

Automated tests

If possible we should run automated tests as soon as the build is made, a failing test should result in a failed build. Most build tools have integration with unit testing tools like NAnt, Ant etc. also batch files and other custom executables can always be used to invoke the test scripts.

Building documentation

If the product includes documentation then it should also be built along with the build, further the documentation should be labeled as well indicating the version of the product for which the documentation is written. Again standard tools like Microsoft Help Compiler, NDoc, SandCastle can be used to generate the documentation.

Release notes

Release notes should be automatically generated for each build, this can be done by either pulling in the list of changes from the source control or by pulling in the task ids that have been implemented since the last build from whatever issue tracking system you are using.

Building installers

For most products a installer will be required that will deploy the product on the users machine/server. Some of the options for building installlers are

Installshield - Very costly and bulky.
Wise - Less expensive and not so bulky.
NSIS- Free, but not as many features as Installshield and Wise
Wix - Microsoft has open sourced the toolkit that allows generation of MSI files from xml.

Whatever tool is used, this should be integrated into the build system so that as a result of a successful build the installer is created as well.

Build archiving

Every labeled build that is created should be archived on some machine so that it can be used at a later stage for testing and debugging.

References

Some good references for build processes and configuration management