aakashd's blog

Thoughts and experiences of aakash dharmadhikari

[aakashvd]

To learn is to change

Mastery, George Leonard

[aakashvd]

If you run a services business, be it a restaurant or a consulting firm like McKinsey, visit one of these 60 nuggets every week. It's likely that you have done something recently that could have used the wisdom in that nugget.

[aakashvd]

You can't TRY to do things, you must simply do them.

Ray Bradbury

[aakashvd]

I just supported who if not WE,when if not NOW? on @ThunderclapIt // @ArvindKejriwal

[aakashvd]

Monthly finance meeting preparation

[aakashvd]

The test that I would like to inherit

To those developers, who are developing or will develop the codebases I will be inheriting in the future.

Inheriting a codebase with bad test-suite is a nightmare. 90% test coverage does not mean anything to me, if I can’t read and maintain the tests.

I want to talk about the perfect test, that I would happily inherit. First, I hope you follow Single Assertion per Test in spirit, if not in letter. Assuming that you do, here is how I would want you to structure my test:

https://gist.github.com/2718348

This is how I use it:

Context - It’s a way to group related tests together. With a context, I can find particular tests easily. Let’s imagine you inherit an e-commerce application, wouldn’t you prefer a context saying “payment settlement”, rather than find the relevant ones through a blob of 100+ tests?

Exact business requirement - A test title, “set status to false”, does not tell me anything I can’t get by reading the assertions. I would appreciate if the test title explains, WHY the code exists, rather than WHAT it is doing. The test body may not be able to explain something like, “a travel itinerary should be archived 3 months after it’s completed.”

<on doing this> - I would like to know the exact steps I need to reproduce the side-effect that we are asserting, nothing more. Having single assertion per test, will make sure that we don’t have bloated setups. This is critical for a maintainable test. 

The emotional energy taken to read & fix a failing test is exponential to the number of lines in that test.

<the assertion>- Apart from “Single Assertion per Test”, following simple readability rules like variable naming, avoiding long statements etc. can make my life so much easier. I would much prefer the second example over the first one:

https://gist.github.com/2718390

or even better

https://gist.github.com/2718428

That’s all I ask of you, focusing on these four aspects of every test. Make sure you write tests like this and I will love you forever.

If you liked this post, please consider…

following @aakashd on twitter

subscribing to this blog

[aakashvd]

Rails and Single Table Inheritance

I was introduced to Single Table Inheritance (STI), using Hibernate, about five to six years ago. I don't remember my exact thoughts, but I have pleasant memories of using it. I recently inherited a codebase that misused STI. Irritated, I sat down to write a blog post titled "Why I hate STI." But as I started writing down the reasons for not using STI, I realized that I hadn't re-evaluated the benefits of STI in the last six years.

Let me put myself in the shoes of the Java developer I was, six years ago. The basic assumption behind STI is, multiple classes having a strong is_a relationship will also have very similar fields; therefore the data for all these classes can be stored in a single table. From application modeling perspective, there is absolutely no benefit of using STI. We could easily store the records in different tables as well, but there are considerable benefits of extending the inheritance to database level. For example:

Ease of DB maintenance:

Here, we assume that the schema for all these classes will evolve together. If we have a single table, the db maintenance activities become easy. Like,

adding/removing columns

change in data-types

building/dropping indexes

writing triggers, foreign-key constraints etc.

Database performance:

Assuming we store the records for all types of payments in a single table, a query for "finding all the payments for an order" becomes a single table query. With a single table, the look-ups will be considerably faster due to the reduced number of joins.

Smaller DB interfacing code:

ORM's have certainly reduced the amount of SQL queries we write. Although, I used to write a lot more custom SQL's, than I do nowadays. With a single table containing data for multiple classes, I have to write/maintain a smaller codebase.

These are all the benefits I can think of, but let's see where these aspects stand in the ActiveRecord world. I am taking Rails/ActiveRecord as an example, but the analogy holds true for any modern framework:

Ease of DB maintenance: I have stopped writing triggers, foreign-key constraints etc. With Rails migrations, the maintenance cost has reduced drastically.

Database performance: Still holds good.

Smaller DB interfacing code: I can't remember the last time I wrote an SQL query. On second thought, I did write one about an year ago; but the point is, amount of DB interfacing code we have to write, has almost vanished.

To summarize, there is only one valid reason for introducing STI in your application: DB performance. Although, DB performance is a very important aspect, STI comes at it's own cost. More on the price you pay for this benefit, in my next blog post.

    If you liked this post, please consider...

following @aakashd on twitter

subscribing to this blog

[aakashvd]

Why you should respect deprecation warnings

I have worked with legacy Java codebases for nearly five years of my career and to be perfectly honest, I quite happily ignored deprecation warnings for the entire period. However, upgrading multiple Rails applications over the last couple of years, I've learned a healthy respect for deprecation warnings, especially in the Ruby ecosystem.

If there is one thing that Rubyists are not afraid of, it's removing backward compatibility. As developers are not afraid of correcting their mistakes in API design, the quality of ruby library APIs keeps on increasing. But this benefit comes at a hefty cost of code change for every upgrade. For any major version upgrade of a gem, I would end up braking a lot of functionality due to the API changes. If I upgrade my application once in an year, it's likely that I have 4-5 gems due for major version upgrades. The size of your codebase and the test coverage will determine exactly how long an upgrade will take, but it can vary from a handful of days to several months.

Many teams find such rework very expensive and if the test coverage is low, it's a common practice to rewrite the application than upgrade it. Wouldn't it be nice to get the benefits of newer libraries without this effort? There is only one answer, deprecation warnings.

Deprecation warning is a way for API authors to tell us about future API changes. If a method call or class is being deprecated, authors would flag usage of such entity with deprecation warning & alternate API suggestions. For every frequent minor upgrade, if you change the API usage as suggested, you will have a smooth major version upgrade. As the cost of development and QA is spread over a complete year, you would hardly notice this cost saving the costly rewrites.

Certain guidelines to follow here would be:

upgrade frequently; check every fortnight if any of the gems in your application can be upgraded

have your build fail on deprecation warnings

never commit code with deprecation warnings

The key discipline to follow here is, "upgrade frequently and never commit with deprecation warnings". You will be surprised, how easy it becomes to keep your applications up to date.

  If you liked this post, please consider...

following @aakashd on twitter

subscribing to this blog

[aakashvd]

Tips and tricks for a maintainable codebase

Recently, after inheriting a codebase that was difficult to read and maintain, I thought of creating a simple list of tips & tricks for developers. The list is very primitive, but just following these practices can guarantee a decent quality codebase.

If you can think of anything that I have missed, I would be happy to add that to the list.

Writing Tests

single assertion per test (ref: http://blog.jayfields.com/search?q=single+assertion)

tests should  be in their proper context (ref: https://github.com/thoughtbot/shoulda-context)

test name should describe the behavior being asserted

build should not give any warning

use libraries (like shoulda) to reduce the verbosity, e.g. should_not allow('invalid').for(:payment_type) rather than writing a complete test

Writing Code

method should not go beyond 7 lines (ref: http://c2.com/cgi/wiki?FewShortMethodsPerClass)

avoid static methods (User.find_by_name is a class method, User.format_date is a static method) (ref: http://blog.jayfields.com/2007/04/ruby-class-methods.html)

do not use magic strings/numbers. (ref: http://c2.com/cgi/wiki?MagicNumber)

do not pass around hashes. Identify the behavior associated with that hash and build appropriate objects.

Process

commit and push every 30 minutes. frequent commits => Less merge conflicts => less rework

never push failing build

never do partial commit. Partial commit is committing only part of the changes on your machine. The build may pass on your machine but not on others.

Some Reading Material

http://c2.com/cgi/wiki?SingleResponsibilityPrinciple

http://c2.com/cgi/wiki?LawOfDemeter

http://c2.com/cgi/wiki?SingleChoicePrinciple

Most Important Practice: Once in a while read your code aloud. If what you are reading makes a lot of sense to you, the developer who is going to read it 6 months down the line might be able to understand it.

  If you liked this post, please consider...

following @aakashd on twitter

subscribing to this blog

[aakashvd]

Should software professionals have Code of Ethics

I am not trying to say that money earning is unethical or immoral. But before I put forward my opinion let me cite few examples I have seen around me.

A freelancer works for 8 hours and creates a mediocre application for Rs.10,000/- (billing $27.77/hour) for a company. The company sells that piece of crap to a Bank for Rs.2,25,000/- (billing $625/hour).

A multinational bank needs simple log parser and web based reporting application working on top of the summary of data derived out of that parsing. Something pretty much possible with simple PHP script with normal desktop as well (and the developer knows it). This consultancy asks for a Dual Quad Core machine, 16 GB ram and develops code in Java so that it looks complex and charges Rs.2,50,000/- for it. The max effort with all beautification and setup is not more than a week. Billing rate $125/hour and am not even considering the infrastructure cost bank will incur.

A freelancer is given a job to be completed in 6 weeks, but its so simple that he completes it in 3 weeks. The company who hired the freelancer, warns him not to do such thing again, as it exposes their inflated estimates to client.

This one is worse - A business analyst tells developer that the application user interface should not look simple and easy to use. It should be very complicated and non intuitive, so as to look difficult. This would make them bid more cost as the product itself looks complicated and difficult to develop.

This is the worst - A company always leaves few known bugs in its products, so that they can charge for maintenance. Also the code is kept cryptic and undocumented (with the kind of developers they have, may not take any effort!) just to make it difficult for any third party to understand it.

Though I cannot tell the real names of developers and companies involved in these frauds (for obvious reasons), you probably know that these things happen. I have also come across colleagues who told me, that I am a moron and should never get into business, for this is the business. Apart from a gut feeling that these things are wrong, I have some serious concerns over how this affects IT Industry and the society as whole. Lets take a scenario and the ill effects it has.

This insane increase in the cost of software development increases the operational cost of our clients. So it shouldn't surprise us when we have to pay more for Hospital bills, pharmaceuticals, transportation, loan interest rates, rent etc. Also when the FD interest rate drops or the Mutual Fund NAV drops; we should applaud our contribution as well.  This list can go on, but crux is, that the inflation we cause will trickle down to us directly or indirectly.

Creating complicated software to charge more, is a subset of previous point but it has another aspect as well. The very essence of automation is to get rid of mundane tasks. If the applications we develop just creates another complexity, it again increases the cost of operations and future enhancements to the application. Such architectures will create information silos, creating big hurdles in the growth path of organization. I have seen a single company storing and maintaining millions of records of same data (pertaining to its business) at more than 5 places, cause the applications were never developed to talk to each other.

Might sound idealistic, but as an IT professional I see myself as a proponent of automation and seamless integration. But these practices would create mental and monitory apprehensions for the acceptance of technology and change in traditional industry. I think its time to change this, either by preaching or by direct action.

  If you liked this post, please consider...

following @aakashd on twitter

subscribing to this blog

[aakashvd]

Using Fleximage with rails to render images of various sizes

At any web site we have images with different sizes for different placements. For example size

50 x 50 for thumbnails

150 X 150 for medium sized images to be displayed in search results etc

400 x 400 for large sized images to be shown in slide show

One way is to process images immediately on upload, where we create all the required sized images synchronously with fixed path/filename. There are loads of ways to do it, so I wont go into it. But let me describe the problems that tag along this approach.

Assume a scenario, where we had decided 50x50 dimension for thumbnails and now we have 2 million thumbnails created over the period of 2 years. Good thing, and good time to revamp our design. As per new design, the images would need to be of size 75x75. OMG! What do we do now. Simple solution is to run a job to change all existing images, DB paths, code base to work with this new dimensions. Those who can imagine the scale would tell you, that its a nightmare.

Do I have a universal solution? NO. But I have implemented a simple, nearly as efficient and much more maintainable approach using Rails and Fleximage. How to use fleximage? go through the wiki. Here is the basic approach...

Let fleximage store the image as it wants (or how you want), the default path is the [./public/images/uploaded/yyyy/mm/dd] and the file name is image id in DB. But you can change these as you want.

In the Image controller when you receive request like /photos/1.png, you render the original image.

When you receive request like /photos/1_50_50.jpg, use the 50 x 50 to create the thumbnail. Sample code given below...

# GET /photos/1 def show idsplit = params[:id].split('_') image_id = idsplit[0] @photo = Photo.find(image_id) if idsplit.length > 1 idsplit.delete_at(0) render :inline => "@photo.operate { |p| p.resize '#{idsplit[0]}x#{idsplit[1]}' }", :type => :flexi end end

Believing that you have enabled caching for show action on PhotosController, the rendered image will be stored in your public folder in whatever cache location you have specified.

The next request to photos/1_50_50.jpg, would directly served out of your cache with performance hassles :)

Thats pretty much how I have done it. The ruby/rails pro guys out there, please pardon the code its taken from one of my spikes with Fleximage. About who makes the decision of, thumbnail for a product should be of 50x50, I have created helpers which give me path to thumbnail images on passing photo object. You can pretty much encapsulate any way you want.

Now when the thumbnail image size needs to change I will change only the helper method (as DRY as you can get), and the whole application works as it is. The new request photos/1_75_75.jpg will be treated as new request and will be processed once for the show action; thus creating a cached image of that size and name. We really don't have to worry about image migration. As per the cleanup of cache I haven't thought about it much, but I might as well clear them manually with a simple rm command.

[aakashvd]

Creating mocks where it matters

I have always had this complaint against the web designers around me. They would use 28 inch iMac and Adobe Photoshop to create the mocks; which are most often seen on 14 inch CRT in IE 6. Everything looks goody goody to them; but when it goes through our IDEs, few of the effects are not as per their expectations. The very definition of what's above fold and below fold changes because of the screen size. Recently I got answer to my second issue of screen size - browser size, another brilliant tool by Google. On providing your web page URL, it would show you a graph as to which part of your page is visible, to how much percentage of users (the dataset is Google visitors). This is useful for ensuring that important parts of a page's user interface are visible to a wide audience. I won't talk about it more, just check the about page. But take it with a pinch of salt, as for India centric site like ours the ratio of users with smaller monitors would increase a lot. About the issue of using Photoshop for designing web designs, I found a nice article by Meagan Fisher. It talks about why making your mocks in browser is the way to go. I would just summarise the benefits...

Clients can see how it looks in different browsers.

Client can check interactions like on hover, visited links behaviour etc.

In case of missing images or disabled JS, what's the impact on pages look and feel.

How the page would look while loading when part of the content is loaded (specifically image and flash components)

Designer gets to check the feasibility of his ideas

CSS is almost ready when mocks are complete

In all, I loved the idea and would certainly advice designers trying this approach.

[aakashvd]

how HTML class should be used - pure selector or CSS tags

I have always seen HTML attribute class used for tagging elements with css class name. To an extend that, I always believed the class attribute was meant to be used for CSS. Over the period, I started noticing something fishy about this class business. First was when I used the method document.getElementsByClassName(). If I am using class attribute for css, then why are we polluting the JS API with something which was meant to be for styling only. My styling changes and then my JS code is broken. That was my first interaction with the term selectors, and I got to know how CSS works. The classes we specify are just used as selectors by the browser, to give particular styling to the elements. So all this time I was using CSS, all I was supposed to be doing, I was just tagging elements. This brings me to next question. If I have a red header in left column, should I tag it as class="header left" or class="header red"? Both approaches have their pros and cons. Let me take the first one, which I will call purist.

Purist : The elements should be tagged based on their role in page and not by their styling requirements.

In case of three column layout, even if both left and right side column headers have same styling, the class names would still be class="left_column header" and class="right_column header" rather than class="header green". This sort of approach would increase the number of classes one needs to define in css file. Every element will be tagged to describe what that element stands for, and css along with other frameworks, uses these tags to identify the element to which style is to be applied. As said before, it will certainly increase the size of the css file, but at the same time makes maintenance very easy. In a scenario, where style of an element is to be changed, only the css file needs to be updated. If we had gone for the "green header" approach, we will need to access every page where this class is mentioned and change it to "red header".  Also where templating needs to be done, such approach is the only option.

Popular : Element is tagged based on the styling requirements and not on their position/role in the page.

Though this approach is never followed to the word, I have seen most of the sites following this approach for most of their pages. The usual explanation is re-usability. If 5 components on a page have same styling, why should there be 5 different classes having same styling, when one can do the job. Also with such styling, hacks are very easy. Developer will clear all inherited styles and give the element its own styling. This amazingly easy to envision and fast to develop approach would prove a maintenance nightmare when styling of one component needs to be changed across site. Now we will have to access all those HTMLs (PHP,JSP,ASP... whatever they are), and change the classes. Or when all headers need to be made light pink from green, a wise developer might decide to change the class green to contain font color pink. It was just to see how it looks; but high chances of this being stuck in code for years to come. And with all this, we haven't even spoken of different frameworks, like Javascript using these values as selectors for validation or dynamic DOM modification or any other magic. If classes were given, considering styling need only, application would prove very difficult for such upgrades. In all, I have a very clear view as to which approach I want to follow. Does any one have a different take on it?

[aakashvd]

Internet Exporer - missing keypress event for delete, end, enter, escape, function keys, home, insert, pageUp/Down and tab

Unlike(?) any good developer, I use Firefox exclusively for my app development rather than IE. I get webdeveloper and firebug at a click to solve all problems, and firefox works well with standard Javascript almost all the time. So here I was happily coding a functionality where I want to track `up arrow`, `down arrow`, tab and enter/return carriage key hits. Depending on the value in input box, I will decide what to do in the event handlers I have registered. Well half a days job, done in hardly 2-3 hours, feeling lucky. But alas, finally I RDP to windows machine (I use Ubuntu), to test the things on IE 6. Everything else works pretty fine till I hit enter. To brief you about the code, if I found value X in input box, I would redirect the page to some other URL and not submit the form, which otherwise would. Even on having X in the input box, the form was getting submitted. After some primitive checks, I realize that there was a flash of yellow in the lower left corner of IE. Ever heard of "IE returns true and submits form, on catching error in onsubmit event handler". The moron assumes that if there is error in onsubmit event handler, it should just submit whatever data it has. So I change settings to "display notification about every JS error", and IE 6 gives me some random line number. Some more time feeling helpless, and then I shift to IE8 for testing. Unfortunately this guy also gives error, but also tells me the line in the included js file it cant understand. From JS 1.3 specification, function object can be executed by a call function.apply(context,[args]). And in a use case, the args was coming as null. This freaked out IE, and it decided to submit the form without validation. No biggie, I fixed it by a simple null check. Now no error, but form still gets submitted. After putting few console.log() using Firebug lite, I find that when I press enter the event handler just doesn't get called. Check this post on quirksmode, "Explorer doesn't fire the keypress event for delete, end, enter, escape, function keys, home, insert, pageUp/Down and tab." After trying JQuery, prototype, I finally accept the reality and went for a hack. I put a clause in onsubmit event handler for the value of the input box. It hurt me putting such hacks, but finally the feature had to go live. I would be really happy if some one can tell me a better way of dealing with this issue. When can we really say RIP IE...

[aakashvd]

AVATAR - may be my expectations were more

I spent 300 bugs on AVATAR last night. Had heard a lot about the movie and specifically had great expectations from writer/director James Cameron. I am not a big fan of Titanic, but I have worshipped Terminator 2 : Judgement Day more than any other movie I can remember. Needless to say I was again expecting to get overwhelmed by the story and cinematography. I certainly didn't get the kind of kick, I got after watching Judgement Day, but I loved the movie none the less. I watched the movie at R-City in Ghatkopar, with 3D effects which were not at all effective. Rather I felt it reduced the clarity of certain scenes and was restricting me from getting completely immersed into story. I don't know if the problem was with print or the cinema infrastructure, but it resulted in less than perfect movie projection for sure. Coming back to story, the Pandora NAVI world was pictured very well and the difference between humans and Navis unfolds in a beautiful way. I would have given 10/10 on the story line, but I wish certain aspects of the Navi world would have been elaborated more. The initial interaction between the Navis and humans has not been shown. Talking about the animation, I couldn't see it being a next mile stone in movie industry. The facial expressions, moves looked more like animated movie than real life scenery. I found LOTR : Return of The King much better than AVATAR in this aspect. Action scenes are choreographed pretty good, but again apart from 2-3 scenes, nothing which would make me drop my jaw. So in all, what I liked most about the movie was the Story line. But I don't see any hook, where second film can be looped; other than next batch of humans coming back to the planet. I guess I will be watching it again to finalize my opinion about the movie.

[aakashvd]

MySQL group by and those 3 hours

It was my pure stupidity that cost me almost 2 days of head banging, when I used inner query and performed group by on the resultset of this query. I wanted to write a query whose out put would contain data/summary from almost 80 % of the tables and ranging to around 60 thousand rows. I wrote one big query and to avoid all tables being loading into memory, wrote few subqueries which would act as resultsets in the from clause. Just to give you idea - select x.*, y.*, count(z.A) from X x, (select ... from ... where ..) y, (select ... from .. where ..) z where ... having ... group by ... Any sane developer would crusify me for writing such nonsense, but I though as these queries would be more efficient as not complete database is being loaded into memory at a time. When we executed this query on production, the mysql temporary space had to be increased from 64 MB to 1 GB. It was after a great effort and few bashing from other folks I realized what was actually happening. As the inner queries were building huge resultsets (though less that actual tables, the number of rows itself was giant), and I was performing group by operation on it, the whole processing was happening on unindexed data. A pure suicide. Finally I wrote a script where all summaries were created on required tables, and put into temporary tables with appropriate indexes. The final query was on all these summary tables without any group by statement. The initial query which took 3 hours to execute was completed in 6 seconds.

[aakashvd]

URL rewrite should happen after/before authentication

URL rewriting is a very regular way of making your URLs user/seo friendly. But so is url based authentication. URL based authentication, allows us to specify url pattern and roles required for these patterns. Both these functionality are request filters, and obviously cannot be executed at the same time. Thus next question, which should be executed first. I saw ppl doing it either way, but without much justification. After thinking a while I decided to use UrlRewriter first and then the auth filter for following two reasons.

As my application evolves and I get richer data, my urls would change to encompass more data. Say I get more detailed geographical data, and I want to add that to my URL (obviously). I auth was to get executed before rewriter I will have to change both auth and rewriter. Thus keeping auth after rewriter would make my authentication impurvious to any changes to user facing Url structures.

On similar grounds, keeping UrlRewriter after auth would force us to mention both the original as well as rewritten URLs in auth configuration. Keeping only user facing urls in auth configuration is just little less than not keeping th authentication module at all.

So by all means without any confusion, URLRewrite happens before authentication.