Thinking Aloud

Mahesh Gattani | Jet.com | Associate Director of Engineering

[methinking]

Moving to Medium

I will be moving my blogging efforts to Medium going forward. You can follow them here https://medium.com/@maheshgattani

[methinking]

The rise of Functional Programming languages

When I was starting college, the language used to teach the first computer science class was SML or Standard ML. It was a simple but powerful language which was strongly functional. But unfortunately I did not know what a functional language was back then. Soon the classes moved to the more main stream imperative languages like C, C++ and Java and I ignored the functional programming paradigm. Fast forward a years of working and functional programming is hard to ignore. 

It is not the latest rage in programming but definitely an on-going development. Functional Programming languages are being adopted at an ever increasing pace and are considered part of  "modern languages" at times. Functional programming has its roots in lambda calculus and Lisp. It was mostly academic for a long time and only in last decade has it began getting used in commercial software development.

Functional programming is a programming paradigm. Hence a lot of languages which are not designed to be functional first can be used to implement functional concepts. Languages like Scala live in the area between functional and imperative languages. F#, for instance, lives on the other extreme and strongly pushes for functional paradigm. All the most profound languages, C++, Java have been getting functional features in their latest upgrades. 

Functional programming has a few primary features. First and foremost being functions as first class citizens. That usually translates into being able to define functions anywhere in code and being able to pass them around as you would any other value.  For example:

def addTwo(number): return number + 2 def addFunction(number, fn): value = fn(number) return value print(addFunction(5, addTwo)) # returns 7

Second biggest feature for me would be pure functions. Pure functions are functions without any side effects. A simple example of no side effect vs side effect would look something like this:

class Number: def __init__(value):   self.value = value def addTwo(number): number.value = number.value + 2 def addTwoNoSideEffect(number): value = number.value + 2 return Number(value)

Functional programming languages are usually also accompanied with higher order functions out of the box. This usually makes most code bases smaller and that is also one of the reason the industry is seeing a movement towards functional languages. For instance, if I want to multiply all the numbers in a list by 2:

nums = [1, 2, 3, 4] #imperative for i in range(len(nums)): nums[i] *= 2 #functional nums = map(lambda x: x*2, nums)

Such higher order functions are also a reason why some of the functional languages are used for scientific purposes in terms of data manipulation and churning.  All these come together to make a powerful combination. But there are some drawbacks too. It is important to know the pros and cons of a choice before making it. To simply note, the no side effect way of doing things can lead to temporary high memory usage. If that is not allowed for your application, maybe if it has extensive memory constraints, then functional programming might not be the correct choice. There are other systematic choices, some handled by compiler optimizations and hence not that impact full and some are more obvious as mentioned above. 

I have been working with functional programming lately with a combination of Scala, Java8, Python and F#. Some are more functional than others but all have the feature set to allow for functional first code. For any developer, I highly recommend checking out functional programming even if your language of choice is not functional first. As everything, knowledge only helps guide our future decisions. 

[methinking]

Event based system design and Event sourcing

When you are working on server side systems day in and day out, you often encounter a problem of wanting to make two network calls, but atomically. For example, if you want to update the last active time stamp of a user on any action made by them, you might have to update two tables. Or if you want to write to a database and a queue in an atomic operations.

The solution to this dilemma is usually an event driven system. Such a system usually first writes to a queuing system followed by a set of consumers. Most modern queuing systems, like Kakfa or RabbitMQ allow for multiple consumers for the same queue. This simple setup can allow for a distributed design where every micro-service consumes from the queue and does one thing well. It can be updating one table or summarizing and producing another message for a different queue. One benefit of such a design, sometimes thought of as asynchronous design, verses a synchronous request-response design, is that a failure does not lead to data loss. As long as the queue is reliable, the event will stay on the queue and any failure on the micro-service will simply lead to a retry. This coupled with monitoring of micro-services for exceptions can lead to a pretty stable system. The big caveat being a stable queuing system. Taking that design to an extreme. If we can get a very stable, scale-able and fast queuing system, we can indeed use it as a main database. This is the idea behind Event Sourcing and Command Query Responsibility Segregation (CQRS). CQRS has a lot more theory to it which I recommend reading but event sourcing tl;dr is, the queuing system is the main data store and the consumers can create the aggregations or snapshots to surface the latest states. The snapshots can be saved in whichever database we need to surface the data best. It can be a key value store or a relational database or a document storage or an in memory database, anything works. The reason for the flexibility is that data store can be re-hydrated from the main queue system whenever needed. It might be expensive to do so and there are work arounds for it but the basic idea has great merit. The event driven system design pattern is widely used and I am seeing event sourcing being picked up as well. But it is important to remember that this is not a silver bullet. This pattern works great for the maintainability, scalability and reliability of a system but it comes with trade offs. The first question you need to answer before using this pattern, can your system be asynchronous. Is it allow to do delayed processing or does it have to provide the side effect of a request immediately. Event sourcing and event driven programming works great for asynchronous programming. Event sourcing is gaining momentum and hence the tool sets around it as well. Kafka can be used to persist snapshots. Azure CosmosDB has a streaming feature and so does AWS DynamoDB. I highly recommend reading more about event driven programming and event sourcing if you are interested in system design. Once a system grows beyond a certain size, there is usually no way to avoid them.

[methinking]

Getting back to methinking

I have been busy with professional and personal stuff for a while now and blogging has taken a back seat. Planning to get back on it!

[methinking]

Building a scalable communications platform with queue consumers

Birchbox is on a mission to help people all over the world discover new beauty and lifestyle products. Our tech team is working hard to make this mission a reality. This requires us to connect with our customers frequently to surprise and delight them. As our user base increases, there are scalability pains, maintainability pains, and reliability pains.

Communication with users via email is an integral part of any ecommerce company and the same goes for Birchbox. In addition to email notifications about purchases, shipments, and deliveries, we also feature monthly specials and features like Sample Choice. To address this while getting important email stats like open rate, bounce rate, etc., we use a third party email service to send our emails. The issue with using a third party is that we need to make sure our users still get those important emails even when the third party service is down. Sending emails is usually an asynchronous task. A few minutes here and there are acceptable as long as they don’t wait beyond a certain threshold. To deal with these reliability issues, we decided to go with a retry solution using the Python retrying library. Something like:

import random from retrying import retry @retry def do_something_unreliable():   if random.randint(0, 10) > 1:       raise IOError("Broken sauce, everything is hosed!!!111one")   else:       return "Awesome sauce!" print do_something_unreliable()

Retrying is a great library. It allows for configurable delays, configurable maximum retries, and retry logic based on exception type, definitely worth looking into. However, the issue with retrying in code like this is that you need to sleep to induce backoffs. Much like how it’s done in the Python retrying library. At various points during our shipping cycle, the volume of outgoing email increases and this sleep-and-retry can cause a bottleneck which ultimately results in failed messages being dropped on the floor.

Queues are coming! Queues are coming!

To fix this, we wanted a system that allowed us to configure non-blocking delays between retries. We started with queues and queue consumers. A system that needs to send an email sends an event to the queue and the queue consumer consumes the event as soon as possible and does the actual sending. The issue with the reliability of third parties is still there but now since things are happening asynchronously we can do something about the failure and still give users a good experience.

We chose RabbitMQ as our queue system. It’s based on AMQP, proven to be scalable and allows for the kind of robust message delivery we were looking for. A couple of additions that RabbitMQ brings to AMQP are DLX (Dead Letter Exchange) and TTL (Time To Live). DLX allows a queue to throw a message to a preconfigured exchange if the message has expired, been NACK’ed (negative acknowledged), or the queue is full. A TTL can be set to allow for message expiration on a per queue basis as well as a per message basis. A big caveat here is that a message is expired only once it reaches the head of the queue. That means a message with a long TTL in the front can stop all the other messages in the queue from expiring.

Using DLX and TTL, we found a solution to our problem. To understand the solution, a basic understanding of RabbitMQ system is required. We encourage checking out RabbitMQ documentation.

Not the easiest of diagrams to follow, we know. Here is how it works in more detail:

The idea is that there are exchanges which receive messages. Queues subscribe to these exchanges with a routing key. Any message coming to the exchange with that key is routed to the subscribed queue. Consumers subscribe to the queues, get messages, and process them. Queues can have a dead letter exchange associated with them as mentioned earlier. RabbitMQ allows creation of exchanges and queues on the fly from consumers.

Say our configurable retry array is [1000, 2000] in milliseconds. This would mean that after the first failure, we wait one second. After the second failure, we wait for two seconds. In the case of more than two failures, the message can be dropped or handled with a hard failure queue to be monitored manually. In the diagram, the PQ (Primary Queue) subscribes to PX (Primary Exchange) using RK (Routing Key) and also a couple more routing keys derived from the retry array. This allows failed messages to roll back into the system seamlessly.

If the consumer fails to process a message because of a third party failure, it will push the message to a waiting exchange with waiting queues subscribed to it. Waiting queues are subscribed with routing keys to make sure all the messages in their first retry go to the first waiting queue and for second retry go to the second queue. Since in RabbitMQ we can create exchanges and queues on the fly, we can create as many waiting queues as we want from the consumer using the retry array and create bindings between waiting exchange and waiting queues.

Messages being posted to waiting exchanges are assigned a TTL based on how many failures occurred and waiting queues are assigned DLX to the PX. This way we induce the message back into the original flow once the timeout expires in the queue. From there on, the message starts following the original course. To figure out which failure it is, RabbitMQ helps us by adding specific error info in the message header which can be read by the consumer of the primary queue.

Waiting queues will have no consumers. They are just meant to allow the messages to wait for a certain amount of time before being re-queued to the primary exchange and become part of the original flow.

All these steps can be seen in action in the Python code here.

This system allows us to configure the number of retries and the delay between each retry. With exchange and queue creations being done on the fly, the configuration can be updated any time and the code can be written to adapt to any configuration. We use a separate queue in case we can’t process the message even after maximum number of retries. This queue is to be looked at manually and fixes made before messages are requeued again.

This is our way of making sure we reach all our customers and provide them with all the surprise and delight we can.

You can also checkout the blog at UnBoxed

[methinking]

Tornado and RabbitMQ

For my initial dive into Tornado and asynchronous programming, check this out. 

Let’s talk RabbitMQ this time, Tornado and RabbitMQ are like match made in heaven. They go together even better if you don’t care about delivery confirmations. 

Here is a Tornado application to post stuff to RabbitMQ that can scale up to tens of thousands of open connections by never blocking the main thread. It’s all event driven, just how Tornado likes it.

Here is what the app looks like

import tornado.ioloop import tornado.web import json import pika_publisher import argparse import pika import yaml PRODUCER_CONFIG_TAG = "producer" RABBITMQ_CONFIG_TAG = "rabbitmq" class PublishHandler(tornado.web.RequestHandler):  def initialize(self, publisher):    self.publisher = publisher  def post(self):    data = json.loads(self.request.body)    self.publisher.publish_message(data)    self.write(data) class PublishApplciation(tornado.web.Application):  def __init__(self, publisher):    handlers = [      (r"/test", PublishHandler, dict(publisher=publisher)),    ]    tornado.web.Application.__init__(self, handlers) def main():  # Parse arguments, especially config file location.  parser = argparse.ArgumentParser(description='Publishes a payload to rabbitmq')  parser.add_argument('--config_file', type=str, required=True, help='Path to the config file. Example: /etc/test-config.yaml')  args = parser.parse_args()  config = yaml.load(open(args.config_file))  producer_config = config[PRODUCER_CONFIG_TAG]  rabbitmq_config = config[RABBITMQ_CONFIG_TAG]  credentials = pika.PlainCredentials(rabbitmq_config["username"], rabbitmq_config["password"])  parameters = pika.ConnectionParameters(rabbitmq_config["host"], rabbitmq_config["port"], rabbitmq_config["virtual_host"], credentials)  publisher = pika_publisher.PikaPublisher(parameters, producer_config["exchange"], producer_config["exchange_type"], producer_config["queue"], producer_config["routing_key"])  application = PublishApplciation(publisher)  application.listen(8888)  ioloop = tornado.ioloop.IOLoop.instance()  ioloop.add_timeout(500, publisher.connect())  ioloop.start() if __name__ == "__main__":  main()

[methinking]

Lessons in Tornado and MySQL

Tornado is python web framework. If you don’t know about it, you should check it out. It’s one of the easier frameworks to use, if you know what you are doing.

Tornado, as the documentation says, is great for applications that require long lived connections to the user like long polling. Tornado becomes great for these applications by using non blocking I/O. 

The main thread in Tornado accepts connections so we need to keep that thread free as much as we can for Tornado to scale as it’s meant to be. This is great when the logic in your application is minimal or you extensively depend upon network based call. In case of I/O, you can initial the I/O and register a callback. This would be what people call event driven programming. 

All this is great but where tornado is not so great is libraries to support event driven programming for basic technologies like MySQL and memcache. That makes it very important to look at what your application will do now or in future before you choose Tornado as your web framework.

In case you do choose Tornado and MySQL is your bottleneck. There are things you can do using python’s asynchronous programming and latest versions of Tornado even help you write asynchronous code looking like it’s synchronous. But if you are stuck to version of python like 2.6 or 2.7, here is my shot at allowing asynchronous MySQL logic. It uses  a pool of workers to whom you send a query and a callback, which is called once the query returns. 

The app looks something like this

import adb import tornado.ioloop import tornado.web import sys from tornado import gen from adisp import process import json class Connect(object):  adb = None  @classmethod  def _connect_mysql(cls, conn_type='ro'):    prefix = '%s_' % conn_type    if cls.adb is None:      cls.adb = adb.Database(driver="MySQLdb", database="test_db", user="user", password="password", host="localhost")  @classmethod  def _get_adb(cls, conn_type='ro'):    if cls.adb is not None:      return cls.adb    else:      cls._connect_mysql(conn_type)      return cls.adb @process def fetch(sql, params, retry=False, conn_type='ro', callback=None):  adb = Connect._get_adb(conn_type)  data = yield adb.runQuery(sql)  if callback is not None:    callback(data)   class MainHandler(tornado.web.RequestHandler):    @tornado.web.asynchronous    def get(self):      fetch("select count(*) from test_table", {}, callback = self.callback)    def callback(self, results):      self.write(json.dumps(results))      self.finish()       application = tornado.web.Application([    (r"/", MainHandler), ]) if __name__ == "__main__":    application.listen(8888)    tornado.ioloop.IOLoop.instance().start()

[methinking]

SQLiteDBArray: Using PHP interfaces.

I have been itching to write this one for a long time.

The idea behind this post is to go through PHP interfaces and see some of the cool things we can do with it. I just wrote something in a few hours so that I can use it as an example.

Checkout https://github.com/maheshgattani/SQLiteDBArray.

Its a simple, in memory, indexed array implementation in PHP using SQLite DB. I may be updating it later to enable associative arrays, more complex data types, file saves, serialization and other addons. What the first cut show is how easy it is to make your own data structure act like an array and the convenience you get by doing so.

I recommend reading about ArrayAccess Interface, Iterator Interface and Countable Interface. These interfaces when implemented, allow your object to act like an array. What I mean by 'act like' is, you will have access to elements its using [] operator. You can use for and foreach on your object and you can call count on the object to find out the number of items it is storing.

The use case: Write a wrapper on SQLite DB allowing array like access.

Benefits: It becomes plug and play alternative for arrays. It can act like a cache which is capable of SQL like queries. It can do all the complex logic SQL queries can, seamlessly. All this in the same class.

Drawbacks: You need to know how you are going to use this array. It is not as dynamic as a PHP array. For example you have to specify the name and type elements you elements you expect upfront. You can only use basic types integer, float and string. No nested arrays. So its beneficial only if you know the use case.

How do we do it?  By extending SQLite3 and implementing ArrayAccess, Iterator and Countable Interfaces in PHP. Its simpler than it sounds. I invite you to take a look at the Github repo. About 150 lines of codes do it. What stands out for me? The ability to swap the array with the new class seamlessly. Plug and play.

Example usage:

$db = new SQLiteDBArray(array('name' => SQLiteDBArray::$TEXT, 'age' => SQLiteDBArray::$INTEGER, 'address' => SQLiteDBArray::$TEXT, 'salary' => SQLiteDBArray::$REAL)); // Example writes $test = array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 15, 20); foreach($test as $t) { $db[$t] = array('name' => 'testName' . $t); } // Example of count var_dump(count($db)); // Example reads using for for($i = 0; $i <= 20; $i++) { var_dump($db[$i]); } // Example reads using foreach foreach($db as $key => $value) { var_dump($key); var_dump($value); } // Example unset var_dump($db[1]); unset($db[1]); var_dump($db[1]);

[methinking]

Play! Framework, Scala and Code coverage

One of the problems with using relatively new languages is the lack of tools around white box automation, unit testing, code coverage etc. These things pile up your technical debt over a period of time.

Play framework and scala do have a good unit testing tool set with ScalaTest, Specs2, Mockito and more. Though I was not able to find a great tool for code coverage. There are tools which do the job, but most of them also profile the play framework code which skews up the results. The best option I found is scct (Scala Code Coverage Tool).

Here is a step by step guide for the least intrusive way to include code coverage in a scala play project and your continuous integration setup. This if for a single project. For multiproject builds, consult scct site.

Add the following code to you plugings.sbt file.

resolvers += "scct-github-repository" at "http://mtkopone.github.com/scct/maven-repo" addSbtPlugin("reaktor" % "sbt-scct" % "0.2-SNAPSHOT")

Update your Build.scala to have these hooks

val scct_settings = Defaults.defaultSettings ++ Seq(ScctPlugin.instrumentSettings: _*) val main = play.Project(appName, appVersion, appDependencies, settings = scct_settings).settings(..

With this, you have the required tool in place for code coverage. Just restart your play project so that the new dependency you have in your plugin file is downloaded. Now run 'scct:test'. If you have unit test cases in play, a report will be generated and put in ../server/target/scala-x.xx/coverage-report/index.html. Use Mozilla/Firefox to open this report and you can see your code coverage.

If you use Jenkins, you can integrate this with your build process enabling you to produce code coverage reports every build, fail the build if code coverage goes below a certain percentage of code and stuff.  To do this, we use Cobertura Plugin in jenkins. The great thing about scct is that it creates a report in cobertura format as well, which enables us to use cobertura plugin in jenkins. 

To start, go to Jenkins->Manage Jenkins and install Cobertura plugin. Then go to the project you want to have code coverage integrated in and click Configure->Add post-build action. You can find this drop down at the bottom most part of the page. Choose Publish Cobertura Coverage Report. This will add a few new things to your build settings. The Cobertura xml report pattern will be something like this "**/target/scala-x.xx/coverage-report/cobertura.xml". Rest of the settings are pretty self explanatory. 

Now run a build and you should see a code coverage report on your Build page. And we are done! :)

[methinking]

Play! Framework and request response times

Play! Framework is a high velocity web framework for Java and Scala.

Its a nice framework. Provides you clear MVC, stateless and web friendly architecture. I have been using it for last few months and we at Lucidchart utilize it extensively. I don't want to go into details but for interested people,  you should try and use Scala and find out what new it bring to the table. Play gives you all that and makes kick starting anything easier.

What I want to discuss here is a problem we faced recently. I wanted to have a dashboard where I can see the request serving time of each of my API end points. The use case is pretty common. This information comes in handy when your system behaves weird. It comes in handy when optimizing your web service. It even comes in handy to see the bottle necks of your system. Well I can go on, but the idea is simple. Its a useful matrix and we should be collecting it.

To implement it, I started goggling what people have been doing to get this data out of play. What I found was not good, there is no straight forward or recommended solution. Digging deep, the easiest and non usable way I found was AccessLog module in play. The module, as the author correctly states, is not production ready. Obviously I had to do some more leg work.

The way I had been getting this data till now for other languages/frameworks/web engines was to parse the access log. Access log usually has the request which the system served, the time it was served, time it took to serve the request in milliseconds, HTTP status and some other info. Parse this log properly and you have all the data you need. Unfortunately, Play! uses JBoss Netty as an in built web server and provides us no access log for that. Sad.

So the possible solution.

1) AccessLog Module for Play! Framework.

2) Put Nginx in front and use its logs. If you are already using Nginx, your work is cut in half. Just parse its logs and boom!

3) Do what this guy did. A nifty solution for synchronous requests, which is what we wanted actually.

We just started using this solution with a required change.

This is what the initial solution is.

import play.api._ import play.api.mvc._ object Global extends GlobalSettings { def ResponseTime[A](action: Action[A]): Action[A] = Action(action.parser) { request => val start = System.currentTimeMillis val result = action(request) println( request + " -> " + (System.currentTimeMillis - start) + " ms.") result } override def onRouteRequest(request: RequestHeader): Option[Handler] = { super.onRouteRequest(request).map { case action: Action[_] => ResponseTime(action) case other => other } } }

Elegant, isn't it? Call another function which in turn calls the action. In between that, take the time diff and log.

Now about the needed change. As you might have figured out, "println" wouldn't work for production setup. What we want is to save the response times without having to do a file I/O every request. There are many ways to do it. Let me point out a couple.

1) Keep the response times in memory. Create a map of request.path to response time. Every few minutes, dump that data to a file or DB using a separate job. The issue with this is, you might miss requests since every time you restart your service or anything like that, your in memory data is lost.

2) Write an actor and pass response time data to that actor. Let the actor do the work asynchronously. This will make sure your API response time is not impacted.

I  like the second solution. And that's what  we implemented. Works nicely.

[methinking]

Curious case of Facebook Hackathon

Who knows about ongoing facebook hackathon?

It is in it's 2nd second year and I had many hopes from it. Last year, in its first avatar, it was almost a disappointment. But since it facebook, even in its disappointment, the problems were good and to solve them was fun.

This year, i must say, i feel more disappointed. Don't take me wrong, the execution of the event was quite good. I am not able to make peace with just one rule. The only rule which makes or breaks your solutions. The rule is, you can submit your solution within a period of 6 minutes after downloading the input. This i feel is madness...utter and complete madness. A bit too dramatic in my head, i sound.

Let me tell you why all this kolaveri..

A person spends quite some time, energy and sleep in case of Indians, just to be a part of such events. I would hope, like any other event of this kind, I am allowed to make some mistakes. 

A person who works fast, makes mistake, and adapts fast, is a good person to have on your side. This is coming from the philosophy of fail quickly, fail fast. But if you tell me, i can submit my solution only once, and thats it. It becomes a problem for me. I spend more time looking at my code for corner cases then actually testing it. 

In my past experience with such events, I have seen this issue addressed by simply allowing the user to upload the code and trying to execute the code on the receivers side. This is the most accepted solution for coding competitions. The other solution, if you want to remove the server side is, providing different inputs every time someone downloads the input and check for the output accordingly. I understand this means dealing with a lot more input downloads and solution uploads, but you will at least get away with blog entries like mine.

This is coming from my experience, single syntax mistake makes your couple of hour work go in vain. Almost makes me not participate anymore. But again, its facebook, the problems are good and its fun to race against time. :)

[methinking]

A Hackathon

Today i want to pin down my experience with hackathons. I have been recently part of two close to close hackathons and the experience was very new to me. I mean i have done coding, quite a bit of it. I have written long programs, sometimes even before pinning down what i will be doing, i have dived into code. But to think of something, finding out what all you need to learn to accomplish your mission and the final execution to create a product, all within 24 to 48 hours. Its exhilarating to say the least. Almost a adrenaline rush. Pushes you to think beyond, makes you feel you are meant for something bigger, better and you have so much more to contribute.

I have never been of the fast kind when it comes to creating something. I like to think about what i want to do, learn what i need for it, then plan it and yes, executing it should be fast. The best part about hackathons is that you have so many people around you to tell you that you are wrong. Correct you, help you and improve you.

Here is mostly what i used to think before i wrote any piece of code.

1) Why? 

2) How? - Select one from multiple solutions available

3) Learn what you found out in step 2.

4) Create a modular framework for whatever you wanna write.

5) Code.

Here is what usually happens in Hackathons

1) Why?

2) How? - Select the first or the quickest solution available.

3) Very less time to learn, google for existing code snippets and learn while you copy.

4) The quickest way goes, no time to think over modularization.

5) Keep coding, which btw you started somewhere around point 2. 

Its not ironic if you see that the events are named hackathons. But the best part is ahead of us. After the hackathon is complete, everyone is a winner.  You are always proud if you did something meant to take long times in 1-2 days by whatever means possible. Whatever you learned in the 1-2 days can never be taken away from you. And you get to meet and bond with people so passionate about what they are doing, that's always a nice experience. 

I think everyone who is techie and wants to have a larger than life experience, should go for a hackathon. 

"You have so much potential, only if you had a platform to reveal them" : Me

[methinking]

DropBox SDK and Android

So recently i worked on Dropbox SDK provided by the Dropbox team and tried to integrate it with an android app. Wouldn't say it was easy but i got through it, and they the result is worth the time spent. :). I am writing this for anybody who wants to use dropbox api and i bothered by the wide spread info.

So here is what i did.

1) Get the idea of your app in place and get the code ready. (apart from DropBox integration part)

2) You need to register yourself on DropBox to be able to use their API`s.

3) You have to add an app there. This will give you an app key and an app secret. There is a small trick though. DropBox doesn't allow you to spread you app without their permission. So with the Keys you just got, you can only use it for development. When you think your app is ready to be distributed, you have to pass it through them to be checked for their standards and best practices.

4) Assuming you have your app registered, you should download the DropBox SDK. This includes the libraries required to use DropBox and it also includes an example app.

5) Go through the example app. Its pretty decent. It doesn't showcase all the functionality available though. You can find out what all you can do with the API here.

This is all you need to get started. After this point, its all you and your app.

How i did it comes next.

One problem with DropBox API is the authentication screen is to created by you. You ask the user for username and password. You pass them along to the respective authentication API. Its not a very good way to do it as the programmer can save the usernames and passwords on the go. Its not allowed though its not clean by any standards.

The API itself is damn simple and easy to use, with functions like "put(file, path)" to put a file in your DropBox and "get(file, path)" to get a file from your DropBox. The problem i faced was to keep a user logged in even as the activities change and not to ask the user to enter his credentials again. But i guess it was more of a lack of knowledge on my side then the problem with the SDK. Once i figured out that a session class can do the job, it was more or less about implementing the session class. Still i feel the ease to as we have in Facebook API is missing. There is a lot of scope to do better and i hope things are in the pipeline. Here are a few screen shots of how i solved my problem of using DropBox for saving a single or multiple photos.

There is still work to be done on the app. But as they say, its the Journey that matters the most, result is just a byproduct.

[methinking]

Testers and Android

Continuing to my last post. Today i want to discuss how to make android apps which are stable, fast and efficient on resources. By now we should understand, mobile platforms are different and need specific tools as well. But whats common is, the need to optimize resource utilization. Again, like for iOS, we discuss the following topics for Android.

Feature Testing

Automation

Profiling for leaks

Profiling for CPU Usage

Profiling for memory usage

Allocations

Before you start reading further, i recommend reading abut the instrumentation class in Android documentation and DDMS. DDMS is a great tool provided by android specifically to profile your application and is for our good luck, very nicely integrated in Eclipse.

Feature Testing is usual manual testing. It doesn't require any tools and so is the same as how you do it on Desktop Application or on iOS.

Automation : There are many tools out there to do automation on android. AutoAndroid, SeeTest, Calculon, Robotium and many more may be. This image taken from Vendor Benchmarking depicts nicely the comparison of the first three tools.

I would personally say, Robotium, having used it, is very easy to start with. Being a library so can be imported directly to your project. You can also create a separate test project along side your app. This is the normal way of using it. In order to use it for one click automation, you need it be a totally independent project. And so you need to write a wrapper above Robotium which can be bit of a pain. All the above mentioned 4 automation tools solve different problems and you need to pick one based on your product and your requirement. This goes without saying that all of these are black box testing tools and may or may not require maintenance as you go along developing.

Lets go to profiling, Leaks in your android are hard to find. I couldn't find any useful tool out there to successfully monitor leaks on android. There is no equivalent of Instrument Leaks of iOS for Android. But there are other ways you can try and monitor. Instrumentation class provides us access to the memory usage of your app at any point of time. You just need to make a call and you can get the memory usage. Now using this function you can track the memory usage before and after the code you want to monitor. I understand its hard to find small leaks through this approach but its the best i got. Let me know if you find something better :).

While we are on memory usage. Lets go through memory utilization as well. There are many many ways to get to know your apps memory utilization. DDMS is the easiest, just select the DDMS view and your app from the left side pane. Then there is Instrumentation class, you can use it to track the memory utilization from your code. Then there is another way, your adb shell. You can use many commands to do so, top, dumpsys meminfo proc, procrank, vmstat. There is another tool named smem but you have to separately install it.The tricky part is, once you run the above commands, we get many memory values and its not very clear which one is useful

Vss = virtual set size

Rss = resident set size

Pss = proportional set size

Uss = unique set size

Different command line functions would give different outputs.

Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. PSS instead measures each application's "fair share" of each shared area to give a realistic measure of the activity of a system during boot. Hence PSS is our best shot for the memory usage by our app. I use procrank as it directly gives me the PSS value. Sometimes if you need more in depth output of dirty pages, you should use dumpsys meminfo.

Profiling for CPU. This is a bad one as i know only one way to do it. Use the top command and see the cpu usage. It works great for me tough because thats all i am looking for :).

Allocations again is something which can be done in multiple ways. I particularly like the DDMS solution for this. You can also use dmtracedump.exe and traceview. Whats good about android allocations is, you can create a dump file for the period you were monitoring your app for, and store it to be viewed later or used later or compared later. its an amazing debugging and testing tool and you should definitely read more about it.

All along the way i have assumed you have the knowledge of what is logcat and how to use it. Its basically a console output of what your device is doing. If you are reading this, you must already understand that for mobile testers, you need to get close to the product from the start and just UI level and feature testing is not enough.

I hope this was helpful. In case i mentioned something wrong or you want to ask anything, drop me a mail at [email protected]

[methinking]

Testers and iOS

Before i say anything else i want to mention that these are my views. So they might not be coherent with yours. Well you just gotta deal with it.

So about testing on iOS. Let me first start with just testing on mobile devices. As a tester, you have a few more responsibilities as compared to testing desktop applications. Let me pin down what i think are the requirements to make up for a comprehensive testing and assuring a stable and optimal product across revisions.

Feature Testing

Automation

Profiling

But when you talk about mobile devices, things unfold in a bit more complex, narrowed way. Also testing has to be done along side development, otherwise it just wouldn't work. Waterfall model of development is not at all suitable for mobile development. For mobile testing, the list goes like this

Feature Testing

Automation (How?)

Profiling for leaks (We wanna be very memory efficient right?)

Profiling for CPU Usage (We don't want to choke the device now do we)

Profiling for memory usage (Again, cant choke the device out of memory)

Allocations (Later?)

Now whats new is that in Desktop applications, you don't need to be so particular about profiling. Fair enough is good. But for mobile apps, you need to be sure you are not doing anything to use extra resources. Reason obviously being we have limited resources and more importantly, there are other apps as well who want those resources. We need to live in harmony remember.

Another thing to remember for a tester on mobile devices. If you go to a developer with "Our app is crashing", "We are using too much memory", "We are using too much CPU". Its not good enough. You need to figure out whats causing that behavior. Is memory the culprit, is CPU or is there some wrong call in the code? And for this you need to profile the builds at the same time you are testing it. You need to get hold of how to use allocations to figure some things. For the testing to be comprehensive, you need to be hand-on with all these things.

Except for manually testing the features, everything else would require tools. This is again something which would be mobile OS dependent. But the bottom line is, we need tools for automation, profiling leaks, CPU, Memory Usage and tracking allocations. Also we need to save the traces (dumps?) of the runs we found problematic, so that developers can look at it and more often then not figure out what the problem is.

Now lets talk iOS testing.

We will keep Automation for the end. The good news is, Apple provides us with tools for all the profiling and allocation tracking in a nifty Mac application called Instruments (remember instrumentation?).It ships with XCode. You have many many tools in Instruments for instrumenting your app. Starting with activity monitor to automation. Lets talk a bit about some of them and how they are useful for mobile testers. You can also go through the list here.

Activity Monitor : Use this to track the system workload and also virtual memory. This is basically a counterpart of Activity monitor for Desktop on iOS. Use it as you use activity monitor.

Allocations : Monitor memory and object allocation pattern.

Leaks : This along with allocations is used to track memory leaks.

CPU Sampler : Use for CPU profiling, you can compare the workload of your app vs the whole system.

And there are many more tools in Instruments which you can use to profile your app. The whole kit is in there. It sure is a pain to start but it makes your life so easy you cant even imagine. Now the best part, you can use these profiling tools on the actual device as well as on simulator. You can create and save the dumps anytime you want, and not to mention, load it later.

Allocations, i want to talk about allocations once again. To start with, it might look like why does a tester need to worry about allocation patterns. Well for mobile devices, you need to help the developers, and in turn the product in finding out whats causing the problem. Allocation lets you do that. Using allocations you can find out which function consumed most amount of time, which function was called most number of times and so on. This information is very useful. Even in case you don't understand it, remember to track this information for your dump, because it will make life easier for the people debugging the code based on your dumps.

Now we come to automation. This is a bit tricky one because of the requirements of automation systems. Sometimes we want one click automation which would do everything for us and generate the reports, sometimes we want the automation system to be in CI (Continuous Integration). Most of the tools available right now don't allow these upfront, well not unless you make some tweaks of your own. Most of the tools are UI automation, and given the scenario, i would strongly suggest doing UI automation, because in mobile apps, its the feel that matters. White box automation is always good to have, but in mobile apps, black box automation is a necessity. I have been through some of the tools like FoneMonkey, Frank Framework by ThoughtWorks and the Automation Instruments (shipped in with XCode by Apple). Where Automation Instrument and FoneMonkey are completely UI driven automation, Frank Framework has the capability to do White box automation. Lets talk about each of them.

FoneMonkey : Easy to start with, record and playback, option to change and save the scripts on device(Yay!!), black box testing, found that a few commands got executed without actions (Ouch!!), required to be added in your source code (Eeee!!!) and not very accurate.

Frank Automation : Assume you have a server sitting inside your app and you can communicate with that server. White box testing, Reasonably Hard to start with and maintain, Can very easily be used in CI. Required to be checked in to your source code, with a server sitting in your app, you can basically do anything in the app using white box testing off course.

Automation Instrument : Black Box Testing, One script at a time execution, No interference with code is required, Profiling tools can be used while running automation instruments leading to automatic generation of dumps,

As i said, none of these are a sure shot winner. All of them can be tweaked to be useful. I would recommend frank for white box testing and Automation Instruments for Black Box.

CI with Automation Instruments : Automation Instruments uses javascript as the scripting language and runs one such script at a time, generating a xml output file on the go. We can use apple script to automate Automation Instrument itself. i can say its very much possible having done it myself. After this you can use Automation Instrument for CI as well.

This is a very small outline of what is the minimum required for testing and what are the option to get it done. This is mainly for iOS thought the concept pass the boundary can carry on for android and other mobile platforms as well. Hope this was useful, for any clarifications, mail me at [email protected].

[methinking]

Experiment with Android

So today`s post is about my first experiment with android development. Its like they say "The best way to learn is to do it yourself" or "The first step is the most hard to take" or in my words "to be the best, you have to pass through good and better first".

Coming to the app and what i went through. The app is as simple as it can be, A Facebook album viewer for you and your friends. Easy to start with, it allowed me to learn two things, Android app development and Facebook SDK for android. Lets talk about each of them.

Android: Oh My, if you have tried your hands on iOS development, it might not be so easy to just switch. The first and the easiest part in iOS is to create your views. You start the Interface Builder and its just drag and drop, as easy as it sounds (well for simple apps atleast). But in android, the drag and drop thing doesnt work as simply, you have to write most of the things in the xml file manually. That`s a pain. And doing it for all the activities takes a lot out of you.

After that things are a lot simpler and similar to the way you code anything mobile. There are lot of view available and you can use any number of them on a single activity. You can hide the not needed ones, create views on fly or remove them. If you are down with the UI, its more about the concept then the code.

A simple UI View can look like this with some image, some button and some text

And a bit more complex can look like this: an image, multiple texts, gallery view, list view.

It takes time to get used to the way Activities behave, how to navigate between them, how to communicate between them. This was the second problem i faced. Once you figure this one out, the concept of adding view becomes simple. I actually became so comfortable that i added multiple new views against my earlier mocks.

Next problem i faced was how to let views interact with each other. What if you want one view to be updated if some changes are made to some other view? This is actually very simple using adapters (notifydatasetchanged() method)and comes in very handy to improve your user experience. In this activity, i was changing the gallery view if the list view changed, and changing the list view if the gallery view changed.

You need to remember, views and activities are two different things. An activity can contain any number of views. And a view can be used as an activity.

You can go through the code at https://github.com/maheshgattani/DownloadFacebook

Now the Facebook SDK. Man o Man, this is the reason Facebook can never be the next Orkut. They just made it so simple for developers. Now its not only Facebook people working for them, they have people like me also. Its just the simplest thing. Go to Facebook Developers page, setup a test app, get an app ID, download the SDK, and you can start writing code. The first thing i figured when i started writing this app was Facebook SDK is the least of my worries, because its just so simple and easy to use. and what more, Facebook has provided tools to test a few things while you are working on them.

[methinking]

Android vs iOS

I have been having these long discussions with friends and colleagues about android and iOS. Which one is better? Which one has a better future? Which device to buy right now, IPhone or Android device? I have heard so many arguments, arguments from hard core android fans and arguments from apple fans.

I for one, believe iOS is in a very very good state as compared to android. Also, when we say iOS, we can assume the device and the OS as the same because of the integration apple provides. When it come to Android, we have to think of the OS separately and of device separately. But lets see what each of them has to offer.

For Developers?

When we talk about the developer view, we need to take care of 2 things. What all we can do and how easily we can do that?

Android is a clear winner in what all you can do. Being an open source, it provides such an environment that you can essentially do anything if your device permits it.

iOS is a restricted environment. Somethings are not meant to be changed because the OS is designed that way (read Flash, Any app which can be used to run other programs is not allowed). Apple has a screening process, so just because you developed an app doesn't mean you can put it on App store. You need to follow some strict guidelines only then you can assume apple would approve your app.

iOS is a winner when it comes to how easily can you do something.

To start with, the toughest part being creating the UI experience. iOS development SDK provides you with a tool for interface building. A very easy to use and effective tool. MVC(Model-View-Controller) implementation might be a bit tricky to start with but is very effective as well. The code goes in Objective C which is just another OOP language and you tend to get used to it as you use it. Also since there are only a small set of devices with iOS, building an app on it takes a lot less of thinking as to what all devices you support.

Android lacks heavily in terms of UI experience. Without any standard interface builder, it is a haunting task to get your UI as you want it to look. Writing whole or part of views in XML might not suit all the users. The code goes in Java which is very well known. Also, with so many android devices in market, it might be a painful experience to figure out which all devices you want to support. And since the OS and device are not so well linked, you also have to think about the device specification when you are building an app.

For Users

For users, things boil down to ease of use, User experience and ability to do more.

iOS, for me, is the winner when it comes to ease of use. The devices with iOS gives you an easy feel to everything. Be it accessing things, be it closing things, be it downloading music and videos, be it downloading new apps, almost everything. For a person who is not very technical, iOS is far far easy to use then it android counterpart.

User experience, again iOS. This is true because of the tight integration between the device and the OS. This is an easy one, just try playing a game in android and same on iOS, the difference is visible. Also, android as a platform isnt as stable as iOS is. The apps on android arnt as stable as they are on iOS (read apple`s filtering process).

Ability to do more? This is tricky, mainly because android as a package does provides you more stuff to do. But iOS provides you more app to do search and use from. The difference isn't very apparent. On the contrary, you can do almost anything on both the OS`s. You would find app for everything on both the platforms. As of now, you have many more devices to select from in android and many more app to select from in iOS. My vote goes to iOS, being more stable and with lot more apps.

Going Ahead

Both the platforms, iOS and android, have great future. iOS 5 has been announced and it includes lot of nice features. Android 3.1 (Ice cream sandwich) is out which is the first in line to supports both tablets and handsets. Though Android is behind iOS right now, i expect it to match iOS sooner than later. With contributions from everywhere, android is developing fast. We have Microsoft "Mango" coming. Mobile field is very exciting right now and rapidly progressing.

What to buy?

My vote goes to iOS mainly because of the tight integration. The iOS devices are high end when it comes to specs. The user experience is amazing as compared to only good for android. Android OS depends a lot on device hardware. Samsung has done some rights with Galaxy series. It basically boils down to

"Freedom of choice OR freedom from choice"

read

"Android OR iOS"