Nirvana Tikku

I'm a technology enthusiast with a passion for software, startups, leadership and innovation. I enjoy building software products, authoring and contributing to open source projects, simplifying development processes by creating tools and hack, a lot. One of my side projects revolves around my love for music, particularly the trance/progressive genre. Check out the Android app ASOTUnofficial. Another side project revolves around my passion around education. Stack-Ed was created with the goal of helping people 'Learn from the Best on Stack Exchange'. There's also an iPhone version. I'm currently working on the innovation team at Nuance building solutions around the next generation of intelligent systems. This blog will contain practical insights on software development and presentations of any data that I obtain. All views are my own.

[nirvanatikku]

Set your OSX storage location for screen captures

I was initially excited about Dropbox's screenshot sharing functionality but honestly its been a PITA. To restore the location simply open terminal, and then

defaults write com.apple.screencapture location ~/Desktop/

(note: trailing slash is important as you'll want to specify the directory that they will be stored in)

killall SystemUIServer

[nirvanatikku]

Inserting rows into a Google Spreadsheet with Python

Below are the steps to get started with the Google Spreadsheets Data API via the GDATA Python Client and perform your first row insertion into a blank spreadsheet. The reason for writing this is to provide insight into a few gotcha's along the way.

1. Go to https://code.google.com/p/gdata-python-client/downloads/list and unzip the client and install

python setup.py install

2. Create your python script and import the python module

import gdata.spreadsheet.service

3. You will need your account information in order to integrate with the SpreadsheetService

email - the account your spreadsheet will be connected to

password 

4. Initialize a link to the client in your script

client = gdata.spreadsheet.service.SpreadsheetsService() client.debug = True # feel free to toggle this client.email = email client.password = password client.source = 'some description' client.ProgrammaticLogin()

5. Construct a dictionary object for each row. The keys are tied to the column names in your spreadsheet and the values are the associated values must be of string type. NOTE: column names must be lower case and are can't have spaces. [a-z].

row = { "id": "0", "title": "First" }

6. Create a new Spreadsheet at http://docs.google.com. Note down the key query param as you will need it. 

spreadsheet_key - this is the key param of the spreadsheet you've created

worksheet_id - the default sheet is 'od6'

7. [important] Insert a row into your blank spreadsheet with the values that represent your dictionary keys. 

If you don't do this and try to insert data into your spreadsheet, you will most likely get the following error

gdata.service.RequestError: {'status': 400, 'body': 'We're sorry, a server error occurred. Please wait a bit and try reloading your spreadsheet.', 'reason': 'Bad Request'}

8. Insert your rows into the spreadsheet by performing the following command

client.InsertRow(row, spreadsheet_key, worksheet_id)

And there you have it. You should see the data in your spreadsheet.

The main gotcha's are:

If you're inserting into a blank spreadsheet, make sure to create a row of column headers, which are the values of the keys in the rows that you insert

Make sure column names that you use, don't have spaces and are lowercase

Represent your row values as strings

For those interested in something to copy/paste, here you go:

#!/usr/bin/env python import gdata.spreadsheet.service email = '' password = '' spreadsheet_key = '' # key param worksheet_id = 'od6' # default def main(): client = gdata.spreadsheet.service.SpreadsheetsService() client.debug = True client.email = email client.password = password client.source = 'test client' client.ProgrammaticLogin() rows = [] rows.append({'id':'0','title':'First'}) rows.append({'id':'1','title':'Second'}) for row in rows: try: client.InsertRow(row, spreadsheet_key, worksheet_id) except Exception as e: print e return if __name__ == '__main__': main()

[nirvanatikku]

Stack-Ed V1.2.0

Added Search functionality for Tags. Also added question history -- clearly distinguish questions you have and have not viewed.

Review took: 7 days

[nirvanatikku]

Google Analytics April Fool's Day Visitors

Another April fool's day joke this time in Google Analytics. Seems like I've got visitor's from the "International Space Station - Control Room" visiting the "April Fool's" page. :)

[nirvanatikku]

ADD NEW AWESOME APPLICATION

Looks like the Android team has added a little Easter egg to the Android Developer Portal :)

Here's the flow in screenshots

...and a few moments later

[nirvanatikku]

Stack-Ed V1.1.0. 

Includes the ability to search through sites, as well as conveniently pick the site that powers Stack-Ed. 

Review took: 7 days

[nirvanatikku]

"Stand on the shoulders of the StackExchange Giants" 

Excited to have my first published my first iPhone app: Stack-Ed, and it's awesome to see it in iTunes! The app review process took 6 days, which is inline with http://reviewtimes.shinydevelopment.com/ and it got approved on a Saturday! Thanks, Apple!

While building the iPhone app, I decided to add the ability to specify which Stack Exchange site Stack-Ed is powered by. I've also made this change to the website http://stack-ed.com. 

If you have an iPhone, check the app out! It's great when you're looking for some casual reading to do or to take a break and learn something new/different. Learn as much as you can!

[nirvanatikku]

Stack-Ed for iPhone. Coming soon.

[nirvanatikku]

Foodler Delivery Reminder Google Apps Script

If you order food online using Foodler.com, you know that you need to check your email for the delivery estimate of when your food is expected to arrive (sent shortly after placing an order).

If you use Foodler frequently enough, you'll realize that the predictions aren't that great; they usually compensate to ensure that my expectations are set to 15 minute longer than it would usually take. Regardless of the estimations, though, wouldn't it be nice to have a calendar reminder for when your order is supposed to arrive (so that you effectively don't have to go fish into your email for the order info)?

I decided to whip together a little Apps Script to help with this. The script itself looks at your Gmail emails, look for those that are from [email protected], parses out the relevant information for the order, i.e. total price, tip paid, restaurant, phone number, date the order was placed, to whom it was placed, and of course, when the delivery should have arrived, and creates reminders in your Google Calendar.

The script performs two cron operations (via 'triggers'). The first checks gmail to see if a new email has come through from foodler. If orders have come in, then we must flag them as needing 'notifications to be created' for that order. The second, looks to see if any orders have been flagged, and if so, two calendar events are created: 

The order from when it was placed to when it will arrive, spanning the entire duration, without an alert

An alert for the time that Foodler estimated arrival

The script as it is today, is essentially the first part of what I think could be a valuable tool. Ideally, I'd be able to follow up with the script and provide information about when the food actually arrived, allowing me to perform some math to predict the arrival of future deliveries. If everyone was using said script, imagine the analysis that one can perform to more accurately determine when your food will arrive. Moreover, you can tie in traffic information based on your location vs. the restaurant, making the system pretty interesting.

Anyhow, Apps Script is an extremely interesting platform, and I came away with my 8 hours hacking on it with some tips/feedback:

One can parse HTML as XML and essentially extract particular nodes using a pseudo-xpath expression and performing an eval on the XMLDoc. i had to visually traverse the tree to ensure i was extracting the right nodes.

Triggers are constantly running, so make sure that while you're in dev mode you don't have them set too frequently (otherwise your logs may get clobbered eventually leaving you confused with why there's hardly anything in your logs)

HTML output is not standard html output - the HTML is preprocessed by Google Caja, so don't expect to be able to import just any library.. so i had to go to a more rudimentary solution for building the html for the webapp view

When performing a query on ScriptDb, you can perform a sort and pass in the sortStrategy, which if you're dealing with numeric data in string form, allows you to specify db.NUMERIC as the strategy. awesome.

If you're in the process of executing a script, and you perform any operation like saving the document, the notification allowing you to cancel or dismiss a running function is lost

Debugging large objects in the AppsScript console is a bit painful for the time being. Use the logger or write out to a google doc.

if you're debugging triggers, set the notifications to be sent immediately. this allows you to sort of debug any issues that may be occurring in those cron jobs

I gave a lightening talk presentation at the Boston Google Developers Group after +Arun Nagarajan gave an awesome presentation on the platform. Here's a link to the presentation: http://ntikku.appspot.com/p/GAS_Foodler_Delivery_Reminder/

I also posted the project on GitHub, so feel free to check it out. https://github.com/nirvanatikku/foodler_delivery_reminder

Moreover, if you use Foodler, install the script! Available in the Chrome Web Store.

[nirvanatikku]

Screenshot of Stack-Ed.

Current tag: JavaScript. Current answerer: Alex. Current question: Debugging JavaScript in IE7

[nirvanatikku]

If you're a developer, there's no doubt that you utilize StackOverflow's vast amount of questions and answers in your technical endeavors. Whether you use it for reference, or use it to learn, the resource is phenomenal and the platform rocks.

Recently I have been addicted to SO and feel great engaging in the very act of giving back. I love sharing knowledge and being able to pass along anything I've learned wins.

I started perusing the top answerers for tags, and their most viewed questions. I found it tremendously useful (there are some *brilliant* people answering questions extremely thoroughly) but found the experience sub-par for learning purposes. 

That weekend I decided to throw a website together (scratching an itch from going from zero to published project in a weekend), which ended up becoming Stack-Ed. In short, Stack-Ed helps one learn about any topic (on the stack-exchange platform, but particularly StackOverflow in this case). As succinctly put in a tweet, 

Stack-Ed seems great for learning programming things you didn't know you didn't know: stack-ed.com

— Jeremy Singer-Vine (@jsvine) January 28, 2013

I plan on writing a follow-up blog post on the rise to #1 on Hacker News but as can be seen in the photos uploaded with this post, after launching and posting to HN 30 minutes went by without any activity, then the following happened:

2.54am - #4 2.59am - #3 3.42am - #2 4.10am - #1

I figured such a hack would resonate with the HN community and it sure did! I managed to get just shy of 19,000 visitors that day alone, with several spikes as a result of tweets from SO engineer Ben Dumke-von der Ehe and a retweet from Jeff Atwood!

This is pretty neat: stack-ed.com

January 28, 2013

The comments were extremely helpful, and I was happy to post the project on GitHub. That week, I also implemented a trending questions/answers TwitterBot - http://www.twitter.com/stack_ed, via the github.com/nirvanatikku/ga_twitterbot project.

The reception from the StackExchange folks was absolutely amazing. Several folks said really kind words and sent me a t-shirt and stickers! Pretty sick. I plan on creating mobile versions soon (and most likely open sourcing those, too), so stay tuned!

[nirvanatikku]

Google Analytics Driven TwitterBot on AppEngine

Gave a talk at the +GDG Boston yesterday on creating a +Google Analytics driven TwitterBot that runs on +Google App Engine.

If you're interested in checking the bot out, it's @ASOTbot; ASOTbot tweets the previous day's top trending tracks (with the associated YouTube video) played on the Unofficial ASOT Android app. #AutomatedSocialMediaMarketing  

Link to the source code for those interested:  https://github.com/nirvanatikku/ga_twitterbot Link to the presentation (also embedded below):  http://www.slideshare.net/nirvanatikku/twitterbot-via-appengine-driven-by-google-analytics

TwitterBot via AppEngine (Driven by Google Analytics) from Nirvana Tikku

[nirvanatikku]

Happy New Year!

[nirvanatikku]

Nooooo! One month later, the ASOTbot has been suspended due to "unsolicited mentions to other users"......... looks like I have to disable the auto-tagging feature. Will still pursue building out the database, though. 

Edit: It's back up.

[nirvanatikku]

AppEngine Download Datastore Entity as CSV

If you've used AppEngine, you're probably familiar with the bulkloader. If you haven't, simply put, Google has built the datastore infrastructure with various helpers to ensure that the data that lives in your application can be uploaded or downloaded efficiently. The bulkloader is an extremely powerful tool and provides the developer with a ton of flexibility (how many threads to run, any bandwidth limitations, file formats, etc).

With this flexibility, however, comes unnecessary complexity (when doing something relatively trivial), or so I feel. That being said, once you demystify what needs to be done, its not that bad.

If you've been reading my earlier posts, I created a twitter bot that lives on AppEngine. The twitter bot grabs data from Google Analytics, uses the Echo Nest's artist extraction method to determine what artists are featured in the given track, and tweets that information out, ideally with the artists tagged by their twitter handles.

Well, it's not much data, but over the course of the past few weeks, there have been 115 artists that the extract method has linked to particular ASOT tracks/artists. As suspected, quite a few of the EDM DJ's didn't have associated twitter accounts. As promised I have been ensuring tagged artists have their twitter handles. It's time to send the EN team this dataset. I was hoping there would be some kind of 'export entity as csv' feature in the AppEngine admin tool, but unfortunately (somewhat surprisingly?) there isn't. Oh well, here's what you will need to do:

First things first, you'll need to build a yaml file describing the contents of your entities on AppEngine. Fortunately Google has hooked us up with a convenient method for doing so. Within the root of your app directory, go ahead and generate the yaml file like so:

$ appcfg.py create_bulkloader_config --filename=bulkloader.yaml --url=http://[your-app].appspot.com/_ah/remote_api

Once this is complete, you'll notice that there is a bulkloader.yaml file (or whatever you called it) with metadata describing your data. Additionally, there are "TODO"'s sprinkled within this file, that you must modify before trying to perform the download. Based on the comment block at the top of the generated file, the following items need to be provided

The 'connector' value for each entity's transformer. I want my data in the CSV format, so all of my entity connector's were simply csv. Other options are simplexml and simpletext.

The 'import_transform' and 'export_transform' properties with their associated objects (particularly seen when you have a 1-many or many-many relationship); simply providing the entities name in place of the TODOs is sufficient

With these modifications to the bulkloader, we have now described whatever data we want to retrieve and the expected output format of the entity. The last step is to actually perform the download. This can be done like so:

$ appcfg.py download_data --config_file=bulkloader.yaml --filename=[whatever-you-want.out] --kind=[entity-name] --url=http://[your-app].appspot.com/_ah/remote_api 

Hopefully once this process is complete, you'll notice the file populated with your entity's data. All in all it takes three steps up front, and subsequently the download_data command will suffice (provided there aren't schema changes).

[nirvanatikku]

Seems as though I didn't catch this when it happened, but two months or so ago, Google merged Insights for Search and Trends. This new service is undoubtedly one of the best sources for market research and a great go-to when trying to find good keywords for ad campaigns, demographic information, some historical context and rising trends. If you're looking to build a product, this is a great starting point.

[nirvanatikku]

CPython and the Global Interpreter Lock

I've been using Python extensively over the past two months. I've also been playing with the Spotify library via the PySpotify project. 

In this post I'd like to share insight into a fix for the PySpotify library that was necessary to prevent a deadlock issue. The credit here goes to Steve Laverty, who helped break this down for me and make sense of CPython's Global Interpreter Lock.

First, let's lay the groundwork for the setup we're dealing with. PySpotify is a Python wrapper around libspotify -- Spotify's C API package. If you're interested in integrating with Spotify using Python, you'll be able to grab the source from GitHub and then perform your standard    python setup.py install    to build the .egg (a zip file containing the compiled source; similar to .jar, .zip, etc) and have it imported in your PYTHONPATH so you can    import spotify    and get to coding.

The PySpotify project is essentially a C Module with bindings from PySpotify to Spotify's C library. I won't go into how the bindings are created, but suffice it to say that there exists C code that defines all of the libspotify objects (i.e. Track, Album, Playlist, Artist, Image, User, Session and so on) utilizing the Spotify library interface, and the PySpotify Python library leverages the C code as a static library. If you're coming from a Java background like myself, I liken this to JNI (Java<-->C) in that a bridge is created between the two languages; in this case Python<-->C. 

The issue cropped when fetching the album art image from Spotify. In order to provide a reasonable user experience around displaying multiple results, one strategy is to prefetch album art in batches. We pursued this and ended up successfully retrieving album art, but sporadically our program would hang in deadlock. The method that was culprit was    session.image_create    in session.py, which delegates to    Session_image_create    in session.c and then to    sp_image_create    in libspotify which is probably performing some blocking IO operation (network request, file IO, whatever). So, how do we debug this? What is the fix?

With an understanding of the setup, it's time to take a detour through threaded programming in Python, the CPython interpreter and the Global Interpreter Lock (GIL). 

Python is an interpreted language. In the case of the CPython byte code interpreter (the most widely used implementation of the Python language) threads do not run concurrently in one process. CPython can sort of support multiple threads with the understanding  of a construct known as the Global Interpreter Lock. The GIL is a mutual exclusion lock held by Python's interpreter thread. This lock was designed to make operations like code execution and garbage collection thread safe as well as the benefit of enhanced performance of single threaded execution. The GIL is specifically an implementation artifact in the CPython interpreter, and it's worth noting that other interpreters like IronPython or Jython do not have this construct and can take advantage of true concurrency. Moreover, the concept of a GIL also exists in other interpreted languages like Ruby.

Understanding the presence of the GIL in this deadlock issue is critical. The C code that delegates to libspotify -- which in turn performs some blocking IO operation, and isn't a thread safe library -- had acquired the GIL and subsequent requests during its execution were cause for the deadlock issues. In order to make fetching the album art resource thread safe, releasing the GIL and then re-acquiring it after libspotify's execution is what solves the issue. In a C extension, there are two helpful macros that accomplish this:    Py_BEGIN_ALLOW_THREADS    and    Py_END_ALLOW_THREADS. In the PySpotify fix, simply wrap the blocking code with these macros.

The following addition in session.c took care of the issues:

static PyObject * Session_image_create(Session * self, PyObject *args) {      byte *image_id;      size_t len;      sp_image *image;      if (!PyArg_ParseTuple(args, "s#", &image_id, &len))           return NULL;      if (len != 20) {           PyErr_SetString(SpotifyError, "Image id length != 20");           return NULL;      }      Py_BEGIN_ALLOW_THREADS;      image = sp_image_create(self->_session, image_id);      Py_END_ALLOW_THREADS;      return Image_FromSpotify(image); }

If you're interested in an in-depth understanding of the GIL and the performance of concurrent programming using CPython be sure to check out this article from Jesse Noller. Hopefully you found this useful.