Fun with Python and Javascript @didip-blog - Tumblr Blog

The easiest way to upgrade your Python

Python 2.7.4 and 3.3.1 had just came out, they offer quite a few performance enhancements as well as bug fixes. Also, upgrading to 2.7.4 seems like the first logical step before moving to Python 3.3.x.

I hope this post can help you upgrade your Python as painless as possible.

1. Stop relying on base Python installation

Installing a different version of Python is very easy now thanks to pythonz.

# 1. ALWAYS preview the code before piping to bash! curl -kL https://raw.github.com/saghul/pythonz/master/pythonz-install | bash # 2. Source pythonz paths source $HOME/.pythonz/etc/bashrc # 3. Install a different version of Python pythonz install 2.7.4

2. Use virtualenv to sandbox all your Python modules

Virtualenv is here to stay, it has already been absorbed as standard library in Python 3.3. It's a great tool to setup Python modules for your application.

This is the example on how to setup virtualenv using pythonz's Python.

# VARIABLES PYTHON27_VERSION='2.7.4' PYTHON27_NAME="CPython-$PYTHON27_VERSION" PYTHON27_BIN="~/.pythonz/pythons/$PYTHON27_NAME/bin/python" PYTHON27_VENV_DIR="~/.pythonz/venvs/$PYTHON27_NAME" # NOTE: # pythonz put all the different pythons here: ~/.pythonz/pythons # We put our venvs under ~/.pythonz/venvs to make organization simple. PROJECT_NAME="example" # Create directory for virtualenv mkdir -p $PYTHON27_VENV_DIR # Create virtualenv virtualenv --no-site-packages -p $PYTHON27_BIN $PYTHON27_VENV_DIR/$PROJECT_NAME

Here's another example, the snippet setup PyPy as well.

3. Activate your virtualenv to install Python modules

source ~/.pythonz/venvs/CPython-2.7.4/$PROJECT_NAME/bin/activate pip install -r requirements.txt deactivate

4. Don't worry about activate() vs deactivate() when running your daemon

Just run the venv Python binary directly.

This approach is very convenient when dealing with cron or Supervisord.

~/.pythonz/venvs/CPython-2.7.4/$PROJECT_NAME/bin/python

#python #administration

tornado-stripe v.1.0.0 is released

I've finally gotten around to release async Stripe library as full fledged open source project. It has been in production for On the Plates for a few months now.

The cool thing about this API is that, it maps to Stripe Curl API URLs exactly one-to-one. For example:

from tornado_stripe import Stripe stripe = Stripe('api_key', blocking=True) stripe.charges # == /v1/charges stripe.charges.id(CHARGE_ID) # == /v1/charges/{CHARGE_ID} stripe.customers # == /v1/customers stripe.customers.id(CUSTOMER_ID) # == /v1/customers/{CUSTOMER_ID} stripe.customers.id(CUSTOMER_ID).subscription # == /v1/customers/{CUSTOMER_ID}/subscription stripe.invoices # == /v1/invoices stripe.invoices.id(INVOICE_ID) # == /v1/invoices/{INVOICE_ID} stripe.invoiceitems # == /v1/invoiceitems stripe.invoiceitems.id(INVOICEITEM_ID) # == /v1/invoiceitems/{INVOICEITEM_ID} stripe.tokens # == /v1/tokens stripe.tokens.id(TOKEN_ID) # == /v1/tokens/{TOKEN_ID} stripe.events # == /v1/events stripe.events.id(EVENT_ID) # == /v1/events/{EVENT_ID}

To install:

pip install tornado-stripe

For more information, visit its GitHub page.

#api #stripe #tornado #python

Playing With Web.go

Finally, curiosity go the best of me. I decided to install Go language and it's web.py like framework called web.go.

My OS Info

OS X Snow Leopard, 2.26 GHz Intel Core 2 Duo, 8 GB 1067 MHz DDR3 RAM

Prerequisites(on OS X)

To install go, simply download the package and double click.

To install web.go:

# You may need to restart your terminal session after installing Go language. sudo go get github.com/hoisie/web

Hello World Page

Follow the instruction on web.go github for creating the hello world page. I copy-pasted the instruction here for convenience:

# Do all 3 steps in the same directory # 1. Create hello.go package main import ( "github.com/hoisie/web" ) func hello(val string) string { return "hello " + val } func main() { web.Get("/(.*)", hello) web.Run("0.0.0.0:9999") } # 2. Compile hello.go go build hello.go # 3. Run the executable ./hello # Visit: http://localhost:9999/world to see the page.

The compilation time is miniscule, but that doesn't say much because our application is tiny. Once we got the hello world page up, like a true tech blogger, we have to benchmark it using AB tool.

# 100,000 requests. 210 simultaneous requests. Keep-alive is on. ab -k -n 100000 -c 210 http://localhost:9999/world

The result is quite astonishing. On fresh boot, our little app server took only 4.1 MB of RAM and after AB is finished, the memory usage climbs to 15.3 MB. Not bad at all.

Now let's look at the AB result:

Server Software: web.go Server Hostname: localhost Server Port: 9999 Document Path: /world Document Length: 11 bytes Concurrency Level: 210 Time taken for tests: 25.870 seconds Complete requests: 100000 Failed requests: 0 Write errors: 0 Keep-Alive requests: 100000 Total transferred: 16700000 bytes HTML transferred: 1100000 bytes Requests per second: 3865.54 [#/sec] (mean) Time per request: 54.326 [ms] (mean) Time per request: 0.259 [ms] (mean, across all concurrent requests) Transfer rate: 630.41 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 1 22.8 0 1000 Processing: 0 54 36.6 51 486 Waiting: 0 54 36.6 51 486 Total: 0 54 42.9 51 1159 Percentage of the requests served within a certain time (ms) 50% 51 66% 57 75% 62 80% 71 90% 102 95% 116 98% 155 99% 175 100% 1159 (longest request)

3865.54 requests per second and 0 failed requests! I am very impressed, this result beats anything I've ever tested in this laptop. And look at that time per request: 54 ms. Wow!

#golang #web.go

Linode NodeBalancer ssl_error_rx_record_too_long

The Situation

You've checked that your SSL certs are valid and HTTPS is still not working on your Linode instance that's behind NodeBalancer.

The Solution

Edit your port 443 NodeBalancer settings. Change protocol HTTP to TCP.

Profit.

#linode #NodeBalancer #SSL

What to do after upgrading Python...

...through prepackaged binary from python.org. As opposed to using virtualenv.

This post is pretty much a self-reminder post. I was upgrading Python to 2.7.2 from 2.6.1 on OS X Snow Leopard.

cd /tmp # fix pip curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py sudo python /tmp/get-pip.py # Fix setuptools # Download the appropriate .egg from here: http://pypi.python.org/pypi/setuptools curl -O http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg#md5=fe1f997bc722265116870bc7919059ea sudo python /tmp/setuptools-0.6c11-py2.7.egg # Fix distribute curl -O http://python-distribute.org/distribute_setup.py sudo python distribute_setup.py

#python

Broken setuptools is lame

Today is finally the day when Python package management is giving me feud. Which is a total surprise because it had always worked reliably.

OS: OS X Snow Leopard

Problem:

/usr/bin/easy_install-2.6:7: UserWarning: Module pkg_resources was already imported from /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc, but /Library/Python/2.6/site-packages is being added to sys.path from pkg_resources import load_entry_point

The error message is so not obvious, by the way.

How to fix:

Just reinstall distribute

curl -O http://python-distribute.org/distribute_setup.py /usr/bin/python2.6 distribute_setup.py

#python

Fabric logger "ssh.transport" woes

Today I finally had the opportunity to use Fabric (v.1.4). True to "hello world" spirit, this is my fabfile.py setup:

from fabric.api import run

def ps_aux(): run('ps aux')

And then fabric freak out and complain about this:

No handlers could be found for logger "ssh.transport"

This type of error usually occur when a module uses logging standard library. So, to make logging module stop making sad face, we need to set the log level on "ssh.transport":

logging.basicConfig() logging.getLogger('ssh.transport').setLevel(logging.INFO)

That's it.

#python #fabric

Rarely Mentioned Benefits of Tornado Framework

What is Tornado? Tornado is a popular event-loop based web framework.

Most people who talk about Tornado usually mentioned how fast it is when hitting "Hello World" using Apache Benchmark. This post is not about that.

Background jobs

In most web application request cycle, it's common for the request handler (or controller) to perform various tasks that are less important than page delivery (and sometimes expensive). Tasks like recording analytic, updating counts, processing uploaded files, etc.

Django applications solve this problem by creating jobs on Celery or Gearman.

Rails applications solve this problem by creating jobs on Resque or various AMQP solutions.

Tornado applications solve this problem through IOLoop.add_callback() within itself, without any external daemons:

def expensive_callback(): pass # your expensive work

with tornado.stack_context.NullContext(): tornado.ioloop.IOLoop.instance().add_callback(expensive_callback)

Scheduled jobs

Web applications sometimes need to perform routine work. Many people solve this problem using cron. Some people try to reinvent cron through scheduler daemon.

Meanwhile, Tornado have built-in solution to this problem:

def run_me_everyday_callback(): pass

tornado.ioloop.PeriodicCallback(run_me_everyday_callback, 24 * 60 * 60 * 1000).start()

node.js also have the same benefit through setInterval()

It's true that the interval doesn't use system clock, but that's solvable by running the callback every X seconds, but check the system clock first.

External dependencies

Tornado has 0 external dependencies. It has its own HTTP parser, epoll.c, template engine, and even web server. It used to depend on curl strictly, but now Tornado shipped SimpleAsyncHTTPClient that's 100% Python.

Having minimal dependencies is very nice. When I encountered any bugs, I can file them under GitHub issue tracking or mentioned them on the mailing list. The core developers can basically fix everything inside Tornado.

Its fast web server also makes deployment simpler. You typically only need nginx/haproxy in front of Tornado instances.

It's a micro framework

Besides having zero external dependencies, Tornado is also small in terms of LoC. Because of its simplicity:

Tornado can run under PyPy.

Tornado applications can use cream of the crop Python modules (e.g. SQLAlchemy).

It's very easy to add/extends any parts of Tornado.

With a careful planning, your Tornado application can use very little memory per instance. Less memory means you can spawn more instances per box.

It supports OAuth authentication to famous social networks

Thanks to the various mixins that Tornado provides, your application can connect to Google, Twitter, or Facebook in no time.

It has global Application object

Unlike Django and very similar to the design of Flask, CherryPy, web.py, or Sinatra+Rack. With this design, it's very easy to instantiate multiple Tornado apps inside one Python runtime. It makes interactive debugging easier.

It has a familiar template engine

Tornado's template engine is very similar to Django template or Jinja2, developers won't have much trouble getting up to speed in it.

#python #tornado #web framework

Good Bye TextMate

It's been fun a ride. We've known each other since 2006. Through good times and bad times. One dotcom to another. I know your shortcuts like the back of my own hands.

But it's time to move on. You are starting to show your age. Accidental Command+Shift+F always killed you while Command+T is too slow on TextMate 2 Alpha.

Meet my new editor, Sublime Text 2. Here are a few tips I just discovered on OS X.

How to: Setup command line executable

Follow this instruction.

How to: Open Python prompt

Press Control+`

How to: Install Package Control

Follow this instruction.

How to: Install Git package

Follow this instruction.

How to: Install any packages

Press Command+Shift+P to open context sensitive menu.

Type packin, it will reveal Package Control: Install Package.

Type whatever package you expect to find.

How to: Use git blame on current buffer file

Install Git package first. See above.

Press Command+Shift+P to open context sensitive menu.

Type gitbl, it will reveal Git Blame.

Press enter.

How to: Select with multiple cursors

Press Command while clicking.

How to: Indent a block of text

Highlight the text.

Indent outward with Command+] or inward with Command+[

How to: go to line in opened file

Press Control+G (This one bugs me a little). It turns out, I can also do: Command+P, then : (colon)

How to: go to method in opened file

Press Command+R

How to: find and replace in opened file

Press Command+Alt+F

That said, I prefer my Command+F to do the job. So I changed the key bindings. To do that:

open Preferences > Key Bindings

reverse the following:

{ "keys": ["super+f"], "command": "show_panel", "args": {"panel": "find"} }, { "keys": ["super+alt+f"], "command": "show_panel", "args": {"panel": "replace"} },

How to: Debug your Flask or Tornado application interactively

Open Python prompt using: Control+`

Import your Application object.

More resources:

http://net.tutsplus.com/tutorials/tools-and-tips/sublime-text-2-tips-and-tricks/

#sublime text #textmate #editor #python

Foursquare OAuth2 Tornado Mixin

The GitHub repo is here. Use this mixin to interact with Foursquare OAuth2 API.

It's very similar to Tornado's auth.FacebookGraphMixin. Below is sample code on how to authenticate to Foursquare:

class FoursquareLoginHandler(LoginHandler, FoursquareMixin): @tornado.web.asynchronous def get(self): if self.get_argument("code", False): self.get_authenticated_user( redirect_uri='/auth/foursquare/connect', client_id=self.settings["foursquare_client_id"], client_secret=self.settings["foursquare_client_secret"], code=self.get_argument("code"), callback=self.async_callback(self._on_login) ) return self.authorize_redirect( redirect_uri='/auth/foursquare/connect', client_id=self.settings["foursquare_api_key"] ) def _on_login(self, user): # Do something interesting with user here. See: user["access_token"] self.finish()

Similar to Tornado's other API setup, don't forget to assign foursquare_client_id and foursquare_client_secret into Application constructor. See below:

import tornado.web class Application(tornado.web.Application): def __init__(self): tornado.web.Application.__init__(self, routes(), **dict( foursquare_client_id = options.foursquare_client_id, foursquare_client_secret= options.foursquare_client_secret, ))

#python #tornado #foursquare #oauth2

Rest In Peace, John McCarthy

What a dark October. John McCarthy (1927-2011), father of Lisp and inventor of garbage collection, passed away today.

$ racket Welcome to Racket v5.0.1. > (displayln "Good bye, John McCarthy. Thank you for Lisp and Garbage Collection")

#computer scientist #john mccarthy

Rest In Peace, Dennis Ritchie

Man, this past weeks have been such a downer. Dennis Ritchie, the co-creator of C and Unix has passed away (1941-2011).

# include <stdio.h>

int main()

{

printf("Thank you so much for creating C and Unix.");

return 1;

}

#dennis ritchie #unix

Rest In Peace, Steve Jobs

This is the first time, ever, in my life that I felt really sad about a person I've never met before.

That guy I saw on TV, around 95-96. Young, rich, good looking & does "computer". The same guy who drilled "American Dream" in me for more than a decade, is no longer with us.

Rest in peace, Mr. Jobs (1955-2011).

#apple #steve jobs #american dream

system clock is too fast? Configure it with ntpdate

The problem that prompted this blog post is this AmazonSES error:

RequestExpired, it must be within 300secs/of server time

Let's check my system clock:

date

Sat Aug 6 09:36:38 PDT 2011

Now let's point my ntp daemon to do different server: pool.ntp.org

sudo ntpdate pool.ntp.org

Now, let's do date again:

date

Sat Aug 6 09:30:31 PDT 2011

Yup. My system clock was 6 minutes too fast. And sure enough that fixed the AmazonSES error.

#amazon #ses #ntp

Running Tornado application inside PyPy

The title says it all, this blog post will demonstrate on how to run Tornado application inside PyPy environment.

Prerequisites:

The OS we'll use is OS X Snow Leopard.

system-wide pip.

Obviously, default python installation (2.6.1).

Virtualenv with PyPy in a nutshell:

Install virtualenv

sudo pip install virtualenv

Download your binary of choice from http://pypy.org/download.html

Extract the compressed file to a location, for example: /tmp. You will see directory that look like this: pypy-c-jit-43780-b590cf6de419-osx64

Go to the desired location of your pypy environment, for example: ~/www

cd ~/www

Create a new virtualenv, let's call it pypy-env. It is best to not include site-packages because we don't really know if pypy has support on your existing eggs.

virtualenv --no-site-packages -p /tmp/pypy-c-jit-43780-b590cf6de419-osx64/bin/pypy pypy-env

Activate your virtualenv

source ~/www/pypy-env/bin/activate

Start installing eggs. All your eggs will be located inside ~/www/pypy-env/site-packages

pip install MySQLdb-python

pip install tornado

Something a little different (my own approach):

In this section, I will suggest my style of setting up pypy environment for each Tornado app. It is heavily influenced by Isolate in Ruby land. If you don't like what you read here, that's cool. Just follow the nutshell above.

I like how Isolate puts all ruby gems under $RAILS_ROOT/tmp/isolate. It makes auditing dependencies easy. So, we'll emulate that by putting pypy-env inside our application directory (Example: ~/www/my-tornado-app/pypy-env). You can automate the whole process by using this bash script.

Don't forget to exclude the pypy-env and pypy binary directories on your source control (.gitignore or .hgignore).

Stumbling blocks:

Unfortunately, not everything is rosy. MySQLdb-python still doesn't work quite right in Tornado in PyPy (See: stacktrace). It seems like copy.deepcopy doesn't like something inside MySQLdb.converters.conversions. To side-step this problem, I added support for pymysql inside tornado.database (See: this commit). Once you apply this patch, simply do: pip install pymysql

Another stumbling block is intermittent SSL error when using tornado.auth library. It is already documented and fixed by other people. Since the fix is recent(2011-05-15), you need to compile PyPy from source to get it. The error was this:

AttributeError: 'SSLObject' object has no attribute 'peer_certificate'

Conclusion:

Besides those two issues, Tornado runs like a champ inside PyPy. Unsurprisingly, the app use a lot more RAM when running inside PyPy (without: 15mb vs with: 45mb).

#python #tornado #pypy

Thinking about using Amazon Simple Email Service?

Don't forget to request production access at your earliest convenience. See documentation:

http://docs.amazonwebservices.com/ses/latest/DeveloperGuide/index.html?InitialSetup.Customer.html

To request production access:

http://aws.amazon.com/ses/fullaccessrequest

Otherwise, you are faced with these terrible limitations:

Emails can be sent only to and from verified email addresses.

You can send a maximum of 200 messages per day.

You can send a maximum of one message per second.

It's unfortunate that users don't get a warning when they signed up for the service, explaining about these limitations.

#amazon #simple email service

Redesigning Reddit with dotjs

What is dotjs? It's a way to customize your favorite website using javascript + jQuery. @defunkt created it. Get it here (firefox)(safari).

For techies, why you might like dotjs instead of Greasemonkey:

It works on Chrome and Safari (Firefox support requires FF4).

It's revisionable using github or dropbox since dotjs is just a folder with a bunch of js files. Write once, effective everywhere on many browsers.

It gives you convenience such as jQuery.

For normals, why you might want to customize your favorite websites:

Have you ever read YouTube comments? Ever wish you never read them, ever?

Have you ever put a duct tape on the right-side of Facebook?

Have you ever wish that Reddit becomes prettier?

Quick forewords

This post is for all of you backend developers who are interested in learning frontend programming. I'm not the authority on this subject, as a matter of fact, I'm learning a lot while doing this.

This is the beginning of many posts on how to learn doing site redesign.

Redesign Reddit

In this redesign, there is a type of user (persona) called "Me". Me is the VIP customer and this is how we describe Me:

He reads Reddit more than 5 times a day.

He always click "next" at least 4 times (He will read page 4 or older).

He never click "next" more than 10 times.

He always read top rated comments.

He could care less about statistics, badges, and list of moderators.

He appreciates gimmick ads on Reddit knowing that they need ads revenue to survive.

He likes to explore multiple subreddits.

He rarely posts.

He comments more than posting.

He will upvote insightfulness.

He really thinks that Reddit footer is ugly and hates it with passion.

Given this persona, we will try to redesign Reddit and limit our scope appropriately. In this redesign, we will assume that Conde Nast is permissive, encouraging, and benevolent.

This is what Reddit site looks like at the moment:

Clearly we have a lot of work ahead of us.

Starting with the footer

Me hates it with passion so its natural if we starts from here. The footer is not only lame but looks out of place from the overall site design.

The brothers and sisters columns are completely unnecessary since each of them is quite popular already, they don't need Reddit juice. Google doesn't advertise FeedBurner on their footer. YouTube doesn't advertise Google Search on their footer.

about us column is important since it's part of revenue strategy, but what's up with the color? I can understand gold, but red? Also, /store should be group together along with other revenue generating links. It seems sensible.

the help column sounds important to help integrate new users inside the hivemind quicker. Unfortunately three of them raise 404 error. We should remove them.

So, how are we going to redesign the footer? Easy. Trim out useless links and splat important links into one line. See result:

Moving on to the listing

This is the meat of Reddit. It's what separates men from boys. Our redesign should enhance Redditors experience and most importantly maintaining its soul.

Number one complain that Me have is that listing item feels too cramp. Too many things going on in such a tight space. We'll try to address this situation by adding generous negative space between items.

Another thing that I will do is to get rid of the enumeration. Me often misread the voting count of a particular item thanks to this useless enumeration. By doing so, we can reclaim some real estate space.

Ideally, we could/should change the link blue color to something soft, easy on the eye (#369 perhaps?). Unfortunately, due to limited resources and perhaps lack of skills I'm unable to make such enhancement (nerd alert: I'd like to style only non :visited links).

Another cool thing I'd like to have is endless pagination. It would be super cool to not have to click next. But that's too mean to Reddit guys. See result:

The sidebar

Me was adamant that the sidebar doesn't provide him any value, thus have to be removed. But when I brainstorm on where to put the rectangle ads, I couldn't come up with better alternative. Thus, the sidebar stays.

I tackle the sidebar by removing all useless features that Me dislike. Soften the search box and make the subscribe button larger since he likes to follow subreddits.

I removed the crayon-like pointy button because Me thinks it looks gimmicky and lame in general. Those buttons stand out among boxy feeling that Reddit have. We also removed the description for these buttons. Why? Reddit is a household name that's mentioned on mainstream TV. Reddit buttons don't need explanation. See result:

The comment section

This section is the most important entertainment value that Me get from Reddit. Me also suggests readability style look on this page. So I took on this challenge by giving comments a generous line-height and slightly bigger font-size.

I also expand the make comment box to allow room for greater thinking process. Hopefully this could help reducing meme-ish, one-liner comment. It would be cool to have markdown text editor to enhance the textarea experience, unfortunately, due to lack of resources we are unable to make such changes. The save button is now uniform to the rest of other buttons. See result:

The header

Finally, we are down to the last piece of redesign. The easy target for redesign is surely Reddit tabs. They are too small and look like something out of 2001. I streamlined that and cleanup a few links on gray header bar as well. I removed saved tab since Me uses a different app to remember links.

I also cleanup the user corner by making it blend together into the main banner. See result:

That's pretty much it

The whole redesign process took about 6 hours. 1 hour planning and 5 hours of jQuery programming. This blog post took 10 hours to write. You can get the redesign code here. See you next time on "Learn site redesign with a backend developer".

Bonus point: Since you guys made it all the way, here's what Reddit looks like 5-6 years ago.

#dotjs #javascript #learning #redesign #jquery

Trending Blogs

Recently Viewed Blogs

Fun with Python and Javascript