Jacky Tian | Blog

Hi, I'm Jacky. I sling code and barbells with similar ease.

[embedly]

Recently, we had an Embedly hack week where internally we played with ideas to make something cool. I made a Reddit discussion network visualization, powered by D3 with previews generated by Embedly jQuery, and UI by Foundation.

[xjtian]

Some really cool visualizations of Reddit comment threads using one of the coolest javascript libraries I've ever worked with (D3). It's awesome how little code is needed for the actual visualizations.

[xjtian]

Taking Technical Notes in LaTeX

Last summer, I had to figure out how to take notes for technical classes electronically before I started school. I tried Evernote, Google Docs, Markdown, and even Microsoft Word with Google Drive, but nothing worked well enough to make the effort worth it. The features I was looking for were:

Taking notes in mathematical notation easily and quickly

Minimal hassle to lay out notes how I want them

Synchronizing across multiple computers

Creating and including figures in notes

I couldn't find anything to satisfy even the first requirement, let alone all four at once. For a while, I was seriously considering just scanning my hand-taken notes to back them up.

Luckily, it was about this time that I discovered LaTeX. When I started learning, everything clicked wonderfully. I loved how easy it was to take notes in mathematical notation, and having the engine do all the typesetting and formatting for me meant that I only had to worry about the content of my notes.

After using LaTeX for transcribing hand-taken notes after class (and some live-TeXing during slower classes), I've developed an efficient and practical workflow for note-taking. In this post, I'll share some of the habits and practices I've adopted that have helped me tremendously.

To see examples of the tips I'm going to share, you can check out the files in the GitHub repo for my class notes, which actually leads to my first point:

1. Use Version Control

I use Git for my notes right now, and I used to use Mercurial. It's absolutely crucial to use some kind of version control for your notes to have a backup on a remote repo (Bitbucket, GitHub, whatever), a history of changes that you can revert if needed, and an easy way to transfer your notes from computer to computer.

Another advantage of using version control is that you can use a branching workflow. When I start a new topic, I create a new branch for the class that the notes are for and merge back into master when I'm finished proofreading. This way, the master branch only ever holds notes that I've finished and can print. For example, here's a portion of the network graph for my notes:

It looks a lot crazier than it actually is because the Git repo was converted from an Hg repo, but it illustrates the general idea that I start a new branch from the latest commit on master for each set of notes I start.

You'll also want to adjust your version control's ignore file (.gitignore, .hgignore, etc.) to exclude compiled PDF's and log files. I've got the following entries in my .gitignore:

*.gz *.aux *.pdf *.log

This means that all I track with Git are .tex source files and whatever images and other assets I embed into my notes.

2. Develop Your Template

TeX users are usually pretty religious about their templates and preambles (see this). For note-taking, you won't need a massive template and preamble, but as you start working, you'll probably find yourself using some packages and commands pretty frequently. Pull those out into a template, and start new notes by copying that template to save yourself the trouble of re-typing your preamble every time.

Here's my template file, which grew pretty organically after 2 semesters of note-taking:

\documentclass[11pt]{article} \usepackage{amsmath, amssymb, amsthm} \usepackage[retainorgcmds]{IEEEtrantools} \usepackage[pdftex]{graphicx} \usepackage{tikz} \usepackage{circuitikz} \usetikzlibrary{intersections} \usepackage{fancyhdr} %Listings stuff \usepackage{listings} \usepackage{lstautogobble} \usepackage{color} \definecolor{gray}{rgb}{0.5,0.5,0.5} \lstset{ basicstyle={\small\ttfamily}, tabsize=3, numbers=left, numbersep=5pt, numberstyle=\tiny\color{gray}, stepnumber=2, breaklines=true, boxpos=t } %Format stuff \pagestyle{fancy} \headheight 35pt %Header info \chead{\Large \textbf{Title}} \lhead{} \rhead{} \begin{document} \section{} % \begin{center} % \begin{tikzpicture} % [scale=3,line cap=round, % %Styles % axes/.style=, % important line/.style={very thick}, % information text/.style={rounded corners,fill=red!10,inner sep=1ex}, % dot/.style={circle,inner sep=1pt,fill,label={#1},name=#1} % ] % % %Colors % \colorlet{anglecolor}{green!50!black} %angle arcs/lines % % %The graphic % \end{tikzpicture} % \end{center} % \begin{figure}[htb] % \centering % \includegraphics[width=0.8\textwidth]{filename.eps} % \caption{Caption.} % \label{fig:figure} % \end{figure} % \def\enotesize{\normalsize} % \theendnotes \end{document}

The packages I include in my preamble are:

amsmath, amssymb, amsthm: For math symbols and notation.

IEEEtrantools: This package has the IEEEeqnarray environment, which is absolutely crucial for proofs and long series of equations.

graphicx: For embedding images and figures in documents, like scans from a textbook.

TikZ: For creating vector graphics with code. More on this later.

listings, lstautogobble: For formatting code. lstautogobble makes tabs in listings blocks relative to the first line so you can preserve the formatting of your TeX source files.

I've also got some commented code at the bottom of the template for things that I use somewhat frequently but can't be bothered to remember every time - my TikZ template for figures, and some boilerplate code for embedding figures and images with graphicx.

3. Use TikZ for Figures

TikZ is great for relatively simple figures because it generates vector graphics that scale perfectly and the figure is defined in code that you can track changes to with version control. You can do some pretty complex stuff with TikZ (see here), but if you're just working on notes, it's probably best to keep things simple.

For example, here's a figure I created in TikZ to illustrate Dijkstra's algorithm, and the code for it:

\begin{tikzpicture} [scale=1.5,line cap=round, %Styles main node/.style={circle,fill=blue!20,draw} ] %The graphic \filldraw[dashed,fill=orange!20] (0, 0) circle (1.2cm); \node[circle,fill=green!20,draw] (s) at (0, 0) {$s$}; \node[main node] (1) at (0, 1) {$1$}; \node[main node] (2) at (2, 2) {$2$}; \node[main node] (3) at (1, 1) {$3$}; \node[main node] (4) at (-.5, -.5) {$4$}; \node[main node] (5) at (0, 2) {$5$}; \path[thick,->] (s) edge (1) edge (4) edge (3); \path[thick,->] (1) edge (5); \path[thick,->] (3) edge (2); \end{tikzpicture}

For the small stuff, I like to use TikZ because I can get a better figure in less time than composing something in Photoshop or Illustrator, and the way TikZ figures are defined in code means that I can make very precise, measured changes that would be a pain to do in a WSIWYG editor.

For more complex figures that would take too long to create with TikZ, I'll usually just scan a figure from my textbook and include it in the document with graphicx.

That's a high-level description of how I use LaTeX for note-taking at school. If you're taking STEM classes and want to learn or already know how to use LaTeX, I'd highly recommend transcribing your handwritten notes. You'll become more familiar with the material from repetition, get some practice with TeX, and end up with beautifully formatted notes that you'll actually be able to read and understand when you're cramming at the end of a semester.

[xjtian]

Hosting Media on Amazon S3 in Django

When I'm working on a Django project that supports user-uploaded media files, I usually prefer to keep the files I use during development stored locally in the project directory and in source control along with associated fixture data. This way, I only have to clone the project repository when I'm provisioning a development VM and I can work on the code even without a working internet connection.

Obviously this isn't the way to do things in production, because it's silly to be keeping user files in source control if they're not associated with any data in your test fixtures. Your repo will just end up becoming way too bloated and slow, and you might even exhaust your 1GB of storage on GitHub!.

My preferred solution to serving media files is to use Amazon S3 using django-storages. Thankfully, changing from local media storage to S3 using django-storages is pretty painless and requires little to no code rewrite depending on how you wrote your file-handling code in development. In this post, I'll go over the process of transitioning from local media files in development to cloud-hosted files on S3.

Installing django-storages

I like to use the S3Boto backend for django-storages, which requires boto. Just pip install boto and pip install django-storages and add 'storages' to your INSTALLED APPS:

INSTALLED_APPS = ( ... 'storages', ... )

Next, you'll need to add the following fields to your desired settings module:

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage' AWS_ACCESS_KEY_ID = '<your access key ID>' AWS_SECRET_ACCESS_KEY = '<your secret access key>' AWS_STORAGE_BUCKET_NAME = '<your bucket name>'

Your access keys will be in the IAM management console, but you'll only be able to see the ID's. If you've lost the secret key, just delete the old one and create a new one.

That's about it for installation. Check to make sure everything's right in the Django shell, if you're on 1.3+, with

>>> from django.core.files.storage import default_storage >>> print default_storage.connection S3Connection:s3.amazonaws.com

Modifying Your Code

If you haven't written any code that opens media files for reading or writing, then django-storages will be a drop-in solution for you. However, if you've written code that deals with files at a lower level than just ImageField or FileField, you're probably going to need to refactor a bit.

Because the files are now hosted online, you can't use open() to access them anymore. You'll need to change any file-handling code from the built-in open() to Django's file storage API.

To refactor your code to work with django-storages, any time you see something like

# 'file' is a FileField open(file.path)

change it to

from django.core.files.storage import default_storage as storage storage.open(file.name)

If you've already been using the storage API to open files, then you won't need to make any changes!

Keeping Development Local

Of course, you'll still need some local media files for development and test fixtures. I keep these in a directory 'media/' in my project root directory, and I add this to my urls.py:

from django.conf import settings from django.conf.urls.static import static ... if settings.LOCAL_MEDIA: urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

I keep the S3-specific settings in the settings module I use for production, and a LOCAL_FILES = False in my base settings that I override in my development and testing settings so that my development environment and tests remain contained. Also, the code with default_storage will still work after going back to local storage.

To recap - to change your media file storage to S3, do this:

Install boto and django-storages, and add 'storages' to your INSTALLED_APPS.

Add the appropriate settings to your settings module(s).

Use the file storage API to open files in your code.

And that's basically it - in 15 minutes you could have your project up and serving media files from an S3 bucket.

[xjtian]

A section of M street in Washington, DC just east of the intersection with 28th street shows up completely grey on Streetview about half a block. Looks like there were some technical difficulties with the cameras on the Streetview car here.

[xjtian]

Testing Scrabble: Fake it 'Till You Make It

The problem with testing a complex system where functions have side effects and depend on other functions is that it's tough to write actual unit tests instead of integration tests. In this post, I'll discuss how I recently ran into this problem when testing move validation for my Scrabble project, and the steps I took to write a bulletproof, easy to maintain test suite.

The process for playing and validating a move in my Scrabble game API involves a number of intermediate steps, and the method calls involved often have side effects. Within the API, playing a move goes like this:

Call set_candidate with the move to play. This method constructs the Move object for the candidate, checks that all tiles are placed in-bounds and that all the letters selected are in the player's rack, and sets the move candidate of the game if the initial checks pass.

Call validate_candidate to validate and score the move candidate. If the move candidate is valid, it will be scored properly on method return. Otherwise, it may be unscored or partially scored.

Call commit_candidate to commit the move candidate to the game, add new tiles to the player's rack, add the move score to the player's score, and pass the turn. If there's a computer player in the game, this will also re-evaluate the cross-sets for anchor tiles.

If I just want to test validate_candidate, I run into the problem that it's written to fit into this integrated situation, not as a standalone black box. To get around this problem, I replaced the dependencies with fakes and fixtures, and came up with a simple configuration file format to make adding test scenarios as easy as possible.

May I Have that in a Sentence, Please?

My first attempts at testing move validation were stumped by me being a really horrible Scrabble player. I wasn't creative enough when it came to coming up with more complex scenarios like long parallel plays or multiple crosses. That's when I decided to fake out the wordlist with something more suitable for my skill level:

from itertools import permutations with open('wordlist1.txt', 'w') as f: word = 'ABCDE' for i in xrange(2, len(word) + 1): for perm in permutations(word, i): f.write(''.join(perm)) f.write('\n')

The fake wordlist contained all permutations of the letters 'ABCDE' of length 2 to 5. This wordlist only had 320 words instead of the 178,000 in OSPD4, and was super easy to pick words from - as long as no letters are repeated in a word, it's valid.

With this new weapon at my disposal, I had no problem at all coming up with different situations to test, but I was still writing a lot of code to set up these situations and clean them up for each test. I had to clean things up a little bit more.

Don't Repeat Yourself... Twice?

The setup code for each testing scenario looked like this at first:

self.game.candidate = None self.game.board = ... word = 'ABCD' # Letters of the move pos = (7, 7) # Starting position of the move horizontal = True # Direction of the move self.game.candidate = Move() self.game.candidate.horizontal = horizontal offset = 0 for i, letter in enumerate(word): if horizontal: x, y = pos[0], pos[1] + i + offset while y < len(board[x]) and board[x][y] not in board.empty_locations: offset += 1 y += 1 else: x, y = pos[0] + i + offset, pos[1] while x < len(board) and board[x][y] not in board.empty_locations: offset += 1 x += 1 self.game.candidate.positions.append(BoardPosition(letter, (x, y))) self.assert_(self.game.validate_candidate())

I'd mocked out set_candidate by basically re-writing the main algorithm without any of the extra validation checks. Not DRY, I know, but at least this way, if I break set_candidate, my tests for validate_candidate won't fail as well.

I pulled this setup block into a method in the test case that took board, word, pos, and horizontal as parameters. This made testing scenarios super easy, but I was still typing too much boilerplate code.

No More Boilerplate [Code]

Eventually, I got really tired of typing out all the boilerplate for each scenario I wanted to test, especially to set up the board, which had to be 15 strings each wrapped in list(). All the setup was cluttering my test case, and I needed to get it out of my code, so I decided to use simple text configuration files for the test fixtures.

The format I decided on in the end looks like this:

BEGIN BOARD #..2...#...2..# [email protected]...@. [email protected]...@.. [email protected][email protected] ....@.....@.... .3...3...3...3. ..2...2.2...2.. #..2...*...2..# ..2...2.2...2.. .3...3...3...3. ....@.....@.... [email protected][email protected] [email protected]...@.. [email protected]...@. #..2...#...2..# END BOARD ABCDE 7, 7 H 1 22

Now the board is in plaintext, sandwiched by "BEGIN BOARD" and "END BOARD". The lines after the board are:

The letters for the move candidate

The start position for the move

The direction of the move ('H' for horizontal, 'V' for vertical)

The validity of the candidate (1 or 0)

The expected score of the candidate if it is valid

I wrote a simple script to parse files with multiple scenarios set up as in the example above, and after plugging it into the setup function I had before, adding test scenarios became a 1-step process. Now, to add a scenario to test, all I have to do is append it to one of the scenario files and run my test suite again. There's still a lot of boilerplate copy-and-paste kind of work to do, but now it's in a really simple plaintext format, and parsed by a really unforgiving script that throws an exception and quits on the first formatting error it finds. Much better than making a mistake in boilerplate code like missing a column in the board and not understanding why the test failed, or wasting time trying to find a non-existent bug!

Was it Worth It?

Yeah, it's a lot of work just to test a single method, but if there's anything I've learned while coding with Python, it's that tests are absolutely crucial for the maintainability of applications written in dynamic languages. Truthfully, I don't mind if it takes me twice as long to write the test for a function than it took me to write the function itself if it means that I can revisit the code months down the road and still understand what it does and how it does it.

Also, considering how crucial move validation is to the project, I consider all the time I spent writing, re-writing, and refactoring tests time well-spent. If validation doesn't work flawlessly, everything down the road with move generation will totally fall apart, so I really want to test it thoroughly.

[xjtian]

Compelling and Affordable Copywriting and Copyediting. Knee deep in code? Keep programming. We'll give your website or app the landing page text and description it deserves. Need a document, long-form work or press release edited or written? We'll spit shine what you have or craft something from scratch.

This looks like a really cool service, and I love the design of the website. Although I do enjoy it, writing has never been my strong point, so I could see myself using this in the future. With the pricing as low as it is, I don't see any reason not to at least give it a spin before launching a product.

[xjtian]

Built-in Password Reset Views in Django

One of the best things about Django is that it ships with an easy-to-use authentication application that you could plug into any site without much difficulty. What's even better is that django.contrib.auth comes with a number of built-in views that you can use to seriously alleviate the amount of code you need to write for projects that require authentication. In this post, I'll go over the built-in views that deal with user-requested password resets and how to use them in your projects.

Setting up Email

The built-in password reset views rely on email, so you'll first have to make sure your settings.py is set up properly for Django to send emails. If you're using gmail, insert this snippet into your settings:

EMAIL_USE_TLS = True EMAIL_HOST = 'smtp.gmail.com' EMAIL_PORT = 587 EMAIL_HOST_USER = '[email protected]' EMAIL_HOST_PASSWORD = 'p@ssword' DEFAULT_FROM_EMAIL = EMAIL_HOST_USER SERVER_EMAIL = EMAIL_HOST_USER

If you're not using gmail, just tweak the SMTP host name and port if necessary for the provider you're using. To test that you've set up email properly, open up the Django shell and try:

>>> from django.core.mail import EmailMessage >>> email = EmailMessage('Subject', 'Message', to=['[email protected]']) >>> email.send() 1

If the command returns 1, you're good to go.

Creating Your Templates

The built-in password reset view is a simple one-field form that asks for an email address to send a reset link to. First, you'll need to provide a template that accepts a context variable form that displays the email form and submits via POST. This defaults to registration/password_reset_form.html, but you can put it wherever you want - if you don't use the default name, you'll just have to supply another parameter to the view function.

Here's an example of a very simple password reset form template:

{% extends 'base.html' %} {% block title %}Reset Password{% endblock %} {% block mainblock %} <form method="post"> {% csrf_token %} {{ form.as_p }} <button type="submit">Submit</button> </form> {% endblock %}

You'll also need an email template and email subject line template for the reset email that Django sends to the user. The email template defaults to registration/password_reset_email.html and the subject line template defaults to registration/password_reset_subject.txt, but again you can use whatever you want. The Django documentation has a good example of a bare-bones email template that has everything you need:

Someone asked for password reset for email {{ email }}. Follow the link below: {{ protocol}}://{{ domain }}{% url 'password_reset_confirm' uidb64=uid token=token %}

After a user gets the reset confirmation email and follows the link, they'll be taken to a password reset confirmation page that asks them to confirm a new password. This template defaults to registration/password_reset_confirm.html and is passed two context variables: form, the form that sets the user's new password, and validlink, a boolean that is True if the reset link is valid or unused and False otherwise. A simple reset confirmation template would look something like this:

{% extends 'base.html' %} {% block title %}Reset Password{% endblock %} {% block mainblock %} {% if validlink %} <form method="post"> {% csrf_token %} {{ form.as_p }} <button type="submit">Submit</button> </form> {% else %} <p>This reset link is no longer valid!</p> {% endif %} {% endblock %}

The only other template you might need is one after a successful password reset, but I typically just redirect the user to some other url (like login). If you need this template, it defaults to registration/password_reset_complete.html and isn't passed any context variables.

Configuring your URL's

Depending on how your project is set up, you may be able to call the built-in views directly from your urls.py. I didn't have any success with this for my recent project because reverse-URL lookups for redirects failed when they were in the URL configuration, but your mileage may vary. I'll go through wrapping the built-in views in your views.py because it's how I did it, but feel free to try passing the built-ins directly in your URL configuration.

Add the following two entries to your urls.py:

from django.conf.urls import patterns, url urlpatterns = patterns('', url(r'^reset/confirm/(?P<uidb36>[0-9A-Za-z]+)-(?P<token>.+)/$', 'app.views.reset_confirm', name='reset_confirm'), url(r'^reset/$', 'app.views.reset', name='reset'), )

Note that the reset confirmation url takes a base-36 encoded user ID (changed to base-64 encoded in 1.6) and a token to check the validity of the password. You need to have regexes in your reset confirmation URL to catch these two parameters, and the ones in my example should work fine.

Creating the Views

Next, you'll need to wrap the built-in view functions in your views.py. This is super-straightforward:

from django.core.urlresolvers import reverse from django.contrib.auth.views import password_reset, password_reset_confirm def reset_confirm(request, uidb36=None, token=None): return password_reset_confirm(request, template_name='app/reset_confirm.html', uidb36=uidb36, token=token, post_reset_redirect=reverse('app:login')) def reset(request): return password_reset(request, template_name='app/reset.html', email_template_name='app/reset_email.html', subject_template_name='app/reset_subject.txt', post_reset_redirect=reverse('app:login'))

If you chose a non-default name for your templates, you'll need to pass the keyword arguments template_name for both views and email_template_name and subject_template_name for the initial reset view. Set post_reset_redirect to wherever you want to send the user after submitting a password reset, and that should be it! If you point your browser to the reset URL for the project and submit the form, you should get an email with a reset link if there's a user account associated with that email.

[xjtian]

One of the least-known and most-kick-butt free products we have is PTVS - Python Tools for Visual Studio. Whether you're just interested in learning Python or you're a hardcore PhD who wants mixed-language Python and C++ debugging or somewhere in between, you gotta check this out. (Seriously, the mixed-mode debugging thing can't be overstressed...)

I used VS 2010 almost exclusively at work last year for C# development, and I absolutely loved it. Apart from the limited refactoring capabilities that came with it out of the box, I couldn't find anything that I didn't like about it. Oh, and IntelliSense is hands-down the best code completion tool I have ever used - it's almost like an extension of your mind when you're coding.

I'm extremely tempted to try out PTVS just because VS is such an awesome IDE, but I work on my projects on both a Windows machine when I'm at my desk and a MacBook Air when I'm away. I don't want to fidget with 2 different development environments, so for now I'll stick with PyCharm and Sublime Text, but it's great to know that if I ever get the itch, I can get PTVS for free.

[xjtian]

Vagrant Tips for Windows Users

I've recently been using Vagrant to manage development environments for my Django projects, and while it's worked very well to simplify my workflow in general, I've run into a few kinks while using it on a host Windows machine. In this post, I'll explain some confusing stumbling blocks that come up when setting up development environments with Vagrant under Windows and some easy ways to work around them.

SSH Client

Since Windows doesn't ship with an SSH client, you'll need to install one to SSH into your Vagrant box. PuTTY is what I use and highly recommend, but you could also use the command line tools that come with Git for Windows, OpenSSH in Cygwin, or whatever else tickles your fancy.

If you do choose to use PuTTY, you can either use the default password for the vagrant account ('vagrant'), or key-pair authentication. If you want to use key-pair authentication (my preferred method), you'll first need to convert the insecure_private_key for Vagrant to PuTTY's .ppk format. To do this, you'll need PuTTYgen. Fire it up, hit "Load", select "All Files" in the file picker, and select the insecure_private_key file. On Windows 7, you'll find it at C:\Users\<account>\.vagrant.d.

Once you load the key, hit "Save private key", and put the .ppk file anywhere you can find it again. I saved mine with the insecure_private_key.

Next, in PuTTY, go to Connection > SSH > Auth in the sidebar, and select your new ppk file to use as the private key. Under "Session", set the host name to vagrant@localhost on port 2222 (if you set up port forwarding to 22 on the guest machine to a different port, use that instead of 2222 here). Once you save your profile, you'll be able to connect to your VM when it's up by loading the profile in PuTTY.

If you use the SSH client that comes with Git (select the option to add the command-line tools to PATH during installation or add the directory yourself), all you have to do is type 'vagrant ssh' in the shell (PowerShell, Command Prompt, Git Bash, etc) and you're in. The same applies for any other SSH client you choose to use.

Shell Provisioning

If you use the shell provisioner to set up your Vagrant boxes, you'll run into some serious issues with CRLF line endings. If you try to provision an Ubuntu VM with a shell script that has CRLF line endings for example, you'll get a bunch of "package not found" errors with apt-get install. The easiest way to fix this is to switch the line endings to Unix-style LF-only with dos2unix or an editor that supports Unix line endings like Sublime Text. If these shell provisioner scripts are kept in Git (they should be!), you should set the core.autocrlf setting in Git to input so you don't end up with a Wall of Pink every time you need to provision a new box.

Another big issue to be aware of is that any interactive prompts during provisioning will either hang indefinitely or default to "no". The biggest implication of this for me is that I have to install libraries in Ubuntu with sudo apt-get install -y, and put commands that need my input into a separate script that I run after the box has been provisioned.

Here's an example provisioning and post-provisioning script setup I have for one of my Django projects:

sudo apt-get update sudo apt-get install -y apache2 python-dev git vim curl dos2unix libapache2-mod-wsgi sudo rm -rf /var/www sudo ln -fs /vagrant /var/www curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.9.1.tar.gz tar xvzf virtualenv-1.9.1.tar.gz cd virtualenv-1.9.1/ sudo python setup.py install cd .. sudo rm -rf virtualenv-1.9.1/ rm virtualenv-1.9.1.tar.gz mkdir venvs/ cd venvs/ virtualenv thetachi_venv cd .. sudo apt-get build-dep -y python-imaging sudo ln -s /usr/lib/`uname -i`-linux-gnu/libfreetype.so /usr/lib/ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libjpeg.so /usr/lib/ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libz.so /usr/lib/ git config --global user.name "Jacky Tian" git config --global user.email "[email protected]" git config --global color.ui true cp /vagrant/after_provision.sh . dos2unix after_provision.sh

sudo apt-get install mysql-server sudo mysql_secure_installation sudo apt-get install libmysqlclient-dev git clone ***.git sudo chown vagrant venvs/example_venv/ -R sudo chgrp vagrant venvs/example_venv/ -R source venvs/example_venv/bin/activate cd ***/ pip install -r requirements.txt echo "Enter mysql root password:" mysql -u root -p -e "create database ***;grant all on ***.* to 'xxxxx' identified by 'xxxxx'" echo "Enter mysql root password again:" mysql -u root -p *** < init_data.sql

Note that MySQL is installed after provisioning because it brings up an ncurses prompt for a root password that seriously wonks out PowerShell after it comes up, and mysql_secure_installation is run in the post-provisioning script because it requires my input. If you wanted to run apt-get upgrade, you'd have to put it in this script as well because it'll bring up an ncurses interface for GRUB (at least for precise32 it will).

Working with Shared Folders

Shared folders in Vagrant under a Windows host can potentially lead to some troubles because of CRLF. Again, I'd recommend using an editor or IDE that allows you to choose the type of line endings you want and to run git config --global core.autocrlf input on your host machine. I've generally not run into any issues with shared folders aside from Walls of Pink from forgetting to change line endings.

In Conclusion...

All in all, Vagrant is pretty easy to use and get the hang of, even with its idiosyncrasies on Windows hosts. Just be careful about your CRLF line endings and shell provisioning scripts and you'll see your productivity skyrocket as you no longer have to worry about how to set up your development environment from computer to computer.

[xjtian]

Serving Static Files in Django: More Complicated than it Should Be

I love Django, and I've used it for a number of personal projects, but what really irks me is how difficult it is to understand how to properly serve static files. I don't think that the documentation makes the process clear enough to new users, and the multitude of questions about static files in Django on sites like stackoverflow shows that this is a pretty common problem for new users.

In this post, I'll explain what all the settings regarding static files actually mean, how to use them, and how to serve static files in development and production.

What the Settings Mean

STATIC_URL is the URL at which your static files are hosted. You can probably leave this setting as /static/ (note: for a relative path like that, you must end it with a forward slash). If you're hosting your static files on a CDN or otherwise outside your project root, then use the URL where the base directory for the files can be found.

STATICFILES_DIRS is a list of all directories where you're keeping your static files during development. If you've got multiple directories for static files, STATICFILES_DIRS should reflect that.

STATIC_ROOT is the local directory where all the static files that Django can find through STATICFILES_DIRS will go when you run ./manage.py collectstatic. When you're developing using Django's development server, you won't have anything to do with this setting.

Static Files During Development

Django's development server will serve static files for you as long as you include all the appropriate static file directories in STATICFILES_DIRS. If you have multiple directories for static files spread across separate apps, then you'll need to include each one of those in STATICFILES_DIRS. For example, in the following directory structure where any occurrence of assets/ contains static files,

my_app/ |----app1/ |----assets/ |----app2/ |----assets/ |----assets/ |----css/ |----js/

You can set your STATICFILES_DIRS to

from Unipath import Path PROJECT_DIR = Path(__file__).ancestor(3) STATICFILES_DIRS = ( PROJECT_DIR.child("assets"), PROJECT_DIR, )

In your templates, when you want to access a static file, use the {% static %} template tag and pass the relative path to the asset from one of the paths defined in STATICFILES_DIRS.

{% load staticfiles %} <!-- This will find base.css in PROJECT_DIR/assets/ --> <link rel="stylesheet" href="{% static 'css/base.css' %}"> <!-- These will fetch app-specific assets --> <link rel="stylesheet" href="{% static 'app1/assets/styles.css' %}"> <link rel="stylesheet" href="{% static 'app2/assets/styles.css' %}">

In development, this will "just work" - the development server's built-in static file server will look through the paths defined in your STATICFILES_DIRS to serve the right assets based on the path you pass in the {% static %} template tag.

Going to production throws a big wrench in all of this though.

Static Files During Production

The method above will only work when you're running the local development server with DEBUG = True. When it's time to push to production, things get a little more hairy. If you try to access your app on a production server with the development static file setup, you'll get a ton of 404 responses to static file requests. Why? Because your STATIC_URL isn't in your urls.py.

To get around this, first you'll need to condense all your static files into one location. Assuming you set STATICFILES_DIRS properly in your development settings, when you run ./manage.py collectstatic, you'll get all of your static files (and Django admin static files) dumped into the directory pointed to by your STATIC_ROOT setting.

Next, if you plan on serving static files from the same webserver that's hosting your Django app, you'll need to set up STATIC_URL to point to STATIC_ROOT. If you're using Apache, your VirtualHost file in sites-enabled would look something like

<VirtualHost *:80> ServerAdmin [email protected] ServerName example.com ServerAlias www.example.com DocumentRoot /home/user/example.com/public/ ErrorLog /home/user/www/example.com/logs/error.log CustomLog /home/user/www/example.com/logs/access.log combined Alias /media/ /home/user/example.com/mysite/media/ Alias /static/ /home/user/example.com/mysite/static/ <Directory /home/user/example.com/mysite/media/> Order deny,allow Allow from all </Directory> <Directory /home/user/example.com/mysite/static/> Order deny,allow Allow from all </Directory> WSGIDaemonProcess mysite python-path=/home/user/example.com/mysite/:/home/user/venvs/mysite_venv/lib/python2.7/site-packages user=user WSGIProcessGroup mysiteg WSGIScriptAlias / /home/user/example.com/mysite/mysite/wsgi.py <Directory /home/user/example.com/mysite/mysite/> <Files wsgi.py> Order deny,allow Allow from all </Files> </Directory> </VirtualHost>

This file is actually taken directly from the Apache config for one of my projects in production. Basically, it maps STATIC_URL, which is set to/static/, to the STATIC_ROOT directory where I collected all my static files (the same is done for media files too, but I won't get into that). This way, you sidestep Django's URL dispatcher which would throw a 404 if you tried to access STATIC_URL. Everything else is just boilerplate Apache config stuff.

Once you get your chosen webserver to serve static files correctly, things should work just fine - there's no need to alter the {% static %} tags in your templates or to adjust any of your static files.

The process for serving static files in production through a CDN or separate webserver is a little different, and outside of the scope of what I want to cover in this post - if you need to figure out how to do that, I suggest reading the bottom half of the documentation on static files.

In summary, working with static files in Django goes like this:

When developing, separate your static assets between apps (if necessary) and add paths to STATICFILES_DIRS as you see fit. How deep you go with these paths will determine how nested your static file collection will be - the more specific your paths in STATICFILES_DIRS are, the flatter your STATIC_ROOT will be.

Use {% static %} tags to fetch assets in your templates.

When you go to production, run collectstatic and set your webserver to serve any requests for files under STATIC_URL from STATIC_ROOT (manual config, unfortunately).

[xjtian]

New PR's in the Weightroom

On Friday, I hit a new 1RM on the squat at 275lb, at 150lb bodyweight. It's a huge milestone for me after 18 months of lifting.

And today, I hit a 305lbx2 deadlift as well. It's my first lift above double bodyweight, and I was really psyched to hit it for a double since volume squats earlier this night went pretty badly.

I think there are a lot of valuable lessons that one could take from weightlifting. The thing with strength training is that if you train smart and consistently, you'll always make progress. Once you start missing workouts or backing off on intensity because you don't want to put in the effort, your numbers fall off precipitously. The guys who are moving 700lb+ weights are the ones who are willing to grind for even the slightest gains - maybe a whole month of training just to increase their squat 1RM by 10 pounds.

There's a lesson to take away from it all that applies to just about everybody: if you want to become better at anything, you have to be willing to work smart and really grind. If you're willing to drill the mundane parts and work efficiently and consistently at it, you'll come out ahead of the impatient guys in the long run.

[xjtian]

Absolutely incredible work to recreate these logos almost perfectly with pure CSS. The creativity in the combinations of shapes used is mindblowing.

Now I'm wondering if it's possible to write a program to automatically do this kind of thing... it's probably not a trivial problem at all, but would be incredible cool.

[xjtian]

A very creative and entertaining bug report for MongoDB. If only all bug reports were as entertaining and detailed as this one

[xjtian]

When writing tests for your code, it can seem easy to ignore your code's dependencies by mocking them out. However, not using mocks can sometimes result in tests that are simpler and more useful.

I've always felt like relying too heavily on mocks for unit testing ends up coupling your unit tests to implementation logic. If you're a serious stickler for strict unit-testing (no dependencies, each test is completely independent, etc.), mocks are really the only feasible option for testing some kinds of functions. But in reality, I think having some not so ideologically strict tests for your code can catch all the same bugs while saving time spent writing the tests. Plus, these pseudo-unit tests won't need to be refactored every time you make some minor tweak to your implementation.

[xjtian]

The first part on a series designed to survey the design and analysis of various numerical methods will look at error propagation.

Having used numpy for some rudimentary scientific applications, I can attest to the horrors of rounding errors in floating-point arithmetic.

Of course, for 99% of uses, you don't ever run into serious problems with floats and doubles, but when you do, it can be devilishly tricky to debug.

[curlylang]

Today I started digging into how lexers work. Lexers are the first step of turning source code into executable. Their job is to take a stream of letters, symbols and whitespace and convert it into chunks of text which represent the various parts of the language. These chunks are called tokens. At...

[xjtian]

I've always been interested in the theory and practice behind compilers, so it's just my luck that I find someone working on their own compiler AND PL at an early stage in the game... real fun to read about in general.

I did a small project with a regex engine based on automata recently, and the tough part was that I had to parse the regexes with a recursive descent parser before I could feed them into the Thompson algorithm. It's almost circumlocutory in that you have to parse the tool that you're going to use to help you parse/lex stuff.

I enjoyed this article about parsing regexes with a recursive-descent parser: http://matt.might.net/articles/parsing-regex-with-recursive-descent/. It drives you in even more of a circle because it uses a CFG to define regexes ;).

I also remember non-minimized DFA's of regexes actually being much slower than the corresponding NFA because the NFA has O(n) states but the corresponding non-minimized DFA can have O(2^n) states. To me, it seemed like it wasn't worth the effort in converting the NFA from the Thompson algorithm to the DFA, but there must be a reason for it that I'm missing.

[xjtian]

The Two Sides of Algorithmic Problems

Last night as I was working on an aspect of move generation for my Scrabble project, I was struck by how messy the code ended up being but how clean the high-level procedure I had outlined was. It got me thinking about how many algorithmic problems in real life are very elegant in theory, but much messier in practice.

The problem I was working on was generating cross sets for valid Scrabble moves. A cross set is the set of letters that if played on the corresponding square, form a valid move. For example, the cross set for the square in blue in the example below contains the letters 'X', 'W', and 'P', among others.

The high-level procedure for computing cross sets for a given move is pretty simple to describe, and is as follows:

Compute the left and right parallel cross sets for the word made by the whole move (in the example above, the horizontal cross sets off both ends of 'HELLO').

For each letter in the move, get the whole crossing word in the perpendicular direction, and compute the appropriate left/right perpendicular cross sets.

But when it comes to implementing this in code, things get messy real quick, thanks to bounds checking when indexing into the board, code duplication because horizontal and vertical moves have to be handled differently, and an important edge case of middle crosses.

When an empty square is sandwiched between two filled squares, the entire procedure for computing cross sets needs to be completely changed when using a GADDAG. When reasoning about the problem at a high level, it's easy to just say that these cases need to be handled differently, but when it comes down to the code, it's just more of a mess on top of the previous solution.

That's the pattern I've found in a lot of algorithmic problems - when you're reasoning about a problem and a proposed solution at a very high level, you can focus on the crux of the problem and avoid getting bogged down by irrelevant or trivial details and corner cases. Personally, I love studying such problems at a higher level of abstraction because the problems and solutions are usually very concise and mathematically elegant.

But when it comes down to getting your fingers dirty and actually coding it, all of those details you once waved away with a nonchalant "it's trivial" come back to bite you, and your once-elegant pseudocode solution starts looking like mom's spaghetti in your IDE. When you start from a high level and move downwards like this, it becomes very tempting to start throwing lots of band-aid fixes on your code to account for corner cases that you missed, spiraling you further down into unmaintainable spaghetti-code until either you snap and rewrite the whole thing from scratch, or it "just works" magically sometime before your breaking point.

And sometimes, no matter how well you write your code and how many careful refactorings you do, it just doesn't come out looking or feeling as elegant as what you had imagined in the first place. And that's absolutely fine, but I'd be lying if I didn't say that it's always just a little disappointing when things don't go according to plan.