page.title

A few of my favourite things

[discoblackout]

nginx + php-fpm - (defaults + documentation) = :(

Today I had to kickstart a web server to host an upcoming project I'm working on. I thought to myself, "hey, I've used Apache almost exclusively in the past, but I hear good things about nginx, let's give it a shot." Well, a few hours later and I'm regretting the decision a bit.

Overall, installing the relevant packages (nginx, php, php-fpm) on CentOS via yum, with the addition of the Software Collections (SCL) Repository for PHP 5.4, was pretty painless. There are quite a few handy tutorials around the web to walk through basic setup and configuration. However...

At first glance, the default configuration for nginx provided via the CentOS package looks like it covers the basics. Unfortunately, it's full of very questionable defaults, which only becomes apparent once you dig through the documentation to figure out what they mean. For example, it includes some commented out configuration for fast-cgi (php-fpm):

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000 # #location ~ \.php$ { # root html; # fastcgi_pass 127.0.0.1:9000; # fastcgi_index index.php; # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name; # include fastcgi_params; #}

The fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name; line looks straightforward enough until you realized /scripts is an absolute path. What happens is that any php script URIs matching the location expression get appended to the /scripts path. The assumption is that you've got a root directory called scripts that will house all your php scripts - I can't believe that's standard practice. A better option is changing the line to fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;, as suggested by most tutorials and the nginx common pitfalls document. Now the script name is appended to the document root as expected.

Next up is the root html; line. It is confusing because it sets the document root to a relative path called html. Relative to what, you might ask? According to official documentation it is relative to whatever parameters nginx was compiled with, or specifically --prefix=path. If this isn't set, it defaults to /usr/local/nginx. For whatever reason, the CentOS nginx package I was using appears to default to /etc/nginx (which I only discovered after getting phpinfo() working). Again, not very intuitive.

My last headache was digging into the "Passing Uncontrolled Requests to PHP" issue outlined on the nginx pitfalls page. A bug was discovered circa 2010 that noticed you could get arbitrary files to get passed through the php interpreter when using the default nginx configuration with php-fpm. In brief, if someone uploads a file to /myFile.jpg and then requests the path /myFile.jpg/fake.php, php/myFile.jpg gets passed through the php interpreter. Voila, compromised web server. Most sites recommend setting cgi.fix_pathinfo=0 modifying the nginx configuration to explicitly block this. Turns out that since PHP 5.3.9, released in January 2012, the php-fpm configuration contains a security.limit_extensions option enabled by default (set to .php). What it does is prevent anything without the matching extension from getting executed, mitigating the bug. Of course, the official php-fpm documentation contains no mention of it, and every tutorial I looked at insisted that cgi.fix_pathinfo=0 was the way to go. Alas...

The power of open source software is an amazing and wonderful thing, but it really gets marred by a poor understanding of the people who attempt to use them. Maybe it's a (possibly elitist) culture of keeping things complicated so that people 'earn' the right to use them, but I can't believe all those wasted hours are so necessary. /rant

[discoblackout]

Blender: Duplicate Curve Points

Lately, I've been working with exported contours from Paraview in Blender. They've been driving me nuts because Paraview exports a lot of duplicate points with its curves. Blender, which happily selects duplicate points in meshes, seems to be missing the same functionality when dealing with curves. Rather than convert back and forth between the two formats, I wrote a quick python script (*ahem*, my first python script to be exact...) to find them for me.

Basically, the script adds all the points in the object to a kd-tree, iterates through each point and looks for anything within a threshold distance away, and then selects it. Quick, dirty, and gets the job done. The threshold distance can be configured, as well as the number of points to check for. Technically, this could be set to the total number of points but this keeps it snappy for a huge number of points.

Hope this helps someone else!

import bpy import mathutils def main(): # Check we have the right object type if not bpy.context.active_object or bpy.context.active_object.type != 'CURVE': print("Function only works on curves") return # Configuration threshold = 0.001 pointChecks = 10 splines = bpy.context.active_object.data.splines points = [] # Build list of points for i in range(0, len(splines)): for j in range(0, len(splines[i].points)): points.append(splines[i].points[j]) # De-select everything splines[i].points[j].select = False # Build kd-tree kd = mathutils.kdtree.KDTree(len(points)) for i in range(0, len(points)): kd.insert(points[i].co.to_3d(), i) kd.balance() # Count duplicates duplicateCount = 0 for i in range(0, len(points)): for (co, index, dist) in kd.find_n(points[i].co.to_3d(), pointChecks): # Skip self if index == i: continue # Skip points we've already seen if dist < threshold and index > i: points[index].select = True duplicateCount += 1 print(duplicateCount, "duplicates found") main()

[discoblackout]

Math Blast! - Part 1

The first step to building an HTML5 canvas-based game is getting some boilerplate up and running.

HTML

I typically use the following to cover most of the basics:

<!DOCTYPE html> <html> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <title>Math Blast!</title> <meta name="viewport" content="width=device-width, initial-scale=1"> <link rel="stylesheet" href="main.css"> </head> <body> <div id="container"> <canvas id="canvas" width="640" height="480"> </div> <script src="codez.js"></script> </body> </html>

Nothing too exciting going on here other than the canvas element, of id "canvas", set to 640 by 480, a script tag pointing to codez.js, and a link to a CSS stylesheet called main.css. The next step is to follow this up with some JavaScript code to set up the basic game loop.

JavaScript Game Loop

**UPDATE**: I originally wrote this section based on my limited experience with writing game loops. After hunting around I discovered far more robust methods to achieve a more stable game loop. That version is what follows.

var curFrameTime = 0; // Time at the start of the frame var lastFrameTime = 0; // Time at the start of the previous frame var deltaTime = 0; // Difference in the previous times var lastFixedUpdateTime = 0; // Last time the fixed update was called var fixedTimeLag = 0; // How far behind the fixed update is var fixedDeltaTime = 33; // The fixed unit of time between fixed updates var startTime = 0; // Time the game started var totalElapsedTime = 0; // Amount of game time elapsed var totalRealElapsedTime = 0; // Amount of actual time elapsed var loop = function() { // Compute time since last frame curFrameTime = Date.now(); deltaTime = curFrameTime - lastFrameTime; lastFrameTime = curFrameTime; // Update real time totalRealElapsedTime = curFrameTime - startTime; // Call fixed update as many times as fits while (fixedDeltaTime <= (curFrameTime - lastFixedUpdateTime) ) { fixedUpdate(fixedDeltaTime); lastFixedUpdateTime += fixedDeltaTime; totalElapsedTime += fixedDeltaTime; } // Keep track of how far behind the fixed time is fixedTimeLag = curFrameTime - lastFixedUpdateTime; update(deltaTime); render(context); window.requestAnimFrame(loop); };

The game loop I've listed here is an adaptation of Glenn Fielder's version that he listed in his article, Fix Your Timestep!. The timing involved in this variation of the tradational game loop has some very nice properties that aren't apparent at first glance.

Essentially, the loop supports two sets of functions: update and fixedUpdate. update is called every frame, or iteration of the game loop. This should look after anything you want to happen as fast as possible, where synchronization issues aren't a problem. On the other side, fixedUpdate executes in fixedDeltaTime increments, a feature that is critical for things like updating a physics engine or synchronizing game content. Every frame, the loops checks to see how much time elapsed since the last loop iteration, and then attempts to fit as many fixedUpdates in as it can before calling regular update. The benefit is that if we use our fixedUpdate to, say, generate future game states, then we can use fixedTimeLag to interpolate between previous and future states to determine what state the game is in at that exact moment. This results in nice and smooth physics and animation that avoids the pitfalls of extrapolation where objects end up in some wacky place.

[discoblackout]

Let's Get Educational! In this upcoming series of posts, I'm going to document my attempts at making a rudimentary HTML5 canvas based clone of the classic Math Blaster game. Tune in for the exciting developments (*cough*)!

[discoblackout]

My first piece of art in a while. Threw this acrylic painting together in about 2 hours using this youtube video as inspiration.

It was a really good reminder that it doesn't take long do a semi-respectable job when you have a composition in mind.

[discoblackout]

My third and final image for my CS2504 Computer Graphics ray tracer. This one demonstrates glossy and non-glossy reflections.

[discoblackout]

This is a video I made as part of my project for CSC2524: User Experiences with Next-generation Input & Output Technologies. Was pretty cool being able to use one of those $10k Surf... sorry, PixelSense digital tabletops (Surface is such a better name!).

[discoblackout]

After moving the light sources around a small amount the "Terminator Problem" is eliminated. From what I've read, increasing the polygon count for the teapot (tessellation) or biasing the origin of the shadow ray off the surface can also work.

[discoblackout]

The infamous Utah Teapot rendered by my ray-tracer. Not quite sure what's up with the shadows on the bottom of it, but otherwise it's looking good. Update: Looks like I've encountered the equally as infamous "Terminator Problem". The triangles opposite the light source are incorrectly generating self-shadows. This is an artifact from interpolating normals but leaving the intersection point on the original flat surface.

[discoblackout]

My attempt at a Cornell Box. 4xAA (stratified), soft shadows (although I should really up the sample count), specular reflections, refraction. Update: Now with better shadows!

[discoblackout]

New features - glass-like materials!

[discoblackout]

New feature - soft shadows! [Updated - now in higher res]

[discoblackout]

Image generated by an early version of my ray tracer for CSC2504. Shiny balls.

[discoblackout]

The Pixels Among Us (Part I)

As atoms are to our universe, so to are pixels to digital images. And like our universe, sometimes we need to stop and wonder just what is a pixel at the end of the day, in order to use them optimally. Are all pixels made the same? Can we assume our digital image will look the same, both in dimension and colour, across all displays? How many pixels are too many, too few? To answer these questions, and more, we need understand how pixels work at the physical level, and then see how well they match up with their virtual counterparts.

Physical Pixels

On the physical level, a pixel is any (typically) light emitting unit that makes up a larger image or display. Our first assumption might be to think that this unit is consistent across all of our displays and monitors, but, alas, this is hardly the case. In fact, the physical composition of a physical varies radically from one display to another. Some are square, rectangular, and even triangular. Some are arranged in square grids, triangular ones, or hexagonal ones. Additionally, the pixels themselves are made of subpixels. Traditionally, we think of red, green, and blue subpixels composing a pixel. Even the subpixels can be inconsistent across pixels. Take for example Super AMOLED screens, where the pixels alternate between containing green/blue subpixels and green/red. In fact, these subpixels may not even occupy the entire area of a given pixel. So all in all, the takeaway message is that all pixels are not created equal, nor can we assume any given display is using one or another.

Different pixel grids as viewed close up

Virtual Pixels

After all that ambiguity with physical pixels, you'd think (or hope) the virtual definition would be far more consistent. Unfortunately, this is not entirely the case. Things like pixel aspect ratios do exist, but for most purposes we can treat a (raster) digital image as relatively standard: a rectangular grid of square pixels. Here, each pixel covers a given colour space in a myriad of ways, from RGB to CMYK, using varying amounts of bits to store the information. Since these virtual pixels have no inherent size, we have to denote a ratio of pixels to physical units via the term pixels-per-inch (ppi), or pixels-per-centimetre (ppcm) for metric folk, to map these to the real world. Not all digital images carry this meta-data however, so these is usually an assumption that one virtual pixel equals one real one. This has very important consequences when we begin to move an image from one display to another.

Digital Displays

Digital displays are typically defined by two metrics: diagonal size (inches or centimetres), and resolution (# of pixels wide by # of pixels tall). These two metrics give rise to the display's pixel density, defined as the number of diagonal pixels divided by the diagonal size, and given as ppi. The inverse of this ppi value is the pixel pitch (traditionally in millimetres), which is the upper bound for the contents of a given pixel. Now consider an image with fixed pixel dimensions of 640x480. On a display with a ppi of 90, this gives an image 7.1" by 5.3". On another display with a ppi of 300, this image is now 2.1" by 1.6". The image on the second display now occupies 9% of the area of the first display. This has important consequences when it comes to viewing it with the human eye, and so this leads to next topic on what limitations exist on our ability to resolve details at a given distances.

Pixel Density and Human Vision

Colloquially, we all know of the concept of 20/20 vision. However, what does 20/20 vision actually mean? Historically, the term came from work done by Hermann Snellen, of Snellen chart fame, who wanted to standardize measures of human visual acuity. His solution was to use the fact that in ideal conditions normal human vision allows a person to visually differentiate between two lines that are one arc minute (1/60th of a degree) apart. This arc minute value works out to 1.75mm at 20ft away, and so sets the bound on the size of an image read at that distance. By using 20ft as a standard distance, all vision metrics are scaled to this point. Thus, 20/20 means you can read at 20ft what normal visual should enable you to read at 20ft. Being unable to read a given line shifts the ratio below 1 and indicates comprised vision. What does all this mean for displays? Well, using the one arc minute value, we can determine the ideal ppi at a given distance from a display:

Steve Jobs' famous decision to switch to 300 ppi displays (so-called Retina Displays) for the iPhone was due to this exact idea. At 10"-12", the ideal ppi falls in the ~300 ppi range. The biggest caveat here is that this considers only those people with average (20/20) vision, viewing the screen in an environment free from glare and other ambient effects that may decrease visual acuity.

To be continued…

[discoblackout]

Enter The Matrix

This was originally going to be a longer post about driving an 8x8 LED matrix with two 8-bit shift registers. I've cut it down to just talking about the LED matrix itself, in case people with similar models happen to be looking for a circuit diagram. I had trouble hunting down a pinout diagram for the pins on it, so I made my own by testing each pin combination.

In my case, I picked up an 8x8 red LED matrix, labelled as a CILE 2088ABEG, that is about 5cm x 5cm in dimension. It has 24 pins, which seems odd at first given that you only need 16 to drive 64 leds. After some googling I realized it must be using the same case/form factor as the dual colour LED matrices, such as the red/green models. I'm guessing it saves on manufacturing costs to put both models in the same package. What it means for this model is that 8 of the 24 pins don't do anything. Additionally, the LEDs are rated at a If of 20mA and a Vf of 2V, putting out a wavelength between 620 and 625nm.

With some testing, I determined the pinout to be as follows:

Hope this helps!

[discoblackout]

Dem Links

I'm discovering the biggest challenge of operating a blog is putting aside time to actually produce content for it. I've had plenty of ideas, mind you, but I've procrastinated to no end in following through. As a consolation, today I'm going to post a few links I came across that really caught my eye:

How To Use Math To Crush Your Friends At Monopoly Like You've Never Done Before

This is a great breakdown of the probabilities behind Monopoly, and helps piece together an optimal strategy when playing the game. The funny part is that the gut-feeling based strategies most people use are pretty close to the mathematically ideal ones. In summary,

Go for the Orange, Red, and Purple - these will get landed on the most - as well as the Railroads and Board Walk.

Jail is the most likely tile to end up on, so use that to your advantage

Most players will end up moving seven spaces on their turn

Building three houses will give you the best ROI

Now in all fairness, this probably makes Monopoly even LESS fun to play as it is, so I suggest choosing the optimal strategy with care lest you lose a few friends along the way.

In Head-Hunting, Big Data May Not Be Such a Big Deal

An interesting interview with Google's senior vice president of people operations, Laszlo Bock. He discusses Google's use of data-mining to assess the value of their legendary hiring process. I found this personally very interesting, having been through the Google hiring process myself and faced their brainteasers.

The short of it, according to Mr. Bock, is that brainteasers and GPA checks have very little predictive value on later success at the company. Or, to quote him directly, "we found that brainteasers are a complete waste of time."

In my experience, all the brainteasers did was make for an unsettling interview, as I found myself scrambling to decipher the riddle instead of conveying anything useful or meaningful about myself and my abilities. It's vindicating to hear one their higher-ups admit that, as I surmised at the time, the process did little to actually gauge anyone's capabilities. Still, it did make for a memorable interview...

A Fully 3D Printable GlaDOS Robotic Ceiling Arm Lamp

Coolest. Lamp. Ever.

Be sure to watch this video of it moving around:

[discoblackout]

Pi-Duino - A Serial Adventure

Having recently acquired an Arduino UNO and Raspberry Pi, I've been beginning to play around with them in order to dream up a project idea. Today, I decided to toy with the idea of connecting the two together via the USB port and doing some serial back and forth talking.

My first approach was to adapt the "Analog Read Voltage" example to print the data to the Raspberry Pi's standard output. On the Arduino side, the code was used as-is:

Arduino

// the setup routine runs once when you press reset: void setup() { // initialize serial communication at 9600 bits per second: Serial.begin(9600); } // the loop routine runs over and over again forever: void loop() { // read the input on analog pin 0: int sensorValue = analogRead(A0); // print out the value you read: Serial.println(sensorValue); delay(1); // delay in between reads for stability }

On the Raspberry Pi end, I used the pySerial library to access the serial port - /dev/ttyACM0 in this case - at a baud rate of 9600.

Raspberry Pi (Python)

import serial import sys ser = serial.Serial('/dev/ttyACM0', 9600, timeout=1) while True: try: x = ser.readline() sys.stdout.write(x) except KeyboardInterrupt: print "Exiting..." ser.close() sys.exit()

Now, this looked great on paper, but after a few test runs it noticed some garbage values being read:

2 28 266 ... 264 ▒ 247 ... 264 ▒365 362

After a bit of googling I managed to figure out two things:

I clearly don't know enough about serial ports and protocols

the Arduino resets every time a serial connection is established

What this means is that every time my python script established a serial connection, the Arduino would reset. During this reset process, the Raspberry Pi was already buffering the output from the Arduino, output that was leftover from prior to the reset and was in the midst of being overwritten, The end result is that the first few reads were completely inconsistent.

Solution(s)

Raspberry Pi (Python)

import serial import sys import time ser = serial.Serial('/dev/ttyACM0', 9600, timeout=1) # give the Arduino time to reset time.sleep(5) # dump any input buffered during that time ser.flushInput() while True: try: x = ser.readline() sys.stdout.write(x) except KeyboardInterrupt: print "Exiting..." ser.close() sys.exit()

One way to mitigate this is to introduce a delay at the beginning of the python script on the Raspberry Pi, and flush its input buffer before reading. Sure enough, this cleans out the garbage values and makes everything much more predictable. The downside, of course, is that the first few lines of output from the Arduino are ignored.

If sequential output is required, an alternative is to use a handshake at the start, so as to hold off outputting values from the Arduino until the Raspberry Pi lets it know it is ready. An example of this can be found in the "Serial Call and Response (handshaking)" sample code. I've included an abridged version here:

Arduino

void setup() { // start serial port at 9600 bps: Serial.begin(9600); establishContact(); } void establishContact() { // wait for data to be sent while (Serial.available() <= 0) { Serial.print('A'); // send a capital A delay(300); } }

Finally, the auto-reset can be disabled outright with a few adjustments; see here