Killipede: Programming A Music Video
A still from the Killipede music video.
[epilepsy warning: this music video and this post feature a lot of flashing colors. After the “keep reading” jump there are many animated gifs.]
I’m Andy Wallace, an independent game designer and professor who lives in NYC. I tend to split my time between systems-based arcade games and experimental, generative experiences. But today I’m talking about a music video.
Fanfare, the new album by Dan Friel just came out and I’m excited to announce that in addition to making the cover for the album, I created a music video for the single Killipede (which is a real noise-pop banger). You can watch the video right here:
This isn’t the first time Dan and I have worked together. In 2017 we released Bleep Space, a harsh sequencer toy which you can check out for free. I can’t recommend his albums enough. When he started wrapping up Fanfare he got in touch about making a music video for one of the songs.
What makes this project really fun is that this video was written entirely in C++. I’m a game designer by trade, and programming things that move is what I do. Rather than switch to standard animation tools, I stuck with what I’m good at and wrote the entire thing in code.
To create the style of the video, and to match Dan’s energetic and chaotic noise, I focused on JPEG corruption to create a set of filters and effects that I used in the video. The techniques used are not stylistic attempts to mimic the look of JPEG corruption; they are honest-to-god damaged JPEG files.
What does it mean to program a music video?
A scene from around 1:34 in the video
The idea might seem confusing. With most animation, somebody draws something on a screen or on paper that then gets brought into some type of animation program. For Killipede, instead of using traditional tools, the images and effects are all generated via code. The program exported each frame as an image, and eventually these images were strung together to make the video.
Here’s a look at my project. People love screenshots of code, right?
No outside assets (images, 3D models, etc) were used. All of the drawing was done with code. Any images used in the glitch process were first generated by the program and saved locally as JPEG files.
The one exception to this was this title card, which I made in Photoshop and which was used as an input image for one of the glitch effects early in the video.
Above you can see the one asset not generated via code (left) and how it was used in the video (right).
To help with the drawing-via-code, I used openFrameworks, a C++ library that I frequently use (PARTICLE MACE and Bleep Space were both written with openFrameworks). It’s a collection of C++ code that makes it a lot easier to draw shapes to the screen. It allows programmers to use commands to draw things like lines, rectangles & circles.
An example of programming a simple scene using openFramework commands.
I’ve done a lot of procedural animation for games and interactive projects using these tools. They may appear basic, but you can be surprisingly expressive with enough experience!
These drawing tools just make the basic image of each frame of Killipede, though. The real magic comes from the glitch techniques.
JPEG Corruption as a Style
Who doesn’t love some glitch art? Things go wrong and sometimes that makes them look great. JPEG is a way of compressing images in order to make the file size smaller. It is what is known as a “lossy-format” because this compression comes at some cost in the quality of the image. JPEG is widely used because it manages to make images very small, typically with very little loss in quality, but it is also susceptible to damage in a lot of ways. If you’ve ever seen a blown out, funky-looking image online, chances are it was saved, resized, posted online and saved again one (or ten) too many times. The quality of the file degraded as little aspects of JPEG compression happened over and over again. It’s not unlike a game of telephone. A little bit gets lost every time the image is repackaged. And once you know a bit about how JPEG compression works, there are little things that can be done to break the compression algorithm in specific and interesting ways.
0:34 in the video.
For Killipede I took advantage of this by having the program draw the initial frame using the basic shape tools built into openFrameworks, save the frame as a JPEG file on my computer, then reload and manipulate that file. This process of reloading, manipulating, and saving was often done many many times for a single frame of animation.
For some effects, the file itself is manipulated by adding or changing individual bytes. And in others, the image is loaded but then specific pixels are modified to do things like change the hue of part of the image.
Many of the effects use a second image, taken from earlier in the video, as the basis to perform some operation. I often used this as a way to transition between scenes in the video.
An example of using an earlier frame as a second image input. The spiral background from earlier persists into a new scene. This can be seen at 1:32 in the video.
This whole process was largely inspired by a single example project that ships with openFrameworks. In a collection of examples about input and output–most of which cover standard utility things like how to load text files or save an image–a project called imageCompressionExample stood out. Every frame it loaded in a photo, resized it and saved it. It also allowed you to insert a random byte into the file and do some real damage.
This started out as a photo of some busses
I found it fascinating and kept it in my back pocket. When Dan came along saying he wanted a glitchy video for his song, I knew I had my opportunity.
The Glitch Stack
While doing preliminary work on the project, I developed a lot of different ways of manipulating and breaking the JPEG images. There is a strong degree of randomness and unpredictability to all of them, and they are each expressive in their own way. I was able to exercise a lot of control in how and when glitches were used, but still be delighted and surprised by what was generated.
Laying the glitches on thick around 1:01 in the video.
Key to this was the ability to layer these effects. To accomplish this, I created a structure that allowed me to instantiate as many of these glitch effects as I wanted. Each effect receives a JPEG, mutilates it in whatever way it needs to, and then saves the result as a new image file. The Glitch Stack makes sure that the output of the first effect is used as the input for the second effect, and so on. Each effect in the stack can have its own settings and values as well, meaning I may use the same effect twice in the stack, but have it behave in slightly different ways.
An example image showing the output of several steps in a Glitch Stack on a single frame of animation.
Naturally, when saving the image for each step in this process, I have my code use the absolute lowest quality setting in order to introduce a little more entropy into the process.
For each scene, I programmed geometric shapes and movement and selected from my toolbox of glitch effects to find the best ways to “ruin” the scene. These effects as well as the basic movement of the objects in the scene were all timed to fit Dan’s music.
Some stacks required hundreds or even thousands of file saves to create the effect. Saving and loading images can be a time consuming operation in code, and this was no exception. I’m used to working in games where everything needs to run in real-time. Luckily, for fixed media like video, processing time is no issue, and I relished the opportunity to write effects that would take hours to produce a few seconds of footage. It allowed me to be greedy in a way I could never be with games!
Meet the Glitches
I wound up creating 10 unique glitch effects that were used in the video. It would be too much to go through all of them, but I want to highlight a few of my favorites. Once you know how they work, try watching the video again and you’ll see them all over the place.
Byte Insert
This is a variant on the effect in that openFrameworks example above that I found so intriguing. The basic idea is that the JPEG is loaded into memory, and then one bit is selected at random and changed to a random value before being re-saved. This creates big, flashy effects as the garbage value introduced often throws off the entire file from that point. This is often seen as a color distortion running horizontally across part of the image and then down everything below that point. Sometimes we get lucky and it shifts part of the image over, making the whole thing appear out of phase!
In this still from 0:20 you can see that the modified byte occurred about halfway through the image file, causing the rest of it to get a green tint. If you look closely you can even see the exact region in the JPEG compression where the byte was changed as it has become a small square of rainbow garbage data.
JPEG Compression (to a certain degree) stores the info for the image left to right and then top to bottom, meaning it describes the image in bands running left to right, moving down to the next band when it hits the right side of the image. This type of corruption displays this structure clearly. Everything to the left and above of the damaged part of the file is unaffected!
Here is the sample frame of animation from above being run through a double Byte Insert. Every frame, two bytes are chosen at random and replaced with garbage data.
This effect is very cool, but also very strong. It’s difficult to use as a filter for more than a fraction of a second because each frame can look wildly different from the one before it. To account for this, I made a second version of the effect that masks out a certain color from the input image (such as the background), and redraws everything else each frame to give the moving parts a bit more consistency. He’s a clip from an early test I did while working on this mask effect.
You can see how the red circle gets redrawn each frame, but the background is effectively transparent. The path of the circle gets covered up pretty quickly by the Byte Insert glitch.
Crunch
This was maybe the simplest of the glitches used. This effect works by taking the image, scaling it up (just 1 or 2 pixels bigger), saving it to disk, reloading, scaling it back down, and saving it again. When done once or twice, there is no real loss in quality, but when done 200 or 1000 times, the image gets very fuzzy.
Here’s the same sample frame of animation going through an increasing number of passes through the Crunch.
You can start to see square or rectangular fuzz. That’s because JPEG compression works partially by breaking the image up into square regions. By resizing the image to be just a pixel or two bigger, these regions get broken, and what would have been a small, difficult-to-notice line of fuzz becomes extreme as it is pushed back and forth over that border point.
Wanting a bit more control, I also wrote a version of this glitch that can be “cropped” so it only affects a specific part of the screen. The entire thing is crunched, but then only pixels from a given region are transferred on top of the source image.
A still from around 0:50 in the video where horizontal bands of Crunch are applied.
I could also set the number of Crunch passes each frame to do effects similar to screen-shake, where it would suddenly get very blurry and crunched and backoff over time. This was a nice effect for things like kick drum hits.
The first shot of the video where the kick drum is especially apparent.
While simple, this effect has the single longest processing time because of how many times it requires loading and saving an image.
Stretch/Resize
Related to the idea of resizing an image to break it, a different effect can be achieved if the resizing process is applied in a weighted way, so that instead of reverting the image to the same size, it gets wider or taller over time (for example, expanding by 4 pixels and then reducing by 2 over and over). In this version, the image experiences some of the region breaking from the Crunch, but it also stretches and distorts in a way that almost looks like digital melting, complete with heavy artifacting.
Here is the Stretch/Resize effect being used on the same sample frame.
Because this effect benefits strongly from being able to see it grow over time, I typically used it along with a mask so that the same image could be stretched while new things were drawn on top. It made for a great transition early in the video.
A clip from 0:16 in the video. To get this to fit on Tumblr, I had to crunch this one even more than most of the gifs in this post. Watch the video to see a clean version.
Cleaning Up
This is just a sampling of a few of the effects I wrote for this video. A lot of time and lines of code went into the 1 minute 45 seconds of Killipede. While certainly not the only video done in this style, I think if you made a pie chart of animation tools, C++ would make up a pretty tiny sliver.
It was a fun challenge to bring my game development and programming sensibilities to a non-interactive piece of media. And on a personal level, it was fun to make a music video again, as that was something I used to do back in the Flash days.
Make sure to check out Dan Friel’s new album, Fanfare. It’s been on heavy rotation in my apartment since it came out a few weeks ago.
Album cover by yours truly.
And if you like this kind of thing, you might love Bleep Space, the sequencer toy that Dan and I created. It’s free for iOS (on the app store) and Mac/PC here.