COPPER HEAD

MATH STUFF, MACHINE LEARNING, NEURAL NETWORKS, ARTIFICIAL INTELLIGENCE

[letgodbeyourgardener]

using recurrent neural networks for image completion

I read a cool paper today.

https://arxiv.org/abs/1601.06759

It talks about using recurrent neural networks for image generation. Recurrent neural nets basically examine a problem sequentially to predict what something should be based on everything that came before it. They have a lot of uses in problems that have an obvious time domain, like speech recognition, or audio processing, or even things like text generation where the overall context of individual pieces of data plays a larger role.

The interesting thing is that a lot of problems that aren’t obviously sequential in nature can be made into sequential problems and approached with these networks. In this paper they sequentially predict the values of individual pixels based on all the pixels that came before.

Here they show it at work completing images which have had their bottom halves covered up. A lot of the completions look pretty plausible, and show an impressive level of awareness about what’s going on in the image as a whole. One of my favorites is the dogs on the bottom row. All the model has to work with in the occluded image is that tiny wedge of land on the right, so it imagines the dogs sort of floating in a weird misty void. Interestingly it also puts forward a few possibilities that involve the dogs sitting on some surface like grass or sand. It had no way of knowing that from the occluded image, but it knows enough from the data set it was trained on to know that dogs are usually sitting on something.

They also use the model to generate some images from scratch.

These images have an interesting texture. It reminds me of datamoshed images, or the stuff you see when you watch a video in VLC and the decoder completely loses its cool and maps a couch from one scene onto a guy’s face in the next scene. They also have a certain jaggedness to them which other models I’ve seen don’t have.

[letgodbeyourgardener]

using artificial neural nets to analyze and model raw sound

The deep mind team have been working on something called WaveNet, and published their results about a month ago. It’s a new method for using convolutional neural networks to model raw audio, sample by sample. It’s pretty crazy to me that it’s possible to get that granular and still walk away with meaningful data, but convolutional neural networks are also used to analyze and model the deep mind pictures pixel by pixel.

https://deepmind.com/blog/wavenet-generative-model-raw-audio/

I think I’m having trouble breaking out of this algorithmic way of thinking. When you get so granular that you’re looking at individual samples in a waveform or individual pixels and color channels in an image, it seems intuitively impossible to get back to any kind of meaningful big picture. It seems like a contradiction to me but that basic idea is the foundation of how these convolutional networks work.

They have maybe 10 or 20 layers of neurons, and slowly build in abstraction as they go along. The first layer deals with regions around individual samples or individual color channels on pixels, basically the most granular and raw interpretation of the data. Then those neurons are pooled together and the next layer deals with the maximal elements of those pools, then those neurons are pooled, and so on until we reach a high level of abstraction. Each successive layer takes the most significant features of the previous layer and builds further connections, which is what allows the model to go from looking at contrast in small regions of pixels to identifying lines, then from identifying lines to detecting outlines and closed regions, and so on until a meaningful concept of what a feature actually is emerges on the other side.

I have no real way of visualizing this or wrapping my head around this concept. I’d like to write a convolutional network myself so I can peel it apart layer by layer and figure out what’s going on, but that feels a little inaccessible to me right now. Networks like this could probably write some interesting music with the right training. There’s a small sample of that toward the end of their blog post.

[letgodbeyourgardener]

These are some of the images created when a neural network for image classification is given an image of white noise as input, then told to amplify whatever it sees and use that new image as the next input. This causes a feedback loop where the net detects the same things it already amplified and amplifies them even further. Ultimately what we’re seeing are examples of all the things the network knows about, generated from random noise.

They’re like heroes of might and magic 3 meets where’s waldo on acid, but they’re also kind of an interesting insight into how the model sees the images its fed. You can get a sense of what elements of shape and structure and contrast are important to the net for classification, and which parts are ignored. The mountains are overwhelmingly cool turquoise, and everything has a weird pastel thomas kinkade vibe. I wonder why all the colors are so mellow, and why we don’t see more piercing neon shades or clashing colors. It could be that the way the images are processed kills a lot of those colors, or maybe the neural net is sticking to some definition of niceness or naturalness that makes the color balance kind of smooth and harmonious. This model was also presumably trained on landscapes and architecture because there aren’t like dog snouts and paws and beaks coming out of everything.

This is the blog post where they explain what’s going on and how the neural net works, and do a bunch of other cool stuff with the model:

https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html

[letgodbeyourgardener]

using neural nets to smoothly manipulate images (and make new ones)

So I saw this a few days ago and it’s incredibly cool. It’s a technique using a neural net to allow you to smoothly edit images into different designs without weird artifacting or distortion. It also allows you to create natural looking images entirely from scratch using extremely simple brush strokes.

The natural image manifold they mention in the title is the secret sauce that makes this all work. The basic idea is that the natural image manifold is a mathematical structure that contains all “natural” images, which basically means all images that satisfy certain conditions of niceness, like not being full of editing artifacts, not having sharp discontinuous transitions, and not being merzbow album covers.

Obviously this is a vanishingly small subset of all possible images, which includes every variety of merzbow album cover and every ugly thing in the universe. The natural image manifold is kind of like a sheet of paper floating in this huge space of all images, embedded inside of it and twisted up.

What they do is find a smooth path on this manifold from one natural image to another natural image. Since they constrain this path to the surface of the natural image manifold itself, every intermediate step along the way must also be a nice natural image.

This is why all the transitions between different shoes and bags in the video look so smooth. When you paint on the empty canvas, the neural net is finding the points on the natural image manifold that most closely correspond to the strokes you’ve painted. I think they look a little weird and kind of blurry and sharpened because of the way that their net is processing and downsampling the images, but I could be totally wrong about that.

The natural image manifold is a very weird and mathy topic which really deserves like 4 posts of its own, which I’ll probably do soon. It’s an incredibly cool idea but also very hard to explain informally because it’s basically a space of possibly hundreds of dimensions twisted and embedded in an even higher dimensional space in a particular way. Trying to talk about 4 dimensions is hard enough and there really isn’t a great way to intuitively visualize it. You have to fall back on more abstract stuff.

[letgodbeyourgardener]

the point of this blog

So recently I’ve been getting super interested in machine learning and artificial neural networks, mainly inspired by some of the stuff I saw from google’s deep mind thing and other interesting image and audio analysis problems. My friend Savannah told me that I should start a blog to sort of track my process learning about all of these things, which are all very new to me. I have a pretty solid math background to understand what’s going on in these systems, and I’m excited enough (at least right now) that I feel like I need to really delve into the topic as deeply as possible.

I’m going to post explanations of some of the concepts in a way that is hopefully interesting and intelligible, and I’m also going to post some of my vague ideas as well as any related projects I might be working on. I’m going to try to keep from getting way too technical at any point but I also want to explain any relevant mathematical ideas as best as I can when they come up.

What really excites me is the potential of these neural networks to generate art, both as an aid to creative expression and as a sort of form of emergent art without a real artist in the traditional sense of the word. I really want to look at the potential of machine learning to be a medium for creating novel art and expressing ideas, dual to its purpose as an analytical tool.