using artificial neural nets to analyze and model raw sound
The deep mind team have been working on something called WaveNet, and published their results about a month ago. It’s a new method for using convolutional neural networks to model raw audio, sample by sample. It’s pretty crazy to me that it’s possible to get that granular and still walk away with meaningful data, but convolutional neural networks are also used to analyze and model the deep mind pictures pixel by pixel.
https://deepmind.com/blog/wavenet-generative-model-raw-audio/
I think I’m having trouble breaking out of this algorithmic way of thinking. When you get so granular that you’re looking at individual samples in a waveform or individual pixels and color channels in an image, it seems intuitively impossible to get back to any kind of meaningful big picture. It seems like a contradiction to me but that basic idea is the foundation of how these convolutional networks work.
They have maybe 10 or 20 layers of neurons, and slowly build in abstraction as they go along. The first layer deals with regions around individual samples or individual color channels on pixels, basically the most granular and raw interpretation of the data. Then those neurons are pooled together and the next layer deals with the maximal elements of those pools, then those neurons are pooled, and so on until we reach a high level of abstraction. Each successive layer takes the most significant features of the previous layer and builds further connections, which is what allows the model to go from looking at contrast in small regions of pixels to identifying lines, then from identifying lines to detecting outlines and closed regions, and so on until a meaningful concept of what a feature actually is emerges on the other side.
I have no real way of visualizing this or wrapping my head around this concept. I’d like to write a convolutional network myself so I can peel it apart layer by layer and figure out what’s going on, but that feels a little inaccessible to me right now. Networks like this could probably write some interesting music with the right training. There’s a small sample of that toward the end of their blog post.