Artist Memo Akten has been exploring methods to generate neural network images in realtime. It started with his #Learningtosee project, with a âblankâ neural network, not trained on anything, visualizing fresh visual input:
This is a deep neural network that has not been trained on anything. It starts off completely blank*. It is literally âopening its eyesâ for the first time and trying to âunderstandâ what it sees. In this case âunderstandingâ means trying to find patterns, trying to find regularities in what itâs seeing now, and with respect to everything that it has seen so far; so that it can efficiently compress and organise incoming information in context of its past experience. Itâs trying to deconstruct the incoming signal, and reconstruct it using features that it has learnt based on what it has already seen â which at the beginning, is nothing. When the network receives new information that is unfamiliar, or perhaps just from a new angle that it has not yet encountered, itâs unable to make sense of that new information. Itâs unable to find an internal representation relating it to past experience; its compressor fails to successfully deconstruct and reconstruct. But the network is training in realtime, itâs constantly learning, and updating its âfiltersâ and âweightsâ, to try and *improve its compressor*, to find more efficient internal representations, to build a more âuniversal world-viewâ upon which it can hope to reconstruct future experiences. Unfortunately though, the network also âforgetsâ. When too much new information comes in, and it doesnât re-encounter past experiences, it slowly loses those filters and representations required to reconstruct those past experiences. Â Â
Following from this, he is using the same realtime method but with a neural net trained on art collections from 150 museums:
#DeepNeuralNet #LearningToSee (havin a hard time cos I'm playin w LR momentum gradclip etc in realtime - no post vfx, all from the DNN)
Here is a link to a video where it re-visualizes a drawing in realtime [EDIT 01/05/17 - Video now uploaded to Vimeo:]
You can follow more updates on the project at his twitter profile here