Depixellation? Or hallucination?
Thereâs an application for neural nets called âphoto upsamplingâ which is designed to turn a very low-resolution photo into a higher-res one.
This is an image from a recent paper demonstrating one of these algorithms, called âPULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelsâ
Itâs the neural net equivalent of shouting âenhance!â at a computer in a movie - the resulting photo is MUCH higher resolution than the original.
Could this be a privacy concern? Could someone use an algorithm like this to identify someone whoâs been blurred out? Fortunately, no. The neural net canât recover detail that doesnât exist - all it can do is invent detail.
This becomes more obvious when you downscale a photo, give it to the neural net, and compare its upscaled version to the original.
As it turns out, there are lots of different faces that can be downscaled into that single low-res image, and the neural netâs goal is just to find one of them. Here it has found a match - why are you not satisfied?
And itâs very sensitive to the exact position of the face, as I found out in this horrifying moment below. I verified that yes, if you downscale the upscaled image on the right, youâll get something that looks very much like the picture in the center. Stand way back from the screen and blur your eyes (basically, make your own eyes produce a lower-resolution image) and the three images below will look more and more alike. So technically the neural net did an accurate job at its task.
A tighter crop improves the image somewhat. Somewhat.
The neural net reconstructs what itâs been rewarded to see, and since itâs been trained to produce human faces, thatâs what it will reconstruct. So if I were to feed it an image of a plush giraffe, for exampleâŠ
Given a pixellated image of anything, itâll invent a human face to go with it, like some kind of dystopian computer system that sees a suspectâs image everywhere. (Building an algorithm that upscales low-res images to match faces in a police database would be both a horrifying misuse of this technology and not out of character with how law enforcement currently manipulates photos to generate matches.)
However, speaking of what the neural netâs been rewarded to see - shortly after this particular neural net was released, twitter user chicken3gg posted this reconstruction:
Others then did experiments of their own, and many of them, including the authors of the original paper on the algorithm, found that the PULSE algorithm had a noticeable tendency to produce white faces, even if the input image hadnât been of a white person. As James Vincent wrote in The Verge, âItâs a startling image that illustrates the deep-rooted biases of AI research.â
Biased AIs are a well-documented phenomenon. When its task is to copy human behavior, AI will copy everything it sees, not knowing what parts it would be better not to copy. Or it can learn a skewed version of reality from its training data. Or its task might be set up in a way that rewards - or at the least doesnât penalize - a biased outcome. Or the very existence of the task itself (like predicting âcriminalityâ) might be the product of bias.
In this case, the AI might have been inadvertently rewarded for reconstructing white faces if its training data (Flickr-Faces-HQ) had a large enough skew toward white faces. Or, as the authors of the PULSE paper pointed out (in response to the conversation around bias), the standard benchmark that AI researchers use for comparing their accuracy at upscaling faces is based on the CelebA HQ dataset, which is 90% white. So even if an AI did a terrible job at upscaling other faces, but an excellent job at upscaling white faces, it could still technically qualify as state-of-the-art. This is definitely a problem.
A related problem is the huge lack of diversity in the field of artificial intelligence. Even an academic project with art as its main application should not have gone all the way to publication before someone noticed that it was hugely biased. Several factors are contributing to the lack of diversity in AI, including anti-Black bias. The repercussions of this striking example of bias, and of the conversations it has sparked, are still being strongly felt in a field thatâs long overdue for a reckoning.
Bonus material this week: an ongoing experiment thatâs making me question not only what madlibs are, but what even are sentences. Enter your email here for a preview.
My book on AI is out, and, you can now get it any of these several ways! Amazon - Barnes & Noble - Indiebound - Tattered Cover - Powellâsï»ż - Boulder Bookstore
stormborns
origamiaround
titsay
ellievsbear
snowghoul
dededos
prguitarman
