MACHINE VISION

The Machine Vision team will study theories and histories of visual technologies and current machine vision, analyse digital art, computer games and narrative fictions that use machine vision as theme or interface, and examine the experiences of users and developers of consumer-grade machine vision apps.

[machinevisionineverydaylife]

Seeing like a state of exception | 例外状态的视角

In a bittersweet combination of personal achievement and global emergency, I moved to Norway and started working on a new research project right when the COVID-19 epidemic started sweeping across the world. When I was settling in my new office and sketching out the next three years of academic careering, friends and relatives were quarantined in Wuhan or bracing for lockdown in other parts of the world. And as I’m writing this post, I’ve moved back to my university residence right before the WHO declared the virus a pandemic and my department building was locked for a few weeks of state-enforced social distancing. Regardless, the project I’m excited to be part of is about machine vision, and I’ll be looking into how this technology is deployed and utilized in Chinese everyday life, which seems to be slowly getting back to a degree of normalcy. In the meantime, China has been using machine vision to look into itself and try to track down the virus as it spread through its citizenry.

Following the outbreak through social media and news headlines, I noticed the diverse roles that machine vision has been playing in this situation, and followed how this technology is embroiled in variegated discussions ranging from the widespread distrust of authoritarian surveillance and the enrolling of homegrown tech industries into state propaganda to enthusiastic paeans to datafication and digital governance. More than a centralized attempt of the government to enforce a degree of legibility onto its citizens – what James Scott would call “seeing like a state” – China’s response to the COVID-19 epidemic seems to be benefiting from a new kind of sensing that is enabled by digital technologies deployed as an integral part of a state of exception. In Benjamin Bratton’s words, this is an important phenomenon because

[t]he optical positions of a state—how it sees the world and its constituents and how its citizens see themselves reflected through the ambient qualitative commons—might bear all the benefits and bankruptcies of earlier forms of communicative reason. (2015, p. 121)

But how did China’s “optical position” vis-à-vis the COVID-19 outbreak consolidate?

In the early days of the contagion, once the (by now disputed) origin of COVID-19 was pinpointed to the Huanan Seafood Wholesale Market in Wuhan, the virus was mostly tracked by concerned citizens via vernacular forms of coveillance: people would take photos of license plates and share them on social media, warning each other about the suspect presence of Wuhan residents in their town or district. In the absence of official measures or recommendations, all that people had at their disposal were mobile cameras, social networks, and the imperfect geolocational information provided by car plates. This strategy was quickly adopted by local authorities, that started tracking the movement of drivers across provinces; partial reports would circulate as copy-pasted news or screenshots, stoking a climate of ambient fear about Wuhanese residents. For example, one WeChat comment alleged that data from the Shenzhen Public Security System and the Center for Disease Control identified 60,000 people entering Shenzhen from Wuhan, worrying the local government and prompting a warning for the local population to stay put and prepare to spend the Spring Festival at home.

Tech companies quickly reoriented their platforms towards this sort of cross-regional surveillance, drawing on intersecting streams of data: as another WeChat message warns, Alibaba had supposedly tracked 6,000 Wuhan residents conducting Alipay transactions in Shanghai over a single day, hinting at a massive exodus from the city and an impending danger for the coastal metropolis. For many, this repurposing of smart city systems and platform urbanism was a foreseeable confirmation of the state’s surveillance capabilities. As Liza Lin documents on the Wall Street Journal, tracking the movement of infected or at risk individuals was a readily available capacity of China’s surveillance apparatus, which extends from roads to subways and from smartphones to social media, combining visual identification with geolocation and other sorts of data trails.

Another intersection between COVID-19 and machine vision is more mundane, and revolves around people’s faces. As the virus spread, citizens resorted to using face masks (for a situated history of this object, see this), which were also made obligatory in the most affected provinces; yet, people quickly realized that wearing masks affected ths convenience and usability of face recognition systems including smartphone locks, community gates and payment terminals. Many noted the uncanny resonance with the proposed ban on face masks during the 2019 Hong Kong protests, which was supposedly motivated by the need to identify protesters; even though pandemic and protest are two radically different states of exception, they spotlight the face as a contested nexus of biopolitical affordances. Regardless, technological fixes often trump convenience in exceptional times: mask-vending machines that use face recognition to identify buyers and ration masks started popping up, and eventually a tech company in Chengdu developed a 3D face recognition system capable of bypassing face masks while also checking a person’s body temperature.

Besides car plates and masked faces, local tech companies have competed to play a central role in facilitating life under quarantine, which conveniently also helped spotlighting their commitment to public service by strengthening governmental responses. Alongside donations, emergency services and information platforms, artificial intelligence has predictably emerged as one of the central contributions of companies like Tencent, Baidu and Alibaba, which have offered their cloud computing platforms and predictive algorithms to researchers for efforts in tracking infection vectors and simulating drug compounds. A broader definition of AI has cemented the impression that this technology is powering China’s response to the epidemic, as exemplified by the many reports marveling at automated health information chatbots, autonomous vehicles used for disinfection and delivery, and a hospital ward “run entirely by robots” in Wuhan.

[machinevisionineverydaylife]

Seeing like a state of exception pt.2

One specific example of how the use of machine vision technologies is both overblown and contested is offered by drones. As the COVID-19 virus spread beyond Wuhan, some residents of small towns and rural areas uploaded drone footage depicting people engaging in activities outdoors being remotely reprimanded via on-board loudspeakers and prompted to wear face masks and head back home. Given the funny lines dubbed over the footage, it is likely that these videos were mainly produced by minor online celebrities as humorous content to be shared on social media or as public service announcements for local officials; and yet, outside of China they were framed as a dystopian example of authoritarian control in a surveillance state, while local state media appropriated them as examples of national innovation. In practice, drones have been used for a broad range of tasks, including holding informational QR codes over traffic, spraying disinfectant and checking temperatures at a distance. While their actual usefulness in these contexts remains questionable, as David Li notes this sort of experimentation is only natural given the size of China’s drone industry and the ready availability of heavy-duty drones for agricultural use across the Chinese countryside. 

QR codes themselves are perhaps the simplest form in which machine vision is utilized across China today, and throughout the outbreak they have been deployed as gateways for a wide variety of purposes – from linking citizens to quarantine declaration forms and cellphone roaming histories to directing digital payments and donations. The most relevant innovation here has been the system launched by Alibaba in Hangzhou that assigns a color-coded QR-code to each citizen, based on a self-declared travel history and health status connected to real-name identification, allowing them to clear checkpoints in public spaces. The green, yellow & red QR codes have subsequently been launched in Shanghai as well, and their integration in Alipay’s ‘City Services’ function showcases the efforts of platforms to become infrastructures. Similarly, ride-hailing apps have also introduced a system requiring users to scan QR codes when they use public transportation or taxis, in order to collect data and share it with health authorities to help track contacts between passengers.

It’s important to note that machine vision also powers screening and diagnostic technologies. Checkpoints set up at airports and stations and building entrances often deploy thermal cameras, and these have reportedly been installed in buses as well, warning drivers about the presence of people with a fever onboard. A short video shared by the Global Times shows thermal cameras embedded in “high-tech smart helmets” worn by police agents in Chengdu, while a robot uses them to check the temperature of passersby in Guangzhou. If temperature is not an entirely helpful metric given COVID-19’s long incubation period, medical imaging procedures are central to its diagnosis, and platform companies like Alibaba and Ping An both claim to have developed automated systems capable of detect coronavirus in CT scans with a higher accuracy than human evaluation in dramatically shorter times. More suspect claims – such as the development of a gait recognition system that could help tracking the virus – also abound, but it seems undeniable that machine learning applied to medical imaging can improve the processing of both citizen flows and medical data.

To answer my initial question: China developed its optical position in confronting the COVID-19 crisis by marshaling preexisting digital infrastructure and the surveillance apparatuses overlaid on top of them, and re-orienting large-scale technical systems (and the private companies behind them) towards the provision of an imperfect hodgepodge of technologies of vision: traffic and thermal cameras, drones and other automated vehicles, smartphones and QR codes, combined and nested in search of useful optics for the evolving state of exception. To be sure, this happened later than it should have, after cover-up attempts and the silencing of concerned medical professionals, with a delay that is costing more and more lives both in China and across the world. And to be even more sure, these technologies have been often tested and showcased in less-impacted cities and regions, while Hubei and its provincial capital Wuhan were struggling with much more dire lack of hospital beds and supplies.

But in general, the widespread use of machine vision in responding to the COVID-19 outbreak – with both the innovation of tried and tested diagnostic technologies and the experimental reconfiguration of large-scale surveillance systems and vernacular usages – raises important questions about machine vision and its often assumed dehumanizing or illiberal potential. Rogier Creemers has sharply identified an ontological shift in China’s digital governance from the model of the panopticon to that of the ‘panspectron’, which aims at making Chinese society “legible and predictable” (2017, p. 89) by intersecting constant streams of ambient data instead of relying on direct surveillance. The question to ask is how much these sorts of optics can contribute the kinds of seeing – at the inhuman sensory scales of viral contagion and global flows – that a state of exception requires to make itself unnecessary; in the case of a pandemic threatening to lock down the entire planet, this should by now be a pressing concern.

References:

Bratton, B. H. (2015). The Stack: On software and sovereignty. MIT Press.

Creemers, R. (2017). Cyber China: Upgrading propaganda, public opinion work and social management for the twenty-first century. Journal of Contemporary China, 26(103), 85–100.

Scott, J. C. (1998). Seeing like a state: How certain schemes to improve the human condition have failed. Yale University Press.

Written by Gabriele De Seta, originally posted on his blog 12.03.2020

[machinevisionineverydaylife]

Pre-histories and futures of Machine Vision

Part 2: Last Friday Linn Heidi Stokkedal went to the Pre-histories and the Futures of Machine Vision symposium as the Victoria and Albert Museum. The symposium explored early moment in the development of computer art and machine vision, from the mid-1960`s onwards. Artist, designers and curators discussed aesthetic and political impact that contemporary computer vision and machine vision technologies have today. The symposium was divided into two sessions. The morning session looked towards the past, and early computer vision and art. The afternoon session was with practicing artists and what they are currently working on and looking into.

The Afternoon session started with artist Anna Ridler: Automated Dreaming: Using AI in a Creative Practise.

Ridler talked about machine learning and what it is. She finds that most people are more fascinated with the algorithm behind it, but it should be more focus on the training sets that feeds the algorithm. Ridler works mainly with GAN`s (Generative Adversial Networks) and it is a form of unsupervised machine learning. We don`t really understand how it works, but we know it loves bright colors! The way it works is that one part of it is the Detective and the other is the Art Forger. Back and forward these two try to process imagery by confirming (The Detective) and producing another image (Art Forger) until it resembles the images its been fed.

Ridler is a big fan of how these images turn out. They have a meandering dream quality and many imperfections. She also thinks they inspire a different way of looking at photographs. We are no longer briefly glancing, like we normally do with images. With generative images we look closer, because we want to find the counterfeit, or study the deepfake.

Returning to the thoughts on why we should be more concerned about datasets, Ridler says they are the difficult part of making GAN`s. Algorithms are often open sourced and easy to get. Datasets are difficult to make, takes a long time and a lot of natural resources. To avoid bias in datasets, the best thing to do is to make your own. At least then you can be in fully control of the classifications of your data. “Classifying is human” she says, and we have always done it. Like this picture below where the world are divided into fourteen categories. Datasets are contemporary encyclopedias and they are linked to memory.

It is boring and time consuming to make your own set, but it is worth if it avoids bias. In her work Mosaic Virus every still is generated using GAN. When she worked on her Myriad project she entered in 10 000 photographs of tulips and it cost 800 euros. She said that when you are in the process of making your own datasets you become obsessed and sensitive. It seems like Ridler almost becomes a part of the algorithm. She states that the algorithm is often considered as the “art work” while the datasets are similar to crafts. She reminds people that information is PHYSICAL. It comes from a physical space, based on physical things and it is also hard physical labour. 

The next speaker was Alan Warburton: RBQFAQ. He is doing his PhD at Birbeck University and is an artist and animator. He talked about post-photographic practise and CGI history. He showed how depth map inside the new cameras works and how images are rendered from from models to looking like photographs.

Machine learning projects synthetic data into the wild and functions as a meditator between the human and machine perspective. Some of the processes he spoke about is similar to Adam Harvey`s V Frame where there is a reversed function of how we normally operate with machine learning. 3D-printing objects for it to be fed into an algoritm, instead of feeding real photos of objects.

The last speakers were Natalie Kane and Tobias Revell: What if Our World is Their Heaven. Kane works as a Curator of Digital Design at V&A and Tobias Revell is an Artist and Designer. Together they form the curatorial project Haunted Machines set up to explore the relationship between myth, magic and monsters in technology. A recent project is focusing on automated technological processes and how they shape notions of futurity and truth. Revell shared so many interesting projects and I will try and refer to many of them below.

An example they shared was a picture by Michael Hahn from 2009 called “Drone” which is not a real photograph but a rendered image based on popular imagination.

Another example is Crystal Bennes project: Development Aesthetics “chronicle the rise and rise of the inane language and visuals used to market new buildings and developments in London”. According to Revell this is foreclosing our arbitrary to imagine our future.

Another example is Liam Young`s Renderlands from 2015 where the artist collected unused 3D rendered landscape and made it into a movie.

A similar project is Hyperlinks or it Didn`t Happen by Cecil B. Evans. According to their website: It is narrated by the failed CGI rendering of a recently deceased actor, PHIL, and follows a group of digital beings—render ghosts, spam bots, holograms—as they search for meaning. Multiple storylines and materials collapse and converge to raise questions about what it means to be materially conscious today and the rights of the personal data we release.

Unreal Estate by Lawrence Lek is another project displaying an alternative world and future.

Revell is concerned about how to use subcultural tropes to change how the future looks. If not, everything imagining the future will look like Minority Report.

Kane talks about internet phenomena and rendered images representing real life. Lil Miquela who is a fictional rendered character on Instagram. Kane found it fascinating when Lil Miquela had an identity crisis when she became aware that she was a constructed person.

Shudu is another fictional character on Instagram, she is a dark skinned model that looks so great, that she actually ends up stealing jobs from real models.

Performance Artist Amalie Ullmann staged a five month perfomance online (Instagram and Facebook) called Excellences and Perfection. During this time she underwent a semi-fictional but quite extreme make-over.

Sondra Perry`s project Graft and Ash from 2016 demonstrates how technology was not good enough at the time to render the tone of her darker skin correctly.

At the last panel there were many good questions, but Joel McKim referred to Flusser answering a question I cannot quite remember but he said Flusser wrote that technical images are not windows, as one might like to think. It is overlaying the reality. Technical images are not revealing or seeing the world a new, but only seeing it through a digital layer.

Another tendency observed by one of the people in the crowd was that the first half of the symposium represented machine vision as playful and creative and a gesture of creation, while the afternoon-session had a more serious tone. McKim answered that it is probably because we are more exhausted by technical images now, because it is everywhere and patterning out everyday life. In the beginning it was more experimental and new. I imagine that in the start we had a different imagination and idea of how machine vision would develop, more akin to what we have read in science fiction novels. Now that we are here it is possibly a bit more dystopic then what we expected back then.

Written by Linn Heidi Stokkedal 04.03.2020

[machinevisionineverydaylife]

Part 1: Last Friday Linn Heidi Stokkedal went to the Pre-histories and the Futures of Machine Vision symposium as the Victoria and Albert Museum. The symposium explored early moment in the development of computer art and machine vision, from the mid-1960`s onwards. Artist, designers and curators discussed aesthetic and political impact that contemporary computer vision and machine vision technologies have today. The symposium was divided into two sessions. The morning session looked towards the past, and early computer vision and art. The afternoon session was with practicing artists and what they are currently working on and looking into.

The first speaker was Douglas Dodds: Vision and Accident: Creating the V&A`s Computational Art Collection. He is a senior curator in the V&A Word and Image Department and is also responsible for developing the digital art collections. His talk was fascinating, showing a wide overview from the beginnings of computer art and up until now. He has curated exhibitions including Chance and Control: Art in the Age of Computers (2018); Barbara Nessim: An Artful Life (2013); and Digital Pioneers (2009-2010). He spoke about a movement called the Algorists, that I have never heard of before. Which is anyone who works with algorithms. Traditionally this has been applied to mathematicians but now also artists. Read more about the movement and the artists behind it here.

To find digital artworks from their collection, go to: collections.vam.ac.uk.

The next speaker was Zabet Patterson: A Portrait by a Computer as a Young Artist. She is an Associate Professor at Stony Brook University and wrote Peripheral Vision: Bell Labs, the S-C 4020, and the Origins of Computer Art, published in 2015. She presented a close-reading of some computer images even before the the first computer art became an art gallery item. The images she presented is from a contest from the trade journal Computers and Automation in 1963 and 1964.

She argues that the images chosen by this journal as “computer art” is “suggesting new relationships - and new regimes - of perception and knowledge. These images also show different attempts to reinvent and reconceive vision as adjunct to the computational machine (extracted from symposium handout). She was referring this images as as an older mode of aesthetics using the ancient greek term: aisthesis: meaning sensation or perception (see senses), in contrast to intellectual concepts or rational knowledge. The images are a another way of making sense and suggesting new perceptions.

This image of a bullet projectile visualization is set up like a Muybridge shoot, with mutiple cameras capturing several parts of the movement. The image was made to be informative and to become a training set for humans. A tool made by machine, for human visions. This next image below is from a project called Celestial Telescope (SCO) and shows UV wavelengths of brightness of stars. Zabet claims that this image if Unheimlich (aka. uncanny) and claims is a image made by machine for the machines.

She explains also what separates these visuals from more modern computer images, is that these are made and produced by giant machine that are very expensive (always based in a military or a government institution). Another point is that the images are also not made by artists, but engineers, which is typical for the beginning of computer art.

The next presenter was Joel McKim: Cybernetic Visions: Early Experiements in Ai by UK Computer Artists. He is a Senior Lecturer in Digital Media and Culture and the Director of the Vasari Research Centre for Art and Technology at Birckberck, University of London. He wanted to talk about 3 different people who had an impact on early machine vision. He started his presentation by introducing David Marr, a neurophysiologists who published “Vision” in 1982. He argued that vision is an information processing system.

It was interesting to see how Marr had made a overview of the processing steps from input and into 3D. Which can be compared to Deep Neural Network processing that we know today.

I found this talk very fascinating because this made the connection between the human vision processing system and how remarkably similar it is to the machines vision.

The second person that McKim introduces is Edward Ihnatowicz, a polish cybernetic art sculptor, producing works especially in the 60s. The first sculpture he built is called SAM (Sound Activated Mobile) which had built in microphones, which made it respond to surrounding sounds. The robot was constructed looking like a flower, and them stem looked similar to a vertebrae that moved towards the origin of the sound being made. Ihnatowicz most significant work is The Senster, a large hydralic robot that responded to both sound and movement. It is the first robot to be controlled by a computer.

and as the last example McKim presents Harold Cohen. An artist that worked a long a robot called AARON for most of his career. Cohen`s art practice was inspired by reading audiences and learn about meaning and how it is being made. He wanted to understand what creativity was and where it came from. He worked with his robot AARON in performative acts for a while (such as Turtle in 1977), before moving away from the spotlight and into the studio. AARON was trained to see foliage and people (representational imagery) and the robot was strictly rule-based. Cohen says that AARON`s art appeared meaningful, but was more “stand-in-forness”. Cohen never meant that he was working with computer art, but in stead it was painting dealing with images about perception.

That concludes the morning session. Afternoon session summary you will find in the next post.

Written by Linn Heidi Stokkedal 02.03.2020

[machinevisionineverydaylife]

Linn Heidi Stokkedal testing the ImageNet Roulette by Trevor Paglen at “Training Humans” Exhibit 12.02.2020.

[machinevisionineverydaylife]

Training Humans

“Our images now look back at us. And we won`t always like what- or how- they see.” - Exhibit statement.

Artificial Intelligence (AI) has changed the way we view the world, but more importantly, how we ourselves are viewed. Machine vision is used to capture most features the human body can display. It can recognize gait, irises, fingerprints, faces and gestures. All these features are analysed and, as this exhibit demonstrates, is then use to classify people by gender, age, race, emotional state and so forth.

“Training Humans” are a curated selection of photographs that has been used to train AI, and it is possibly the first exhibit of its kind. This photography exhibit is also unusual in its almost complete absence of “traditional artists”. Here you will find vernacular photographs from everyday life, such as family events, ID photos and birthdays. Crawford states in the exhibit book (#26) that it is essentially “accidental art” made by humans.

If you are training different kinds of AI systems, you need datasets. If a company want to build a facial recognition system, they have to train the AI to learn to see faces. Which means you need a great deal of photos of faces and identification metadata built in to optimize the system.

Different examples from selected datasets has now been extracted from its servers, and are now curated beautifully on the windows and walls of Osservatorio Fondazione Prada in Milan. The curators Trevor Paglen and Kate Crawford are giving us an impression of how easy the images of ourselves are harvested and labelled.

The first floor of the exhibit presents different examples of “early” datasets. The earliest set is from 1963 with the “A Facial Recognition Report” funded by the CIA and this was from before the term Artificial Intelligence was as widespread as it is today. Even still, they experimented with automated systems within the context of the military. A more modern dataset is the Multiple Encounter Dataset-II (MEDS-II). This set consists of mugshots of deceased criminals supplied by the FBI to the National Institute of Standards. Because the people in these photos were arrested multiple times, they wanted to learn how people age and especially how a face can change over time. The worst thing about this is that the arrested people in the photos never gave their consent for the images to be used, and the photos were done before they were convicted of any crimes. The other datasets are studying gaits, irises and how different lighting can change the appearance of a face. Generally, we find that the early datasets explores training of AI within the bigger institutions and research laboratories.

What I found interesting is the difference between the “early” sets (1990s) and sets from recent years. The U.S Department of Defense Counterdrug Technology Development Program Office developed automatic face recognition for intelligence and law enforcement personnel called the FERET set. This set consists of portraits of 1199 people. The portrayed people have put make-up on and their nicest shirt and earrings. It appears very formal and bears resemblance to portraits from anthropological expeditions (like the portraits made by Sophus Trombolt of the Sami people in the 1900s).

When you enter the second floor this tendency is gone. Except for the JAFFE set (Japanese Female Facial Expression) and Cohn-Kanade Au-Coded Expression Database, all the images in the datasets are now “found” and “collected” images. Gathered from platforms such as Flickr and Instagram, the formal appearance is now gone, and is much more intimate and personal. The modern datasets also bears another visual trait. The curators made a visual point to underline the massive shift in scale. When you see the enormous walls with miniature photographs, you become overwhelmed by the amount of data presented.

These sets consists of billions photos extracted from internet and social media. Since it became a part of everyday life, the amount of online photography has exploded. With bigger servers and better algorithms, machine vision as a technology is growing quickly and the problems and responsibility that comes with it as well. People`s private photos has been labelled, categorized and harvested without their knowledge or consent.

The curators seem eager to highlight the issues of labeling people within these massive datasets. The simple measurements and metadata are becoming problematic moral judgements. This is especially apparent in the ImageNet by Li Fei-Fei and Kai Li from Stanford and Princeton Universities “to map out the world of objects” The dataset contains 14,197,122 images organized into 21,841 categories. Paglen and Crawford chose to focus on the humans, which amounts to a million images sorted into over two thousand categories. Browsing the curated selection of people, my first impression is how absurd this appears. It is so massive and so full of error. I would be sad to find my own image up there, among the different cruel and unscientific labels such as “bad person”, “failure” and “looser”.

The last section is actually an art installation by Trevor Paglen himself. He developed “Image Roulette” in 2019 and it reveals how neural networks work and discusses the classifying of people in machine learning systems. The American girl with the red hair before me was recognised as “Irish”. I was recognised mainly as a non-smoker and my age: 25-32, gender: female emotion: fear (0,99). This would change rapidly by slightly moving your head.

“Training Humans” is both intimate and terrifying. All the precious moments and selfies people share with their loved ones ends up as data in an industrious dataset. The exhibit is showing how images of us are represented (how the machine sees us) and how we have been defragmented into binaries. Paglen and Crawford demonstrates the obvious flaws, but also how powerful machine vision can appear. Data is the new currency and big corporations utilizes machine vision technology with a complete disregard of bias within the datasets. The need to classify humans go way back, but it has hardly ever been successful in its task. Crawford wonders if AI will produce a shift in how we understand ourselves and our relationship to others, and society itself. Very few people benefit from these technologies so far, they are only subjected to it. You leave the exhibition thinking twice before uploading another selfie.

Photos and text by Linn Heidi Stokkedal 13.02.2020

[machinevisionineverydaylife]

We asked Overhaus to design us some pictograms to use in presentations and workshops and we all fell in love with the image generation pictogram. We are looking forward to 2020 and all the exciting conferences, exhibits and workshops that will happen. We have also started planning a exhibit hosted by the Natural History Museum in Bergen, where we will curate artworks that discusses machine vision, but we will also host a big LARP as a way of engaging thoughts and ideas around technologies. Read more about it here.

[machinevisionineverydaylife]

Last week we had a wonderful workshop at Solstrand going through how we tag works in the database! We are intrigued by the new system and we hope to implement it into the database very soon.

[machinevisionineverydaylife]

ARS Electronica 2019 – Understanding AI - Part 3

Seoul LiDARs from Sookyun Yang on Vimeo.

Kim Albrechts Artificial Senses approaches artificial intelligence as an alien species trying to understand it by visualizing raw sensor data that our machines collect and process. In his work he “tames” the machine by slowing it down to be grasped by human sensory. We move away from the vision to other senses, however, the when sensor data of location, orienting, touching, moving and hearing are visualized the work brings me to the question of what is machine vision.In a digital world in which machines actually just see numbers any sensor input can be translated to visuals. Hence what the “machines sees” are data visualizations and any sense can be made visible for the human eye. However, the machine dose not need this translation. Kim questions the similarity of the images:

Source: https://artificial-senses.kimalbrecht.com/

Both sensing and mirroring human behavior is demonstrated in Takayukis SEER: Stimulative Emotional Expression Robot. In this work the machine is using face detection and eye tracking to create an interactive gaze creating an illusion that the robot is conscious of its surrounding. Facial motion tracking mirrors our emotions towards the robot and expresses it back simply using an soft elastic wire to draw a curve the robots eyebrows. According to the artist the experiment is not to answer the philosophical theme “Will a robot (or computer) obtain a mind or emotions like mankind”, rather it is reflecting back a human produced emotion.

The exhibition would not be complete if it would not raise the question: Can AI be creative? Followed by the recognizable Neural Aesthetics. Among others these section features projects such as Pix2Pix Fotogenerator, Living Portraits of e.g. Mona Lisa and Marilyn Monroe, and interactive installations such as Memon Akten’s Learning to See: Gloomy Sunday (2017) and ShadowGAN developed by the Ars Electronica Future Lab. When such tools generate new unique artifacts it is tempting to read machine creativeness into it, however several artist statements describe how human and machine creativity intertwine and that the artist is still doing a lot of decisions. This would rather lean towards being a new generative art technique and we will for sure see more tools such as NVIDIA’s GauGan which also could be tested in the Ars Electronica center.The development of such tools are directed to motion/graphic designers in advertisement agencies to produce cost efficient images. Drawing photo realistic landscapes or generating realistic photos of non-existing people helps agencies to avoid image and model royalties and expensive trips to photo shoot in exotic locations.

Last (depending of how one navigates through the exhibition), but not least, a wall dedicated to the wonderfully complex Anatomy of an AI system (2018) illustrations by Kate Crawford and Vladan Joler.

Photo credits of this blog post goes to our lovely Machine Vision project assistant Linn Heidi Stokkedal and some of the photos are taken by me as well. 

Written by Linda Kronman and orginally posted on Kairus

[machinevisionineverydaylife]

ARS Electronica 2019 – Understanding AI - Part 2

Opposite to the “face wall” a piece called GLOW developed by Diederik P. Kingma, Prafulla Dhariwal; OpenAI, gives a interactive experience linking data sets with characteristics. A set of 16 characteristics can be adjusted for the generated image. Playing around with the thing for a while showed for example that when maximizing the young parameter and glasses the glasses were very faded, however when going towards older the glasses got more visible showing that in the trained data set people tagged old were wearing glasses are more often.

Just the section of faces could have been an own exhibition, however we are just starting to get warm. Another big part of the exhibition was dedicated to explain various AI networks e.g. modeled on moths olfactory system or intelligent navigation systems modeled on rats. Interactive displays were also used to explain how current neural networks are modeled on neurons in the human brain. Here I learned about the scientific image technique “Brainbows”.

A section was dedicated to explain research in the field of cognition and consciousnesses. On one of the screens I got immersed into a work of Kurzgesagt (In a Nutshell) called The Origin of Consciousness – How Unaware Things Became Aware, and it turns out they have a whole channel of shot animations dealing with philosophical and scientific questions.

Further on one could experience training of networks on several levels. From how the weights and biases of single neurons change when training a network to training a system to recognize the predators of a mouse. This part of the exhibition was very well done, however, it was like an updated version of a science museum rather than a art exhibition. Though this is very much in style what Ars Electronica center is. An info layer of short texts and images also explained machine learning terminology e.g. what is the difference between Generative Adversarial Network, Recurrent neural network and a Convolutional neural network or explaining terms such as  Latent space or Activiation Atlas.

“How machines see” was also a topic in several works in the exhibition. A wall of screens depicted how various layers in a neural network recognized various features. The work visualized live layers, convolution and filters,  and how the artifacts shown in front of a web-camera finally was recognized and labeled  either successfully or not. Hence visitors could test the accuracy of the trained network and at the same time get insight onto how current AI “sees” and “understands” objects.

This work on Neural Networks Training developed by The Ars Electronica Future Lab was the most impressive attempt to open the black box of machine learning in computer vision.

How machines “see” environments e.g. is of course a big issue in computer vision. A section of the exhibition was exploring the challenges to develop self driving cars and obviously in traffic it is crucial that the communication between pedestrians and eventually self driving cars is smooth. Various aspects of the challenges to design an autonomous car was on display ranging from “Looking Through the Eyes of a Tesla Driver Assistant” to MIT’s Moral Machine a platform for gathering human perspective on moral decisions made by machine intelligence such as self-driving cars.

Korean collective Seoul LiDARs uses industrial technology in various artistic experiments. Their Volumetric Data Collector a bulky wearable is using 3D laser sensors (Lidar) which are usually implemented in autonomous vehicles. A video work illustrates how such tech extends the sensory organ of a human body while walking in Seoul e.g. on the shamanistic mountain Ingwasan.

[machinevisionineverydaylife]

ARS Electronica 2019 – Understanding AI - Part 1

Earlier in the autumn the basement of Ars Electronica Center had been reopened with two totally new exhibition, both relevant for the Machine Vision Project: Understanding AI and Global Shift. We visited these exhibitions both for the festival pre-opening  program and late for a deeper dive into the very information rich and detailed exhibitions. All together we probably spent 5 hours there without really able to cover everything in depth. 

Understanding AI What is artificial intelligence? And what do we actually know about human intelligence? How intelligent can artificial intelligence be in comparison? And more importantly: what effects will the advances in this field have on our society? https://ars.electronica.art/center/en/exhibitions/ai/

Understanding AI combines artworks, research, visualizations and interactive experiences trying to explain what AI is now as in Machine Learning. The exhibition emphasizes on the positives of AI, however, between the lines also issues of privacy, biases and other “highly questionable” use of AI is brought forth. Whereas the Uncanny Values exhibition in Vienna was bringing fort social, economical, ethical and democratic issues from a discourse around ethical/fair use of AI the Ars Electronica exhibition was very didactic and aimed to explain the technology in a very detailed way.

Following some highlights from the perspective of my research:

When entering the exhibition the first thing one faces is facial recognition. Later one realizes being part of the WHAT A GHOST DREAMS OF (2019) artwork by artist collective H.O. According to the artist the artwork questions: What do we humans project into the digital counterpart we are creating with AI? It is getting to know our world without prior knowledge and generating data that never existed. What are the effects of using AI to produce works of art? Who holds the copyright? And what is AI, the “ghost,” dreaming about, and what does that mean for us as human beings?

Getting down the stairs to the basement we naturally continue to explore questions about facial and emotion recognition. Mixed on a wall are research papers, websites, industry apps, viral deep fake videos and (artistic) research such as Adam Harvey’s Mega Pixels  or Gender Shades by Joy Buolamwini team.The mix brings applications of facial and emotion detection both showing how advanced the technology however questioning the ethical aspects of it as well as. Research shows how “fake” faces produced and getting more realistic bringing also forth problematic uses of deep fakes.

On the long wall one can try Affectiva’s emotion detection app. One can also watch Lucas Zanottos film in which he used very simple everyday objects to express feelings, together with the emotion detection app these works raises questions of how emotions are mapped to our facial expressions.

Further the visitor can read how neural networks were trained to recognize criminal faces in a Chinese study, and how e.g. Amazon Face recognition failed to mach lawmakers faces.

One section of the wall questions what can be read in a face followed by examples such as face2gene and faception the first an app to recognize genetic syndrome from a child’s picture and the second profiling faces into categories such as High IQ, Poker face, Terrorist or Pedophile. Even if info layers refer to Physiognomy here a sharper tone on condemning such applications and revealing their dangers would have been beneficial. Also the the highly alerting study on training AI to recognizes sexual orientation from facial images is only paired with a question if AI understands what it sees, rather than implications of making such an study or application in the first place. How ever a section of the wall is dedicated to problematic use of facial recognition in predictive policing and oppressive use in a surveillance state such as China. The anecdote on the wall referring to public shaming of jail walkers.

One section on the wall is turning our attention to the generation of faces using neural networks. The “Obama Deep Fake” by Researchers at the University of Washington and AI generated news anchors are paired with artist Matthias Nießner face2face in which he plays the puppet master with Trumps face. In addition a research on generating very realistic fake faces by a company called NVIDIA shows how fast such fakes are getting harder to recognize. Here I would like to seen a segment on how spoofing fakes is also very important as described e.g. by artist Kyle McDonald

[machinevisionineverydaylife]

Ars Electronica 2019 – Global Shift

In the newly renewed Ars Electronica center we visited two exhibitions Understanding AI and Global Shift. Understanding AI was very relevant for our research in the Machine Vision project find a detailed blog-post about it here. The Global shift was interesting from the Machine Vision project because it was a representation of sensing the Earths surface through various technologies and of course satellite images and satellite technology were very prominent in it.

Global Shift Humans have made far-reaching changes to the natural living environment for their own purposes. Apart from the concrete wastelands and the refuse we produce, there are many other examples that testify to this. But it hasn’t always been that way. https://ars.electronica.art/center/en/exhibitions/globalshift/

Several of the artworks in this exhibition was using satellite images and imagery from the orbit to “witness” the Anthropocene, how we humans have been shaping the Earth.

An impressive work, by Seán Doran, ORBIT – A Journey Around Earth in Real Time is a  real time video reconstruction of time lapse photography taken on board the International Space Station by NASA’s Earth Science & Remote Sensing Unit, was to be viewed on a huge high definition screen. This epic travel around the Earth talks about the beauty of the earth , however, it also implies the fragility of our ecosystem and how our current presence is the biggest threat to destroy it.

Terra Mars by SHI Weili is a imagination of Mars like Earth. The artist trained an artificial neural network with topographical and satellite imaginary from Earth and then applied it on topographical data of Mars. This remapping creates a visual improbable(?) hallucination of a new planet.

In terms of visualizing the invisible materialities of machine vision from the orbit two works were shown that map all the “stuff” humans have launched into space. Stuff in Space and Quadrature‘s Orbits visualize man-made objects in space, relating to discourses of satellite debris.

There was also a Digital Citizen part in this exhibition with e.g. Joanna Molls and Tactical Techs The Dating Brokers a million online dating profiles from USDate which the artist payed 136 euros for. This “black market” of profiles including several images make me wonder if they end up fueling machine learning for face recognition. As Adam Harvey has shown in Megapixels the collection of training data is far from ethical or respecting the privacy of people.

—-Acknowledgements—- Photo credits of this blog post goes to our lovely Machine Vision project assistant Linn Heidi Stokkedal and some of the photos are taken by me as well. 

Written by Linda Kronman and originally posted on Kairus

[machinevisionineverydaylife]

This month we attended the Nordmedia conference in Malmø, where we hosted a workshop on our database. We introduced the project and gave an introduction to our tagging system and also showed how we can use Gephi to visualize the data. We also presented questions regarding our methods and about our tagging system, and we received a lot of valuable feedback.

[machinevisionineverydaylife]

The Machine Vision team having a last group meeting before the summer holiday starts. We are preparing our database to be presented at a workshop in Malmø in August and we are very excited about it. Happy summer!

[machinevisionineverydaylife]

The Machine Vision team had a database workshop at Solstrand Hotel last week. We invited scholar Annette Markham to look over our database with fresh eyes and together we came up with a whole new approach for entering data. We are looking forward to test the database further and to see how we can work with the new data sets.

Our goal is to have a database with over 500 entries that we can present at Normedia in Malmø in August.

[machinevisionineverydaylife]

Everybody Jam!

Drones or UAV (Unmanned Aerial Vehicles) are becoming almost a household item and they are creating headlines in the news where they disturb or create major disruption, like at Gatwick airport last December.

New laws are continuously being made to deal with the challenges that come with new technology. It is illegal to shoot down a drone in most countries because it is considered an aircraft. Now you can get a weapon that jams the signal of the drone, where you can force it to land or drive it out of the area you are in (and want to protect). Some jammers are also able to shut down the camera (image signal).

Drones operate on many different levels in terms of their use. Most people associate them with warfare, but also delivering packages on your door or planting trees. Drones are a tool for the state and government for mapping or surveillance, but can also be used by activists to provide footage in a demonstration against powerful corporations.

Anyone who is anti-drone probably already looked into acquiring one. But it does not appear to be available for sale for other than “professionals” since the companies making them are marketing the jammer as perfect for no-fly zones, key institutions, important facilities and large scale events.

The design of this Defender Series UAV Jammer is interesting because the company made the most obvious silhouette similar to a machine gun or rifle, I am certain that the function would still work in a different shape. 

But this will be explained better when you watch youtube videos of men trying out the jammer. All the action of this handheld device is similar to the action of shooting a gun. You AIM at your target, then SHOOT.

It will be interesting to see where this technology will be in just a couple of years, if jamming signals from drones and other intrusive gadgets will be as normal as putting a wash on, or any other everyday chore. Will jamming signals be prohibited or will everybody jam?

Written by Linn Heidi Stokkedal

[machinevisionineverydaylife]

Maskinsyn i spill, eller hvordan jeg ble spillforsker

Toppbilde: Sims Community

Hvor mange spill finnes det der man styrer droner? Plasserer spill spilleren for å se eller bli sett når overvåkningskamera er en viktig mekanisme? Gjør det noe med måten vi oppfatter verden på når vi spiller Augmented Reality-spill som Pokémon Go? Hva skjer egentlig når spill igjen og igjen legger spillerperspektivet i riflas sikte? Hvordan påvirker perspektiv og mediert visuell informasjon vanskelighetsgrad? Skal Ragnhild slutte å skrive spørsmål snart?

Machine Vision-prosjektet

(Nesten) alt dette faller under paraplyen vi kaller maskinsyn. Maskinsyn er når maskiner og algoritmer registrerer, analyserer og representerer visuell informasjon. Disse teknologiene er overalt i samfunnet vårt: Overvåkningskamera, Snapchat-filter, deepfakes (som når Yama Wolasmal snakker flytende mandarin og samisk uten å kunne språkene),  følelsesgjenkjenning, ansiktsgjenkjenning og medisinske bilder er å nevne noen.

Snapchats nyeste filtre brukt på et pauset bilde av Aloy fra Horizon Zero Dawn. Kodinga av hva som er «feminint» og «maskulint» er slående.

Teknologiene i seg selv er interessante, men bruken og forståelsen av de er av spesiell interesse for forskningsprosjektet jeg nå er en del av. De positive sidene med Yamas nyfunnede språkekspertise (eller for å ta noe litt mer kjent: General Leias datagenerering) åpner for interessante spørsmål om hvordan vi bruker og stoler på visuell informasjon.

Maskinsyn som vi møter i samfunnet viser seg nemlig gjentatte ganger å reprodusere fordommer. Såpedispensere som ikke reagerer på mørk hud og algoritmer som tagger mennesker med gorilla er bare noen få eksempler. Dette oppsummeres vakkert og dønn ærlig i Joy Buolamwinis «AI, Ain’t I a Woman»?

Noe har gått skeis i algoritmens klassifisering her… Bildeutsnitt fra: Medium

Disse eksemplene viser at mangfold i teknologiutvikling er viktig – tilfellet med såpedispenseren kan i alle fall unngås om man brukertester på flere hudfarger. Algoritmene er litt mer sammensatte, men når maskiner i større og større grad tar over prosesser som tidligere har blitt gjort av mennesker, er det grunn å spørre seg hvordan de maskinene er programmerte. Roboter og algoritmer opererer innenfor strenge system, men disse er lagd av mennesker. Det er altså snakk om en formbarhet (og ofte manipulering) som ofte er skjult for brukerne.

Målet til hele prosjektet (ledet av Jill Walker Rettberg) er å utvikle et teoretisk rammeverk for å forstå hvordan vanlig bruk av maskinsyn  påvirker oss. Vi har en programmerer, en forskningsassistent, snart en antropolog, og tre stipendiater som ser på hvert vårt medium: narrativ, digital kunst og spill. Psst: Jeg har spill!

For mer informasjon, følg prosjektet på Facebook og Twitter!

En helt vanlig dag på jobb.

Maskinsyn i spill

I spill kan maskinsyn være både grensesnitt og tema. Et eksempel på det første er når prosjektgruppa mi spiller Just Dance 2019, og kameraet registrerer og visualiserer bevegelsene våre for å se hvor godt de passer til maskinens regler. Da brukes maskinsyn i selve spillmekanismen – vi «utsettes» for dette. Når maskinsyn er tematisk framstilt, er det ikke spilleren selv som får oppleve det, men det presenteres i spillverdenen på en eller annen måte. I denne kategorien finner vi overvåkningskameraene i WATCH DOGS 2 eller Aloys Focus-teknologi i Horizon Zero Dawn.

Aloys «Focus» lar henne få en AR-versjon av verden rundt henne, der hun blant annet ser patruljeringsruter og svakheter til fiender. Bilde: RPGSite

I mitt prosjekt fokuserer jeg nå på tematisk maskinsyn, spesielt på maktrelasjoner og forholdet mellom menneske og maskin, men alle som har prøvd å skrive en større oppgave eller gjennomføre et forskningsprosjekt vet at det man tenker i starten ikke nødvendigvis er det man ender opp med til slutt…

Enn så lenge koser jeg meg med å bli kjent med mange fine kolleger, lese masse spennende litteratur, diskutere definisjoner av begrep med resten av Machine Vision-teamet og, selvfølgelig, spille spill. I et øyeblikks galskap har jeg streamet noe av dette på Twitch. Jeg spilte Night Trap fra 1992, hvor vi møter en operatør av overvåkningskamera i et hus og målet er å redde tenåringsjenter fra vampyrer.

Intet nytt under solen, altså.

100 vampyr-zombie-militære, et slumberparty, litt mer vampyrer, hus med masse koder og feller… høres helt innafor ut. Bilde: Gaming Shogun

I Night Trap presenteres maskinsyn åpenbart som et redskap som kan hjelpe, mens nyere spill har tendenser til å være mer kritisk i representasjonen av visuelt mediert informasjon. Kanskje. De skjulte spørsmålstegnene i det jeg skriver er mange, det er derfor jeg forsker på det!

But how?

Maskinsynteknologier viser oss både muligheter og fallgruver: vi kan effektivisere, nå lengre, multitaske, men også videreføre forutinntatthet og potensielt sette bort arbeid som ikke teknologien kan takle ennå. Hvordan vi tenker om og bruker disse teknologiene får konsekvenser, for eksempel når tenåringer redigerer bilder til det ugjenkjennelige for at de skal være «godkjent» for publisering i sosiale medier.

Vi påvirker og påvirkes, så spørsmålet er: Hvordan?

Eller sagt annerledes:

Se for deg Papers, Please, bare at det er en maskin som gjør hele utvelgelsesprosessen om du får slippe inn i landet eller ei.

Da er det plutselig veldig interessant hvordan den maskinen opererer.

Har du møtt maskinsyn i dataspill? Databasen begynner å bli stor, men kan gjerne bli større, så tips mottas med takk! Legg igjen en kommentar her eller send de til @ragnsolb Written by Ragnhild Solberg Originally posted on Spillpikene