#visualimagereconstruction

[kamitani-lab]

ピエール・ユイグ Liminal: 人間のいない世界を表現する｜神谷之康（Yukiyasu Kamitani）

現代美術家ピエール・ユイグと私の研究室との間で、新たなコラボレーションが実現した。 2023年夏、ユイグから私宛てに一通のメールが届いた。それは、ヴェネチアのPunta della Doganaで開催予定の個展に向けた依頼だった。Punta della Doganaは、イタリア

[kamitani-lab]

Oriai/織合い - aya kawato

Oriai / 織合い　2017　2500×2000mm　Acrylic on wooden panel / 木製パネルにアクリル絵具 This painting, created by Aya Kawato, was inspired by the science fiction novel Facettes by Samantha Bailly, a French author. The story of Facettes revolves around the protagonist, a female neuroscientist who designs clothes. The emotions of the wearer are projected in real time through the clothes that are created. The novel ends with the invention of a dress that combines tradition and technology. Through her grid-painting style, Kawato combined technology with tradition in such a way that they coexist. The title of the painting is Oriai ( 織 合 い ). The common Japanese word, pronounced oriai ( 折 合 い ), means reconciliation or an agreement. The artist intentionally used the kanji 織 (which means weaving) in her title; this character is read the same as the rst part, ori ( 折 ), of the word as it is generally written. Bailly’s novel prompted Kawato to realize that nding a harmonious arrangement is the key for us when we face something unusual or di erent from us. Creating reconciliation, agreement, or seeking harmony is just like weaving a fabric. Kawato incorporated this idea into the title of her painting, which looks like a fabric of di erent kinds of threads. The image in Figure 1 is taken by the artist and symbolizes technology. It is an image showing what a person was seeing, reconstructed from an analysis of neural signals from the brain, using new technology called visual image reconstruction. This cutting-edge technology has potential for applications in creating visualizations of more subjective experiences, such as dreams and mental imagery. Because Kawato found that the image was a good foundation for her painting as it shows how technology and tradition coexist, she used it to print a pattern onto the lower part of the painting through silk-screening. これは、フランスの SF 作家 Samantha Bailly の『Facettes』という小説をモチーフに制作された作品です。『Facettes』は女 性の脳科学者が感情をリアルタイムに映し出す洋服を作るお話で、小説の最後には、テクノロジーと伝統を融合したドレスが 生まれます。川人はこのストーリーからインスピレーションをもらい、テクノロジーと伝統が共存する絵画を描きました。 この絵画のタイトルは『織合い / Oriai』です。本来の日本語のおりあい(折合い)ではなく、「織る」という漢字を使いました。 川人は『Facettes』を読んだ後、私達が何か異なるものや、普通でないものに直面した時、協調や和解を探ることがとても大 切だと改めて感じるようになりました。折りあいをつけることは、まるで異なる種類の糸で布を織るような行為です。織物の ように見えるこの絵画のタイトルには、そんなメッセージが込められています。 Figure 1 は、川人にとって、テクノロジーを象徴するイメージです。visual image reconstruction と呼ばれる技術を使って脳 からの信号を分析し、人が実際に見ている画像を再構成したものです。このテクノロジーは、その原理を応用することで、心 的イメージや夢のような主観的体験を画像化する可能性をも秘めています。テクノロジーと伝統が共存するペインティングに 適したイメージとして、そのパターンを画面の下方に、シルクスクリーンを使ってプリントしました。 Fig.1 ©Kamitani Lab, ATR

[kamitani-lab]

Attentionally modulated visual image reconstructions from brain activity

Attentionally modulated reconstructions of visual images from human brain activity measured by fMRITo perform the image reconstruction, we first decoded (tra...

[kamitani-lab]

Yuki Kamitani on Twitter

“Visual images can be reconstructed from fMRI brain signals. Are they just about stimuli or about subjective percept? In this new preprint, work led by Horikawa-san, we show that given overlapping images as a stimulus (left), attention (to 'red') alters the reconstruction (right) https://t.co/QqKzw01hEq https://t.co/V5wb41kuXd”

[kamitani-lab]

Visual image reconstruction from brain activity produces images whose features are consistent with the neural representations in the visual cortex given arbitrary visual instances [[1][1]–[3][2]], presumably reflecting the person’s visual experience. Previous reconstruction studies have been concerned either with how stimulus images are faithfully reconstructed or with whether mentally imagined contents can be reconstructed in the absence of external stimuli. However, many lines of vision research have demonstrated that even stimulus perception is shaped both by stimulus-induced processes and top-down processes. In particular, attention (or the lack of it) is known to profoundly affect visual experience [[4][3]–[8][4]] and brain activity [[9][5]–[21][6]]. Here, to investigate how top-down attention impacts the neural representation of visual images and the reconstructions, we use a state-of-the-art method (deep image reconstruction [[3][2]]) to reconstruct visual images from fMRI activity measured while subjects attend to one of two images superimposed with equally weighted contrasts. Deep image reconstruction exploits the hierarchical correspondence between the brain and a deep neural network (DNN) to translate (decode) brain activity into DNN features of multiple layers, and then create images that are consistent with the decoded DNN features [[3][2], [22][7], [23][8]]. Using the deep image reconstruction model trained on fMRI responses to single natural images, we decode brain activity during the attention trials. Behavioral evaluations show that the reconstructions resemble the attended rather than the unattended images. The reconstructions can be modeled by superimposed images with contrasts biased to the attended one, which are comparable to the appearance of the stimuli under attention measured in a separate session. Attentional modulations are found in a broad range of hierarchical visual representations and mirror the brain–DNN correspondence. Our results demonstrate that top-down attention counters stimulus-induced responses and modulate neural representations to render reconstructions in accordance with subjective appearance. The reconstructions appear to reflect the content of visual experience and volitional control, opening a new possibility of brain-based communication and creation. ### Competing Interest Statement The authors have declared no competing interest. [1]: #ref-1 [2]: #ref-3 [3]: #ref-4 [4]: #ref-8 [5]: #ref-9 [6]: #ref-21 [7]: #ref-22 [8]: #ref-23