Brain decoding: Reading minds
Jack Gallant perches on the edge of a swivel chair in his lab at the University of California, Berkeley, fixated on the screen of a computer that is trying to decode someoneâs thoughts.
On the left-hand side of the screen is a reel of film clips that Gallant showed to a study participant during a brain scan. And on the right side of the screen, the computer program uses only the details of that scan to guess what the participant was watching at the time.
Anne Hathawayâs face appears in a clip from the film Bride Wars, engaged in heated conversation with Kate Hudson. The algorithm confidently labels them with the words âwomanâ and âtalkâ, in large type. Another clip appears â an underwater scene from a wildlife documentary. The program struggles, and eventually offers âwhaleâ and âswimâ in a small, tentative font.
âThis is a manatee, but it doesnât know what that is,â says Gallant, talking about the program as one might a recalcitrant student. They had trained the program, he explains, by showing it patterns of brain activity elicited by a range of images and film clips. His program had encountered large aquatic mammals before, but never a manatee.
Groups around the world are using techniques like these to try to decode brain scans and decipher what people are seeing, hearing and feeling, as well as what they remember or even dream about.
Media reports have suggested that such techniques bring mind-reading âfrom the realms of fantasy to factâ, and âcould influence the way we do just about everythingâ. The Economist in London even cautioned its readers to âbe afraidâ, and speculated on how long it will be until scientists promise telepathy through brain scans.
Although companies are starting to pursue brain decoding for a few applications, such as market research and lie detection, scientists are far more interested in using this process to learn about the brain itself. Gallantâs group and others are trying to find out what underlies those different brain patterns and want to work out the codes and algorithms the brain uses to make sense of the world around it. They hope that these techniques can tell them about the basic principles governing brain organization and how it encodes memories, behaviour and emotion (see âDecoding for dummiesâ).
Applying their techniques beyond the encoding of pictures and movies will require a vast leap in complexity. âI donât do vision because itâs the most interesting part of the brain,â says Gallant. âI do it because itâs the easiest part of the brain. Itâs the part of the brain I have a hope of solving before Iâm dead.â But in theory, he says, âyou can do basically anything with thisâ.
Brain decoding took off about a decade ago1, when neuroscientists realized that there was a lot of untapped information in the brain scans they were producing using functional magnetic resonance imaging (fMRI). That technique measures brain activity by identifying areas that are being fed oxygenated blood, which light up as coloured blobs in the scans. To analyse activity patterns, the brain is segmented into little boxes called voxels â the three-dimensional equivalent of pixels â and researchers typically look to see which voxels respond most strongly to a stimulus, such as seeing a face. By discarding data from the voxels that respond weakly, they conclude which areas are processing faces.
Decoding techniques interrogate more of the information in the brain scan. Rather than asking which brain regions respond most strongly to faces, they use both strong and weak responses to identify more subtle patterns of activity. Early studies of this sort proved, for example, that objects are encoded not just by one small very active area, but by a much more distributed array.
These recordings are fed into a âpattern classifierâ, a computer algorithm that learns the patterns associated with each picture or concept. Once the program has seen enough samples, it can start to deduce what the person is looking at or thinking about. This goes beyond mapping blobs in the brain. Further attention to these patterns can take researchers from asking simple âwhere in the brainâ questions to testing hypotheses about the nature of psychological processes â asking questions about the strength and distribution of memories, for example, that have been wrangled over for years. Russell Poldrack, an fMRI specialist at the University of Texas at Austin, says that decoding allows researchers to test existing theories from psychology that predict how peopleâs brains perform tasks. âThere are lots of ways that go beyond blobology,â he says.
In early studies1, 2 scientists were able to show that they could get enough information from these patterns to tell what category of object someone was looking at â scissors, bottles and shoes, for example. âWe were quite surprised it worked as well as it did,â says Jim Haxby at Dartmouth College in New Hampshire, who led the first decoding study in 2001.
Soon after, two other teams independently used it to confirm fundamental principles of human brain organization. It was known from studies using electrodes implanted into monkey and cat brains that many visual areas react strongly to the orientation of edges, combining them to build pictures of the world. In the human brain, these edge-loving regions are too small to be seen with conventional fMRI techniques. But by applying decoding methods to fMRI data, John-Dylan Haynes and Geraint Rees, both at the time at University College London, and Yukiyasu Kamitani at ATR Computational Neuroscience Laboratories, in Kyoto, Japan, with Frank Tong, now at Vanderbilt University in Nashville, Tennessee, demonstrated in 2005 that pictures of edges also triggered very specific patterns of activity in humans3, 4. The researchers showed volunteers lines in various orientations â and the different voxel mosaics told the team which orientation the person was looking at.
Edges became complex pictures in 2008, when Gallantâs team developed a decoder that could identify which of 120 pictures a subject was viewing â a much bigger challenge than inferring what general category an image belongs to, or deciphering edges. They then went a step further, developing a decoder that could produce primitive-looking movies of what the participant was viewing based on brain activity5.
From around 2006, researchers have been developing decoders for various tasks: for visual imagery, in which participants imagine a scene; for working memory, where they hold a fact or figure in mind; and for intention, often tested as the decision whether to add or subtract two numbers. The last is a harder problem than decoding the visual system says Haynes, now at the Bernstein Centre for Computational Neuroscience in Berlin, âThere are so many different intentions â how do we categorize them?â Pictures can be grouped by colour or content, but the rules that govern intentions are not as easy to establish.
Gallantâs lab has preliminary indications of just how difficult it will be. Using a first-person, combat-themed video game called Counterstrike, the researchers tried to see if they could decode an intention to go left or right, chase an enemy or fire a gun. They could just about decode an intention to move around; but everything else in the fMRI data was swamped by the signal from participantsâ emotions when they were being fired at or killed in the game. These signals â especially death, says Gallant â overrode any fine-grained information about intention.
The same is true for dreams. Kamitani and his team published their attempts at dream decoding inScience earlier this year6. They let participants fall asleep in the scanner and then woke them periodically, asking them to recall what they had seen. The team tried first to reconstruct the actual visual information in dreams, but eventually resorted to word categories. Their program was able to predict with 60% accuracy what categories of objects, such as cars, text, men or women, featured in peopleâs dreams.
The subjective nature of dreaming makes it a challenge to extract further information, says Kamitani. âWhen I think of my dream contents, I have the feeling Iâm seeing something,â he says. But dreams may engage more than just the brainâs visual realm, and involve areas for which itâs harder to build reliable models.
Decoding relies on the fact that correlations can be established between brain activity and the outside world. And simply identifying these correlations is sufficient if all you want to do, for example, is use a signal from the brain to command a robotic hand (see Nature 497, 176â178; 2013). But Gallant and others want to do more; they want to work back to find out how the brain organizes and stores information in the first place â to crack the complex codes the brain uses.
That wonât be easy, says Gallant. Each brain area takes information from a network of others and combines it, possibly changing the way it is represented. Neuroscientists must work out post hocwhat kind of transformations take place at which points. Unlike other engineering projects, the brain was not put together using principles that necessarily make sense to human minds and mathematical models. âWeâre not designing the brain â the brain is given to us and we have to figure out how it works,â says Gallant. âWe donât really have any math for modelling these kinds of systems.â Even if there were enough data available about the contents of each brain area, there probably would not be a ready set of equations to describe them, their relationships, and the ways they change over time.
Computational neuroscientist Nikolaus Kriegeskorte at the MRC Cognition and Brain Sciences Unit in Cambridge, UK, says that even understanding how visual information is encoded is tricky â despite the visual system being the best-understood part of the brain (see Nature 502, 156â158; 2013). âVision is one of the hard problems of artificial intelligence. We thought it would be easier than playing chess or proving theorems,â he says. But thereâs a lot to get to grips with: how bunches of neurons represent something like a face; how that information moves between areas in the visual system; and how the neural code representing a face changes as it does so. Building a model from the bottom up, neuron by neuron, is too complicated â âthereâs not enough resources or time to do it this wayâ, says Kriegeskorte. So his team is comparing existing models of vision to brain data, to see what fits best.
Devising a decoding model that can generalize across brains, and even for the same brain across time, is a complex problem. Decoders are generally built on individual brains, unless theyâre computing something relatively simple such as a binary choice â whether someone was looking at picture A or B. But several groups are now working on building one-size-fits-all models. âEveryoneâs brain is a little bit different,â says Haxby, who is leading one such effort. At the moment, he says, âyou just canât line up these patterns of activity well enoughâ.
Standardization is likely to be necessary for many of the talked-about applications of brain decoding â those that would involve reading someoneâs hidden or unconscious thoughts. And although such applications are not yet possible, companies are taking notice. Haynes says that he was recently approached by a representative from the car company Daimler asking whether one could decode hidden consumer preferences of test subjects for market research. In principle it could work, he says, but the current methods cannot work out which of, say, 30 different products someone likes best. Marketers, he says, should stick to what they know for now. âIâm pretty sure that with traditional market research techniques youâre going to be much better off.â
Companies looking to serve law enforcement have also taken notice. No Lie MRI in San Diego, California, for example, is using techniques related to decoding to claim that it can use a brain scan to distinguish a lie from a truth. Law scholar Hank Greely at Stanford University in California, has written in the Oxford Handbook of Neuroethics (Oxford University Press, 2011) that the legal system could benefit from better ways of detecting lies, checking the reliability of memories, or even revealing the biases of jurors and judges. Some ethicists have argued that privacy laws should protect a personâs inner thoughts and desires as private, but Julian Savulescu, a neuroethicist at the University of Oxford, UK, sees no problem in principle with deploying decoding technologies. âPeople have a fear of it, but if itâs used in the right way itâs enormously liberating.â Brain data, he says, are no different from other types of evidence. âI donât see why we should privilege peopleâs thoughts over their words,â he says.
Haynes has been working on a study in which participants tour several virtual-reality houses, and then have their brains scanned while they tour another selection. Preliminary results suggest that the team can identify which houses their subjects had been to before. The implication is that such a technique might reveal whether a suspect had visited the scene of a crime before. The results are not yet published, and Haynes is quick to point out the limitations to using such a technique in law enforcement. What if a person has been in the building, but doesnât remember? Or what if they visited a week before the crime took place? Suspects may even be able to fool the scanner. âYou donât know how people react with countermeasures,â he says.
Other scientists also dismiss the implication that buried memories could be reliably uncovered through decoding. Apart from anything else, you need a 15-tonne, US$3-million fMRI machine and a person willing to lie very still inside it and actively think secret thoughts. Even then, says Gallant, âjust because the information is in someoneâs head doesnât mean itâs accurateâ. Right now, psychologists have more reliable, cheaper ways of getting at peopleâs thoughts. âAt the moment, the best way to find out what someone is going to do,â says Haynes, âis to ask them.â
Reference: Nature 502, 428â430 (24 October 2013) doi:10.1038/502428a
