Most of us have at one point or another had a dream in which we could neither speak nor move; waking up from such a nightmare—and remembering what it’s like to be able to use your voice freely—feels like liberation. Now Meta says he’s getting closer to helping people who actually live with the crippling condition communicate with artificial intelligence that decodes brain waves.
On Monday, the company presented Brain2Qwerty v2his latest attempt to translate noisy brain activity into coherent text: think of it as a rudimentary form of algorithmically mediated mind reading. Although the research is still in its early stages, it offers a glimpse into a perhaps not-so-distant future in which patients suffering from anarthria, locked-in syndrome, amyotrophic lateral sclerosis (ALS) and other crippling neurodegenerative disorders will be able to communicate through thought without the need for neuroprosthetics, which usually require extremely invasive, complex and expensive surgery to brain.
“We believe that this research has the potential to make a real difference to the millions of people who suffer from brain injuries that prevent them from communicating,” Meta wrote in his announcement. The base code for Brain2Qwerty v2, as well as its predecessor, v1, has been made available online. “We hope that this work, carried out in the open, will advance neuroscience to detect, diagnose and treat neurological disorders faster than in silos,” the company wrote, repeating the claim movement in the AI industry to give scientists access to open source AI in the name of accelerating the pace of discovery.
How Meta Trained Brain2Qwerty v2
The new training model, conducted at the Basque Center for Cognition, Brain and Language in San Sebastián, Spain, involved nine healthy volunteers aged 25 to 56 who were asked to type more than 2,500 sentences over ten sessions. During these sessions, their brain activity was monitored using magnetoencephalography (MEG), which measures the tiny electrical fields created by neuronal activity in the brain. All of these typed sentences and brain scans then served as the raw training data fed into Brain2Qwerty.
In its most successful experiment, Brain2Qwerty v2 achieved word accuracy — meaning that more than half of the sentences that were decoded based on brain activity contained no more than one word error — of 78%. In contrast, Brain2Qwerty v1 (which was released last year) achieved a score of 48% in the most successful case.
The researchers also found that the accuracy of the new system’s decoding ability increased with the amount of training data it was fed, suggesting that simple scaling laws could be applied to create more capable systems in the future: “if long-term training with non-invasive MEG data could eventually obviate the need for neurosurgery,” the researchers write in their technical paper“this would represent a transformational shift in patient care.”
From Brainwaves to LLM to Communication
Brain2Qwerty v2’s unprecedented decoding accuracy has been achieved in large part by using the same pattern recognition technology behind chatbots such as ChatGPT and Meta’s Llama. In the first two steps of the decoding process, subjects’ MEG-measured brain waves were translated by artificial intelligence into tokens representing individual characters, after which another AI system, called an aligner, organized the individual characters into complete words. A large language model takes over, turning another AI’s jumble of signs and words into coherent sentences.
The results represent the first successful application of LLM to transform noisy brain activity into structured, comprehensible sentences. It may also offer a valuable new model for future researchers trying to build new brain-machine interfaces, physical or virtual, in which multiple artificial intelligence systems are used to hierarchically and collaboratively decode brain activity.
Along with this multi-layered AI-driven decoding system, Brain2Qwerty also relies on a contingent of “self-study” AI agents whose job it is to autonomously hone the decoding process to improve its accuracy and efficiency; think of them as worker bees, constantly making structural improvements to the hive so that all the vital activity that takes place inside can continue unhindered. The agents were trained to “iteratively modify our code base to invent new, better architectures,” the researchers wrote in the paper, providing “significant improvement” in the verbal error rate (WER).
However, the paper also notes that while agents have been useful in identifying new optimization strategies, they have been far from fully replacing human researchers: “while AI agents can serve as a powerful force multiplier, human research remains an important part of the scientific process.”
