In 2008, biologist Gerry Rubin, director of the Howard Hughes Medical Institute's Janelia Research Campus, assembled a team to construct the fruit fly connectome — a map of all the connections between an animal's brain cells. The first-ever connectome completed was of the tiny C. elegans roundworm in 1986. That connectome proved essential to research on this model organism, helping researchers understand how its nervous system controlled its movements, its reactions to light, temperature and chemicals, and its mating behavior. Rubin wanted to construct the connectome of an animal with more complex behaviors than the worm (i.e., the fruit fly), hoping that it could lead to insights about basic functions, such as learning and memory, and brain diseases. Though still far from the complexity of a human brain, the fruit fly brain has 400x more neurons than a roundworm, and would be a major step forward for the field.

"People were skeptical that this was worth doing," Rubin says. "They said that you would make a wiring diagram and it would be a mass of spaghetti, that we wouldn't be able to figure it out." Rubin was determined to try. His team used electron microscopes to take detailed images of the fly’s brain, and computers to reassemble the neurons back into a three-dimensional composite image. They'd then use computer vision to follow the paths of each neuron through this 3D image stack.

However, computer vision was not very accurate when the project first began, so Rubin had to hire a large team of human proofreaders to audit the generated data. Early in the project, Rubin estimated that it would take 2,000 person-years to complete this task. In other words, it would take a team of 100 scientists 20 years to audit the data generated by computer vision programs.

Around this time, AI advancements led to the development of the deep learning method, which revolutionized computer vision, greatly improving its accuracy. Viren Jain, a computer vision scientist at Janelia, decided to leave Janelia to join the newly formed Connectomics team within Google Research to work on deep learning.

The first project the Google Research Connectomics team embarked on was a collaboration with Rubin and Janelia Research Campus to map the fruit fly connectome. Jain's team began using Google Research's powerful computing resources to develop more accurate AI models to "segment," or trace each individual neuron through the 3D brain volumes produced at Janelia.

“We would provide data and proofreading, Google would provide the results of their computer vision-generated neuron structures, and we will go around in a circle and try to improve the results,” Rubin remembers. "That is how it got set up."

The arrangement led to what Rubin calls a "tremendously productive collaboration" that continues today. "Viren built the go-to group in the world for people in the field," Rubin says.

Google Research created flood-filling networks, state-of-the art algorithms for segmenting connectome data. Then it created a self-supervised learning technology, SegCLR, which automatically extracts key insights from segmented volumes, such as identifying cell type (e.g., pyramidal neuron, basket neuron, etc.) and parts of each neuron (e.g., axon, dendrite, etc.).

Google later created connectomics infrastructure, such as TensorStore for storing connectomics data and Neuroglancer for visualizing it. As a result, Rubin’s team at Janelia made much faster progress than expected on mapping the fly brain connectome. Google Research's innovations reduced the cost and effort to proofread the fly connectome by a factor of 20.

"It goes from being not feasible to feasible," Rubin says,

In 2020, Rubin, Jain and collaborators released the connectome of half of the fruit fly brain. Just as Rubin predicted, the fly connectome data have transformed research on learning, memory, and behavior in this model animal. Scientists have published more than 40 papers reporting new findings based on this connectome, prompting one leading theoretical neuroscientist to refer to the “BC” and “AC” eras of fruit fly research, for “before connectome” and “after connectome.”

The collaboration between Janelia and Google Research to map the fruit fly connectome proved the promise of connectomics, inspiring others to embark on even more complex projects, such as the effort to map the mouse connectome. In other words, the completion of the fruit fly connectome has proven far more valuable than the "mass of spaghetti" that early doubters predicted.

Gwyneth Card, a neuroscientist who leads a lab at Columbia University's Zuckerman Mind Brain Behavior Institute, is fascinated by how signals in an animal's brain trigger behavior and movements, such as leaping away from a predator. As a college student, she wanted to study this question in large mammals, such as kangaroos. However, she realized their brains were so complex that it would take more than one scientist's lifetime to unravel their brain-body connections. Instead, Card decided to study the Drosophila fruit fly, a smaller animal with a brain that had already been mapped by scientists.

Card’s studies focus on how fruit flies react when they see the looming shadow of a potential predator. When you attempt to swat a fly, for instance, the fly sees your approaching hand as a dark shadow that grows larger and larger as it advances towards the fly. The fly's brain receives this visual signal, processes it, then sends messages to nerves in its leg muscles. The muscles respond by launching the fly off the ground, and away from your hand.

All of this takes less than a second: almost instantaneously, the fly senses where a threat is coming from, then jumps away to safety. If a shadow approaches from the front, the fly leaps backward to escape. And if a shadow approaches from behind the fly, it springs forward and away from danger.

To understand how the fly converts the visual signal of a threat into a quick physical escape response, Card’s lab team created an experimental device that shows flies "scary movies," and studies how they respond. Card's lab has used these movies to find the specific nerve cells in a fly's brain, or "looming detectors," that sense approaching shadows. Until recently, it wasn't possible to understand how those looming detectors correctly tell the fly where to jump to avoid predators.

The Connectomics team at Google Research and the Howard Hughes Medical Institute's Janelia Research Campus have worked together over nearly a decade to map the connections between every cell in the fruit fly brain. Card's lab used that information to understand how flies interpret and respond to visual threats. "That unique database, from a powerhouse collaboration between industry scientists and a basic research institution, has proven to be invaluable," Card said.

The fly connectome revealed that looming detectors facing the back of a fly had more connections with neurons that triggered the fly to jump forward. And the looming detectors facing the front of a fly had more connections with neurons that triggered the fly to jump backward. In other words, the fly's brain was hard-wired to respond correctly to attackers from different directions.

"The connectome is one of the pieces that's really made me incredibly optimistic that there are things that are solvable in the scale of a single career, and given me a lot of hope and motivation for really digging into this," Card says. "Because it's at the breadth that we really need to be able to understand this mysterious organ, to really actually start making sense of it."

Card also says that, while flies and humans obviously have very different brains, some of our instinctive behaviors, such as escaping predators, may actually share similarities. For instance, Card often shows her students a video of a man walking down the street. All of a sudden, a car careens up on the sidewalk, and the man jumps out of the way of the car. "If you watch this video in slow motion, the exact way he moves his body, it's just so similar to the fly's strategy," Card says. "And what I think that underscores in a very visual way for people is that there are a lot of actions, a lot of behaviors that are very similar, even in animals as distant as flies and humans."

In the future, Card hopes that she'll be able to unravel more underlying processes that flies use to transform sensory information into action. She'd especially like to be able to study connectome data from multiple flies, which would reveal which parts of the fly connectome are universal, and which vary among individuals.

Jeff Lichtman, a neuroscientist who leads the Lichtman Lab at Harvard University, is astonished by how little we know about the human brain. “One might argue that our knowledge of outer space is probably better than our knowledge of inner space and what's going on in our own heads,” says Lichtman.

Lichtman is among the vanguard of scientists who are using new technologies to peer more closely into the brain than ever before. They’re motivated to learn more about how the human brain works to help improve our knowledge of and treatments for mental illnesses and other diseases, such as dementia. 

“When it comes to brain illnesses, we are still sort of stuck in the mud,” Lichtman says.

Lichtman specifically wants to understand how connections between brain cells are involved in learning and memory. How does the physical structure of the brain encode our lived experiences — not only storing the information, but using it to change the course of our lives? To explore this question, Lichtman needs to see the physical structure of the brain in extreme detail, including all the neural connections between individual brain cells. However, the human brain is massively complex, and gathering data from the entire brain and analyzing it isn’t yet possible. 

To that end, Lichtman partnered with Google Research to create the first-ever connectome of a small sample of human brain tissue.  Despite mapping less than 1 percent of the brain, the sample still contained 50,000 cells and 150 million synapses, or connections between brain cells. That’s a huge amount of data to sort and analyze — 1.4 petabytes, to be exact. 

Google Research leveraged the machine learning software it had developed for other connectomics  projects, such as the fruit fly connectome and flood filling networks. The resulting H01 dataset is the largest sample of brain tissue imaged and reconstructed in this level of detail, in any species, and is available to browse via the Neuroglancer tool.

Lichtman says that even the findings from this one segment of the human brain have been powerful. For instance, the H01 dataset has challenged scientists’ understanding of how neurons process information. Each neuron has many connections to other neurons, which are called synapses. Neurons send messages to each other through these synapses. Lichtman found that some neurons form as many as 50 synapses with other neurons, which makes them especially powerful.

Now, Lichtman and Google Research are joining other collaborators to map a tiny fraction (2-3%) of the mouse brain as part of a $150 million National Institutes of Health program called BRAIN Initiative Connectivity Across Scales (BRAIN CONNECTS). Mapping a mouse connectome will help researchers develop more efficient tools and technologies that could one day be used to tackle the much more complex human brain. 

Ultimately, the goal is to use the mouse data to understand more about how the human brain works. But already, Lichtman says that what he has seen of the human brain connectome has changed his perspective. “For me, it has been an opportunity to witness the magnificence of natural processes,” Lichtman says. 

“When you see that the machine that you're using to think is much more complicated than the thoughts that come out of your mind, it's a deeply philosophical humbling.”