Understanding Brain Function and Development in 3D Microscopy
Three migrating neurons are segmented and tracked from a time-lapse 3-D volume of two-photon excitation microscopy data.
CVLAB actively collaborate with a group of neuroscientists in an effort to better understand how the brain develops and
functions. Today, neuroscientists employ many types of microscopy techniques in their research, ranging from simple
brightfield and confocal microscopes
to more advanced phase contrast techniques to state-of-the-art 2-photon
and scanning electron microscopes. While every microscopy technique has certain advantages
to suit its purpose, they all share a common drawback: The massive amounts of data produced cannot be individually analyzed by humans. CVLAB researchers are working with
neuroscientists to develop techniques to automatically process this data, allowing neuroscientists to conduct experiments and analyze
data at scales that were previously not possible.
Our research focuses of four tasks:
Detecting neurons (and components of neurons) in microscopic data, such as nuclei, mitochondria, and synapses;
Detecting neurons, or components of neurons, in microscopic images is an critical problem related to several important tasks.
First, detecting neuron nuclei allows neuroscientists
to estimate the density of neurons in a particular sample. Nuclei detections also provide robust seed locations for segmenting the structure of
the entire neuron, and help to resolve ambiguities between nearby neurons. Detecting cellular components of the neuron, such as mitochondria and
synapses, facilitates the study of neural development and synaptic plasticity.
Detection results for mitochondria in FIBSEM imagery (left) and neuron nuclei in 2-photon imagery (right) using an Adaboost-based detector trained with the Rays feature set.
We have developed a generalized detector capable of finding irregularly shaped blob-like objects in microscopic data, such as neuron nuclei and
mitochondria. Using a machine learning approach, we train a cascaded Adaboost classifier on a labeled data set using a combination of Haar-like
features and a new class of image features that considers image characteristics at distant contour points. Given a location and a direction,
these novel features capture information such as relative distance to the nearest contour in 2 directions, absolute distance to the nearest contour,
gradient direction at the nearest contour, and gradient magnitude at the nearest contour. Because these features are defined at a variable distance
from the given location, they are robust to irregularities in the shape of an object, which is an important characteristic for describing cells.
Four basic image measurements defined at distant contour points make up the Rays feature set, which is a powerful tool for detecting irregularly shaped objects in images.
Many of the characteristics and behaviors of interest to neuroscientists require some sort of segmentation as an intermediate
step. This may involve segmenting sub cellular structures such as mitochondria, dendrites, axons, butons, synapses, etc., or it may imply segmenting the entire
structure of the neuron from the background while resolving the segmentations between several neurons in a volume. While it is difficult to define a completely
general segmentation approach for all types of tasks, machine learning provides a means to learn from the data which can be applied to many similar
segmentation tasks. Researchers at CVLAB are exploring the use of several machine learning approaches including boosting, SVM classification, and K-nearest
neighbors for various tasks such as segmenting flourescent-marked neurons in 2-photon imagery from the background; segmenting neurites, dendrites, and axons; and
segmenting sub cellular structures such as mitochondria, vesicles, and synapses.
Segmentation critically depends on the descriptive power of the features used to describe the data.
We are currently investigating new and more
powerful features to provide extremely fine and robust segmentations, including work with the Hessian Matrix and its associated eigenvalues, high-order steerable filters,
texture descriptors such as textons, and other image characteristics.
State-of-the-art approaches to detecting linear structures
rely on ideal models of their appearance and noise processes. They are usually optimized to
find ideal lines or tubular structures with smooth profiles.
However, real linear structures often fail to conform to this model, which can drastically impact
performance. Researchers at CVLAB use the rotational properties of Gaussian derivatives to achieve rotational invariance, however, instead of
steering the filters we replace the optimality criteria by a machine-learning algorithm that learns from training data. Because the training data
encompasses deviations from the ideal model, the resulting algorithm is more robust to irregularities and can be trained to detect not only simple
linear structures but also junctions and crossings.
Fourth-order Gaussian derivatives can be rotated to arbitrary orientations. A linear combination of these Gaussian derivatives approximating an
image patch can be classified by an SVM as filament-like or not.
Once we have trained a classifier on filament-like data rotated to a canonical orientation, the SVM is applied
to classify voxels as filament-like or not. Results for blood vessels appearing in retinal imagery and neurons in brightfield microscopy data
indicate an improved ability to segment blood vessels, axons, and dendrites over state-of-the-art methods.
Blood vessels in retinal images from the drive database.
(left) Segmentation of the entire filament structure. (right) Detailed view. Red indicates true positive segmentations, green
indicates false positives, and blue indicates false negatives.
(left) Minimum intensity projection of brightfield image of a neuron and its dendritic tree. (center) Segmentation performed using an SVM on features based on fourth-order Gaussian
derivatives. (right) Detailed view of the segmentation.
The maximum intensity projection (MIP) of a neuron in a 3D image stack collected by a two-photon microscope (left) and the resulting 3D segmentation
using the eigenvalues of the Hessian matrix (right).
Reconstructing the shape, or morphology, of neurons is a necessary step towards understanding brain functionality. When
presented with noisy brightfield microscopy data, reconstruction of a neuron's dendritic tree can be an intimidating task. Due to irregularities in the staining process
and other sources of noise, the appearance of dendrites and axons are often ambiguous. While some commercial products provide sophisticated interfaces for
manual delineation of dendritic trees, the process is tedious and time-consuming, taking up to ten hours to process a single neuron image stack.
Researchers at CVLAB have developed a novel approach to handling the difficulties inherent in the process of reconstructing
the dendritic tree. Without assuming an a priori dendrite model used by previous techniques, we rely on statistical learning techniques to construct
models as we go. We first train a classifier to distinguish dendrite-like voxels from the background using a limited amount of expert-labeled ground truth,
then we construct a minimum spanning tree connecting dendrite-like voxels. Then, we use EM to determine which edges of the graph correspond to dendrites,
and we prune and rebuild the final dendritic tree.
The minimum intensity projection (MIP) of neuron and its dendritic tree imaged using brightfield microscopy (left) and the resulting segmentation and
tree reconstruction (right). Note the irregularities in the staining process leading to the non-ideal appearance of dendrite structures.
Certain areas of the mouse brain constantly regenerate, even in adulthood. Immature neurons migrate from the sub ventricular zone (SVZ) along the rostral migratory system (RMS) to the olfactory bulb.
Migrating neurons move in a characteristic manner, elongating in the direction of travel.
While most neurons are born during the
embryonic and postnatal periods, it is now well accepted
that some regions of the brain keep producing new neurons
throughout adulthood. It is of great interest to understand
how processes that govern the birth and development
of neurons might be regulated, as this could lead to future
treatments of degenerative disorders using adult neural stem
cells. The automatic tracking of migrating neurons allow neuroscientists quantify useful data such as cell morphology,
speed, etc. when studying these processes.
Our collection process begins by constructing a
lentivirus vector. In vivo stereotaxic injection of the
lentivirus in the subventricular zone of the brain causes
newly born neurons to express Green Fluorescent Protein.
The neurons migrate along the rostral migratory system (RMS) towards
the olfactory bulb, where they will integrate into the network. At this point,
a slice is prepared for imaging with a sophisticated 2-photon microscope.
Migrating neurons move in a characteristic manner. The nucleus extends a neurite, at the end of
which is a growth cone, which the neuron uses as an anchor
to pull itself forward. Because the structure of the entire cell is highly
deformable and irregular, they are difficult to track. For this
reason, we perform tracking on the nucleus, which retains a
more consistent shape and can be readily detected.
Starting from a set of detected nuclei, we use a batch MCMC data association approach
to efficiently estimate the MAP of the filtering distribution. For more efficient inference, we have defined two general
constraints to limit the search space. The first constraint relaxes the standard assumption that target motion and appearance
are independent. Using a GMM and spoke-based appearance model, we learn a correlation between the appearance of a neuron and its
motion - the neuron nucleus elongates in the direction of motion (increasing elongation with speed). The second constraint enforces,
in a formal probabilistic manner, the targets to enter and exit the scene at logical locations (usually scene boundaries). For more details,
see the flash movie describing this work.
Neuron nuclei detections are tracked in 3D (projected here to 2D) using a batch MCMC approach.
Two general constraints are employed to improve efficiency, the first exploiting the correlation between nucleus appearance and direction of motion, the second enforcing that
neurons enter and exit the volume at logical locations.
