Image Processing & Analysis

TR&D4: Image-derived modeling

(This project was TR&D3 in the previous funding period.)

The overall goal for TR&D4 is to make maximal use of the information in biological images to enable the construction of predictive, multiscale models of structure and dynamics at the subcellular and cellular level.

Vision

We will create and enhance tools intended to be useful for understanding how cell organization is created and maintained and how that organization differs from cell type to cell type and during disease. As a unifying principle for these efforts, we construct generative models of cell organization, rather than simply describing specific aspects. We seek to have these models capture the reality underlying collections of images or movies, such that computing using the models can produce new results much as computing on molecular structure models can be used to obtain results on protein dynamics (as discussed in TR&D1). Generative models have distinct advantages over the discriminative and descriptive approaches usually used. They attempt to make use of all information in images, rather than to just extract selected descriptors (features). Further, features are dependent upon the specifics of image acquisition, while generative models can be compared across different microscopes and laboratories. They are also combinable and reusable, in that models can be linked together to make predictions about new relationships, and models for organelle shape and distribution learned for one cell type can be provisionally extended to new cell types.

The proposed work makes use of best available methods in machine learning and computer vision, including advanced inference methods and convolutional neural nets (so called “deep learning” methods). The work builds on the extensive progress that has been made under what was Aim 1 of TR&D3 in the prior funding period, which resulted in eleven publications that acknowledged P41 support.

Background and Motivation

The tools to be developed and improved will be especially useful for studies of how cell organization is created and maintained and how that organization differs from cell type to cell type and during disease. While existing software primarily provides descriptions of images, the focus of this project is on construction of generative models of cell organization. Generative models are learned from a collection of images and are capable of producing new images that are statistically equivalent to the images used for training.  These models have distinct advantages over discriminative or descriptive approaches. They attempt to make use of all information in images, rather than to just extract selected descriptors or features. While features are not useful for comparing and communicating results between different laboratories due to their dependence upon the specifics of image acquisition, generative models capture the underlying reality that gave rise to images and can therefore be compared across different microscopes and laboratories. They are also combinable and reusable, in that models can be linked together to make predictions about new relationships, and models for organelle shape and distribution learned for one cell type can be provisionally extended to new cell types.

Work during the prior funding led to the development of extensive generative model capabilities that were incorporated into the open source CellOrganizer system. We propose to build upon this work to build new capabilities for constructing models that consider the extensive interrelationships between organelles and structures in cells, and for modeling the dynamics of proteins and organelles.  In conjunction with TR&D3, we will also develop new methods for using images to constrain estimation of the affinities between components of a biological system.  Lastly, we will develop new approaches for constructing models from both electron and fluorescence microscope images.

 
FigIV.3

Example synthetic 3D image for HeLa cells The image shows the nuclear membrane in red, the cell membrance in blue, and lysosomal membranes in green. It was randomly generated from a model learned from real 3D microscope images using the approach described in Peng and Murphy. (Peng,T. and R.F.Murphy. 2011. Image-derived, three-dimensional generative models of cellular organization. Cytometry A 79:383-391.)


Specific aims

  1. To develop and extend methods for learning image-derived deep dynamic generative models of cell organization

    This aim will extend capabilities created under Aim 1 of TR&D3 of the prior grant. A major focus will be on developing approaches to learn the full set of dependencies between multiple organelles from only selected measurements (we refer to these as deep models because there can be many layers of dependencies). A second focus will be learning dynamic models either from short movies or from large collections of cells at individual time points. We will also facilitate CellOrganizer use by providing ability to run it on PSC servers and by producing a Python version; this will also facilitate integration with software from other TR&Ds.

    Subaims

    1.1. Develop methods for constructing deep point process models that more completely capture dependencies between organelle or structure locations

    1.2. Develop improved methods for comparing generative cell models

    1.3. Develop improved methods for modeling and inferring dynamics

    1.4. Establish pipelines for automated analysis of movies of two-color transcription in living cells

    1.5. Implement methods for constructing models of heterogeneous cell populations

    1.6. Develop Galaxy workflows for important CellOrganizer functions for deployment under the PSC Bridges system

  2. To develop software to estimate molecular affinities from fluorescence microscope (FM) movies

    In combination with proposed new developments in TR&D1 and TR&D3, this aim will provide an important new capability for cell modeling and simulation: the ability to guide model choice and parameter estimation with image-derived models. The first subaim seeks to identify interactions that may be important to the process being studied as a way of reducing the number of species that need to be considered and produce a tentative reaction diagram to be used during the second subaim. That subaim then seeks to estimate the apparent affinities of these interactions. The software produced by each subaim can be useful without the other: learning putative causal relationships can motivate studies of underlying mechanisms not involving simulations, and estimating affinities using images can be done for reaction diagrams created from independent knowledge of likely interactions (such as from immunoprecipitations).

    Subaims

    2.1. Develop analysis tools for experimentally testing models of putative spatiotemporal dependencies

    2.2. Incorporate methods for constructing models of vesicle composition and movement

    2.3. Develop methods for estimating apparent affinities between components by inverse modeling using constraints.

  3. To develop software to create generative cell models combining information from electron microscope (EM) and FM images

    This aim will bridge the gap between the higher spatial resolution of EM and the high throughput and live cell capabilities of FM. The goal will be to learn a generative model for high resolution structures in EM images that can be inserted into places in which FM images indicate that that structure is likely to be present.

    Subaims

    3.1. Semantic nonlinear regression of high-resolution EM structures from FM data

    3.2. Learning low-dimensional embeddings of organelle shape

    3.3. Constructing regression pipeline that makes use of unpaired EM / FM images.

 

 

arp3inHelperTCellBeforeSynapseFormationarp3inHelperTCellAtSynapseFormation

 

An illustration of a model of the distribution of Arp3 in the helper T cell [being built in collaboration with DBP4]. Left: 40 seconds before immunological synapse formation. Going across rows left to right and progressing top to bottom, each subimage shows a cross section of the 3D protein distribution averaged across 17 cells. Each slice is perpendicular to the synapse (the face of the half ellipsoid template) and shows the synapse as the top edge of the shape shown. The synapse has not yet formed, so actin branching is occurring uniformly all around the periphery of the cell. Right: Model at the time of immunological synapse formation. Actin branching appears to be concentrated near the synapse as expected.

 

Image Processing & Analysis Research Highlights

distancecell

Tools for determining the spatial relationships

An important task for understanding how cells are organized is determining which components have spatial patterns that are related to each other. Read more

 

4d rtd

Pipeline for creation of spatiotemporal maps

Using a combination of diffeomorphic methods and improved cell segmentation, we developed a CellOrganizer pipeline for use in DPB4 to construct models of the 4D distributions of actin and 8 of its regulators during the response of T cells to antigen presentation. Read more

 

fig2cropped

Development of models of cell and nuclear shape

We added a major new capability to our open source CellOrganizer system, the ability to construct diffeomorphic models of cell and nuclear shape. The models were developed because of the needs of DBP4... Read More 

 

Mouse visual cortex
Anatomy and Function of an Excitatory Network in the Visual Cortex

MMBioS researcher Greg Hood’s collaboration with Wei-Chung Allen Lee of Harvard University and R. Clay Reid of the Allen Institute for Brain Science concerning the reconstruction of an excitatory nerve-cell network in the mouse brain cortex at a subcellular level using the AlignTK software has been published in Nature. Read more

  

CellOrganizerCellorganizer 2.0 Major Release

A major new release of the CellOrganizer system for creating image-derived models of cell shape and organization has just been published.  Read more.


View all Research Highlights

 

Copyright © 2020 National Center for Multiscale Modeling of Biological Systems. All Rights Reserved.