February 12, 2020

Sperry, Darwin, and the Evolution of Reference Frames

It's all about deforming the phenotype. COURTESY: DiPaola, Evolving Darwin's Gaze (click to enlarge).

For this year's Darwin Day (February 12), I will be discussing a classic Neuroscience experiment by Roger Sperry [1]. This was discussed recently on Twitter (Figures 1 and 3), along with a movie (Movie 1) that demonstrates the behavioral effect of this manipulation. In the movie, we can see before and after behaviors with respect to prey capture. Before the manipulation, the frog seamlessly captures flies with a flick of the tongue. Afterward, the frog flicks in precisely the opposite direction of the fly. What is the biology behind this manipulation, and why does the manipulation produce seemingly maladaptive behavior?  

Figure 1. Tweet on Sperry's eye rotation experiment and chemosensory hypothesis (click to enlarge). 

Movie 1. Video demonstration of Sperry's eye rotation experiment (click to enlarge). 

In conducting this experiment, a frog's eyes are surgically rotated 180 degrees about each socket (see Figure 2). Due to this treatment, the normal course of axonogenesis between the eyeball and the tectum (part of the frog brain) is distorted [2]. As a result, the connections are shifted and the visual information is mapped to different regions of the tectum. The tectum serves as a visuospatial map of the environment, and maps visual stimuli to a reference frame used to generate motor behavior. As the reference frame is than systematically rotates, so is the frog's movement behavior. Thus, our manipulated frog produces a tongue flick that is 180 degrees in the opposite direction of the prey it is trying to capture. 

Figure 2. Cartoon demonstrating chemosensory hypothesis and behavioral effects of eye rotation experiment (click to enlarge). 

What is known as the chemosensory hypothesis (see Figure 2) also provided support for another concept, that of experience-dependent plasticity. The second tweet (below) discusses how this concept explains (and does not explain) what we see in the frog tectum and its modified behavior.

Figure 3. Part of response to tweet in Figure 1, with an assessment of the functional consequences (click to enlarge). 

Roger Sperry is also known for another set of experiments conducted a few years later [3] that asked whether neuroplasticity was a real phenomenon (as opposed to an epiphenomenon). This was done by either abnormally innervating muscle or placing end-effectors (limbs) in maladaptive locations on the body. If the organism could overcome these changes, such changes could be overcome via adaptation. As in the case of the eye rotation experiment, motor patterns are not plastic, even when neuronal connections are non-specific. This is why the eye rotated frog cannot adjust its behavior to adaptation in the spatial representation of visual input. 

Figure 4. A demonstration of conduction delay in the quadruped hindlimb in relation to multiple components of the sensorimotor loop. COURTESY: Figure 1 in [7] (click to enlarge).

So what does this have to do with evolution by natural selection [4]? It turns out that there are scaling laws that govern the coordination of the nervous system and phenotype as they both emerge in development [5]. Specifically, there are characteristically proportional relationships between motor neuron innervation and target tissue size in different parts of the organism [5, Note 1]. This developmental relationship (which holds true across related species) leads to interesting functional consequences. More et.al [6] suggests a trade-off in sensorimotor systems between responsiveness (temporal respond to stimuli) and resolution (sensory discrimination translated into muscle force production) results from size variation across phylogeny. 

Figure 5. Scaling of various sources of delay across species. Scaling comparison is in terms of mass (kg) versus delay (ms). COURTESY: Figure 2 in [7] (click to enlarge).

This relationship between size and a responsiveness-resolution trade-off also affects behavior. More and Donelan [7] show that conduction delay (an indicator of reaction time) also scales with variation in organismal size (Figure 4). The delay in force production (behavioral output) can be explained mostly in terms of nerve conduction delay rather than a delay in the sensory or synaptic components of the sensorimotor loop (Figure 5). This suggests a fundamental constraint on motor behavior that is independent of sensory inputs or their neural representation. But notice what is said about frog tectum in Figure 3: while eye saccades and tongue movement that produce the movement itself are not controlled by the tectum, a triggering threshold results from the active representation of visual information. 

Through the efficiency of population coding [8], this representation determines the timing of movement execution, which occurs in spatial context along with an appropriate amount of force. Perhaps the rotated eye manipulation (and associated phenomena like the prism experiment) presents an interesting exception to the responsiveness/resolution trade-off. Perhaps an intervening variable, representational alignment, also affects the linearity of the primary trade-off for specific movement behaviors. Across different species with common ancestry [9], this could become quite variable, and even provide an evolutionary-based account of neural plasticity.

[1] Sperry, R.W. (1943). Effect of 180 Degree Rotation of the Retinal Field on Visuomotor Coordination. Journal of Experimental Zoology, 92(3), 263–279.

[2] the reference to chemoaffinity in the first tweet refers to the process of axons finding their way to a target tissue. This is the basis for Sperry's "chemoaffinity hypothesis". Please see: Meyer, R.L. (1998). Roger Sperry and his chemoaffinity hypothesis. Neuropsychologia, 36 (10), 957–980.

[3] Sperry, R.W. (1945). The Problem of Central Nervous Reorganization After Nerve Regeneration and Muscle Transposition. Quarterly Review of Biology, 20(4), 311-369.

[4] For more on this topic, please see the following Synthetic Daisies posts:

[5] Striedter, G.F. (2004). Principles of Brain Evolution. Oxford University Press, Oxford, UK.

[6] More, H.L., Hutchinson, J.R., Collins, D.F., Weber, D.J., Aung, S.K.H., and Donelan, J.M. (2010). Scaling of sensorimotor control in terrestrial mammals. Royal Society of London B, 277(1700), 3563-3568.

[7] More, H.L. and Donelan, J.M. (2018). Scaling of sensorimotor delays in terrestrial mammals. Royal Society of London B, 285(1855), 20180613.

[8] Shamir, M. (2014). Emerging principles of population coding: in search for the neural code. Current Opinion in Neurobiology, 25, 140-148 AND Pouget, A., Dayan, P., & Zemel, R. (2000). Information processing with population codes. Nature Reviews Neuroscience, 1, 125–132.

[9] Quian Quiroga, R. (2019). Neural representations across species. Science, 363(6434), 1388-1389.

January 17, 2020

Work With Me in 2020

The Google Summer of Code (GSoC) is once again accepting applications from students to work on a range of programming-oriented projects over the Summer of 2020. Orthogonal Research and Education Laboratory and the OpenWorm Foundation have contributed a number of projects to the list. Here are links to the project descriptions (login to INCF Neurostars required):

DevoWorm Group:

Project  #15: OpenDevoCell Integration

Orthogonal Laboratory:

I am the contact person for both the Orthogonal Laboratory and DevoWorm Group projects. If you have any questions about the application process or want to have be review your application before submission, please feel free do so. According to this year's schedule, the proposal deadline is March 31, and the community bonding period starts on April 27. Stay tuned!

Join us once again on the "Road to GSoC"!

December 12, 2019

Google Summer of Docs congratulations!

Congrats to Casper daCosta-Luis (and co-mentors Bradly Alicea and Chee-Wai Lee) for successfully completing the inaugural Google Season of Docs! Casper's project involved automating project documentation (using Continuous Integration) at the OpenWorm Foundation. His final project report can be found here.

Thanks to our sponsor INCF for supporting our application. Speaking of Google Seasons, applications for Google Summer of Code (GSoC) 2020 will be opening soon. Once again, I am hosting two projects: one through the DevoWorm group (OpenWorm Foundation), and the other through Orthogonal Research and Education Laboratory. More information to come.

October 30, 2019

Pre-trained Models for Developmental Biology

Authors: Bradly Alicea, Richard Gordon, Abraham Kohrmann, Jesse Parent, Vinay Varma
This content is cross-posted to The Node Developmental Biology blog. 

Our virtual discussion group (DevoWormML) has been exploring a number of topics related to the use of pre-trained models in machine learning (specifically deep learning). Pre-trained models such as GPT-2 [1], pix2pix [2], and OpenPose [3] are used for analyzing many specialized types of data (linguistics, image to image translation, and human body features, respectively) and have a number of potential uses for the analysis of biological data in particular. It may be challenging to find large, rich, and specific datasets for training a more general model. This is often the case in the fields of Bioinformatics or Medical Image analysis. Data acquisition in such fields is often restricted due to the following factors:

* privacy restrictions inhibit public access to personal information, and may impose limits on data use.

* a lack of labels and effective metadata for  describing cases, variables, and context.

* missing data points, which require a strategy to normalize and can make the input data useless.

We can use these pre-trained models to extract a general description of classes and features without requiring a prohibitive amount of training data. We estimate that the amount of required training data may be reduced by an order of magnitude. To get this advantage, pre-trained models must be suitable to the type of input data. There are a number of models specialized for language processing and general use, but options are fewer within the unique feature space of developmental biology, in particular. In this post, we will propose that developmental biology requires a specialized pre-trained model. 

This vision for a developmental biology-specific pre-trained model would be specialized for image data. Whereas molecular data might be better served with existing models specialized for linguistic- and physics-based models, we seek to address several features of developmental biology that might be underfit using current models:

* cell division and differentiation events.

* features demonstrating the relationship between growth and motion.

* mapping between spatial and temporal context.

Successful application of pre-trained models is contingent to our research problem. Most existing pre-trained models operate on two-dimensional data, while data types such as medical images are three-dimensional. A study by Raghu et.al [4] suggests techniques specified by pre-trained models (such as transfer learning by the ImageNet model) applied to a data set of medical images provides little benefit to performance. In this case, performance can be improved using  data augmentation techniques. Data Augmentation, such as  adding versions of the images that have undergone transformations such as magnification, translation, rotation, or shearing, can be used to add variability of our data and improve the generalizability of a given model.

One aspect of pre-trained models we would like to keep in mind is that models are not perfect representations of  the phenomenology we want to study. Models can be useful, but are often not completely accurate. A model of the embryo, for example, might be based on the mean behavior of the phenomenology. Transitional states [5], far-from-equilibrium behaviors [6], and rare events are not well-suited to such a model. By contrast, a generative model that considers many of these features might generally underfit the mean behavior. We will revisit this distinction in the context of “blobs” and “symbols”, but for now, it appears that models are expected to be both imperfect and incomplete.

The inherent imperfection of models is both good and bad news for our pursuit. On the one hand, specialized models cannot be too specific, lest they overfit some aspects of development but not others. Conversely, highly generalized models assume that there are universal features that transcend all types of systems, from physical to social, and from artificial to natural. One example of this is found in complex network models, widely used to represent everything from proteomes to brains to societies. In their general form, complex network models are not customized for specific problems, relying instead on the node and edge formalism to represent interactions between discrete units. But this also requires that the biological system be represented in a specific way to enforce the general rules of the model. For example, a neural network’s focus on connectivity requires representations of a nervous system to be simplified down to nodes and arcs. As opposed to universality, particularism is an approach that favors the particular features of a given system, and does not require an ill-suited representation of the data. Going back to the complex networks example, there are specialized models such as multi-level networks and hybrid models (dynamical systems and complex networks) that solves the problem of universal assumptions.

Another aspect of pre-trained models is in balancing the amount of training data needed to produce an improvement in performance. How much training data can we save by applying a pre-trained model to our data set? We can reformulate this question more specifically to match our specific phenomenon and research interests. To put this in concrete terms, let us consider a hypothetical set of biological images. These images can represent discrete points in developmental time, or a range of biological diversity. Now let us suppose a developmental phenotype for which we want to extract multiple features. What features might be of interest, and are those features immediately obvious? 

In the DevoWorm group (where we mostly deal with embryogenetic data), we have approached this in two ways. The first is to model the embryo as a mass of cells, so that the major features of interest are the shape, size, and position of cells in an expanding and shifting whole. Last summer, we worked on applying deep learning to

* Caenorhabditis elegans embryogenesis. Github: https://github.com/devoworm/GSOC-2019.

* colonies of the diatom Bacillaria paradoxa. Github: https://github.com/devoworm/Digital-Bacillaria.

While these models were effective for discovering discrete structural units (cells, filaments), they were not as effective at directly modeling movement, currents, or transformational processes. The second way we have approached this is to model the process of cell division and differentiation as a spatial and discrete temporal process. This includes the application of representational models such as game theory [7] and cellular automata [8]. This allows us to identify more subtle features that are not directly observable in the phenotype, but are less useful for predicting specific events or defining a distinct feature space. 

Our model must be capable of modeling multiple structural features concurrently, but also sensitive to scenarios where single sets of attributes might yield more information. Ideally, we desire a training dataset that perfectly balances “biologically-typical” motion and transformations with clearly masked shapes representing cells and other phenotypic structures. Generally speaking, the greater degree of natural variation in the training dataset, the more robust the pre-trained model will turn out to be. More robust models will generally be easier to use during the testing phase, and result in a reduction in the need for subsequent training. 

Finally, specialized pre-trained models bring up the issue of how to balance rival strategies for analyzing complex processes and data features. Conventional artificial intelligence techniques have relied on a representation which relies on the manipulation of symbols or a symbolic layer that results from the transformation of raw data to a mental framework. By contrast, modern machine learning methods rely on data to build a series of relationships that inform a classificatory system. While a combination of these two strategies might seem obvious, it is by no means a simple matter of implementation [9]. The notion of “blobs” (data) versus “symbols” (representations) draws on the current debate related to data-intensive representations versus formal (innate) representations [10-12], which demonstrates the timeliness of our efforts. Balancing these competing strategies in a pre-trained model allows us to more easily bring expert knowledge or complementary data (e.g. gene expression data in an analysis of embryonic phenotypes) to bear.

We will be exploring the details of pre-trained models in future discussions and meetings of the DevoWormML group. Please feel free to join us on Wednesdays at 1pm UTC at https://tiny.cc/DevoWorm or find us on Github (https://github.com/devoworm/DW-ML) if you are interested in discussing this further. You can also view our previous discussions on the DevoWorm YouTube channel, DevoWormML playlist (https://bit.ly/2Ni7Fs2).

[1] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI, https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.

[2] Isola, P., Zhu, J-Y., Zhou, T., Efros, A.A. (2017). Image-to-Image Translation with Conditional Adversarial Nets. Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR).

[3] Cao, Z., Hidalgo, G., Simon, T., Wei, S-E., and Sheikh, Y. (2018). OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. arXiv, 1812.08008.

[4] Raghu, M., Zhang, C., Kleinberg, J.M., and Bengio, S. (2019). Transfusion: Understanding Transfer Learning for Medical Imaging. arXiv, 1902.07208.

[5] Antolovic, V., Lenn, T., Miermont, A., Chubb, J.R. (2019). Transition state dynamics during a stochastic fate choice. Development, 146, dev173740. doi:10.1242/dev.173740.

[6] Goldenfeld, N. and Woese, C. (2011). Life is Physics: Evolution as a Collective Phenomenon Far From Equilibrium. Annual Review of Condensed Matter Physics, 2, 375-399. doi:10.1146/annurev-conmatphys-062910-140509.

[7] Stone, R., Portegys, T., Mikhailovsky, G., and Alicea, B. (2018). Origins of the Embryo: Self-organization through cybernetic regulation. Biosystems, 173, 73-82. doi:10.1016/j.biosystems.2018.08.005.

[8] Portegys, T., Pascualy, G., Gordon, R., McGrew, S., and Alicea, B. (2016). Morphozoic: cellular automata with nested neighborhoods as a metamorphic representation of morphogenesis. In “Multi-Agent Based Simulations Applied to Biological and Environmental Systems“. Chapter 3 in "Multi-Agent-Based Simulations Applied to Biological and Environmental Systems", IGI Global.

[9] Garnelo, M. and Shanahan, M. (2019). Reconciling deep learning with symbolic artiļ¬cial intelligence: representing objects and relations. Current Opinion in Behavioral Sciences, 29, 17–23.

[10] Zador, A.M. (2019). A critique of pure learning and what artificial neural networks can learn from animal brains. Nature Communications, 10, 3770.

[11] Brooks, R.A. (1991). Intelligence without representation. Artificial Intelligence, 47, 139–159.

[12] Marcus, G. (2018). Innateness, AlphaZero, and Artificial Intelligence. arXiv, 1801.05667.

* Model Zoo: pre-trained models for various platforms: https://modelzoo.co/

* DevoZoo: developmental data for model training and analysis: https://devoworm.github.io/

* Popular papers on medical image segmentation along with code:  https://paperswithcode.com/area/medical/medical-image-segmentation

October 25, 2019

OAWeek: share your own case study!

This post is part of a series published over the course of OAWeek 2019.

Do you use, share, or have an opinion about open data? The Data Reuse Initiative would like to hear from you! In honor of OAWeek 2019, we are looking for personal and research group testimonials on how you share or otherwise practice open data. Submit at your leisure (there is no deadline), but we would like to hear from you!

* submit a testimonial (under 200 words) by submitting a pull request to our Github repository or submit to this Google Form.

* if you choose to submit an image (screenshot, diagram, or cartoon), please issue a pull request on Github.

* if you cannot access either of the links, or need help with your submission, please [contact us](mailto:balicea@openworm.org).