Merging electronics and biology: the future of touch

The sense of touch, or haptic perception, is a key feature of animal behavior. For humans, haptic perception and proprioception is a subtle but fundamental sensory modality that helps us move, plan, and interact with our environment. Recent developments in virtual environments and conformal electronics may allow us to build artificial sensory systems that fully integrate into the user’s neural function. But first, we must bring together a number of emerging themes in the scientific literature.

Multisensory Perception and the Sensory Milieu
To understand how the nervous system will deal with inputs from artificial touch sensors, we need to understand how the touch systems of the body behave when perturbed. Virtual reality systems specialized for creating sensory illusions are needed to uncover how visual and haptic information are collated at the body’s surface and merged in the brain (for example, see Figure 1). The integration of visual and touch cues can be explored in an active touch task [1]. In this example (Figure 1), force feedback is provided that corresponds with presented visual stimuli. Multisensory integration can also be explored and decoupled by examining changes in postural sway [1a] when the frequency of visual and touch cues are incongruent [2]. The integration of these signals in the mammalian brain is a stochastic process, involving integration sites such as the superior colliculus (SC) and posterior parietal cortex (PPC) [2a].

Studies examining the cell biology of cutaneous touch receptors at the skin surface have found that different populations of cells have different thresholds for activation [3], which is likely related to the complexity of information that the touch sensory modality provides. Much as the brain abstracts away the key pieces of information needed to perform specific tasks, robotic models have also been constructed to understand the core contributions of touch to multisensory perception [4]. This issue of minimal bandwidth requirements will be an important consideration in the design of future artificial touch systems. While relatively low amounts of information may actually be required for straightforward tasks (such as picking up a cup), a much greater amount of information (and perhaps noise as well) may be required for outcomes such as self-awareness or balance.

Figure 1. Experiment demonstrating the importance of coordinating and 

integrating vision and touch in exploration of unfamiliar objects. COURTESY: Figure 1 of [1].

The sensory milieu refers to myriad interactions between the major sensory systems of the brain (e.g. visual, touch, auditory) and the physiological mechanisms which underlie perception and action. The term "milieu" implies that homeostatic mechanisms are at play in these processes. Considering the emergent order and role of autoregulation among sensory inputs during integration and information processing is a novel way to view the neurobiology of sensation and perception. However, it may be useful to the design and maintenance of artificial systems for two reasons:

1) while it is clear what environmental information is required for statistically optimal perceptual performance [4a], in engineering contexts we may want to achieve sub-optimal performance or robust responses.

2) the perturbation (e.g. decoupling of visual and haptic information) of a multisensory system does not result in a total loss of perceptual integration, particularly over time. Therefore, there must be some type of homeostatic mechanism at play with respect to incoming sensory information [4b].

Artificial Touch is a Matter of Compliance

The interaction with the body’s surface is another key feature of artificial touch systems. To this extent, research that characterizes interactions between the physical properties of surfaces and touch systems is important (see Figure 2). Recent investigations into interactions between the epidermal ridges on the fingertip and the peaks and troughs of a rough surface show that surface forces become more variable with increases in roughness [5]. This leads to an increase in friction during sliding and other fine manipulation behaviors. This has consequences not only for the design of artificial touch systems, but for the tribological design of materials at multiple scales as well [5a].

Figure 2. The interaction between surface reaction forces and the dynamic biomechanics of touch (e.g. the interaction between the finger’s skin and the surface roughness). COURTESY: Figure 1 of [5].

Artificial touch systems require a range of novel material types. In robotics, it has become increasingly clear that the use of compliant (e.g. soft, biocompatible) materials are required for effective interaction [6]. Figure 3 demonstrates how the results shown in Figure 2 can be modulated through the manipulation of surface texture. By controlling the scale and geometry of surface features, friction due to sliding can be minimized. Looking forward (and to animal models), specialized surface features at multiple size scales can enable quite amazing functional behaviors [7].

Figure 3. An example of how surface texture can modulate surface reaction forces encountered at the fingertip. The phenomenon leads to our perception of different surface qualities (e.g. edges, orientation, smoothness). COURTESY: Figures 1 and 2 in [8].

In [8], an experiment using a sliding cantilever and a patterned elastomer demonstrated that the sliding motion of a fingertip can act to modulate the surface reaction force encountered during finger strike. The forces involved are small relative to forces encountered while walking or running [9]. However, the fingertip encounters strains which, when coupled with the complex curves of fingertip geometry, makes for an interesting design challenge. What is needed is a thin film that conducts electrical potentials, and that can bend or buckle without damage to the surface. The group in [10] have developed a carbon nanotube (CNT)-based solution to this problem. Embedded in their thin film piezoelectric device are CNT structures which store strain forces applied to the surface like a spring - and releases this potential after loading. The result is a stretchable capacitor that has the characteristics of a biological viscoelastic (e.g. skin-like) surface.

Figure 4. An summary of piezoelectric characterization for an electronic sensor device that covers the hand. COURTESY: Figure 5 in [11].

Figure 4 shows how such a thin, stretchable film might be used as a skin surrogate. The authors of [11] have engaged in extensive characterization of these types of surfaces. When the thumb is bent, a piezoelectric signal is generated that could be interfaced with the nervous system.

Conformal Electronics

In the past few years, compliant materials have been effectively used as a substrate for nanoelectronic arrays.  One of the leading innovators in this field is the Rogers group at UIUC, who have published a number of papers on fabricating (see Figures 5 and 6) and demonstrating the function (Figure 7) of these materials. One  commonly used fabrication technique involves the use of the silicon polymer Polydimethylsiloxane (PDMS) as a substrate. PDMS has many desirable properties for a device that will be embedded into the human body. That being said, there are other (and perhaps better) organic materials that could be used as a substrate. However, PDMS is the most common. For a presentation on the promise of tissue fabrication and potential methods, see [12].

Figure 5. An overview of how conformal electronics are fabricated. COURTESY: Figure 1 in [13].

Figure 6. The fabrication of microstructured PDMS films. COURTESY: Figure 1 in [14].

The fabrication process of a conformal electronic device can be seen in Figure 6, and shows how the thin film is molded and etched (embedded with features). In the end, we have a fully-functional and bendable array. Bendable electronics can be used for a range of applications, and is the enabling technology behind flexible OLED displays (perhaps a staple of third and fourth generation e-readers). Theoretically at least, a similar process could be used to fabricate multi-layered embeddable replacement skins for humans, which could not only restore the function of touch but also augment our sensory capabilities. A number of challenges would have to be overcome, including interfaces with the body's endocrine (hormones), circulatory (blood), and excretory (sweat) systems, the importance of which is discussed in [15].

Figure 7. An overview of the deformational and functional properties of conformal electronics. COURTESY: Figure 2 in [13].

The Future

Through the use of compliant and biocompatible materials, algorithms that can decode nervous systems activity, and a good understanding of the role touch and proprioception play in human self-awareness, artificial touch systems might become as common a technology as touch screen computers are today. 

References:

[1] Drewing, K. and Ernst, M.O.   Integration of force and position cues for shape perception through active touch. Brain Research, 1078, 92–100 (2006).

[1a] For some interesting examples, please visit the Jeka Lab at University of Maryland-College Park and the Multisensory Perception and Action group at Bielefeld University.

[2] Kiemel, T., Oie, K.S., and Jeka, J.J.   Multisensory fusion and the stochastic structure of postural sway.

Biological Cybernetics, 87, 262-277 (2002).

[2a] For examples of multisensory integration in the SC, please see: Stein, B.E. and Meredith, M.A.   The Merging of the Senses. MIT Press, Cambridge, MA (1993). 

For an example of multisensory integration in the PPC, please see: Pasalar, S., Ro, T., and Beauchamp, M.S.   TMS of posterior parietal cortex disrupts visual tactile multisensory integration. European Journal of Neuroscience, 31(10):1783-1790 (2010).

[3] Li, L., Rutlin, M., Abraira, V.E., Cassidy, C., Kus, L., Gong, S., Jankowski, M.P., Luo, W., Heintz, N., Koerber, H.R., Woodbury, C.J., and Ginty, D.D.   The Functional Organization of Cutaneous Low-Threshold Mechanosensory Neurons. Cell, 147, 1615–1627 (2011).

[4] Duenas, J., Chapuis, D., Pfeiffer, C., Martuzzi, R., Ionta, S., Blanke, O., and Gassert, R.   Neuroscience robotics to investigate multisensory integration and bodily awareness. Proceedings of the IEEE Engineering and Medicine in Biology Society, 8348-8352 (2011).

[4a] For more information, please see: Ernst, M.O. and Banks, M.S.   Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429-433 (2002).

[4b] For an outline of this idea (with experiments!) please see: Alicea, B.  Performance Augmentation in Hybrid Systems: techniques and experiment. arXiv, 0810. 4629 [q-bio.NC, q-bio.QM] (2008). 

[5] Mate, C.W. and Carpick, R.W.   A sense for touch. Nature, 480, 189-190.

[5a] For reflections on the use of hierarchical material scaffolds to realize multiscalar material design, please see: Cranford, S.W. and Buehler, M.J.   Shaky foundations of hierarchical biological materials. Nano Today, 6, 332-338 (2011).

[6] Quake, S.R and Scherer, A.   From Micro- to Nanofabrication with Soft Materials. Science, 290, 1536-1540 (2000).

[7] Gao, H., Wang, X., Yao, H., Gorb, S., and Arzt, E. (2005). Mechanics of hierarchical adhesion structures of geckos. Mechanics of Materials, 37, 275-285.

[8] Wandersman, E., Candelier, R., Debregeas, G.,  and Prevost, A.   Texture-Induced Modulations of Friction Force: The Fingerprint Effect. Physical Review Letters, 107, 164301 (2011).

[9] Dixon, S.J., Collop, A.C., and Batt, M.E. (2000). Surface effects on ground reaction forces and lower extremity kinematics in running. Medicine and Science in Sports and Exercise, 32(11), 1919-1926.

[10] Lipomi, D.J., Vosgueritchian, M., Tee, B.C-K., Hellstrom, S.L., Lee, J.A., Fox, C.H., and Bao, Z.   Skin-like pressure and strain sensors based on transparent elastic films of carbon nanotubes. Nature Nanotechnology, (2012).

[11] Ying, M., Bonifas, A.P., Lu, N., Su, Y., Li, R., Cheng, H., Ameen, A., Huang, Y., and Rogers, J.A.   Silicon nanomembranes for fingertip electronics. Nanotechnology, 23, 344004 (2012).

[12] Alicea, B.   Nano-enabled Biological Materials. Nature Precedings, npre.2010.5448.1 (2010).

[13] Kim, D-H., Ghaffari, R., Lu, N., and Rogers, J.A.   Flexible and Stretchable Electronics for Biointegrated Devices. Annual Review of Biomedical Engineering, 14, 113-128 (2012).

The Rogers lab has come up with a number of interesting advanced applications in the area of confomal electronics, including wearable (tattooable) transistors and dissolvable devices (implantable that dissolve harmlessly once no longer needed) For more information on this application, please see:

Hwang, S-W., Tao, H., Kim, D-H., Cheng, H., Song, J-K., Rill, E., Brenckle, M.A., Panilaitis, B., Won, S.M., Kim, Y-S., Song, Y.M., Yu, K.J., Ameen, A., Li, R., Su, Y., Yang, M., Kaplan, D.L., Zakin, M.R., Slepian, M.J., Huang, Y., Omenetto, F.G., and Rogers, J.A.   A Physically Transient Form of Silicon Electronics. Science, 337, 1640 (2012).

Dissolvable thin film transistor partially immersed in solvent. COURTESY: iMedicalApps.com.

[14] Mannsfeld, S.C.B., Tee, B.C-K., Stoltenberg, R.M., Chen, C.V.H-H., Barman, S., Muir, B.V.O., Sokolov, A.N., Reese, C., and Bao, Z.   Highly sensitive flexible pressure sensors with microstructured rubber dielectric layers. Nature Materials, 9(9), 1-6 (2010).

[15] Slominski, A.T., Zmijewski, M.A., Skobowiat, C., Zbytek, B., Slominski, R.M., and Steketee, J.D. (2012). Sensing the environment: regulation of local and global homeostasis by the skin's neuroendocrine system. Advances in Anatomy, Embryology, and Cell Biology, 212, 1-115.