Virtuality: a new view on virtual experience

The first images when you Google "virtuality". Products from the Virtuality Group. LEFT: Virtual Reality Society, RIGHT: Wikimedia (Dr.  Jonathan D. Waldern/Virtuality Group).

At Neuromatch 2.0, I gave a talk on the concept of virtuality titled "Computational Virtuality as a Form of Artificial Intelligence" (slides available on Figshare). Virtuality is a means to summarize all brain and body systems involved in a virtual experience. This talk features two big ideas, the second of which the Saturday Morning NeuroSim group will be building on in the coming months.  

1) "Being Virtual" as Human Performance (click to enlarge).

2) Differentiating Allostasis machine (click to enlarge). Allostatic load is induced collectively by stimulus [1], leads to dysregulation of system and ultimately to a new homeostatic setpoint.

Overall, virtuality can be characterized as a constructive sensory experience (h/t to Anson Lim). The first big idea stems from the observation that current theories are too limiting to properly characterize virtual experiences at the systems-level. Indeed, the very notion of objective reality is based in part on the synchronization of perception and action. When this synchronization is disrupted or perturbed in any way, we observe virtuality. This does not even require a virtual environment -- virtuality can be experienced in our interaction with the physical world [for an example from fly-human interaction, see 2]. But virtual environments are the most common way to elicit the allostatic load required to observe virtuality.

There are two aspects of embodied performance essential to understanding the virtuality effect. The first is the idea of cognitive gaps, or disruptions to spatiotemporal representations caused by the perturbation of reality. The second involves inducing allostatic load in the internal model the represents perception-action coupling. The latter has wide-ranging effects in the homeostat (nervous system), and can lead to many different effects of varying length scales.

But where does the artificial intelligence part come in? In the second part of the talk, I conclude that the effect sizes for human experiments are too small and formal experimental design is too limiting to demonstrate virtuality. This is particularly true for observing virtuality over long periods of time. Therefore, we can use computational agents! To propose a general approach, we want to identify two attentional and three sensorimotor features essential for any agent to exhibit virtuality.

Naturally, we want our agents to be embodied in some manner. We propose working with two types of agent: Morphogenetic Agents, a novel agent type that can exhibit both pattern recognition and morphogenesis (percepetion-action), and the well-known Braitenberg Vehicle, in this case experiencing incongruous environmental physics which leads to gaps in the perception-action loop. 

NOTES:

[1] this requires a ecological view of perception. For a short introduction, please see: Lobo, L., Heras-Escribano, M. and Travieso, D. (2018). The History and Philosophy of Ecological Psychology. Frontiers in Psychology, doi:10.3389/fpsyg.2018.02228.

[2] Alicea, B. (2013). Perceptual Time and the Evolution of Informational Investment. Synthetic Daisies blog, September 24.

Deep Reading Brings New Things to Life (Science)

Here is an interesting Twitter thread from Jacquelyn Gill on 'deep reading':

The basic idea is that exploring older literature can lead to new insights, which in turn lead to new research directions. The new research of our era tends to focus on the most relevant and cutting-edge literature [1]. This recency bias excludes many similarly relevant articles, including articles that perhaps inspired the more recent citations to begin with [2]. 

I have my own list of deep reads that have influenced some of my research in a similar fashion. These references can be either foundational or so-called "sleeping beauties" [3]. Regardless, I am doing my part to maintain connectivity [4] amongst academic citation networks:

1) Woodger, J.H. The Axiomatic Method in Biology. 1937.

An argument for biological rules, an influence on cladistics (developed in the 1960s), and a natural bridge to geometric approaches to data analysis and modeling. While there is a strong argument to be made against the axiomatic approach [5], this directly inspired much of my thinking in the biological modeling area. 

2) Davis R.L., Weintraub H., and Lassar A.B. Expression of a single transfected cDNA converts fibroblasts to myoblasts. Cell 51, 987–1000. 1987.

This was the first proof-of-concept for direct cellular reprogramming, and predates the late 2000's Nobel-winning work in stem cells by decades. In this case, a single transcription factor (MyoD) was used to convert a cell from one phenotype to another without a strict regard for function. More generally, this paper helped inspired my thinking in the area of cellular reprogramming to go beyond a biological optimization or algorithmic approach [6].

3) Ashby, W.R. Design for a Brain. 1960.

"Design for a Brain" serves as a stand-in for the entirely of Ashby's bibliography, but this is the best example of how Ashby successfully merged explanations of adaptive behavior [7] with systems models (cybernetics). In fact, Ashby originally coined the phrase "Intelligence Augmentation" [8]. I first discovered Ashby's work while working in the area of Augmented Cognition, and has been more generally useful as inspiration for complex systems thinking.

4) Schroeder, M. Fractals, Chaos, and Power Laws: minutes from an infinite paradise. W.H. Freeman. 1991. AND Kauffman, S. At Home in the Universe: search for the laws of self-organization and complexity. Oxford University Press. 1995.

Not so much a couple of sleeping beauty as easy reading technical reference guides for all things complexity theory.

5) Bourdieu, P. Outline of a Theory of Practice. Cambridge University Press. 1977 AND Alexander, C., Ishikawa, S., and Silverstein, M. A Pattern Language: towns, buildings, construction. Oxford

University Press. 1977.

This is a bonus, not because the references are particularly obscure or even from the same academic field, but because they partially influenced my own view of cultural evolution. This is yet another piece of advice to young researchers: take things that appear to be disparate on their surface and incorporate them into your mental model. If nothing else, you will gain valuable skills in intellectual synthesis.

UPDATE (11/17):

Here is another example of old (classic, not outdated) work influencing new scholarship.

NOTES:

[1] Evans, J.A. (2008). Electronic Publication and the Narrowing of Science and Scholarship. Science, 321(5887), 395-399 AND Scheffer, M. (2014). The forgotten half of scientific thinking. PNAS, 111(17), 6119.

[2] related topics discussed on this blog include distributions of citation ages and most-cited papers.

[3] van Raan, A.F.J. (2004). Sleeping Beauties in Science. Scientometrics, 59(3), 467–472.

[4] Editors (2010). On citing well. Nature Chemical Biology, 6, 79.

[5] For the semantic approach (which had been influential to my more recent work), please see: Lloyd, E.A. (1994). The Structure and Confirmation of Evolutionary Theory. Princeton University Press, Princeton, NJ.

[6] Ronquist, S. et.al (2017). Algorithm for cellular reprogramming. PNAS, 114(45), 11832–11837.

[7] Sterling, P. and Eyer, J. (1988). Allostasis: A new paradigm to explain arousal pathology. In "Handbook of life stress, cognition, and health". Fisher, S. and Reason, J.T. eds. Wiley, New York. 

[8] Ashby, W.R. (1956). An Introduction to Cybernetics. Springer, Berlin.

Are the Worst Performers the Best Predictors?

What is the link between performance and a priori predictions? At the individual level, this question has important implications for areas as diverse as sports [1] and job [2] performance potential (including the so-called "Moneyball" approach). This may also be useful for understanding the effects of exercise and technological augmentation on human populations. 

Working Hypothesis:  The performance of sports teams that are perennial winners or losers are much easier to predict than other teams (e.g. those that exhibit parity).

In an attempt to form theoretical insights based on this question, I conducted a rudimentary analysis on how informative PredictWise and BetFair's prognostications for the 2013 MLB season were with respect to the final regular season standings [3]. A similar analysis was done on NFL data to see if these results hold across types of data. 

PredictWise [4] is an aggregator of likelihoods for purposes of betting on outcomes. Their predictions include contests in the realm of politics, sports, and entertainment. The likelihoods are updated as the event unfolds, but the comparison of a priori predictions provide interesting comparisons with the final outcome. These predictions are not entirely naive, but do rely upon a fair number of assumptions.

The first graph shows the difference in rank-order position between the likelihood of winning the world series (generated a priori) and the regular-season won-loss record. The "difference from prediction" was then calculated for the top, middle, and bottom tercile on teams based on their regular-season record.

Interestingly, many of the winningest teams were not predicted to finish strongly. By contrast, the bottom tercile was equally represented by teams that had the least chance of winning it all and teams that were supposed to finish more strongly. With a few exceptions, the middle tercile was represented by underachieving teams, and the most consistent performances (smallest deviations from prediction) were among the lowest achieving teams.

The next two graphs show the magnitude of deviation from prediction (observed vs. predicted). This results in an index (value: 0-1) based on a team's deviation from prediction relative to the maximum and minimum of all teams in the league. The third graph (two panels) breaks this down into teams that finished better than and worse than expected.

Finally, the fourth graph demonstrates how the deviation from prediction is related to the total number of wins a team had during the season. This plot lends no additional support to but is consistent with the notion of "worst performers, best predictors".

To compare these tendencies across sports and odds-making enterprises, I used the Sporting News a priori predictions for the NFL 2013 season [5]. In this example, I compared a team's n-to-1 odds of winning the Super Bowl with the final season standings (e.g. similar methodology to the MLB analysis, but with a different source of predictions). 

From this exploratory graph [6], a similar trend of "worst performers, best predictors" emerges, albeit with more outliers on the lower end. Recapitulating the difference from prediction analysis done for the MLB data, the NFL data shows more deviations from prediction for every stratum of the dataset. However, again, there is a slight tendency for the bad teams to be predicted correctly and the best performing teams to be poorly-predicted. In the case of the NFL data, there is a countervailing "dynasty" effect as well: teams that have been winning consistently were also predicted to do well. As they met this expectation, they were easier to predict correctly.

So are there better means to predict outcomes than making odds? PredictWise uses a combination of a priori odds-making and individual wagering. When people are willing to wager on an outcome, a diversity of mental models are used to inform the prediction. We can also use real-time surveys that make predictions in a manner similar to a logistic regression model [7]. However, whether such approaches can ameliorate the "surprise" factor of unexpected levels of performance (good or bad) is questionable.

NOTES:

[1] Morey, D.   The elephant on the court. Economist, April 20 (2012).

[2] Armstrong, J.S.   Predicting Job Performance: the Moneyball factor. Foresight, Spring (2012).

[3] Sohmer, S.   PredictWise: aggregating the wisdom of crowds. Hypervocal, October 11 (2011).

[4] The Linemakers   Odds to win 2014 Super Bowl. Sporting News, February 4 (2013).

[5] The dataset (predictions vs. MLB and NFL standings from 2013 seasons) can be found at Figshare (doi:10.6084/944542).

[6] The x-axis is defined as the final won-loss record centered upon a .500 (8-8) record. The formula is (WINS)-(LOSSES)+(TIES*0.5). The y-axis is an index based on the odds ratio, where the lowest odds are set to 1.0. The formula is ((ODDS)/(LOWEST ODDS))¹ Rank-orderings of these metrics (and distances between these rank-orderings) were also used to generate the graphs.

[7] Ulfelder, J.   Using Wiki surveys to forecast rare events. Dart-throwing Chimp blog, August 11 (2013).

Fitting the Science to the Hype

This is a response to a recent article in Forbes by Steve Kotler [1], co-author of the book "Abundance" and techno-optimist. Even though it is intended as a blue-sky vision of an emerging technology, it also suffers from something I will call "fitting science to hype" [2], which happens a lot in media coverage of emerging technologies. That being said, I had a number of problems with this article, from its provocative title to its conclusions.

COURTESY: Shutar.

Point 1: It is actually inconclusive as to whether or not video games are addictive, at least in the same way as heroin. They may be addictive in the sense that a person prone to addictions will use video games as an addiction, but that is putting the cart before the horse (in a causal sense). To start off with this premise, and then build an argument off of this "self-evident truth" using an array of neurochemical facts (but not directly relevant evidence) is a bit misleading.

Point 2: The idea that "video games causes dopamine stimulation, therefore they are addictive" is likewise misleading. Aside from being a basis of addictive experiences, dopamine is also a very general signaling molecule for behavioral motivation. For example, dopaminergic signaling serves as the basis for economic transactions (e.g. rewards), so generalizing this to addiction is a large step indeed. Can we say then that because they are tightly associated with increases in dopamine, virtual environments will necessarily become essential economic tools? Probably not.

Dopamine machines, or dopaminergic-friendly activity (so to speak)? For more on the potential addictive aspects of financial trading and risk-taking, see [3].

Point 3: Video games are indeed dopamine-production machines. So are sports cars, jet skis, motorcycles, and guns. I'm not quite sure that one could make the argument that sports cars and jet skis cause the dopaminergic excesses associated with addiction. Aside from the dodgy reasoning of "machine-therefore-dopamine-therefore-addiction", human variation is an important determinant of addiction [4].

But the real problem here is that there is no null hypothesis to compare against. For example, the logical extreme of this argument would be that video games are always addictive. This, of course, would sound foolish. A proper way to disentangle this would be to look at dopamine production across the process of addiction, and then compare this with dopamine production in people who play video games but do not become addicted to them.

Point 4: Why will the our current lack of knowledge regarding how to control neurochemistry change? And why would we want to create virtual world addicts? This is how the sixth paragraph reads to me. There are areas of research such as Augmented Cognition [5] and allostatic regulation [6] and that would be relevant here, but alas nothing was discussed.

Addiction in the context of allostasis and allostatic load. COURTESY: Figure 5 in [6].

Point 5: As a source of explaining addiction as a result of over-allotted human cognition, Kotler's use of flow theory is oddly seductive. It allows for a higher state of awareness during focused activities (e.g. a magical mechanism) to be substituted for what is not well-known about cognitive processes like multisensory integration, attention, and memory. Or perhaps there is a selective interface between, say, flow and attention which allows for some "super" response. In any case, this is a unique form of inductive brain science, the ultimate result of which will be more questions than answers.

While this might be a bit of positivist bias on my part, it is worth keeping in mind that flow always needs to be rigorously defined in its application. Paragraph eight provides an example: the claim is made that flow results in (or from, which is not clear) an extremely potent neurochemical cocktail. This mimics a rapid hit of illicit drugs, and flow itself is the source of intrinsic motivation. A claim like this makes the direction of causality (e.g. enhanced neurochemistry result in flow, or flow results in enhanced neurochemistry) quite important.

Point 6: When I see the words "Moore's Law", I reach for my gun. Or my keyboard to protest. This is not an issue of too much dopamine production, but an issue of gross overuse. In any event, it's almost certain that Moore's Law does not apply here, as what is needed to understand the effects of video games on brains are better models to interpret the neurological data, not increasingly greater amounts of data on its own.


I think the idea of deep embodiment, while well-placed in this article, also masks the magnitude of challenges inherent in this type of work. From my own work, I would predict that deep embodiment might only be achieved through the use of a multivariate, closed-loop interface. Closed-loop control allows for entrainment between the environment and physiological dynamics.

The substrate involved with the setting of physiological state includes many moving parts, including nonlinear control components such as delays and superadditive responses. So take the latest hyperrealistic video game title. A human might experience deep embodiment for awhile as their physiology learns the intensity of the experience. This is a proto-learning that is similar to a more generalized plastic response. Yet, as with all forms of learning, there will be diminishing returns and acquisition will saturate. Thus, the person will adapt over time.

While one could say the person has been "sensitized" to the virtual environment, in reality we are witnessing a form of accomodation (complex as it is) mimicking the effects of addiction. Unless the person is explicitly trained, there will be a large effect between sensory performance in the real versus virtual world.

A model of brain-VR interactions without the pretense of parsimony.

UPDATE (3/1/2014): Here is a new paper [7] on the neural correlates of internet addiction. Given individuals who are identified as addicted to the internet, neuroimaging reveals the brain structures associated with addictive stimuli.

Neural Correlates of Internet Addiction. COURTESY: Figure 1 in [7].

NOTES:
[1] Kotler, S. and Edwards, L.A.   Legal Heroin: is virtual reality our next hard drug? Forbes, January 15 (2014).

[2] Fitting the science to the hype (or putting hype before the data) is much like putting models before the data, except that hype resembles an intentionally distorted model. On RationalWiki, this type of hype before the data is classified as "Science Woo". While I would not go that far, blue-sky hype is always something to be skeptical about.

[3] Coates, J.   The Hour Between Dog and Wolf. Penguin Books. YouTube video (2012).

[4] Volkow, N.D., Wang, G.J., Fowler, J.S., and Tomasi, D.   Addiction circuitry in the human brain. Annual Reviews in Pharmacology and Toxicology, 52, 321-336 (2012) AND Kuss, D.J.   Internet gaming addiction: current perspectives. Psychological Research and Behavioral Management, 6, 125-137 (2013).

[5] Schmorrow, D. and Fidopiastis, C.M.   Foundations of Augmented Cognition. Lecture Notes in Computer Science, Volume 8027 (2013).

[6] Koob, G.F. and Le Moal, M.   Drug Addiction, Dysregulation of Reward, and Allostasis. Neuropsychopharmacology, 24, 97-129 (2001).

[7] Gallinat, J. and Kuhn, S.   Brains online: structural and functional correlates of habitual Internet use. Addiction Biology, doi:10.1111/adb.12128 (2014).

Perceptual Time and the Evolution of Informational Investment

We tend to think of the flow of time in the context of evolution and biology as a fairly consistent thing [1]. We are used to the conceptual mechanisms of molecular clocks, thermodynamic entropy, and circadian rhythms. All of these mechanisms maintain regularity with respect to the flow of time. However, this order may not be as universal as we would like to believe. In fact, there may be a form of perceptual relativism enabled by evolution, physiology, and (increasingly) technology that unseats this order in many unexpected ways. This post will utilize two recent papers as a means to explore this issue.

A recent paper entitled "Metabolic rate and body size are linked with perception of temporal information" by Kevin Healy and colleagues [2] demonstrates that differences in visual sampling across species living in different ecosystems are constrained by the metabolic rate of the organism. Visual sampling can be measured in terms of an organism's critical flicker fusion (CFF) frequency. CFF frequency is the sampling rate (or rather, the minimum sampling rate) at which images captured by the retina are integrated by the brain into coherent visual scenes. A very-high or very-low CFF frequency may lead to differences in how the flow of environmental events is perceived by the organism [3], and can lead to other differences in performance (see Figure 1 for example).

Figure 1. Human trying to swat a fly.

As a sidenote, this paper has elicited an interesting set of reactions in the news media. In some cases, this is being sold as a suggestion that flies experience a lifespan equivalent to that of humans, even though the human lifespan (in terms of biological processes) is much longer [4]. Regardless of the speculation, the potential for relativistic time-keeping [5] across species may also be interesting from an evolutionary standpoint. Is the CFF frequency determined solely by the requirements of ecological niche [6], or is the CFF frequency constrained by metabolic rate, and why? It is well-known that metabolic rate scales allometrically with body size [7], for reasons that are clearly due to energetic efficiency. But might this also extent to CFF frequency?

Figure 2. CFF frequency, explained using a classic reel-to-reel movie projector.

The authors suggest that CFF frequency is merely constrained but not determined by the metabolic rate. This pattern is predicted by the expensive tissue hypothesis [8]. This hypothesis suggests that amount and structure of neural tissue in an organism must be highly-optimized due to the high energetic cost of electrical activity/excitability. In general, the more neural tissue used by the organism (e.g. bigger brains, more elaborate eyes), the higher the energetic cost. If the cost is high enough, this clearly serves as an absolute constraint on the size of an organism’s neuronal architecture. But what are the consequences of this with respect to cognitive complexity [9]?

Figure 3. The principle behind the expensive tissue hypothesis and metabolic scaling, visualized. COURTESY: http://universe-review.ca/R10-35-metabolic.htm

At this point, I would like to propose something called the "animation bottleneck" hypothesis, and is similar to one role ascribed to early attentional selection mechanisms in human consciousness [10]. The animation hypothesis suggests that while higher frequency visual sampling of the environmental provides an advantage for identifying and executing very-high frequency events (being able to catch insect prey or the beginnings of an explosion), lower frequency visual sampling may have other advantages that result in an evolutionary tradeoff. In the case of CFF frequency, a lower sampling rate might result in a greater need to make proper inferences as to what will happen in between the samples. This could result in bigger brains. However, it might also have other consequences, such as the evolution of attentional capacity [11]. If so, variation in the sampling rate within a species might have unique fitness consequences.

So what happens when you have variation in the environment that far exceeds the baseline ability of perception? In natural populations, the findings of [2] demonstrate that environmental stimuli are less of a selective pressure than often assumed [6]. But in the technological environment, there might be signals that exceed what are typically found in the natural world (e.g. ultrafast extreme events). Such is the case with HFT (high-frequency trading) [12]. HFT is defined by [13] as computer-driven algorithm-based trading at speeds measured in millionths of a second.

Figure 4. Trading of stocks according to the HFT model [14]: thirty (top) and five-hundred (bottom) millisecond advantages. COURTESY: New York Times.

A recent paper on HFT and decision-making called "Abrupt rise of new machine ecology beyond human response time" considers just this possibility [15]. In this case one can use artificial agents to explore whether or not the information sampling capacity discussed in [2] is merely a constraint of animal visual systems or if it is a consequence of fitness and selection. This is an intriguing paper, but does not fits cleanly into the context of biological evolution and/or human performance. Nevertheless, it raises a few key points about the evolution of neurobiological information processing.

Figure 5. Illustration of ultrafast extreme events in the context of HFT. COURTESY: Figure 1 in [15].

The authors model this as a competitive process of trading agents using their own ecological perspective. A simulation is used to better understand the relationship between strategies employed by a population of agents and ultrafast extreme events (e.g. the high-frequency trading of shares or, in some cases, a so-called flash-crash). In [16], HFT-reliant trading behavior enables three key advantages, which stem from both having access to very-high-frequency environmental samples and the ability to act upon them [15]. These include: better access to the market, a major speed advantage, and a greater understanding of the market's temporal microstructure. Another ecological explanation suggests that computers trading at very-high frequencies safeguard the market against irrational or inexperienced traders [17].

By now, it may seem self-evident that stimuli related to HFT provide an incentive for higher sampling rates, if not some sort of intra-generational selective advantage. Yet it may also be that most of these ultra-high-frequency events are not the information they are made out to be. Rather, the real information in markets may reside in longer interval periods. Individuals that sample the markets at HFT-like frequencies may actually be aliasing (or oversampling) their environment [18]. In fact, it may be that the purported advantages of HFT are largely driven by noise (e.g. benefits derived from chance) and not information. Returning to the fly visual system, it could also be the case that flies deal with large amounts of visual noise, which may act to suppress any perceptual (or fitness) advantage they may gain from being able to detect things at very high frequencies.

Figure 6. The relationship between the collective action of agents, share price, and strategies employed by agents. COURTESY: Figure 6 in [15].

In short, are visual systems and cognitive complexity selected for the right amount of information in the environment, or are they constrained by other factors? And what consequences does this have for the evolution of signaling and perception? Perhaps very-high and very-low frequency events can be inferred by the visual system and brain in a way that is "good enough" not to be detrimental to fitness. Or the other hand, perhaps very-high and very-low frequency events provide an opportunity to create new niches and hide communication from predators/prey and/or conspecifics. In both cases, these type of events have a subtle effect on cognition and behavior that is largely mysterious in nature. The deployment of evolutionary simulations might provide us with some answers.

UPDATE (6/18/2014): This post has been re-published with slight modifications at Machines Like Us.

NOTES: 

[1] For a alternate philosophical and theoretical view, this book might be interesting: Vrobel, S., Rossler, O.E., Marks-Tarlow, T.   Simultaneity: Temporal Structures and Observer Perspectives. World Scientific (2008).

[2] Healy, K., McNally, L., Ruxton, G.D., Cooper, N., and Jackson, A.L.   Metabolic rate and body size are linked with perception of temporal information. Animal Behavior, 86, 685-696 (2013).

[3] Assuming that CFF frequency is the only component of visual perception, and that the functions of these components are not linked. For more on this, please see: Skorupski, P. and Chittka, L. Differences in photoreceptor processing speed for chromatic and achromatic vision in the bumblebee Bombus terrestris. The Journal of Neuroscience, 30: 3896–3903 (2010).

[4] The popular media have spun this paper a number of different ways (e.g. Google: "Healy" + "Animal Behavior" + "Fly" + "Hz"), and seems to be a lesson in what the public takes away from a research paper. A few examples

a) Silverman, R.   Flies see the world in slow motion, say scientists. The Telegraph, September 16 (2013).

b) Slo-mo Mojo. Economist, September 21 (2013).

c) Time is in the eye of the beholder: time perception in animals depends on their pace of life. ScienceDaily, September 16 (2013).

[5] There are a number of interesting parallels between the percpetual dilation of visual cues in time and the opportunities afforded by virtual world simulations. For more information, please see: Alicea, B.   Relativistic Virtual Worlds: a emerging framework. arXiv, 1104.4586 (2011).

[6] A competing hypothesis (strong ecological) predicts that the response dynamics of retina are ecosystem-specific. For more information, please see: Autrum, H.   Electrophysiological analysis of the visual system in insects. Experimental Cell Research, 14(S5), 426-439 (1958).

[7] Brown, J.H., Gillooly, J.F., Allen, A.P., Savage, V.M., and West, G.B.   Towards a Metabolic Theory of Ecology. Ecology, 85(7), 1771-1789 (2004).

[8] Aiello, L.C. and Wheeler, P.   The Expensive-tissue hypothesis: the brain and the digestive system in human and primate evolution. Current Anthropology, 36(2), 199-221 (1995).

[9] Assuming these comparisons can be made for CFF frequency in the first place. For more on this, please see: Chittka, L., Rossiter, S.J., Skorupski, P. & Fernando, C. (2012). What is comparable in comparative cognition? Philosophical Transactions Of The Royal Society, 3671: 2677-2685.

[10] For more background on early attentional selection and the connections between vision and the construction of conscious percepts, please see:

a) Zhaoping, L. and Dayan, P.   Pre-attentive visual selection. Neural Networks, 19, 1437-1439 (2006).

b) Van Rullen, R. and Koch, C.   Is perception discrete or continuous? Trends in Cognitive Sciences, 7(5), 207 (2003).

c) Ogman, H. and Breitmeyer, B.G.   The First Half-Second: the microgenesis and temporal dynamics of unconscious and conscious visual processes. MIT Press (2006).

[11] For more on the evolution and life-history variability in attentional capacity, please see:

a) Kruschke, J.K. and Hullinger, R.A.   Evolution of attention in learning. In N.A. Schmajuk (ed.) Computational Models of Conditioning. pgs. 10-52, Cambirdge Press, Cambridge, UK (2010).

b) McAvinue, L.P., Habekost, T., Johnson, K.A., Kyllingsbek, S., Vangkilde, S., Bundesen, C., and Robertson, I.H.   Sustained attention, attentional selectivity, and attentional capacity across the lifespan. Attention, Perception, and Psychophysics, 74(8), 1570-1580 (2012).

c) Humphreys, G.W., Kumar, S., Yoon, E.Y., Wulff, M., Roberts, K.L., and Riddoch, M.J.   Attending to the possibilities of action. Philosophical Transactions of the Royal Society B, 368, 20130059 (2013).

[12] Ritholtz, B.   What Happens During 1 Second of HFT? Big Picture blog, May 7th (2013).

[13] Patterson, S. and Rogow, J.   What's behind high-frequency trading? Wall Street Journal, August 1 (2009).

[14] Duhigg, C.   Stock traders find speed pays, in Milliseconds. NYTimes, July 23 (2009).

[15] Johnson, N., Zhao, G., Hunsader, E., Qi, H., Johnson, N., Meng, J., and Tivnan, B.   Abrupt rise of new machine ecology beyond human response time. Scientific Reports, 3, 2627 (2013).

[16] Lopez, L.   A high-frequency trader explains his three basic advantages. Business Insider, September 20 (2012).

[17] Smith, N.   A healthy side-effect of high-frequency trading? Not Quite Noahpinion blog, August 11 (2013).

[18] Alicea, B.   Economic trace, pondered. Synthetic Daisies blog, November 12 (2011).

Human Augmentation Short Course -- Part III

The next two #human-augmentation flash lectures from my micro-blog, Tumbld Thoughts will feature several potential implementations of Intelligence Augmentation (IA), Augmented Cognition (AugCog), and its integration with smart devices. This includes two topical areas: I (Bio-machine Symbiosis and Allostasis), and II (Augmentation of Touch).

I. Bio-machine Symbiosis and Allostatis

The book "The Symbiotic Man" by Joel DeRosnay can be used to frame a graphical discussion on bio-machine symbiosis (e.g. human-smart home interaction) and the concept of mixed allostatic networks. In this case, the symbiotic relationship is between a biological system and a technical one. While there are fundamentally different dynamics between these two types of systems, the fusion of their interactions are not only possible but essential.

As discussed in previous slides, measurements from a human can be used to provide intelligence to the house (in this case, scheduling and other use information). A mitigation strategy can be used to extract information from the collected data and provides instructions for machine learning.

Measurement of the human can be taken on physiological state (e.g. measurements of brain activity or state monitoring of other organs). This can be done using microelectronics, and the measurements must cross a semi-permeable boundary which is selective with respect to available information. Nevertheless, this network allows us to construct a consensus approximation of the body's homeostatic control mechanisms.

This allows us to construct mixed allostatic networks. A mixed allostatic network includes elements from both the house (e.g. appliances) and the human body (e.g. organs). This has already been done by integrating body area networks and domotic networks.

The key innovation here is to unite the function of both networks under global, allostatic control. When the allostatic load of this network becomes too great, this information can be used to modify the mitigation strategy. This may be done in a manner similar to DeRosnay's Symbionomic Laws of Equilibrium.

II. Applications related to the Augmentation of Touch

In this installment of the #human-augmentation tag, we will discuss an assortment of applications that have the potential to augment the sense of touch and upper body mobility. 

The first technology was recently featured in IEEE Spectrum's startup spotlight. The Italian startup Prensilia [1] is working on a robotic hand called Azzurra. The fully artificial hand mimics human grip by using underactuated movements. Inside the hand, the rotary motion generated by a motor is translated to linear actuation to produce biological (e.g. muscle generated) types of motion.

The second technology features the DARPA initiative to create better prosthetic arms. In this video from IEEE Spectrum, the work of Dean Kamen and his group at DEKA Research is profiled. This type of prosthetic arm uses bioelectric signals from chest muscles in combination with servo motors to enable both fine motor and ballistic movements.

The third technology is simulated touch, which unlike the last two does not explicitly involve artifacts. Touch is a physical phenomenon, as contemplated in this Minute Physics video. However, touch also involves human perception, as discussed previously on Tumbld Thoughts. A thourough understanding of this sense allows us to build better ways to interact with virtual environments and robots using touch [2]. 

COURTESY: Chapter 4 from [2b].

NOTES:

[1] Cipriani, C.   Startup Spotlight: Prensilla developing robot hands for research, prosthetics. IEEE Spectrum, July 18 (2013).

[2] The second image from bottom is a LilyPad Arduino project. For more information on the engineering of touch, please see these two books:

a) McLaughlin, M.L., Hespanha, J.P., and Sukhatme, G.S.   Touch in Virtual Environments: haptics and the design of interactive systems. Prentice-Hall, Upper Saddle River, NJ (2002).

b) Bicchi, A., Buss, M., Ernst, M.O., and Peer, A.   The Sense of Touch and its Rendering. Springer, Berlin (2008).

Human Augmentation Short Course -- Part II

I have been continuing to introduce an area of science and engineering called human augmentation to a broader audience using the "flash lecture" format discussed a few posts back. I have using my micro-blog Tumbld Thoughts as a test site for posting these lectures, and the social networking function of Tumblr (e.g. Tumblr radar) has garnered some sporadic direct interest (in the form of likes and re-posts). 

In this portion of the course (four lectures), we move beyond the basics and towards both more detailed phenomena related to augmentation and practical implementations of the technology.

I. Extending the Phenotype

In previous #human-augmentation posts, I briefly touched on the potential role of augmentation (e.g. wearing a prosthetic limb or see-though, head-mounted display) on physiological regulation. This may leave some readers puzzled, because often times these technologies do not directly interface with the nervous system. Nevertheless, once a technology provides a stand-in (or enhancer) of something the body does, it becomes incorporated into the body's physiology and representation of the world [1].

While technologies from rakes to nests have been found to represent an extended phenotype, an intelligent technology (one that includes an adaptive mitigation strategy) can actually serve to enhance or work in concert with an individual's ability for environmental adaptation [2]. Much like the diet and exercise regimen that helps a person lose weight, this ability exhibits great variation across individuals, which may be explained by pre-existing phenotypic or even genotypic differences.

The extent to which the technology participates in the physiological millieu depends of course on how much the augmentation assists in or takes over function. To understand this better, we can turn to the ergonomics definition of symbiosis, which defines "ergonomy" as the degree of coupling between human and mechanical device [3]. This ranges from tightly-coupled systems (implants that are seamlessly integrated into normal function) to ill-fitting systems (poorly-designed interfaces or computer mice).

II. Instrumented Motorcycle Helmet

Here is an example of performance augmentation in the form of an intelligent, see-through, and heads up display integrated into a motorcycle helmet. Brought to you by a start-up called LiveMap. The information presented in the field of view enables the wearer to improve their navigation ability and improve their riding experience. 

The first article (from Mashable - [4]) highlights the components of the helmet, which includes ambient information from multiple types of sensor (e.g. light sensor, microphone, GPS). This information is then fused and presented in a single location (in this case, the helmet) [5].

II. Wired Science Live Chat on Bionic Augmentation

Last week, I attended a live chat called "Our Cyborg Future", hosted by Wired Science. This was a live chat with two scientists in the field: John Rogers from UIUC and Michael McAlpine from Princeton [6]. These researchers work in a field called "bionics", where biological systems are augmented with electronics or other technology to either restore function or provide new sensory or performance capabilities.

The talk featured a number of visions for the future of bionic technologies and technologies for human augmentation. Fundamentally, the major challenge is to merge the language of electronics (e.g. electrons, phonons, and heat) with the language of biology (e.g. ions, proteins, and enzymes). While John Rogers is working towards electronically-augmented organs [7], Michael McAlpine is working towards using piezoelectric materials to harvest energy from biological motion and print 3-D structures such as tissue scaffolds. 

Three of the most interesting ideas [8] discussed during the talk:

* advances such as flexible [7] and bio-compatible electronics might be used to infuse a biological system with distributed electronics at the cellular and subcellular scale. This could include a range of components from silicon diodes to LEDs.

* the increases in brain-machine interface bandwidth due to flexible, laminated skin-like devices.

* the development of emerging technologies such as stretchable batteries, glucose fuel cells, implantable micro-heaters, and mechanical energy harvesting.

IV. The Role of Attention, Training, and critical Meta-Analysis

In a previous #human-augmentation post, I pointed to one experimental paradigm (environmental switching) that may serve as a natural (e.g. non-computational) filter for eliminating (e.g. mitigating) non-optimal performance due to environmental stresses or other challenges. This case is illustrative for two reasons: 

1) in cases where performance response curves are very complex and cannot be characterized by a simple mathematical function (e.g. a "U" shaped curve), a mitigation strategy involving physical chaos (rather than computational control) or other environmental manipulations may be more effective. In the first image (top), the potential dynamic effects of perturbation on attentional shifts during an episode of "Star Trek" is used as an example.

2) fully understanding the effects of mitigation may require more systematic experimental evaluation. One example of this involves the claim that long-term expertise with action video games improves cognitive abilities [9]. A meta-analysis of such studies [10] questions this assumption on several grounds, particularly with respect to the magnitude of improvement (e.g. effect size).

In this case of action video game expertise, two types of effect have been reported [10, 11]. The first involves relative expertise based on cross-sectional comparisons, which evaluate differences between gamers and non-gamers. The second involves acquired expertise (training in action video game play) as having numerous cognitive benefits.

The second type of effect is partially due to an effect called transfer of training (see image, lower left), in which skills acquired in one context can be transferred to another context. This effect also plays a role in human augmentation, and may exhibit a large degree of individual variation. However, there are three caveats raised in [10] that must be kept in mind not only for future action video game studies, but human augmentation studies as well:

* in studies that evaluate the effects of training, an adequate baseline for untrained performance must be used. 

* results should be generalizable to different settings and population (e.g. exhibit a high degree of experimental reproducibility).

* while there may be several improvements to performance/cognition attributable to prior training or experience with the activity in question, there may be many more outcomes that are unaffected by the treatment (action video games) or mitigation (human augmentation). 

NOTES:

[1] A few examples of this extension of the phenotype includes examples from humans (i), social insects (ii, iii), and animals (iv):

(i) Maravita, A. and Iriki, A.   Tools for the body (schema). Trends in Cognitive Science, 8(2), 79-86 (2004).

(ii) Turner, J.S.   The Extended Organism: The Physiology of Animal-Built Structures. Harvard University Press, Cambridge, MA (2002).

(iii) Turner, J.S.   Extended Phenotypes and Extended Organisms. Biology and Philosophy, 19, 327–352 (2004).

(iv) Schaedelin, F.C. and Taborsky, M.   Extended phenotypes as signals. Biological Reviews of the Cambridge Philosophical Society, 84(2), 293-313 (2009). 

[2] This ability, or adaptability, can be characterized using a parametric landscape as shown in a previous post.

[3] Licklider, J.C.R.   Man-Computer Symbiosis. IRE Transactions on Human Factors in Electronics, HFE-1, 4-11 (1960).

[4] Murphy, S.   A Motorcyclist's Dream: Google Glass in helmet form. Mashable, June 17 (2013).

[5] Mike E. New motorcycle helmet with augmented reality and navigation technology built in. VR-Zone, June 17 (2013).

[6] A transcript of the talk can be found here. Also see the McAlpine Research YouTube channel.

[7] For more on flexible robotics, please see this: Zheng, Y., He, Z., Gao, Y., and Liu, J.   Direct Desktop Printed-Circuits-on-Paper Flexible Electronics. Scientific Reports, 3, 1786 (2013).

[8] For more interesting ideas related to cyborgs and bio-inspired robotics, see these:

Rowe, A.   Top 10 Cyborg Videos. Wired Science, November 15 (2009)

When Animals Learn to Control Robots, You Know We’re in Trouble. Wired Science, March 21 (2013).

[9] Green, C. and Bavelier, D.   Learning, Attentional Control, and Action Video Games. Current Biology, 22(6), R197-R206.

[10] Boot, W.R., Blakley, D.P., Simons, D.J.   Do action video games improve perception and cognition? Frontiers in Psychology, 2, 226 (2011).

[11] Simons, D.   Think video games make you smarter? Not so fast..... Daniel Simons blog, December 30 (2012).