Are the Worst Performers the Best Predictors?

What is the link between performance and a priori predictions? At the individual level, this question has important implications for areas as diverse as sports [1] and job [2] performance potential (including the so-called "Moneyball" approach). This may also be useful for understanding the effects of exercise and technological augmentation on human populations. 

Working Hypothesis:  The performance of sports teams that are perennial winners or losers are much easier to predict than other teams (e.g. those that exhibit parity).

In an attempt to form theoretical insights based on this question, I conducted a rudimentary analysis on how informative PredictWise and BetFair's prognostications for the 2013 MLB season were with respect to the final regular season standings [3]. A similar analysis was done on NFL data to see if these results hold across types of data. 

PredictWise [4] is an aggregator of likelihoods for purposes of betting on outcomes. Their predictions include contests in the realm of politics, sports, and entertainment. The likelihoods are updated as the event unfolds, but the comparison of a priori predictions provide interesting comparisons with the final outcome. These predictions are not entirely naive, but do rely upon a fair number of assumptions.

The first graph shows the difference in rank-order position between the likelihood of winning the world series (generated a priori) and the regular-season won-loss record. The "difference from prediction" was then calculated for the top, middle, and bottom tercile on teams based on their regular-season record.

Interestingly, many of the winningest teams were not predicted to finish strongly. By contrast, the bottom tercile was equally represented by teams that had the least chance of winning it all and teams that were supposed to finish more strongly. With a few exceptions, the middle tercile was represented by underachieving teams, and the most consistent performances (smallest deviations from prediction) were among the lowest achieving teams.

The next two graphs show the magnitude of deviation from prediction (observed vs. predicted). This results in an index (value: 0-1) based on a team's deviation from prediction relative to the maximum and minimum of all teams in the league. The third graph (two panels) breaks this down into teams that finished better than and worse than expected.

Finally, the fourth graph demonstrates how the deviation from prediction is related to the total number of wins a team had during the season. This plot lends no additional support to but is consistent with the notion of "worst performers, best predictors".

To compare these tendencies across sports and odds-making enterprises, I used the Sporting News a priori predictions for the NFL 2013 season [5]. In this example, I compared a team's n-to-1 odds of winning the Super Bowl with the final season standings (e.g. similar methodology to the MLB analysis, but with a different source of predictions). 

From this exploratory graph [6], a similar trend of "worst performers, best predictors" emerges, albeit with more outliers on the lower end. Recapitulating the difference from prediction analysis done for the MLB data, the NFL data shows more deviations from prediction for every stratum of the dataset. However, again, there is a slight tendency for the bad teams to be predicted correctly and the best performing teams to be poorly-predicted. In the case of the NFL data, there is a countervailing "dynasty" effect as well: teams that have been winning consistently were also predicted to do well. As they met this expectation, they were easier to predict correctly.

So are there better means to predict outcomes than making odds? PredictWise uses a combination of a priori odds-making and individual wagering. When people are willing to wager on an outcome, a diversity of mental models are used to inform the prediction. We can also use real-time surveys that make predictions in a manner similar to a logistic regression model [7]. However, whether such approaches can ameliorate the "surprise" factor of unexpected levels of performance (good or bad) is questionable.

NOTES:

[1] Morey, D.   The elephant on the court. Economist, April 20 (2012).

[2] Armstrong, J.S.   Predicting Job Performance: the Moneyball factor. Foresight, Spring (2012).

[3] Sohmer, S.   PredictWise: aggregating the wisdom of crowds. Hypervocal, October 11 (2011).

[4] The Linemakers   Odds to win 2014 Super Bowl. Sporting News, February 4 (2013).

[5] The dataset (predictions vs. MLB and NFL standings from 2013 seasons) can be found at Figshare (doi:10.6084/944542).

[6] The x-axis is defined as the final won-loss record centered upon a .500 (8-8) record. The formula is (WINS)-(LOSSES)+(TIES*0.5). The y-axis is an index based on the odds ratio, where the lowest odds are set to 1.0. The formula is ((ODDS)/(LOWEST ODDS))¹ Rank-orderings of these metrics (and distances between these rank-orderings) were also used to generate the graphs.

[7] Ulfelder, J.   Using Wiki surveys to forecast rare events. Dart-throwing Chimp blog, August 11 (2013).

Space Bits, "Red-sky" Thinking, and Vader the Clown

As usual, cross-posted from Tumbld Thoughts. This time, it's loosely-associated features on space science, space speculation, and science fiction. I, II, and III, respectively.

I. Space (Exploration) Simulation

Capture, stretch, and consume. The simulation discussed in this NY Times article [1] shows that the gas cloud G2 [2] is about to collide with the massive black hole that forms the center of the Milky Way galaxy (Sagittarius A*). 

II. Space (Futurism) Boosterism

Once the Singularity occurs (or perhaps slightly before), we will be able to terraform and colonize Mars. At least according to the optimists. See this Humanity+ article [3] by Nicole Willett from the Mars Society and this article in Current Biology [4] for more.

III. Space (Fictional) Humor

A day in the life of Darth Vader. From top: a morning stroll with the colleagues, pensively waiting in line at the local mall, and being put to sleep by Spock the home nurse after milk and cookies (not shown). And what roast of Darth Vader would be complete without a tip of the pen to Hanus [5] by the artist Jim Koch?

Yay, Hanus! That's all, Folks!

NOTES:

[1] Cowen, R.   It's snack time in the cosmos. NY Times, February 17 (2014).

[2] Montet, B.   Seeing black holes with a gas cloud. Astrobites blog, June 28 (2013).

[3] Willett, N.   How to Terraform Mars. Humanity+ Magazine, February 19 (2014).

[4] Gross, M.   The past and future habitability of planet Mars. Current Biology, 24(5), R175-R178 (2014).

[5] Hill, S.   Awesome Vader project puts the "art" in Darth. Wired Underwire blog, July 9 (2010).

Notes on Tangibility and Circularity (informal)

These notes are being cross-posted from Tumbld Thoughts. Some reflections on assorted readings. I and II are about tangibility, economic value, and cognitive bias. III is about circular reasoning and ideology. 

I. Tangibility and Bias: Production

This is my take on the idea [see 1,2] that land is a special form of capital which is non-reproducible and (under Western law) entirely privatizable. This suggest that production (or production that is most completely rewarded) is biased towards things that are private and tangible.

The graphic above shows various forms of production and how they size up on a continuum between private, tangible production and public, intangible production (given examples are ad hoc). The direction of change (or the fight against cultural bias) is noted with an arrow. In a future post, I will detail the consumption side of these continua.

Each axis of this schematic represents a potentially revolutionary tension in modern society, but like the tension between the ancien regime and the bourgeoisie did in the French Revolution.

II. Tangibility and Bias: Consumption

As promised, here is schematic demonstrating consumption on the tangibility, public-private continuum introduced in my previous post on production. Some notes on both schematics:

[A] "public" refers to what are traditionally public goods. In these examples, most examples are a mix of public and private. The relative position is a first-pass approximation at the ratio of that mix.

[B] "tangible" refers to the physical or easy-to-enclose objects (such as land or ingots of ore).

[C] "consumption" means to accept the value of (e.g. engage in a transaction to acquire) whatever it is you are consuming.

III. Circles in the Sands of the Mind

Circular logic all the way down. Is circular logic in the service of an iron-clad debating position [3] ultimately a sign of paranoia [4], or an intractable form of infinite regress [5]? Or is it a quasi-religion (e.g. a means to an end that cannot end) [6]? This reading list might help. Circular logic all the way down (this is not the answer to this question, I just like saying that)!

"The single biggest problem in communication logic is the illusion that it has taken place" -- Me and George Bernard Shaw

NOTES:

[1] Kaminska, I.   The tyranny of land. Dizzynomics, February 5 (2014).

[2] Smith, K.   Not all forms of wealth are equally pernicious. FT Alphaville, February 4 (2014).

[3] Lyngar, E.   Why I fled libertarianism -- and became a liberal. Salon, December 28 (2013).

[4] Sunstein, C.   How to Spot a Paranoid Libertarian. Bloomberg Opinion, January 30 (2014).

[5] Is infinite regress of logical causation possible? Is infinite regress of logical causation necessary? Philosophy Stack Exchange (2013).

[6] Dunn, D.J.   Three reasons why market liberalism is a religion. David J. Dunn blog, October 23 (2012).

Edison and Darwin Day(s) Items of Interest

February 11 is Edison Day and February 12 is Darwin Day. In honor of both (tech innovator and naturalist), I will cross-post some materials from Tumbld Thoughts, in addition to some interesting Darwin-Day related items courtesy of the Center for Scientific Inquiry (CFI).

February 11: Happy Edison Day!

Taken in part from a Chicago Ideas Week poster.

In honor of Edison Day (his posthumous 167th birthday), here are a few readings from IEEE Spectrum on the D-Wave quantum computer. Or pseudo-Quantum computer, depending on how you interpret the evidence.

The D-wave uses a 512-qubit architecture [1] to perform quantum annealing (a form of combinatorial optimization). This should allow for quantum computing-like capabilities upon scale-up (e.g. faster computation, exceeding Moore's Law) [2].

However, its current form is actually slower than conventional (e.g. classical digital) computers, and may not provide the theoretically-predicted advances in computational power [3, 4]. In fact, it may not be a quantum computer at all, a seemingly straightforward fact nobody can seem to verify [5].

NOTES:

[1] Hsu, J.   Scientists confirm D-wave's computer chips compute quantum mechanics. IEEE Spectrum, July 3 (2013).

According to this article, the company has kept the details of how the D-Wave functions shrouded in mystery. They have taken a "show but don't tell" (e.g. celebrate the black box) approach, which was a tactic employed by Edison in one infamous instance.

[2] Hsu, J.   D-wave's Year of Computing Dangerously. IEEE Spectrum, November 26 (2013).

[3] Hsu, J.   D-wave's quantum computing claim disputed again. IEEE Spectrum, February 10 (2014).

[4] Guizzo, E.   Loser: D-wave Does Not Quantum Compute. IEEE Spectrum, December 31 (2009).

[5] Mirani, L. and Lichfield, G.   Why nobody can tell whether the world's biggest quantum computer is a quantum computer. Quartz, April 15 (2014).

The Next Day: Darwin's Legagy, one year older, one year better!

In honor of this year's Darwin Day, I bring you the "Art of Darwin" [*] along with a diversity of evolutionary-oriented readings (on 10 distinct topics) from my reading queue. These should highlight the manner in which Evolutionary Science has grown since Darwin's lifetime.

1) Human Genetics: Loh, P-R., Lipson, M., Patterson, N., Moorjani, P., Pickrell, J.K., Reich, D., and Berger, B.   Inferring Admixture Histories of Human Populations Using Linkage Disequilibrium. Genetics, 193, 1233-1254 (2013).

2) Evolution of Sociality: Waters, J.S., Holbrook, C.T., Fewell, J.H., and Harrison, J.F.   Allometric Scaling of Metabolism, Growth, and Activity in Whole Colonies of the Seed-Harvester Ant Pogonomyrmex californicus. American Naturalist, 176(4), 501-510 (2010).

3) Evo-Devo (animals): Keller, R.A., Peeters, C., and Beldade, P. Evolution of thorax architecture in ant castes highlights trade-off between flight and ground behaviors. eLife, 3, e01539 (2014).

4) Experience-dependent Plasticity (plants): Gagliano, M., Renton, M., Depczynski, M., and Mancuso, S.   Experience teaches plants to learn faster and forget slower in environments where it matters. Oecologia, doi:10.1007/s00442-013-2873-7.

5) Evolution of Phenotypes: Tobias, J.A., Cornwallis, C.K., Derryberry, E.P., Claramunt, S., Brumfield, R.T., and Seddon, N.   Species coexistence and the dynamics of phenotypic evolution in adaptive radiation. Nature, doi:10.1038/nature 12874 (2013).

6) Evolution of Genomes: Wu, X. and Sharp, P.A.   Divergent Transcription: A Driving Force for New Gene Origination? Cell, 155, 990-996 (2013).

7) Genetic Regulation: Stergachis, A.B. et.al   Exonic Transcription factor binding directs codon choice and affects protein evolution. Science, 342, 1367 (2013).

8) Artificial Life (Robustness and Evolvability): Payne, J.L., Moore, J.H., and Wagner, A.   Robustness, Evolvability, and the Logic of Genetic Regulation. Artificial Life, 20(1), 111-126 (2014).

9) Evolutionary Biomechanics: Witton, M.P. and Habib, M.B.   On the Size and Flight Diversity of Giant Pterosaurs, the Use of Birds as Pterosaur Analogues and Comments on Pterosaur Flightlessness. PLoS One, 5(11), e13982 (2010).

10) A dissenter (sort of): Ruse, M.B.   Why I'm not celebrating Darwin Day. Chronicle of Higher Education Brainstorm blog, February 9 (2014).

 

[*] representations of Darwin and his legacy from around the web. Sources in order (from top to bottom): A, B, C, D, E, F.

Extra Goodies, courtesy of CFI

Here are some extra Darwin Day goodies, courtesy of the CFI. The first is a poster. While similar to their Carl Sagan poster series, they only have one style (although they also have a customized Facebook page cover shown below). The second is a link to the audiobook version of "Origin of the Species" (courtesy Librivox).

Thought (Memetic) Soup: Gonzo Cognition edition

This content is cross-posted to Tumbld Thoughts. This edition of thought (memetic) soup is about "Gonzo Cognition". This includes a proposition of something I am calling resistance behaviors (I), a critical review of the bubble concept in economics (II), and selection of most banal ideologies (III).

I. Resistance behaviors everywhere, but not enough to make sense

Denialism, Fundamentalism, Contrarianism! Why? It seems like a standard enough question, given the current parameters of the culture war. I have proposed something called cultural behavior, which is the individual contributors to a meta-cognitive response to cultural change and alien social phenomena. Here is an example of that focuses on simple resistance to an idea or a trend. For resistance, there are three scales of response (mental, behavioral mechanisms):

* Cognitive Taxis: underlying biases that orient the resistance to some set of ideas. This is similar to the "taxis" (chemotaxis, phototaxis) that can influence so-called lower-level perception. This usually favors in-group cohesion in some way.

* Resistance Behaviors: a decision-making heuristic that is contingent upon cognitive taxis. Resistance is based on cognitive taxis in the absence of agreeable information, then construct a post hoc rationale for doing so. This is particularly useful in cases of high uncertainty.

* Resistance Structures: social structures that are co-opted or coupled based on collective resistance behaviors (e.g. religion and politics, co-option of the economic system). Entire political strategies (e.g. government shutdowns) can be based on grafting of resistance behaviors onto existing social structures, thus creating resistance structures. This also allows for naive theories [1] of events and the world to be formed using abductive reasoning and the "facts" derived from resistive behaviors.

This theoretical framework might also be able to synthesize recent events in the world and scientific findings. Some examples:

* recursive culture wars. The recent Westboro Baptist Church protest of the show "Duck Dynasty" (or, more generally, the rightward trend within the GOP) is but one example. While the Duck Dynasty people are conservative and religious, they are still guilty of not being radically conservative enough, hence the protest. Extreme (and increasingly horrific) denialism such as that associated with Sandy Hook also falls into this category.

* using short-term weather trends as evidence for global climate change. A recent paper [2] suggests that people often use exceeding warm days as evidence for climate change (and vice versa), even though climate change is a longer-term phenomenon. This may be due to recency priming, or to people's inability to synthesize information from delayed outcomes [3].

* political orientation (liberal, conservative) might be shaped by cognitive taxis. Another recent paper [4] suggests that liberal and conservative outlooks are shaped by a tendency to embrace uniqueness and consensus, respectively. Resistance behaviors and structures follow from this, as basic orientations need to be made consistent from the bottom up.

II.  Bubbles are everywhere and nowhere, simultaneously

Bubbles, bubbles, everywhere. Bubbles are the most overused term in modern economic discourse. David Kesterbaum [5] offers interviews with Schiller and Fama (the most recent Economics Nobel winners) on their views. On the one hand, Fama thinks that you cannot predict the onset of bubbles, nor their duration. Thus, bubbles cannot be explained nor predicted. By contrast, Schiller thinks that all bubbles are behavioral phenomena, and as such we can know them when we see them.

Unfortunately, this does not give us much of a working definition. In [6], a working definition is offered as "gains that will come to a precipitous end, price of an asset has risen well above its fundamental or intrinsic value".

Furthermore, the Schiller view is considered to be merely a diagnosis, not a definition, as the mechanisms of rapid price increases are not self-explanatory. The idea of intrinsic or fundamental value comes up a lot in this discussion. The current consensus seems to be that bubbles can be defined as a deviation from fundamental value. Fundamental value itself can be discovered from a collection of variables, and bubbles may result from a misforecast of asset values [7].

But is this an issue of collective behavior in context, or something more context independent? For example, Shiller thinks that social contagion and similar factor drives bubbles outside of the financial domain (such as so-called gluts in labor markets) [6]. But perhaps "bubble" is the wrong concept to use. Then again, perhaps that is exactly what define the "fundamentals" of value.

III. Survival of the most Superficial 

Here is a set of critiques about the "dumbing down" of our culture. Well, not really dumbing down, but sort of a regression to the mean. I'll be the first to admit that this article [8] has the typical rhetorical flaws, but points out why the biggest and smartest ideas are often overlooked in favor of less sophisticated (and less thoughtful) ones.

But is this simply the dominance of anti-intellectualism, or something else? In [9], it is suggested that TED talks represent this tendency even more -- advances in thinking are so often encouraged as 15-minute problem-solving exercises that make us feel good in the end [10]. An emotionally-driven meta-intellectualism if you will. So enjoy, and remember, just because it's "simple" or "common" doesn't make it better.


NOTES:

[1] Larkin, J., McDermott, J., Simon, D.P., and Simon, H.A.   Expert and Novice Performance in Solving Physics Problems. Science, 208, 1335-1342 (1980).

[2] Zaval, L., Keenan, E.A., Johnson, E.J., and Weber, E.U.   How warm days increase belief in global warming. Nature Climate Change, DOI: 10.1038/NCLIMATE2093 (2013).

[3] Spence, A., Poortinga, W., Butler, C., and Pidgeon, N.F.  Perceptions of climate change and willingness to save energy related to flood experience. Nature Climate Change, DOI: 10.1038/NCLIMATE1059 (2011). Bottom image is Figure 1 from this reference.

[4] Stern, C., West, T.V., and Schmitt, P.G.   The Liberal Illusion of Uniqueness. Psychological Science, DOI: 10.1177/0956797613500796 (2013).

[5] Kesterbaum, D.   What’s a Bubble? Planet Money, NPR, November 15 (2013).

[6] Fox, J.   What's that you're calling a bubble? HBR Blog Network, January 8 (2014).

[7] Garber, P.   Famous first bubbles. Journal of Economic Perspectives, 4(2), 35-54 (1990).

[8] Simic, C.   Age of Ignorance. NY Review of Books blog, March 20 (2012).

[9] Bratton, B.   We need to talk about TED. Guardian, December 30 (2013).

[10] there seem to be strong parallels with the emotional satisfaction factor of religious belief (discussed in [2]) and the "cult of happiness".