Are the Worst Performers the Best Predictors?

What is the link between performance and a priori predictions? At the individual level, this question has important implications for areas as diverse as sports [1] and job [2] performance potential (including the so-called "Moneyball" approach). This may also be useful for understanding the effects of exercise and technological augmentation on human populations. 

Working Hypothesis:  The performance of sports teams that are perennial winners or losers are much easier to predict than other teams (e.g. those that exhibit parity).

In an attempt to form theoretical insights based on this question, I conducted a rudimentary analysis on how informative PredictWise and BetFair's prognostications for the 2013 MLB season were with respect to the final regular season standings [3]. A similar analysis was done on NFL data to see if these results hold across types of data. 

PredictWise [4] is an aggregator of likelihoods for purposes of betting on outcomes. Their predictions include contests in the realm of politics, sports, and entertainment. The likelihoods are updated as the event unfolds, but the comparison of a priori predictions provide interesting comparisons with the final outcome. These predictions are not entirely naive, but do rely upon a fair number of assumptions.

The first graph shows the difference in rank-order position between the likelihood of winning the world series (generated a priori) and the regular-season won-loss record. The "difference from prediction" was then calculated for the top, middle, and bottom tercile on teams based on their regular-season record.

Interestingly, many of the winningest teams were not predicted to finish strongly. By contrast, the bottom tercile was equally represented by teams that had the least chance of winning it all and teams that were supposed to finish more strongly. With a few exceptions, the middle tercile was represented by underachieving teams, and the most consistent performances (smallest deviations from prediction) were among the lowest achieving teams.

The next two graphs show the magnitude of deviation from prediction (observed vs. predicted). This results in an index (value: 0-1) based on a team's deviation from prediction relative to the maximum and minimum of all teams in the league. The third graph (two panels) breaks this down into teams that finished better than and worse than expected.

Finally, the fourth graph demonstrates how the deviation from prediction is related to the total number of wins a team had during the season. This plot lends no additional support to but is consistent with the notion of "worst performers, best predictors".

To compare these tendencies across sports and odds-making enterprises, I used the Sporting News a priori predictions for the NFL 2013 season [5]. In this example, I compared a team's n-to-1 odds of winning the Super Bowl with the final season standings (e.g. similar methodology to the MLB analysis, but with a different source of predictions). 

From this exploratory graph [6], a similar trend of "worst performers, best predictors" emerges, albeit with more outliers on the lower end. Recapitulating the difference from prediction analysis done for the MLB data, the NFL data shows more deviations from prediction for every stratum of the dataset. However, again, there is a slight tendency for the bad teams to be predicted correctly and the best performing teams to be poorly-predicted. In the case of the NFL data, there is a countervailing "dynasty" effect as well: teams that have been winning consistently were also predicted to do well. As they met this expectation, they were easier to predict correctly.

So are there better means to predict outcomes than making odds? PredictWise uses a combination of a priori odds-making and individual wagering. When people are willing to wager on an outcome, a diversity of mental models are used to inform the prediction. We can also use real-time surveys that make predictions in a manner similar to a logistic regression model [7]. However, whether such approaches can ameliorate the "surprise" factor of unexpected levels of performance (good or bad) is questionable.

NOTES:

[1] Morey, D.   The elephant on the court. Economist, April 20 (2012).

[2] Armstrong, J.S.   Predicting Job Performance: the Moneyball factor. Foresight, Spring (2012).

[3] Sohmer, S.   PredictWise: aggregating the wisdom of crowds. Hypervocal, October 11 (2011).

[4] The Linemakers   Odds to win 2014 Super Bowl. Sporting News, February 4 (2013).

[5] The dataset (predictions vs. MLB and NFL standings from 2013 seasons) can be found at Figshare (doi:10.6084/944542).

[6] The x-axis is defined as the final won-loss record centered upon a .500 (8-8) record. The formula is (WINS)-(LOSSES)+(TIES*0.5). The y-axis is an index based on the odds ratio, where the lowest odds are set to 1.0. The formula is ((ODDS)/(LOWEST ODDS))¹ Rank-orderings of these metrics (and distances between these rank-orderings) were also used to generate the graphs.

[7] Ulfelder, J.   Using Wiki surveys to forecast rare events. Dart-throwing Chimp blog, August 11 (2013).