Prediction and explanation in social systems

See allHide authors and affiliations

Science  03 Feb 2017:
Vol. 355, Issue 6324, pp. 486-488
DOI: 10.1126/science.aal3856


  • Fig. 1 A single question may correspond to many research designs, each yielding different answers.

    (Top) A depiction of the many choices involved in translating the problem of understanding diffusion cascades into a concrete prediction task, including the choice of data source, task, evaluation metric, and data preprocessing. The preprocessing choices shown at the terminal nodes refer to the threshold used to filter observations for regression or define successful outcomes for classification. Cascade sizes were log-transformed for all of the regression tasks. (Bottom) The results of each prediction task, for each metric, as a function of the threshold used in each task. The lower limit of each vertical axis gives the worst possible performance on each metric, and the top gives the best. Dashed lines represent the performance of a naive predictor (always forecasting the global mean for regression or the positive class for classification), and solid lines show the performance of the fitted model. R2, coefficient of determination; AUC, area under the ROC curve; RMSE, root mean squared error; MAE, mean absolute error; F1 score, the harmonic mean of precision and recall.

  • Fig. 2 Schematic illustration of two stylized explanations for an empirically observed distribution of success.

    In the observed world (top), the distribution of success is right-skewed and heavy-tailed, implying that most items experience relatively little success, whereas a tiny minority experience extraordinary success. In “skill world” (bottom left), the observed distribution is revealed to comprise many item-specific distributions sharply peaked around the expected value of some (possibly unobservable) measure of skill; thus, conditioning correctly on skill accounts for almost all observed variance. In contrast, in “luck world” (bottom right), almost all the observed variance is attributable to extrinsic random factors; thus, conditioning on even a hypothetically perfect measure of skill would explain very little variance. [Adapted from (11)]