The first time I read this paper, reporting on a massive collaboration of hundreds of scientists focused on the common task of predicting life outcomes using a single, common dataset, I thought it was an elaborate April fools. There was something about the way the abstract was phrased that immediately made me think “not only is this ridiculous, but it’s ridiculous in a way that’s not even fun”. It didn’t help that the link was posted the to me on April 1st, or that the paper too conveniently was published online March 30. Even as I recognized some of the names among the long list of authors, all I could think of was “OK, they are part of this massive joke. I get it now – you got me. I guess it was kind of fun”. But then when the ones who posted the link started talking about the paper as the real deal, I realized that I was in the wrong – this was no joke. Which is good, because I believe social science (and humanities) could do with more of these massive collaborations focused on clearly defined problems to establish, once and for a long time, what should count as the current state of the art on a phenomenon. So, instead of a disdainful giggle, I felt excitement; I would definitely read this (sometime after finishing that grant application, and that paper, and that other thing).

Now that I have finally had time to read it, I must confess it was a bit of a disappointment.

Sure, the set up, design and scale of collaboration is both impressive and inspiring. Using the common task method, whereby a prediction task is designed and then a large and diverse group of researchers is recruited to provide solutions to this task, i.e., predicting the outcomes using the same data, is something that could definitely be used more often in the social sciences. The prediction task was in this case to predict six outcomes in the 6th wave of the Fragile Families study, using information from prior waves (12,942 variables about 4,242 families) and the six outcomes for half of the 6th wave families – including some information from this wave – as training data. The other half of the 6th wave was the holdout data. The six outcomes was chosen to include variables of different type and scale (each outcome being represented by a variable).

The disappointment came when I couldn’t find what kind of social theory that informed the attempted predictions. In other words, it was not clear to me what we would gain in terms of knowledge regardless of whether predictions would be successful or not. To be harsh, and to use the conceptual apparatus of Gaston Bachelard, it seems to me that this work was more in the spirit of clear positivist empiricism than discursive rationalism.

Moreover, it is not at all clear to me that the authors’ concern for life trajectory research – that it must reconcile their understanding of life trajectories with the fact that any prediction is essentially inaccurate – is valid. Specifically, the entire article lacks a clear definition of analytic level and scope of claims. The fact that individual level outcomes on six specific variables, measured at one instance, are not predictable based on other variables does not mean that there aren’t any regularities in life trajectories on a more aggregate level. As Yves Gingras keeps telling us, the law of gases on a macroscopic scale is not invalidated by the fact that atoms move in a completely random and unpredictable way.

But most of all, I’m probably just jealous.

machine learning