Fraud, bias, negligence and hype in the lab – a gallery of thugs
Science Fiction: How Fraud, Prejudice, Neglect, and Hype Undermine the Search for Truth Stuart ritchie Metropolitan (2020)
If “you’ve heard of the idea that if you take a bigger plate off the buffet you’ll eat more than you otherwise would,” writes psychologist Stuart Ritchie, then “you’ve heard of Professor Brian indirectly. Wansink ”. Wansink, a nutritional psychologist, spent two years as director of the Center for Nutrition Policy and Promotion at the United States Department of Agriculture and has published extensively which formed the evidence base of the “Smarter Lunchrooms” movement in American schools. The validity of his research has since been called into question, in what is now one of the best-known cases in nutrition research. At least 18 of his papers were retracted, 6 in a single day.
Fraud, prejudice, neglect and hype are the themes of Science fiction. Some of the cases presented by Ritchie, like that of Wansink, are intriguing and disturbing combinations of the four. His examples of challenged findings range from psychic precognition, psychological priming and the benefits of a ‘power pose’ to tracheal transplants, gut microbiome and autistic-like characteristics in mice, and forms life based on arsenic. All the stories of replication failure and scientific misconduct you’ve ever heard are here, along with others you haven’t heard yet. Together, they increase the tension between engaged scientific criticism and maintaining confidence in science.
Ritchie opens with the reassuring line that he just “praised science, not burying it.” In many ways, the book is a defense of ideals that he thinks we’ve strayed from. At the heart of these are the “standards” of science codified in 1942 by sociologist Robert Merton: universalism, disinterestedness, community and organized skepticism. Yet Ritchie fails to recognize that even in what we might consider paradigmatic breakthroughs, most scientists have not followed such standards. This uncritical presentation might confuse those interested in modern philosophy of science.
Ritchie prefaces his rogue gallery by introducing a few nuts and bolts, including the structure of the newspaper article and terms like office rejection and peer review. With its overview of the replication crisis, this introduction would be useful for undergraduates or readers in general.
Cognoscenti can dive directly into the middle section. This comprehensive collection of incidents, mischief and cautionary tales is the great strength of Ritchie’s offering. There are examples from nutrition and social psychology (of course), but also from inorganic chemistry, evolutionary biology, genetics, cancer biology, economics, public health. and education, demonstrating the disciplinary scale of the reproducibility problem. This will help build bridges for those in the metascience realm hoping to tackle the causes.
Ritchie’s four themes sculpt intricate and interconnected issues in natural joints and allow his case studies to shine. Sometimes I was slightly frustrated that this came at the cost of separating out the issues that would ideally be presented together: P values and statistical power are explained in different chapters, and the discussion of publication bias is again elsewhere. In the end, however, the compromise is worth it.
He concludes by addressing the problems of incentive structures in science culture and open science initiatives to address them. It clearly and effectively articulates issues such as the flawed metrics we use to assess the quality of research – from h-indices of impact factors – and a culture of publication or destruction. And it sets out possible solutions such as pre-registration of studies and increased transparency. It ends with a plea for “Make science boring again”, which is funny – despite the historical inaccuracy.
Ritchie attempts to recognize science as a social and human enterprise, referring to the philosopher Helen Longino and even asserting that “science is a social construct”. So I was puzzled that after presenting the arguments of the philosopher Cordelia Fine for including feminist perspectives in science, he did not relate them to Longino’s concept of collective objectivity; instead, it rejects Fine’s points.
The heart of Longino’s idea is that objectivity in science is a collective endeavor, rather than just an individual one. To maintain it, the scientific community must be diversified, to help cancel out individual biases. Feminist perspectives are an example of the type of diversity Longino intends to include, which is important in balancing the biases hidden in the status quo. Ritchie appears to find Fine’s arguments inconsistent with other bias mitigation strategies, such as pre-enrollment and blinded trials. I don’t agree: they come from the same toolbox.
No golden age
I support Ritchie’s argument that researchers have adapted to perverse incentives, publication practices, and inappropriate metrics by engaging in P– hacking, overfitting, and other problematic activities, such as dismissing reviews, neglecting error detection, and fraud. But sometimes it rests too heavily on the idea that there were once golden days when science was a pure truth-seeking enterprise. For example, he laments that we have allowed science “to become so tarnished and its progress to be so severely blocked”.
He goes on to say that the problems started “somewhere along the way, between Boyle and modern academia.” Maybe these are just rhetorical flourishes, and it might be unfair to infer any particular historical commitment, but I wonder if Ritchie is saying that there was a time when scientists lived up to Merton’s standards. If so, I suspect most science historians would respectfully disagree.
Towards the end of the book, rightly imploring us to take responsibility for the mess, Ritchie recalls a time “when the scientific community gave its collective approval to these underpowered studies.” Yet low-power studies have been the norm since tests of statistical significance entered the social and life sciences after World War II. Fraud, prejudice, and neglect (less hype, perhaps) have also been with us forever. This “just for” undercurrent doesn’t ruin Science fiction, but that prevents him from developing any more in-depth information on how we got to where we are now, and whether the suite of fixes presented in the last chapter will really be enough to get us through.