Monthly Archives: April 2018

On Questionable Research Practices

Ecologists and evolutionary biologists are tarred and feathered along with many scientists who are guilty of questionable research practices. So says this article in “The Conservation” on the web:
https://theconversation.com/our-survey-found-questionable-research-practices-by-ecologists-and-biologists-heres-what-that-means-94421?utm_source=twitter&utm_medium=twitterbutton

Read this article if you have time but here is the essence of what they state:

“Cherry picking or hiding results, excluding data to meet statistical thresholds and presenting unexpected findings as though they were predicted all along – these are just some of the “questionable research practices” implicated in the replication crisis psychology and medicine have faced over the last half a decade or so.

“We recently surveyed more than 800 ecologists and evolutionary biologists and found high rates of many of these practices. We believe this to be first documentation of these behaviours in these fields of science.

“Our pre-print results have certain shock value, and their release attracted a lot of attention on social media.

  • 64% of surveyed researchers reported they had at least once failed to report results because they were not statistically significant (cherry picking)
  • 42% had collected more data after inspecting whether results were statistically significant (a form of “p hacking”)
  • 51% reported an unexpected finding as though it had been hypothesised from the start (known as “HARKing”, or Hypothesising After Results are Known).”

It is worth looking at these claims a bit more analytically. First, the fact that more than 800 ecologists and evolutionary biologists were surveyed tells you nothing about the precision of these results unless you can be convinced this is a random sample. Most surveys are non-random and yet are reported as though they are a random, reliable sample.

Failing to report results is common in science for a variety of reasons that have nothing to do with questionable research practices. Many graduate theses contain results that are never published. Does this mean their data are being hidden? Many results are not reported because they did not find an expected result. This sounds awful until you realize that journals often turn down papers because they are not exciting enough, even though the results are completely reliable. Other results are not reported because the investigator realized once the study is complete that it was not carried on long enough, and the money has run out to do more research. One would have to have considerable detail about each study to know whether or not these 64% of researchers were “cherry picking”.

Alas the next problem is more serious. The 42% who are accused of “p-hacking” were possibly just using sequential sampling or using a pilot study to get the statistical parameters to conduct a power analysis. Any study which uses replication in time, a highly desirable attribute of an ecological study, would be vilified by this rule. This complaint echos the statistical advice not to use p-values at all (Ioannidis 2005, Bruns and Ioannidis 2016) and refers back to complaints about inappropriate uses of statistical inference (Armhein et al. 2017, Forstmeier et al. 2017). The appropriate solution to this problem is to have a defined experimental design with specified hypotheses and predictions rather than an open ended observational study.

The third problem about unexpected findings hits at an important aspect of science, the uncovering of interesting and important new results. It is an important point and was warned about long ago by Medewar (1963) and emphasized recently by Forstmeier et al. (2017). The general solution should be that novel results in science must be considered tentative until they can be replicated, so that science becomes a self-correcting process. But the temptation to emphasize a new result is hard to restrain in the era of difficult job searches and media attention to novelty. Perhaps the message is that you should read any “unexpected findings” in Science and Nature with a degree of skepticism.

The cited article published in “The Conversation” goes on to discuss some possible interpretations of what these survey results mean. And the authors lean over backwards to indicate that these survey results do not mean that we should not trust the conclusions of science, which unfortunately is exactly what some aspects of the public media have emphasized. Distrust of science can be a justification for rejecting climate change data and rejecting the value of immunizations against diseases. In an era of declining trust in science, these kinds of trivial surveys have shock value but are of little use to scientists trying to sort out the details about how ecological and evolutionary systems operate.

A significant source of these concerns flows from the literature that focuses on medical fads and ‘breakthroughs’ that are announced every day by the media searching for ‘news’ (e.g. “eat butter”, “do not eat butter”). The result is almost a comical model of how good scientists really operate. An essential assumption of science is that scientific results are not written in stone but are always subject to additional testing and modification or rejection. But one result is that we get a parody of science that says “you can’t trust anything you read” (e.g. Ashcroft 2017). Perhaps we just need to repeat to ourselves to be critical, that good science is evidence-based, and then remember George Bernard Shaw’s comment:

Success does not consist in never making mistakes but in never making the same one a second time.

Amrhein, V., Korner-Nievergelt, F., and Roth, T. 2017. The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research. PeerJ  5: e3544. doi: 10.7717/peerj.3544.

Ashcroft, A. 2017. The politics of research-Or why you can’t trust anything you read, including this article! Psychotherapy and Politics International 15(3): e1425. doi: 10.1002/ppi.1425.

Bruns, S.B., and Ioannidis, J.P.A. 2016. p-Curve and p-Hacking in observational research. PLoS ONE 11(2): e0149144. doi: 10.1371/journal.pone.0149144.

Forstmeier, W., Wagenmakers, E.-J., and Parker, T.H. 2017. Detecting and avoiding likely false-positive findings – a practical guide. Biological Reviews 92(4): 1941-1968. doi: 10.1111/brv.12315.

Ioannidis, J.P.A. 2005. Why most published research findings are false. PLOS Medicine 2(8): e124. doi: 10.1371/journal.pmed.0020124.

Medawar, P.B. 1963. Is the scientific paper a fraud? Pp. 228-233 in The Threat and the Glory. Edited by P.B. Medawar. Harper Collins, New York. pp. 228-233. ISBN 978-0-06-039112-6

On the Loss of Large Mammals

The loss of large mammals and birds in the Pleistocene was highlighted many years ago (Martin and Wright 1967, Grayson 1977, Guthrie 1984 and many other papers). Hypotheses about why these extinctions occurred were flying left and right for many years with no clear consensus (e.g. Choquenot and Bowman 1998). The museums of the world are filled with mastodons, moas, sabre-tooth tigers and many other skeletons of large mammals and birds long extinct. The topic has come up again in a discussion of these extinctions and a prognosis of future losses (Smith et al. 2018). I do not want to question the analysis in Smith et al. (2018) but I want to concentrate on this one quotation that has captured the essence of this paper in the media:

“Because megafauna have a disproportionate influence on ecosystem structure and function, past and present body size downgrading is reshaping Earth’s biosphere.”
(pg. 310).

What is the evidence for this very strong statement? The first thought that comes to mind is from my botanical colleagues who keep reminding me that plants make of 99% of the biomass of the Earth’s ecosystems. So, if this statement is correct, it must mean that large mammals have a very strong effect on plant ecosystem structure and function. And it must also imply that large mammals are virtually immune to predators, so no trophic cascade can occur to prevent plant overgrazing.

I appreciate that it is very difficult to test such a statement since evolution has been going on for a long time before humans arrived, and so there must have been a lot of other factors causing ecosystem changes in those early years. Humans have a disproportionate love for biodiversity that is larger than us. So, we revel in elephants, tigers, bears, and whales, while at the same time we pay little attention to the insects, small mammals, most fish, and plankton. Because of this size bias, we are greatly concerned with the conservation of large animals, as we should be, but much less concerned about what is happening to the small chaps.

What is the evidence that large mammals and birds have a disproportionate influence on ecosystem structure and function? In my experience, I would say there is very little evidence for strong ecosystem effects from the collapse of the megafauna. DeMaster et al. (2006) evaluated a proposed explanation for ecosystem collapse caused by whaling in the North Pacific Ocean and concluded that the evidence was weak for a sequential megafauna collapse caused by commercial whaling. Trites et al. (2007) and Wade et al. (2007) supported this conclusion. Citing paleo-ecological data for Australia, Johnson (2010) and Rule et al. (2012) argued in another evaluation of ecosystem changes that the human-driven extinction of the megafauna in Australia resulted in large changes in plant communities, potentially confounded by climate change and increases in fire frequency about 40K years ago. If we accept these controversies, we are left with trying to decide if the current losses of large mammals are of similar strength to those assigned to the Pleistocene megafauna, as suggested by Smith et al. (2018).

If we define ecosystem function as primary productivity and ecosystem structure as species diversity, I cannot think of a single case in recent studies where this idea has been clearly tested and supported. Perhaps this simply reflects my biased career working in arctic and subarctic ecosystems in which the vast majority of the energy flow in the system rotates through the smaller species rather than the larger ones. Take the Great Plains of North America with and without the bison herds. What aspect of ecosystem function has changed because of their loss? It is impossible to say because of human intervention in the fire cycle and agricultural pre-emption of much of the landscape. It is certainly correct that overgrazing impacts can be severe in human-managed landscapes with overstocking of cattle and sheep, and that is a tragedy brought on by economics, predator elimination programs, and human land use decisions. All the changes we can describe with paleo-ecological methods have potential explanations that are highly confounded.

I think the challenge is this: to demonstrate that the loss of large mammals at the present time creates a large change in ecosystem structure and function with data on energy flow and species diversity. The only place I can see it possible to do this experimentally today would be in arctic Canada where, at least in some areas, caribou come and go in large numbers and with relatively little human impact. I doubt that you could detect any large effect in this hypothetical experiment. It is the little chaps that matter to ecosystem function, not the big chaps that we all love so much. And I would worry if you could do this experiment, the argument would be that it is a special case of extreme environments not relevant to Africa or Australia.

No one should want the large mammals and birds to disappear, but the question of how this might play out in the coming 200 years in relation to ecosystem function requires more analysis. And unlike the current political inactivity over the looming crisis in climate change, we conservation biologists should certainly try to prevent the loss of megafauna.

Choquenot, D., and Bowman, D.M.J.S. 1998. Marsupial megafauna, Aborigines and the overkill hypothesis: application of predator-prey models to the question of Pleistocene extinction in Australia. Global Ecology and Biogeography Letters 7: 167-180.

DeMaster, D.P., Trites, A.W., Clapham, P., Mizroch, S., Wade, P., Small, R.J., and Hoef, J.V. 2006. The sequential megafaunal collapse hypothesis: testing with existing data. Progress in Oceanography 68(2-4): 329-342. doi:10.1016/j.pocean.2006.02.007

Grayson, D.K. 1977. Pleistocene avifaunas and the Overkill Hypothesis. Science 195: 691-693.

Guthrie, R.D. 1984. Mosaics, allelochemics and nutrients: An ecological theory of late Pleistocene megafaunal extinctions. In: Quaternary Extinctions: A Prehistoric Revolution ed by P.S. Martin and R.G. Klein. University of Arizona Press Tucson.

Johnson, C.N. 2010. Ecological consequences of Late Quaternary extinctions of megafauna. Proceeding of the Royal Society of London, Series B 276(1667): 2509-2519. doi: 10.1098/rspb.2008.1921.

Martin, P.S., and Wright, H.E. (eds). 1967. Pleistocene Extinctions; The Search for a Cause. Yale University Press, New Haven, Connecticut. 453 pp.

Rule, S., Brook, B.W., Haberle, S.G., Turney, C.S.M., Kershaw, A.P., and Johnson, C.N. 2012. The aftermath of megafaunal extinction: ecosystem transformation in Pleistocene Australia. Science 335(6075): 1483-1486. doi: 10.1126/science.1214261.

Smith, F.A., Elliott Smith, R.E., Lyons, S.K., and Payne, J.L. 2018. Body size downgrading of mammals over the late Quaternary. Science 360(6386): 310-313. doi: 10.1126/science.aao5987.

Trites, A.W., Deecke, V.B., Gregr, E.J., Ford, J.K.B., and Olesiuk, P.F. 2007. Killer whales, whaling, and sequential megafaunal collapse in the North Pacific: a comparative analysis of the dynamics of marine mammals in Alaska and British Columbia following commercial whaling. Marine Mammal Science 23(4): 751-765. doi: 10.1111/j.1748-7692.2006.00076.x.

Wade, P.R., et al. 2007. Killer whales and marine mammal trends in the North Pacific – a re-examination of evidence for sequential megafaunal collapse and the prey-switching hypothesis. Marine Mammal Science 23(4): 766-802. doi: 10.1111/j.1748-7692.2006.00093.x.