non hypothesis driven research

Thinking outside the box: fostering innovation and non-hypothesis-driven research at NIH

Affiliation.

1 Office of Biorepositories and Biospecimen Research, Center for Strategic Scientific Initiatives, National Cancer Institute, Bethesda, MD 20892, USA. [email protected]
PMID: 21325614
DOI: 10.1126/scitranslmed.3001742

The National Institutes of Health (NIH) has long been known as an institution that supports biomedical advances through hypothesis-driven research. Another aspect of NIH, however, has received comparatively little attention and may be critical to advancing translational science beyond its current limitations. Specifically, this aspect of NIH focuses on supporting innovation through the development of high-risk technologies that have the potential to empower research.

Biomedical Research / economics*
National Institutes of Health (U.S.)
Research Support as Topic / economics*
Translational Research, Biomedical / economics*
United States

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Publications
Account settings
Advanced Search
Journal List

Dogmatic modes of science

Roy s hessels, ignace t c hooge.

Author information
Article notes
Copyright and License information

Roy S. Hessels, Utrecht University, Heidelberglaan 1, 3584 CS, Utrecht, The Netherlands. Email: [email protected]

Issue date 2021 Nov.

This article is distributed under the terms of the Creative Commons Attribution 4.0 License ( https://creativecommons.org/licenses/by/4.0/ ) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access page ( https://us.sagepub.com/en-us/nam/open-access-at-sage ).

The scientific method has been characterised as consisting of two modes. On the one hand, there is the exploratory mode of science, where ideas are generated. On the other hand, one finds the confirmatory mode of science, where ideas are put to the test ( Tukey , 1980 ; Jaeger & Halliday , 1998 ). Various alternative labellings of this apparent dichotomy exist: data-driven versus hypothesis-driven, hypothesis-generating versus hypothesis-testing, or night science versus day science (e.g., Kell & Oliver , 2004 ; Yanai & Lercher , 2020 ). Regardless of the labelling, the dichotomy of an “idea-generating” versus an “idea-testing” mode seems pervasive in scientific thinking.

The two modes of science appear to be differentially appreciated. For example, exploratory research may carry the stink of “merely a fishing expedition” ( Kell & Oliver , 2004 ), or may be considered “weak” and yield unfavourable reviews (see the discussion of Platt [ 1964 ] in Jaeger & Halliday, 1998 ). Confirmatory research, on the other hand, seems to be considered as the holy grail in many areas of psychology (and vision science). Whether the appreciation for hypothesis-testing in psychology has been a reaction to the critique that theories in “soft areas of psychology” are “scientifically unimpressive and technologically worthless” ( Meehl, 1978 , p. 806) is an interesting question for debate. Nevertheless, the quintessential question in modern psychology is: “What is your hypothesis?” The correct answer one is expected to produce is a sentence at the level of a statistical analysis. Any other answer is wrong and yields the following response: “Ah, I see. You do exploratory research.” In Orwellian Newspeak “hypothesis” means “that which is to be decided on statistically” (cf. Yanai & Lercher , 2020 ), whereas “exploratory” means “descriptive” or even “unscientific.”

That the confirmatory mode of science is held in such high esteem is intriguing. Confirmation suggests that hypotheses or theories can be verified, a position diametrically opposed to that of, for example, Karl Popper, who claimed that theories can never be verified or confirmed, only refuted (e.g., Popper, 2002a ). Note that this cuts right into the heart of discussions on whether science can be inductive and rational or not ( Lakatos , 1978 ). It is not trivial semantics! But one does not find that a “refutatory” mode of science holds sway. Rather, refutation (or disconfirmation) is commonly avoided by the construction of ad-hoc auxiliary hypotheses when the data do not match with the theory (cf. the practice of Lakatosian defence, Meehl , 1990 ). Although sometimes frowned upon, ad-hoc hypotheses are not without merit. The observation of the planet Neptune by Galle in 1846 followed the ad-hoc hypothesis by Le Verrier and Adams (as discussed in Gershman , 2019 ): a great success for science. That ad-hoc hypotheses may also fail is evident from the hypothesised planet Vulcan by the same Le Verrier. That planet was never observed, although the discrepancy it addressed later proved to be relevant for Einstein’s theory of general relativity.

The discussion of confirmation versus refutation aside, the two-mode view of science is not merely a theoretical fancy that researchers debate about. It pervades increasingly more of the practicalities that researchers are faced with. The pre-registration movement, for example, seems to be built on this strict dichotomy. The Center for Open Science 1 writes that “Preregistration separates hypothesis-generating (exploratory) from hypothesis-testing (confirmatory) research” and this “planning improves the quality and transparency of your research.” Note the explicit normative statement here. But is a strict dichotomy of exploratory research (or data-driven or hypothesis-free) versus confirmatory research (or hypothesis-testing) sensible at all?

Is there such a thing as hypothesis-free exploration? Consider the case of a person sitting in their yard, peering at a pond through binoculars. Can we claim that this person is observing the world without hypotheses? According to Popper (2002b ), we cannot. He states: “observation is always observation in the light of theories” (p. 37). This need not be a formalised hypothesis according to the hypothetico-deductive method. Science is a human affair after all, it piggybacks on perception and cognition, which thrive through instinct, intuition, hunches, anticipation, quasi-informed guesses, and expertise (cf. Brunswik , 1955 ; Riegler , 2001 ; Chater et al. , 2018 ). Such proto-hypotheses ( Felin et al. , 2021 ) do not always lend themselves neatly to verbalisation or formalisation, or are fuzzy at best ( Rolfe , 1997 ). Thus, the decisions of where to sit, what direction to peer in, what binoculars to use, how long to wait are hardly hypothesis-free.

At the other extreme, one can ask whether hypothesis-testing is possible in the absence of exploration. Clearly, exploration in Tukey’s sense is crucial for forming a hypothesis in the first place: “Ideas come from previous exploration more often than from lightning strokes” (Tukey, 1980 , p. 23). However, devising a critical experiment to put a hypothesis to the test inevitably involves exploration. Exploration of where in the stimulus-space to measure, which parameters to use for signal processing, and so forth. This should strike a chord with the experimental scientist. Theoretically, one might be able to conceive of an experiment that can be considered as purely “hypothesis-testing.” Yet, at best it would be the hypothetical limit on a continuum between the exploratory and confirmatory modes of science.

Thus, a strict two-mode view of science is too simplistic. Nevertheless, the practical implications of such a view may be substantial, also to those who abstain from initiatives such as pre-registration. In our experience, the strict two-mode view of science permeates the thinking of e.g., institutional review boards, ethics committees, and local data archiving initiatives. The procedures derived from this strict two-mode thinking tend to take on a Kafkaesque atmosphere: The bureau of confirmatory science will see you now. It will be most pleased to guide you on your way to doing proper science.

We are happy to concede that scientific studies may be characterised as being of a more or less exploratory nature and that some studies may be characterised as clear attempts to refute or decide between scientific hypotheses. We also understand that some procedures taken up by institutional review boards, ethics committees, journals (pre-registration), and so forth, are meant to counter phenomena such as “HARKing” (the evil twin of the ad-hoc auxiliary hypothesis), “p-hacking”, blatant fraud, or to increase the replicability of science (e.g., Nosek et al. , 2012 ; Open Science Collaboration , 2015 ). Good intentions do not solely validate the means, however. What we vehemently oppose is the adoption and dogmatic use of a simplistic model of science and the scientific method that all research should adhere to. Dogma has no place in science, nor has it proved particularly effective throughout the history of science ( Feyerabend , 2010 ).

In our view, the dogmatic two-mode view of science obscures a deeper discussion—that of the goal or purpose of science. According to the influential paper by the Open Science Collaboration (2015 ) it is “that ultimate goal: truth” (p. 7). This contrasts starkly with a quote from Linschoten (1978 ):

The statement that science seeks truth is meaningless. The word “truth” either means too much or too little. It has no scientifically relatable meaning, unless truth is equivalent to relevant knowledge. Knowledge is relevant when it allows us to explain, predict, and control phenomena (p. 390). 2

If one considers hypotheses to be true or false, scientific findings to be true or false, and theories to be true or false, then a purely confirmatory way of thinking makes sense. All efforts to replicate—that is, to decide on which findings are really true—using the right statistical ( Gigerenzer , 2018 ) or methodological rituals ( Popper , 2002b ) will inevitably bring one closer to that truth. If one is less concerned with truth, and more with predicting tomorrow, then the exploratory versus confirmatory dichotomy is not all that relevant. One would rather have meaningful discussions about generalisability ( Yarkoni , 2020 ) or representativeness ( Brunswik , 1955 ; Holleman et al. , 2020 ). Anything that will yield a better prediction of tomorrow is useful, whether arrived at through Popper’s hypothetico-deductive methods, a hunch, a fishing trip, or counterinductively. According to Feyerabend ( 2010 , p. 1), “Science is an essentially anarchic enterprise: theoretical anarchism is more humanitarian and more likely to encourage progress than its law-and-order alternatives.” Science needs no dogma.

Acknowledgements

The authors thank Andrea van Doorn and Jan Koenderink for inspiring discussions and helpful comments.

https://www.cos.io/initiatives/prereg , accessed 14 May 2021.

In the original Dutch: “De uitspraak dat wetenschap waarheid zoekt, is zinloos. Het woord ‘waarheid’ betekent te veel of te weinig. Het geeft geen wetenschappelijk verbindbare betekenis, tenzij waarheid gelijkluidend is met relevante kennis. Kennis is relevant wanneer ze ons in staat stelt verschijnselen te verklaren, te voorspellen, en te beheersen.”

Declaration of Conflicting Interests: The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding: The authors received no financial support for the research, authorship and/or publication of this article.

ORCID iD: Roy S. Hessels https://orcid.org/0000-0002-4907-1067

Supplemental material

Supplemental material for this article is available online.

Brunswik E. (1955). Representative design and probabilistic theory in a functional psychology. Psychological Review, 62(3), 193–217. 10.1037/h0047470 [ DOI ] [ PubMed ] [ Google Scholar ]
Chater N, Felin T, Funder D. C., Gigerenzer G, Koenderink J. J., Krueger J. I., Noble D, Nordli S. A., Oaksford M, Schwartz B, Stanovich K. E., Todd P. M. (2018). Mind, rationality, and cognition: An interdisciplinary debate. Psychonomic Bulletin & Review, 25, 793–826. 10.3758/s13423-017-1333-5 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
Felin T, Koenderink J, Krueger J. I., Noble D, Ellis G. F. (2021). The data-hypothesis relationship. Genome Biology, 22(57), 1–6. 10.1186/s13059-021-02276-4 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
Feyerabend P. (2010). Against method. (4th ed.). Verso. [ Google Scholar ]
Gershman S. J. (2019). How to never be wrong. Psychonomic Bulletin & Review, 26(1), 13–28. 10.3758/s13423-018-1488-8 [ DOI ] [ PubMed ] [ Google Scholar ]
Gigerenzer G. (2018). Statistical rituals: The replication delusion and how we got there. Advances in Methods and Practices in Psychological Science, 17, 251524591877132. 10.1177/2515245918771329 [ DOI ] [ Google Scholar ]
Holleman G. A., Hooge I. T. C, Kemner C, Hessels R. S. (2020). The ‘real-world approach’ and its problems: A critique of the term ecological validity. Frontiers in Psychology, 11, 721. 10.3389/fpsyg.2020.00721 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
Jaeger R. G., Halliday T. R. (1998). On confirmatory versus exploratory research. Herpetologica, 54, S64–S66. https://www.jstor.org/stable/3893289 [ Google Scholar ]
Kell D. B., Oliver S. G. (2004). Here is the evidence, now what is the hypothesis? the complementary roles of inductive and hypothesis-driven science in the post-genomic era. BioEssays, 26(1), 99–105. 10.1002/bies.10385 [ DOI ] [ PubMed ] [ Google Scholar ]
Lakatos I. (1978). The methodology of scientific research programmes. Volume 1. Philosophical papers. Cambridge University Press. [ Google Scholar ]
Linschoten J. (1978). Idolen van de Psycholoog. Erven J. Bijleveld. [ Google Scholar ]
Meehl P. E. (1978). Theoretical risks and tabular asterisks: Sir karl, sir ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834. 10.1037/0022-006X.46.4.806 [ DOI ] [ Google Scholar ]
Meehl P. E. (1990). Appraising and amending theories: The strategy of lakatosian defense and two principles that warrant it. Psychological Inquiry, 1(2), 108–141. 10.1207/s15327965pli0102_1 [ DOI ] [ Google Scholar ]
Nosek B. A., Spies J. R., Motyl M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615–631. 10.1177/1745691612459058 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. 10.1126/science.aac4716 [ DOI ] [ PubMed ] [ Google Scholar ]
Platt J. R. (1964). Strong inference: Certain systematic methods of scientific thinking may produce much more rapid progress than others. Science, 146(3642), 347–353. 10.1126/science.146.3642.347 [ DOI ] [ PubMed ] [ Google Scholar ]
Popper K. (2002a). Conjectures and refutations. Routledge. [ Google Scholar ]
Popper K. (2002b). The logic of scientific discovery. Routledge. [ Google Scholar ]
Riegler A. (2001). The role of anticipation in cognition. In: AIP Conference Proceedings , volume 573. Liege (Belgium): AIP, pp. 534–541. 10.1063/1.1388719 [ DOI ]
Rolfe G. (1997). Science, abduction and the fuzzy nurse: An exploration of expertise. Journal of Advanced Nursing, 25(5), 1070–1075. 10.1046/j.1365-2648.1997.19970251070.x [ DOI ] [ PubMed ] [ Google Scholar ]
Tukey J. W. (1980). We need both exploratory and confirmatory. The American Statistician, 34(1), 23–25. 10.1080/00031305.1980.10482706 [ DOI ] [ Google Scholar ]
Yanai I, Lercher M. (2020). A hypothesis is a liability. Genome Biology, 21(231), 1–5. 10.1186/s13059-020-02133-w [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
Yarkoni T. (2020). The generalizability crisis. Behavioral and Brain Sciences, 27. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
View on publisher site
PDF (534.0 KB)
Collections

Add to Collections

Twitter Icon
Facebook Icon
Reddit Icon
LinkedIn Icon

Newton didn’t frame hypotheses. Why should we?

The success of a grant proposal shouldn’t hinge on whether the research is driven by a hypothesis, especially in the physical sciences.

“Not hypothesis driven.” With those words and a fatal grade of “Very Good,” a fellow reviewer on a funding agency panel consigned the proposal we were discussing to the wastebasket. I listened in dismay. Certainly the proposal had hypotheses, though it didn’t have boldface sentences beginning “We hypothesize . . .” as signposts for inattentive readers. Then I remembered the famous words from Isaac Newton’s Principia : Hypotheses non fingo . “I do not frame hypotheses.” If that approach worked for Newton, why do we have such a mania for hypothesis-driven research today?

The emphasis on hypothesis-driven research in proposals is strangely embedded in the scientific community, with no obvious origin in funding agencies. The word hypothesis appears nowhere in the NSF guide to writing and reviewing proposals, and only once in the National Institutes of Health proposal guide. Yet grant-writing experts universally stress that proposals should be built around hypotheses and warn that those not written this way risk rejection as “fishing expeditions.”

In recent years, a few voices in the biosciences community have questioned this exclusive focus on hypothesis-driven research, even as the mania spreads to the physical sciences. Allow me to add my voice. Evaluating grant proposals is hard, but shoehorning every proposal into the language of hypothesis testing benefits neither the prospective grantee nor the evaluator. It can also hinder scientific progress.

Hypothesis history

Today many high school teachers present the scientific method as synonymous with hypothesis testing. Yet hypotheses are just ideas about how nature works, or what 19th-century scientist and philosopher William Whewell called “happy guesses.” Hypotheses organize our thinking about what might be true, based on what we’ve observed so far. If we have a guess about how nature works, we do experiments to test the guess. In quantitative sciences, the role of theory is to work out consequences of the guess in conjunction with things we know.

Perhaps the most famous hypothesis in all of science is that new species arise from the action of natural selection on random mutations. Charles Darwin based his hypothesis on observations of a few species during his famous voyage to the Galápagos. Charged with predictive power, Darwin’s hypothesis applies to all life, everywhere, at all times. Generations of biologists have tested and built on Darwin’s hypothesis with a vast array of new discoveries. The theory of evolution is now firmly established as the central pillar of biology, as well supported by evidence as any theory in science.

But what would Darwin have written had he been obliged to write a proposal to fund his voyage on HMS Beagle ? He didn’t have the hypothesis of natural selection yet—it grew out of the very observations he was setting out to make. If he wrote, truthfully, that “the isolated islands we will visit are excellent natural laboratories to observe what becomes of species introduced to a new locale,” it would be judged by today’s standards as a fishing expedition without a strong hypothesis.

What did Newton hypothesize, despite his protests to the contrary? He identified the right variables for the problem of planetary motion: force and momentum. Newton’s grant proposal might have read: “I hypothesize that momenta and forces are the right variables to describe the motion of the planets. I propose to develop mathematical methods to predict their orbits, which I will compare with existing observations.” That’s not quite a guess about how nature works, but rather the best way to describe motion mathematically, which by its widespread success grew into intuitive concepts of force, momentum, and energy.

Newton wrote hypotheses non fingo because of what he didn’t hypothesize. He wrote in reaction to vortex theories of gravity originated by René Descartes and Christiaan Huygens. They imagined that so-called empty space was actually filled with swirling vortices of invisible particles that swept the planets along in their orbits. The vortex idea is certainly a guess about how gravity works; it’s just not a very helpful guess. The idea of invisible particles that only reveal themselves by effects on unreachably distant planets is too elastic a notion. It’s not specific enough to make testable predictions. In the language of 20th-century philosopher of science Karl Popper, it’s not readily falsifiable.

Newton didn’t provide a just-so story, a fanciful mechanism for why momentum was conserved or how gravity arose. Instead he formulated simple rules that describe how the planets move—and as it turns out, how nearly everything else moves under ordinary circumstances. Powerful as Newton’s insight was, his description of gravity had the unsettling feature of “spooky action at a distance” of the Sun on the planets, and indeed every mass on every other mass. It took another 250 years for an explanation of the physical origin of gravity.

Albert Einstein’s hypothesis about gravity, unlike Newton’s, was mechanistic: Mass curves space, which is slightly elastic; as a result, straight lines bend near massive objects, including the path of light from distant stars passing near the Sun on its way to our telescopes. It took years for Einstein to develop the math to show that Newton’s description, which was consistent with so many observations, was only an approximation—and to make astounding predictions of things that happen to huge masses (collapse into black holes) or when big masses move really fast (gravitational waves).

Setting physical science apart

So why is present-day funding so focused on hypothesis-driven research? A clue is that hypothesis-driven experimental design is best suited to certain influential fields, especially molecular biology and medicine. Researchers in those fields study complicated, irreducible systems (living organisms), have limited experimental probes, and are often forced to work with small data sets. Unavoidably, the most common experimental protocol in these fields is to poke at a complex living system by giving it a drug or chemical and then measuring some indirectly related response. Those experiments live and die by the statistical test. When a scatter plot of stimulus versus response looks like a cloud of angry bees, the formal discipline of testing the null hypothesis is essential.

That is an overly narrow paradigm for what experiments can be. In the physical sciences, we are more able to manipulate and simplify the system of interest. We also enjoy more powerful experimental techniques, in many ways extensions of human senses, allowing us to see into a material, to listen to how it rings in response to being pinged with electromagnetic fields, to feel how it responds to a gentle push on the nanoscale. When you can do those things, experiments can be so much more than testing whether changing X influences Y with statistical significance. In fact, the history of science can be viewed as the development of new ways to probe nature. The Hubble Space Telescope was not driven by a hypothesis but rather by a desire to see deeper into the universe. Observations from Hubble and other modern telescopes enable new hypotheses about the early universe to be formulated and tested.

Progress in science often depends on advances in how to measure something important. A century after Einstein, ultrasensitive detectors brilliantly confirmed his prediction of gravitational waves. Those detectors rely on clever ideas for using lasers and interferometry to measure extremely tiny changes in the distance between two points on Earth. That work was not hypothesis driven, except in the obvious sense that general relativity predicts gravitational waves. Likewise, progress in quantitative sciences often relies on advances in our ability to compute the consequences of hypotheses that already exist.

Hypotheses are all well and good. But in evaluating research proposals, the key criterion should be: Will the proposed work help us answer an important question or reveal an important new question we should have been asking all along?

Scott Milner is William H. Joyce Chair and Professor of Chemical Engineering at the Pennsylvania State University.