Proposing a theory committee at the Psychological Science Accelerator

By Peder M. Isager & Nicholas A. Coles

August 25, 2019

(written in collaboration with Nicholas A. Coles)

The Psychological Science Accelerator (PSA) is a global network of psychological laboratories that organizes large-scale empirical research projects. The organization aims “to accelerate the accumulation of reliable and generalizable evidence in psychological science” (Moshontz et al. 2018). To succeed in this mission, the PSA has created several committees that work together to address important challenges in producing high quality research, such as having access to an inferentially meaningful sample size, reducing the reliance on WEIRD participants, and improving the methodological quality of designs and analyses. However, one task critical to fulfilling the mission of the PSA has not yet been addressed: assessing the theoretical quality of research proposals (and by extension, the quality of theories in psychology).

Christopher R. Chartier (2017) stated that the PSA would like “… to direct the network’s resources towards the best possible research questions in 2018 and beyond.” To succeed in this, and to follow the mission statement of the organization, we must be able to determine what makes a research question valuable to pursue, and we must be able to assess whether a research design will be theoretically innovative. We therefore propose establishing a committee or sub-committee devoted to theoretical development and assessment within the PSA.

Substantive arguments for focus on theoretical quality

A strong grounding in theory within the field is perhaps the largest difference between the PSA and CERN, the organization on which the PSA was modelled. The CERN Large Hadron Collider was primarily designed to answer very specific theoretical questions in physics (e.g. Zeppenfeld et al. 2000), and it appears that many physicists agreed that these questions would in fact be answered by the experiments conducted at the Large Hadron Collider. In comparison, the PSA is not designed to address any particular theory within psychology. This makes sense as psychology is not nearly as theoretically coherent as physics. Thus, in our field there are many theories, not obviously connected, that may be important to consider for large scale replication. Nonetheless, in order to reduce the distance between truth and our current understanding, we will require strong theory and sound empirical operationalization of those theories.

Sir Karl Popper, Thomas Kuhn, Imre Lakatos, and other prominent philosophers of science have all made it clear that the relationship between observation, theory and truth is complicated (e.g. de Groot 1969; Kuhn 2012; Lakatos 1976; Popper 2002). This is perhaps especially true in the field of psychology. As one example, Meehl (Meehl 1967, 1978, 1990) has outlined major problems with theorizing specific to psychology for half a century, and these criticisms still apply to the field to this day. In many areas of psychology, theory remains underdeveloped (e.g. Fiedler 2004; Muthukrishna and Henrich 2019). If the mission of the PSA is to generate highly informative studies then we have a problem, because the informativeness of a study is often directly linked to the quality of the theory it is derived from (de Groot 1969; Meehl 1990). Fortunately, there exist a number of promising proposals for how to improve the current state of affairs (e.g. Landreth and Silva 2013; Smaldino, Calanchini, and Pickett 2015; Muthukrishna and Henrich 2019) which the PSA theory committee could evaluate and help implement.

Pragmatic argument for focus on theoretical quality

The PSA is in a unique position to spearhead meaningful theoretical development in psychology because of the diversity and size of its collaborative network. Theoretical progress in our field may depend on specialist knowledge of the subfield the theory is being developed in, as well as formal training in logic and philosophy of science, scientific methodology, model construction, different subfields within psychology, and different fields within science. It may be challenging, if not impossible, for one researcher to acquire all the knowledge necessary to make real theoretical progress. Therefore, increased knowledge sharing between specialists will likely be crucial for accelerated theory development in psychology. Due to the scale and diversity of its member network, the PSA is well-suited to facilitate such collaboration.

The PSA also have more selfish reasons for facilitating work on theory development. Studies conducted by the PSA have large costs that are most justifiable if they lead to theoretical innovation. For example, there are currently 166 labs participating in the “Global Valence Dominance” project. Thus, if this study does not lead to theoretical innovation, it would be mean a 166-fold waste of resources compared to a single lab conducting the same study. For factors such as precision and generalizability, the current operation strategy more or less guarantees value for the resources invested. But even a highly precise and generalizable result can be theoretically uninteresting. Furthermore, even a test of a highly interesting hypothesis might be fatally weakened by theoretical flaws in the research design (e.g. Scheel et al. 2018).

A theory committee would also be symbolically important, both to the field and to potential funders. To the extent that the PSA wants to be a model of exemplary research in psychology, the establishment of a theory committee might inspire the field as a whole to seriously address the need for more rigorous theory - much like the “Reproducibility Project: Psychology” inspired the field to seriously address the issue of replicability.

Potential responsibilities of a theory committee

The proposed theory committee would serve in an advisory role, providing input during the assessment of submitted research proposal, and assisting with the design of experiments. Its primary foci could be: (a) identify and update the PSA on promising theoretical advancements in psychology, (b) help researchers working with the PSA derive theory-driven, falsifiable predictions, and (c) produce and publish research on the state of theory within the field of psychology, as well as methods to facilitate theoretical development. Concrete responsibilities of the committee could include:

  • Define and recommend strategies for optimizing research designs to provide the best possible corroboration and/or falsification of theories.
  • Develop criteria for identifying progressive vs degenerative research programmes.
  • Develop criteria for assessing the theoretical quality of proposed studies.
  • Engineer cumulative/meta-analytic approaches to formally relate research findings to theory (e.g. Landreth & Silva, 2013).
  • Help construct and justify grant proposals to secure funding for the PSA.
  • (During proposal evaluation) collect reviews from, and facilitate discussion between, a diverse set scholars with specialist knowledge related to the theory/subfield in question.

A separated but related task that could be designated to this committee is to investigate opportunities for diversifying the theoretical background of members within the PSA, and for fostering interdisciplinary collaboration. Although it is named the Psychological Science Accelerator, psychological science is tightly connected with different branches of science, and would likely benefit from at least occasional infusion of knowledge from other fields (see e.g. Facilitating Interdisciplinary Research 2004). Other large scale collaboration projects are already exploring ways of accessing a more interdisciplinary knowledge base. As one example, NASA is currently applying citizen science to optimize solutions to various engineering problems (see this video for an explanation by Steven Rader). This would be an interesting model to employ in the social sciences. Compared with individual labs, interdisciplinary outreach from a large scale collaboration such as the PSA might be perceived as more legitimate by researchers from fields outside psychology. This would hopefully increase engagement from these communities. Interdisciplinary engagement would also nicely dovetail with the within-psychology outreach that is already facilitated when a PSA project is launched.


The PSA is an organization committed to producing high-quality psychological research in representative samples. However, empirical studies are only as informative as the theories they inform. If the hypotheses that spawn a PSA study are vague or uninformative, the results of the study will be as well, no matter how large and representative the sample is. Consequently, we propose establishing a committee or working group at the PSA that will be devoted to theory development and assessment.


Chartier, Christopher R. 2017. “The Psychological Science Accelerator. Rapid Progress. More Help Needed.” Christopher R. Chartier.
de Groot, Adriaan Dingeman. 1969. Methodology (Methodologie, Engl.) Foundations of Inference and Research in the Behavioral Sciences. MTH.
Facilitating Interdisciplinary Research. 2004. Washington, D.C.: National Academies Press.
Fiedler, Klaus. 2004. “Tools, Toys, Truisms, and Theories: Some Thoughts on the Creative Cycle of Theory Formation.” Personality and Social Psychology Review 8 (2): 123–31.
Kuhn, Thomas S. 2012. The Structure of Scientific Revolutions: 50th Anniversary Edition. University of Chicago Press.
Lakatos, Imre. 1976. “Falsification and the Methodology of Scientific Research Programmes.” In Can Theories Be Refuted?, 205–59. Dodrecht: Springer.
Landreth, Anthony, and Alcino J. Silva. 2013. “The Need for Research Maps to Navigate Published Work and Inform Experiment Planning.” Neuron 79 (3): 411–15.
Meehl, Paul E. 1967. “Theory-Testing in Psychology and Physics: A Methodological Paradox.” Philosophy of Science 34 (2): 103–15.
———. 1978. “Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology.” Journal of Consulting and Clinical Psychology 46 (4): 806–34.
———. 1990. “Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles That Warrant It.” Psychological Inquiry 1 (2): 108–41.
Moshontz, Hannah, Lorne Campbell, Charles R. Ebersole, Hans IJzerman, Heather L. Urry, Patrick S. Forscher, Jon E. Grahe, et al. 2018. “Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network.” Advances in Methods and Practices in Psychological Science, July.
Muthukrishna, Michael, and Joseph Henrich. 2019. “A Problem in Theory.” Nature Human Behaviour 3 (3): 221–29.
Popper, Karl R. 2002. Conjectures and Refutations: The Growth of Scientific Knowledge. Routledge Classics. London ; New York: Routledge.
Scheel, Anne M., Stuart J. Ritchie, Nicholas J. L. Brown, and Steven L. Jacques. 2018. “Methodological Problems in a Study of Fetal Visual Perception.” Current Biology 28 (10): R594–96.
Smaldino, Paul E., Jimmy Calanchini, and Cynthia L. Pickett. 2015. “Theory Development with Agent-Based Models.” Organizational Psychology Review 5 (4): 300–317.
Zeppenfeld, D., R. Kinnunen, A. Nikitenko, and E. Richter-Wa̧s. 2000. “Measuring Higgs Boson Couplings at the CERN LHC.” Physical Review D 62 (1): 013009.
Posted on:
August 25, 2019
8 minute read, 1655 words
PSA theory research practice
See Also:
Meta-research at the Psychological Science Accelerator
Ganging Up on Psychological Science