Meta-research at the Psychological Science Accelerator

(This is a reposting of a blog post originally posted to: https://metaresearch.nl/blog/2019/12/16/meta-research-at-the-psychological-science-accelerator)

Friday November 22, 2019, the Meta-research center at Tilburg University (https://metaresearch.nl/) organized the meta-research day. Around 90 researchers attended the meta-research day that involved three plenary lectures, by John Ioannidis (who received an honorary doctorate from Tilburg University a day earlier), Ana Marušić, and Sarah de Rijcke, and seven parallel sessions on meta-research (https://www.tilburguniversity.edu/about/schools/socialsciences/organization/departments/methodology-statistics/colloquia/meta-research-day). One of these sessions was titled How can meta-research improve the Psychological Science Accelerator (PSA) and how can the PSA improve meta-research?, and was led by Peder Isager and Marcel van Assen. Nineteen participants attended this session.

The Psychological Science Accelerator (PSA) is a standing network of more than 500 laboratories that collect large-scale, non-WEIRD data for psychology studies (https://psysciacc.org). The PSA is currently running six many-lab projects, and a number of proposed future projects are currently under review. Of particular interest to the meta-science community, the PSA has established meta-science working group that is currently examining both how the PSA can best interface with the meta-research community, and how meta-science can help bolster the quality of research projects conducted at the PSA.

The session began with an overview of PSA’s organization, presented by Peder, and a discussion of the importance of many-lab studies, presented by Marcel. The slides for these presentations can be found at https://osf.io/wnyga. Afterwards, the majority of the session was devoted to discussing seven predetermined topics related to how the meta-research field and the PSA may learn from each other. Participants could either independently provide their suggestions on the seven topics in a google doc (https://bit.ly/2KIUHTW) or on paper. After about half an hour independently working on the topics, we discussed the participants’ suggestions in the remainder of the session.

The first topic was PSA’s policy (under development) to allow meta-researchers to submit “piggy-back” studies to run on top of accepted studies. Participants welcomed this idea and option as it provides a more efficient way to collect data and the potential to answer more relevant research questions, but participants found many details unclear. Who determines what can be “piggy-backed”, and what criteria are used for this selection? How will researchers get to know about the many-lab projects and the option and criteria to “piggy-back”? A protocol and clear and inclusive communication to potentially interested researchers by the PSA may be needed.

The second topic was the use of PSA data by the meta-research community. The PSA has potential access to lab/researcher level information from hundreds of labs while they are conducting research. Participants indicated that lab/researcher level information (lab size, location, gender and expertise of researchers, etc etc.) is useful to explain possible heterogeneity of true effect size across labs. Moreover, these data together with information on estimates of true effect size and the research field may be used to create a prior distribution of effect size in the long run.

The third topic was on how the PSA could implement knowledge from meta-science at several stages of the research cycle, such as theory formulation, study design, data collection, data analysis, inference/interpretation, etc. Particularly the theory stage elicited many responses. Suggestions included explicitly indicating how theory and hypotheses are derived, identifying boundary conditions (constraints of generality), and including potentially contextual or other factors in the design that are derived from theory. The idea is that theory should be important, perhaps also when selecting which phenomena to focus on in PSA research. Perhaps a separate theory committee of the PSA would be a good idea (see https://pedermisager.netlify.com/post/psa-theory-committee/ for a proposal). A person with formal expertise in modelling/philosophy of science may help the lead author team of each PSA project tie their project strongly to theory, and thus design a more theoretically informative study. Also when proposing measurement instruments of the main variables in the study, the role of theory should be made explicit. The design of a PSA study may also include different measurement instruments to explicitly examine the sensitivity to measurement.

Many suggestions with respect to data analysis were also given, such as (i) potentially including a multiverse-analysis at the level of labs, (ii) performing individual participant data (IPD) meta-analysis rather than the suboptimal regular meta-analysis, (iii) independent team data-analysis (i.e., the same analysis is conducted by other independent people as well, or by people who are not involved in the project), and (iv) systematically testing measurement invariance of measurement instruments. Moreover, some checks on the analyses carried out at the lab level is recommended. Finally, one interesting proposed option was to use sequential testing procedures, that is, potentially including more labs only when average effect size estimation or heterogeneity estimation would profit from it.

In any case, PSA should provide good guidelines on requirements and criteria for a “good many-lab study”, with respect to all stages of the research.

Fourth, what issues should the PSA meta-science sub-committee be most concerned with? Mentioned were requiring a focus on theory and informative study designs, the evaluation of protocols and their standardization, organization of the project and communication of its essentials to both the participating labs and community, and collecting variables at the lab level. The fifth topic was the question what meta-research projects would be most fruitful to run through the PSA. One idea is to examine effect size heterogeneity as a function of variability in context, variables, or measurement designs (by design). Two other ideas are neuroscientific research, where the costs of research are shared by many labs each examining a small sample of respondents (as is also done in genome-wide association studies [GWAS]), and a network analysis of co-authorship of PSA studies to examine if the PSA leads to the formation of lasting collaborator ties within and across disciplines.

The penultimate topic was what information/resources meta-researchers actually require to interact with the PSA system. Some participants found the website registration confusing, as well as the questionnaire (e.g., what does it mean to indicate interest in working with a specific committee?). Vital to the community is that the analysis code and data are easily accessible and in a standard format such that re-analysis of the data is as straightforward as can be.

The last topic concerned the risks and challenges the PSA (and other team science collaborators) should be aware of. Mentioned were transparency (who is doing and deciding what), securing involvement of enough non-Western labs, and storage of all data from all projects in a standard format at a secure location, and spending resources wisely (e.g., not too many labs for a certain project, which can be prevented by doing appropriate power analysis).

The following conclusions can be drawn from our discussion. There are multiple ways in which the PSA could contribute to meta-science research (e.g. by providing access to lab data and project-level data for conducted studies, and by allowing researchers to vary properties of research designs - like the measurement tools - to study effect size heterogeneity, and advance theory by examining boundary conditions). There are multiple issues within the meta-science field that seems relevant to the PSA. Issues related to theory, measurement and sample size determination were emphasized in particular.

Meta-researchers seem interested in contributing to the PSA research endeavor, but emphasize a lack of both general information about the PSA organization and specific information about what contributions could/would entail (e.g. what volunteer efforts one could contribute to and what studies would be relevant for the “piggy-back” submission policy).

In summary, there seems to be much enthusiasm for the PSA within the meta-research community, and there are many overlapping interests between the PSA and the meta-research community. The points raised in this session will be communicated to the PSA network of researchers, with the hope that it will help facilitate more communication between the two research communities in the future.