Bartosz Maćkiewicz, Joanna Gęgotek, Monika Gurak, Franciszek Kittel and Katarzyna Kuś
Affiliation: Faculty of Philosophy, University of Warsaw
The integrity of science relies on the assumption that studies and their results are adequately described in research papers. This assumption is supported by several practical reasons. First, when studies and their results are underdescribed, it becomes impossible for other researchers to independently verify the authors’ conclusions and derive new insights from the available data. Second, studies that are poorly described or lack details about key methodological aspects suffer from low replicability. Third, precise and detailed reporting ensures that a study’s results can be integrated into secondary analyses, such as meta-analyses, which statistically combine data from multiple studies.
One of the most common challenges that precludes the possibility of conducting meta-analyses on existing studies is the absence of the specific information required to apply meta-analytical techniques. Historically (cf. Dar, Serlin, & Omer 1994 for clinical psychology; Alhija & Levy 2009 for educational research) and contemporarily (cf. Wei, Hu, & Xiong 2019 for applied linguistics; see also Farmus et al. 2022 for improvements in social personality research), empirical papers often lack details about effect sizes and their variances. This issue would be mitigated if authors included original raw data or sufficiently detailed descriptive statistics that would allow researchers to compute appropriate effect size measures. Unfortunately, many disciplines still systematically fall short in this regard (see Brown et al. 2014 for experimental psychology, Hardwicke et al. 2020 for the social sciences, and Tedersoo et al. 2021 for a recent assessment of data availability across fields). Beyond statistical reporting, it is equally important that essential methodological details—such as experimental design, recruitment procedures, sample structure, and experimental protocols—are adequately described. Without such details, key study-level covariates cannot be incorporated into meta-analytical models, potentially obscuring important sources of variance.
Experimental philosophy (x-phi) is a relatively new field that combines traditional philosophical questions and frameworks with empirical methods from psychology, cognitive science, and the social sciences. While meta-research on compliance with best reporting practices is routinely conducted in fields closely related to experimental philosophy (cf. Carp 2012; Grant et al. 2013; Brown et al. 2014; Hardwicke et al. 2020), experimental philosophy itself has not yet received the same attention. In this talk, we present the results of an ongoing meta-research project in experimental philosophy. We have adapted existing meta-research tools to the specific characteristics of papers typical in experimental philosophy, resulting in the development of the X-Phi Transparency and Quality of Reporting Inventory. Modeled after Brown et al.’s (2014) Replicability and Meta-Analytic Suitability Inventory, our tool consists of approximately 100 questions covering various aspects of methodological and statistical reporting, divided into five sections:
In the present study, we use this tool to assess 400 experimental philosophy papers randomly selected from journal articles indexed under ‘experimental philosophy’ category in the PhilPapers database and published between 2007 and 2024. This approach allows us to identify which aspects of reporting in experimental philosophy need more attention and to examine whether reporting standards improve as the field matures. Specifically, we aim to address the following research questions:
Since completing this project is a major undertaking, we will present preliminary findings based on an analysis of 300 papers published between 2007 and 2021 during the conference. Our findings will provide empirical insights into the current state of reporting in experimental philosophy and offer concrete recommendations for improving transparency and replicability in the field.
References
Alhija, F. N. A., & Levy, A. (2009). Effect size reporting practices in published articles. Educational and Psychological Measurement, 69(2), 245–265. Brown, S. D., Furrow, D., Hill, D. F., Gable, J. C., Porter, L. P., & Jacobs, W. J. (2014). A duty to describe: Better the devil you know than the devil you don’t. Perspectives on Psychological Science, 9(6), 626–640. Carp, J. (2012). The secret lives of experiments: methods reporting in the fMRI literature. Neuroimage, 63(1), 289–300. Dar, R., Serlin, R. C., & Omer, H. (1994). Misuse of statistical tests in three decades of psychotherapy research. Journal of Consulting and Clinical Psychology, 62(1), 75–82. Farmus, L., Beribisky, N., Martinez Gutierrez, N., Alter, U., Panzarella, E., & Cribbie, R. A. (2022). Effect size reporting and interpretation in social personality research. Current Psychology, 1–11. Grant, S. P., Mayo-Wilson, E., Melendez-Torres, G. J., & Montgomery, P. (2013). Reporting quality of social and psychological intervention trials: a systematic review of reporting guidelines and trial publications. PloS one, 8(5), e65442. Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. (2020). An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017). Royal Society Open Science, 7(2), 190806. Tedersoo, L., Küngas, R., Oras, E., Köster, K., Eenmaa, H., ... & Sepp, T. (2021). Data sharing practices and data availability upon request differ across scientific disciplines. Scientific Data, 8(1), 192. Wei, R., Hu, Y., & Xiong, J. (2019). Effect size reporting practices in applied linguistics research: A study of one major journal. SAGE open, 9(2), 2158244019850035.