Strengths and Limitations

To our knowledge, our study is the first to evaluate the quality of sharing in healthcare DES. A strength of our work is that our best practice audit is simple, effective, and quick to apply to new studies. Given its simplicity, and the large amount of guidance on open science and coding available for free online, it is important to recognise that it does show clear deficiencies in the last four years of the literature. For example, in open licensing, model testing (an expectation spelled out in any simulation textbook), and in long term storage and access.

Another strength is that we have followed open working practices in this study. All code and data are available, openly licensed, and deposited in Zenodo. We have also taken the additional step of providing an online companion book where code can be run to reproduce the results of the study. This is also deposited in Zenodo to ensure long term availability.

Our study cannot make any definitive statements about the reproducibility of the studies that shared their code versus those that used a reporting guideline versus those that did neither. Our aim instead was to focus on the practice of sharing and the current deficiencies as defined by gold standard guides such as the Turing Way and Open Modelling Initiative.

Our findings are based on information we found in the publication. We recognize that the model code may have been published online but not mentioned in the article. We feel our approach is the most appropriate as articles are the primary means by which researchers find research studies and their artifacts.

In order to scope computer model and code sharing practices in the DES healthcare literature, we attempted to conduct a broad and inclusive search of well known databases. While we cannot be certain that we have found all shared computer models we can attempt to compare our included studies to those included in previous healthcare DES reviews. A direct comparison of the number of studies included with other healthcare DES reviews is difficult due to differing aims, search terms, inclusion criteria and reporting. We can say that our review included more than double the number of health DES articles of a 2021 review covering 1994 to 2021 (n = 231) [Vázquez-Serrano et al., 2021]. For the years that overlapped (2019-2021) we cross-checked our included studies with those included in the 2021 study [Vázquez-Serrano et al., 2021] and found no additional DES articles. Our results also compare favourably to an earlier 2018 article covering healthcare DES between 1997 to 2016 (n = 211) [Zhang, 2018]. This study used very similar search terms to our own, but only included WoS and PubMed in their databases, was unable to cross check with other reviews, and did not conduct backwards and forwards citation chasing.

References

VSPGCB21(1,2)

Jesús Isaac Vázquez-Serrano, Rodrigo E. Peimbert-García, and Leopoldo Eduardo Cárdenas-Barrón. Discrete-Event Simulation Modeling in Healthcare: A Comprehensive Review. International Journal of Environmental Research and Public Health, 18(22):12262, January 2021. Number: 22 Publisher: Multidisciplinary Digital Publishing Institute. URL: https://www.mdpi.com/1660-4601/18/22/12262 (visited on 2023-04-03), doi:10.3390/ijerph182212262.

Zha18

Xiange Zhang. Application of discrete event simulation in health care: a systematic review. BMC Health Services Research, 18(1):687, September 2018. URL: https://doi.org/10.1186/s12913-018-3456-4 (visited on 2023-04-03), doi:10.1186/s12913-018-3456-4.