Background: The reporting quality of participant eligibility criteria in retrospective studies significantly affects research reproducibility and result interpretation. However, standardized guidelines for writing eligibility criteria in retrospective studies are lacking. We aim to systematically evaluate the quality of eligibility criteria reporting in retrospective studies published in high-impact factor medical journals, develop evidence-based recommendations for standardization, and provide supplementary guidance for relevant reporting guidelines. Methods: We conducted a cross-sectional analysis of retrospective studies published in the top 40 nonreview medical journals listed in the Journal Citation Reports (JCR) from January 2023 to September 2024. We extracted article characteristics (journal, author, objective, and study type) and eligibility criteria components. Two independent reviewers did the quality assessment of literature, which focused on clarity of retrospective nature (temporal framework), purposefulness (alignment with research objectives), and logical consistency between inclusion and exclusion criteria. Results: Among the top 40 nonreview medical journals in the 2023 JCR rankings, 11 journals contained 78 retrospective studies that were analyzed, of which 2.6% (2/78) demonstrated unclear retrospectivity and purposefulness in eligibility criteria. Logical contradiction between exclusion and inclusion criteria was found in 11.5% (9/78) of articles. Inter-rater reliability for quality assessment was substantial (κ = 0.857). Conclusion: The reporting quality of participant eligibility criteria in retrospective studies published in high-impact factor medical journals was flawed. On the basis of our systematic evaluation, we propose a structured framework for formulating eligibility criteria that emphasizes temporal precision, diagnostic clarity, and logical consistency between inclusion and exclusion criteria to supplement existing research reporting guidelines.

1. Hess DR. Retrospective studies and chart reviews. Respir Care. 2004;49(10):1171-1174.

2. Glasziou P, Vandenbroucke JP, Chalmers I. Assessing the quality of research. BMJ. 2004;328(7430):39-41.    DOI: 10.1136/bmj.328.7430.39    PMID: 14703546

3. Mathes T, Pieper D. Study design classification of registry-based studies in systematic reviews. J Clin Epidemiol. 2018;93:84-87.    DOI: 10.1016/j.jclinepi.2017.09.016    PMID: 28951107

4. Baumgartner PC, Haynes RB, Hersberger KE, Arnet I. A systematic review of medication adherence thresholds dependent of clinical outcomes. Front Pharmacol. 2018;9:1290.    DOI: 10.3389/fphar.2018.01290    PMID: 30524276

5. Ciulla MM, Vivona P. Time arrow in published clinical studies/trials indexed in MEDLINE: a systematic analysis of retrospective vs. prospective study design, from 1960 to 2017. PeerJ. 2019;7:e6363.    DOI: 10.7717/peerj.6363    PMID: 30723632

6. Thwaites S, Abrahams J, Thewlis D, Rickman M. The absence of reporting standards and a lack of objective, performance-based outcomes following intramedullary nailing of tibial shaft fractures: findings from a scoping review into 179 articles. Eur J Trauma Emerg Surg. 2024;50(1):59-70.    DOI: 10.1007/s00068-023-02338-1    PMID: 37555990

7. Wang SV, Pinheiro S, Hua W, et al. STaRT-RWE: structured template for planning and reporting on the implementation of real world evidence studies. BMJ. 2021;372:m4856.    DOI: 10.1136/bmj.m4856    PMID: 33436424

8. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495-1499.    DOI: 10.1016/j.ijsu.2014.07.013    PMID: 25046131

9. Vandenbroucke JP, von Elm E, Altman DG, et al. STROBE Initiative. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Int J Surg. 2014;12(12):1500-1524.    DOI: 10.1016/j.ijsu.2014.07.014    PMID: 25046751

10. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594.    DOI: 10.1136/bmj.g7594    PMID: 25569120

11. Khozin S, Blumenthal GM, Pazdur R. Real-world data for clinical evidence generation in oncology. J Natl Cancer Inst. 2017;109(11):10.    DOI: 10.1093/jnci/djx187    PMID: 29059439

12. Tooth L, Ware R, Bain C, Purdie DM, Dobson A. Quality of reporting of observational longitudinal research. Am J Epidemiol. 2005;161(3):280-288.    DOI: 10.1093/aje/kwi042    PMID: 15671260

13. Negrini S, Armijo-Olivo S, Patrini M, et al. The randomized controlled trials rehabilitation checklist: methodology of development of a reporting guideline specific to rehabilitation. Am J Phys Med Rehabil. 2020;99(3):210-215.    DOI: 10.1097/PHM.0000000000001370    PMID: 31851008

14. Tan DT, Liang H, Yu YB, et al. Quality evaluation of reports on inclusion and exclusion criteria in diagnostic tests. Hunan Univ J Chin Med. 2022;42(11):1916-1921.    DOI: 10.3969/j.issn.1674-070X.2022.11.023

15. Lobos E, Catanzariti A, McMillen R. Critical analysis of retrospective study designs: cohort and case series. Clin Podiatr Med Surg. 2024;41(2):273-280.    DOI: 10.1016/j.cpm.2023.09.002    PMID: 38388124

16. Ruan XX, Huang ZY, Liu FX, et al. Clinical research paper writing based on the perspective of doctors in diagnosing and treating patients. Chin J Endourol (Electronic Edition). 2024;18(4):397-401.    DOI: 10.3877/cma.j.issn.1674-3253.2024.04.016

17. White RG, Hakim AJ, Salganik MJ, et al. Strengthening the Reporting of Observational Studies in Epidemiology for respondent-driven sampling studies: "STROBE-RDS" statement. J Clin Epidemiol. 2015;68(12):1463-1471.    DOI: 10.1016/j.jclinepi.2015.04.002    PMID: 26112433

18. Sounderajah V, Ashrafian H, Aggarwal R, et al. Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: The STARD-AI Steering Group. Nat Med. 2020;26(6):807-808.    DOI: 10.1038/s41591-020-0941-1    PMID: 32514173

19. Blanco D, Altman D, Moher D, Boutron I, Kirkham JJ, Cobo E. Scoping review on interventions to improve adherence to reporting guidelines in health research. BMJ Open. 2019;9(5):e026589.    DOI: 10.1136/bmjopen-2018-026589    PMID: 31076472

20. Aggarwal R, Ranganathan P. Study designs: part 2—descriptive studies. Perspect Clin Res. 2019;10(1):34-36.    DOI: 10.4103/picr.PICR_154_18    PMID: 30834206

21. Eikenboom EL, Moen S, van Leerdam ME, et al. Metachronous colorectal cancer risk according to Lynch syndrome pathogenic variant after extensive versus partial colectomy in the Netherlands: a retrospective cohort study. Lancet Gastroenterol Hepatol. 2023;8(12):1106-1117.    DOI: 10.1016/S2468-1253(23)00228-5    PMID: 37865103

22. Levy D, Saura O, Passarelli MT, et al. Thrombolysis before venoarterial ECMO for high-risk pulmonary embolism: a retrospective cohort study. Intensive Care Med. 2024;50(8):1287-1297.    DOI: 10.1007/s00134-024-07501-9    PMID: 38913095

23. Kittai AS, Bond D, Huang Y, et al. Anti-CD19 chimeric antigen receptor T-cell therapy for Richter transformation: an international, multicenter, retrospective study. J Clin Oncol. 2024;42(17):2071-2079.    DOI: 10.1200/JCO.24.00033    PMID: 38552193

24. Arthur A, Orton MR, Emsley R, et al. A CT-based radiomics classification model for the prediction of histological type and tumour grade in retroperitoneal sarcoma (RADSARC-R): a retrospective multicohort analysis. Lancet Oncol. 2023;24(11):1277-1286.    DOI: 10.1016/S1470-2045(23)00462-X    PMID: 37922931

25. Wang J, Xiu J, Farrell A, et al. Mutational analysis of microsatellite-stable gastrointestinal cancer with high tumour mutational burden: a retrospective cohort study. Lancet Oncol. 2023;24(2):151-161.    DOI: 10.1016/S1470-2045(22)00783-5    PMID: 36681091

26. Zeng Q, Klein C, Caruso S, et al. Artificial intelligence-based pathology as a biomarker of sensitivity to atezolizumab-bevacizumab in patients with hepatocellular carcinoma: a multicentre retrospective study. Lancet Oncol. 2023;24(12):1411-1422.    DOI: 10.1016/S1470-2045(23)00468-0    PMID: 37951222

27. Karpinski MJ, Hüsing J, Claassen K, et al. Combining PSMA-PET and PROMISE to re-define disease stage and risk in patients with prostate cancer: a multicentre retrospective study. Lancet Oncol. 2024;25(9):1188-1201.    DOI: 10.1016/S1470-2045(24)00326-7    PMID: 39089299

28. Wong CKH, Lau KTK, Au ICH, et al. Viral burden rebound in hospitalised patients with COVID-19 receiving oral antivirals in Hong Kong: a population-wide retrospective cohort study. Lancet Infect Dis. 2023;23(6):683-695.    DOI: 10.1016/S1473-3099(22)00873-8    PMID: 36796397

29. Ehmann MR, Mitchell J, Levin S, et al. Renal outcomes following intravenous contrast administration in patients with acute kidney injury: a multi-site retrospective propensity-adjusted analysis. Intensive Care Med. 2023;49(2):205-215.    DOI: 10.1007/s00134-022-06966-w    PMID: 36715705

30. Saux P, Bauvin P, Raverdy V, et al. Development and validation of an interpretable machine learning-based calculator for predicting 5-year weight trajectories after bariatric surgery: a multinational retrospective cohort SOPHIA study. Lancet Digit Health. 2023;5(10):e692-e702.    DOI: 10.1016/S2589-7500(23)00135-8    PMID: 37652841

31. Häggström I, Leithner D, Alvén J, et al. Deep learning for [¹⁸F] fluorodeoxyglucose-PET-CT classification in patients with lymphoma: a dual-centre retrospective analysis. Lancet Digit Health. 2024;6(2):e114-e125.    DOI: 10.1016/S2589-7500(23)00203-0    PMID: 38135556

32. Cragg WJ, McMahon K, Oughton JB, Sigsworth R, Taylor C, Napp V. Clinical trial recruiters’ experiences working with trial eligibility criteria: results of an exploratory, cross-sectional, online survey in the UK. Trials. 2021;22(1):736.    DOI: 10.1186/s13063-021-05723-6    PMID: 34689802

33. Porzsolt F, Wiedemann F, Becker SI, Rhoads CJ. Inclusion and exclusion criteria and the problem of describing homogeneity of study populations in clinical trials. BMJ Evid Based Med. 2019;24(3):92-94.    DOI: 10.1136/bmjebm-2018-111115    PMID: 30567938

34. Patino CM, Ferreira JC. Inclusion and exclusion criteria in research studies: definitions and why they matter. J Bras Pneumol. 2018;44(2):84.    DOI: 10.1590/s1806-37562018000000088    PMID: 29791550

35. Cobo E, Cortés J, Ribera JM, et al. Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial. BMJ. 2011;342:d6783.    DOI: 10.1136/bmj.d6783    PMID: 22108262

36. Gonzalez P, Wilson GS, Purvis AJ. Peer review in academic publishing: Challenges in achieving the gold standard. J Uni Teach Learn Pract. 2022;19(5):1.    DOI: 10.53761/1.19.5.1

37. Bruce R, Chauvin A, Trinquart L, Ravaud P, Boutron I. Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis. BMC Med. 2016;14(1):85.    DOI: 10.1186/s12916-016-0631-5    PMID: 27287500

38. Paliya N, Jha SS. Quality indicators for academic journals: What makes a journal trust worthy? IP Indian J Anat Surg Head, Neck Brain. 2024;10(4):83-91.    DOI: 10.18231/j.ijashnb.2024.019