Blinding Assessments in Neonatal Ventilation Meta-Analyses: A Systematic Meta-Epidemiological Review

Randomized controlled trials and meta-analyses of these are the gold standards for evaluating the effectiveness and efficacy of medical interventions [1]. Overall, in medicine, the guidelines and good practices have promoted the use of blinding to reduce possible bias in clinical studies [2]. A key part of the critical appraisal of evidence and synthesis is to assess the risk of bias in the included studies [3]. Typically, in randomized trials, this is performed using the Cochrane’s risk of bias original tool and the revised RoB 2.0 tool [4, 5]. In the original RoB 1.0 tool, blinding is assessed more clearly as it contains its own domains for blinding of the participants and personnel, and blinding of the outcome assessment [4]. In RoB 2.0, blinding assessment is part of the domain named as the measurement of the outcome [5]. However, in both guidance documents, the rationale is that the blinding is contextualized to the nature of the outcome, whether the outcome is objective for patient, caregiver, outcome assessor, all of these or none.

The objective nature of the outcome is a critical aspect of the risk of bias in the blinding assessment as the lack of blinding may influence the effect sizes particularly for subjective outcomes [6]. Previous meta-epidemiological studies have had conflicting results from Cochrane registries on the impact of different risk of bias domains on the review outcomes. An older meta-epidemiological review of studies published in 2011 reported that blinding was associated with increased heterogeneity with subjective outcomes [7]. However, a more recent meta-epidemiological review found no evidence that blinding would influence effect estimates [8]. Most recently, a systematic review of meta-epidemiological studies stated that blinding influences the effect estimates for subjective outcomes; however, the impact of health-care providers did not show evidence of such [6].

In the field of neonatology, it is generally assumed that patients are unaware of the interventions they receive, and thus, the focus should be on the outcome assessors only. Previous meta-epidemiological studies have criticized many neonatal trials on the lack of outcome assessment blinding [9, 10], and, actually, neonatal ventilation trials are seldom assessor-blinded, although this is theoretically possible with appropriate investments [11, 12]. However, these studies had not considered the nature of the outcome in their assessments. In cases where the outcome is clearly objective, such as mortality or bronchopulmonary dysplasia (BPD), the blinding of the outcome assessor is unlikely to have any impact, as the assessment is not dependent on or influenced by the knowledge of the received intervention. This has been previously shown in adult intensive care trials [13]. However, in the context of neonatal ventilation, the knowledge of the intervention may influence the assessment of other outcomes such as the ventilation duration or the decision to re-intubate a patient during the neonatal intensive care unit hospitalization, if attending clinicians are also the unblinded assessors. An older meta-epidemiological study of neonatal randomized trials found that only bias due selective reporting was only risk of bias domain associated with treatment effect estimates, whereas blinding domains were not [9].

We were interested in examining how the authors of systematic reviews and meta-analyses have estimated the risk of bias due to blinding and have noted the objectivity of the outcomes in their assessments. Finally, as the risk of bias is part of the evidence certainty assessment [12], we aimed to assess whether the possibility of improper blinding assessment may have influenced the evidence certainty assessment.

We conducted a systematic meta-epidemiological review of systematic reviews and meta-analyses. We have reported this review according to meta-epidemiological adaptation of the Preferred Reporting Items in Systematic Reviews and Meta-analyses (PRISMA) guideline [14, 15].

We searched PubMed and Scopus databases in November 2023 using the following key words and/or MeSH terms: (neonate or newborn or infant) AND (“respiratory support” or ventilation) AND (meta-analysis or systematic review). We hand searched the reference lists of the included studies for possible additional reviews to be included. We did not search grey literature. There was no language limitation, but we restricted the search to past 5 years to have an overview of the most recent literature, and as the RoB 2.0 guidance was published in 2019 and implemented into use simultaneously. Two authors (I.K. and K.R.) performed the screening process in the Covidence software and disagreements were solved by discussion between them.

We included reviews focused on neonatal patients (0–28 days of life) regardless of the gestational age, where the analyzed intervention was represented by any respiratory support technique. We defined these as reviews focusing on either different respiratory support strategies (for example, comparing two noninvasive respiratory support modes) or different pressure limits or oxygen concentrations in the treatment. We did not prespecify comparators and included the reviews regardless of the comparators used in the analyses. As additional inclusion criterion, reviews needed to have either mortality or BPD, diagnosed with any BPD definition, as an outcome. If we found two versions of the same Cochrane review, we included the latest version of it.

We excluded qualitative systematic reviews without statistical synthesis and reviews that focused on surfactant delivery via different ventilation modes or strategies. Finally, we also excluded reviews including only observational studies, or not reporting observational studies separately from randomized trials.

Our main outcome was the proper assessment of blinding. We considered the blinding assessment properly done, when the authors mentioned whether the outcome was subjective or objective, and the risk of bias assessment was performed per outcome and not per study. As a secondary outcome, we evaluated how the assessment of blinding influenced the GRADE evidence certainty reported by the authors.

Basic data on each review and related demographics were extracted by one author (I.K.). The consensus on the proper reporting of blinding was performed independently by two authors (I.K. and K.R.) and then cross-verified. Discrepancies were resolved through discussion between them. If data were lacking or further clarifications were needed, we planned to contact the authors with at least two emails sent to authors 2 weeks apart. We recorded the following information in a dedicated Excel spreadsheet which was tested with the data of two randomly chosen reviews before the actual data collection: title, journal, authors, country, review main outcome, mortality and BPD, risk of bias assessment tool, blinding assessment, evidence certainty assessment for BPD and/or mortality outcomes, influence of blinding assessment on the certainty evaluation, accuracy of the blinding assessment.

We compared the proper assessment of blinding considering the used risk of bias tools, the nature of the review (Cochrane and non-Cochrane), the authors’ country, and publication year, by using χ² or Fischer’s exact test, as appropriate. Subsequently, all these factors were inserted in a multivariable linear regression model to verify their relative effect on the proper assessment of blinding: results were expressed as beta-coefficient (95% confidence interval) and multicollinearity was evaluated with variable inflation factor, as previously described [16]. p values <0.05 were considered as statistically significant.

Protocol was registered to Prospero database (ID: CRD42023484598) and is available from: https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42023484598.

We screened 494 abstracts, and further assessed 57 full reports, of which 40 were included for final analysis (Fig. 1) [17‒56]. Included studies are available in the supplementary material (online suppl. Table 1; for all online suppl. material, see https://doi.org/10.1159/000539203). Basic characteristics of the included reviews are summarized in Table 1.

Overall, we found that 13 (32.5%) of the 40 reviews assessed blinding properly. The proportions of correctly assessed blinding were similar between Cochrane and non-Cochrane articles and per publication year (Table 2). Australian and European authors were most likely to perform correct assessment of the blinding (p = 0.03) and the use of RoB 2.0 tool was also associated with proper assessment (p < 0.001) compared to original RoB tool. Upon multivariate regression the use of RoB 2.0 was the only factor significantly associated with a proper assessment (Beta 0.57 [95% CI: 0.29–0.99], p < 0.001).

GRADE ratings were performed for mortality or BPD in 25 (62.5%) of the 40 included reviews. The reported GRADE ratings for the objective outcomes varied between very low and moderate, and none of the included reviews assessed evidence certainty as high. Of the 25 reviews, the authors downgraded the evidence certainty due to risk of bias in 20 (77.8%), and all of these reviews did not perform the blinding assessment correctly. We further estimated the impact of proper assessment of blinding and in 19 (76%) cases, the proper assessment of blinding could have potentially influenced the risk of bias assessment and had an impact to the GRADE rating (online suppl. Table 2). However, it must be noted that the downgrading of evidence certainty is not always only due to risk of bias due lack of blinding, and thus we have labeled these as could potentially influence the evidence certainty rating.

Overall, we found that blinding was often incorrectly assessed in recent meta-analyses focusing on neonatal ventilation. Two-thirds of the analyzed meta-analyses inadequately classified the original studies to a given risk of bias due to the lack of outcome assessment blinding. They did not consider the objective nature of the main outcome, such as mortality or BPD. Furthermore, we found that in approximately 70% of cases the evidence certainty was wrongly downgraded due to risk of bias, due to the influence of blinding assessment.

An illustrative example of this problem may be given by a recent Cochrane review on noninvasive ventilation strategies in preterm neonates, which reported both subjective (need for mechanical ventilation, mechanical ventilation duration), and objective (mortality, BPD) outcomes [37]. In their evidence rating, authors stated that the evidence certainty was downgraded for mortality to “moderate” due to lack of blinding. As Cochrane studies have been typically classified as being a rigorous source of evidence [57], this issue may also concern several neonatal meta-analyses focused on other fields. Nonetheless, in our sample of neonatal ventilation meta-analyses, we found that this issue was equally distributed between Cochrane and non-Cochrane articles.

We also found that country and the use of RoB 2.0 tool were associated with better blinding assessment quality. The RoB 2.0 tool provides signaling questions and a flowchart for each question with clear guidance to the outcome assessment, which clearly directs the assessment of the blinding to be outcome-dependent instead of study-dependent. However, a previous meta-epidemiological study examined the proper use of RoB 2.0 tool between 2019 and 2021 in all medical systematic reviews and found that the adherence to proper use of the tool was below 70%, and it was associated with the overall quality of the systematic review [58]. The use of RoB 2.0 has been increasing annually, and our data support the importance of implementing this tool in the neonatal epidemiological and clinical research [59].

The rating of evidence certainty is currently a key part of the evidence synthesis process, and GRADE is the most used method to uniformly conduct the assessment. Risk of bias is among the domains which guide the overall evidence certainty rating [60]. The GRADE assessment is by default always performed for each outcome separately; therefore, the risk of bias should be assessed for each outcome as well. This was clearly stated yet in the original publication of the guideline elaboration in 2011, but based on our results and previous meta-epidemiological studies, it is still often wrongly performed [58].

It is unclear if the incorrect blinding assessment is related to the lack of specific funding for neonatal ventilation trials: this would require a deeper analysis of the cost of a neonatal ventilation strategy to inform a better allocation of resources and prioritize more important issues [61]. Future research is needed to further estimate the impact of the proper assessment in evidence certainty also in other neonatal meta-analyses, outside of the ventilation field. Furthermore, for example, a Delphi consensus study among clinicians, researchers, and methodologists would be needed to determine the objectivity in some neonatal outcomes. The pathobiological plausibility shall also be considered as it lacks or is unclear for several multifactorial neonatal disorders: high certainty of evidence is hard to believe in absence of plausibility [62]. This is also worrying for BPD and ventilation since this latter is only one of the many risk factors for BPD and, in fact, several neonates who have never been ventilated can indeed be diagnosed with BPD [63]. A good beginning could be to determine the objectivity of previously suggested core outcome set for neonatal trials [64]. This kind of research effort would uniform the reporting practices and strengthen the evidence-based appraisal quality in future neonatal evidence syntheses.

We acknowledge some limitations. First, we are dependent on the reporting of the original meta-analyses, as they may have considered the influence of the subjectivity and objectivity of the outcome without mentioning it in their review methods. Second, the assessment of the proper reporting of blinding was a consensus between two authors and this may have a certain degree of subjectivity. Third, the assessment of the impact of inadequate risk of bias rating to GRADE rating without the original authors is solely dependent on the reporting information provided in by the original authors. Nonetheless, these limitations are well-known issues of meta-epidemiological project.

We found that the assessment of risk of bias due to blinding is limited in meta-analyses focusing on neonatal ventilation. Wrong assessment of blinding may lead to an unjustified downgrading of evidence certainty, and thus, may cause direct influence on clinical practices globally. Future meta-analyses should provide a properly conducted risk of bias assessment, and consider the peculiar context of some objective neonatal outcomes, when assessing the influence of blinding.

Statement of ethics is not applicable because this study is based exclusively on published literature.

I.K. and K.R. have none to report. D.D.L. received consultancy and lecture fees from Chiesi Farmaceutici, Getinge, Vyaire, Radiometer, Medtronic, Astra Zeneca, Boehringer Ingelheim, Airway Therapeutics, Natus, Masimo; he also has equity options from Ophirex Ltd. All these were unrelated to the present work. M.R.G. received a lecture fee from Sanofi, unrelated to this work.

None to report related to this work.

I.K. and D.D.L. had the original idea. I.K. and D.D.L. drafted the protocol. I.K. and K.R. performed the literature screening and data extractions. I.K. and D.D.L. performed analyses. M.R.G. provided methodological guidance to these. I.K. wrote the initial draft. All authors participated for commenting and critical revisions of the manuscript. All authors have approved the final version to be submitted.

All data generated during the review process is provided in the supplementary materials. Further inquiries can be directed to the corresponding author.