Background: Assessing the quality of clinical research is a key evidence-based practice skill. Clinicians, guideline producers, policy makers, service commissioners, and families need to have a sense of the validity, applicability, and certainty of research evidence when determining how it should inform their decision-making and practice. Methods: We consider the various methodological and study design factors that contribute to the validity and applicability of clinical research findings. We describe the “Grading of Recommendations Assessment, Development and Evaluation” (GRADE) methodology and discuss how this approach is used to assess and report certainty of evidence and strength of recommendations. Results: The randomized controlled trial (RCT) is the gold standard method for assessing interventions because randomization balances prognostic characteristics between comparison groups. The GRADE approach considers evidence from RCTs as high quality, but acknowledges that the quality and level of certainty of trial evidence may be “downgraded” based on consideration of threats across 5 domains: risk of bias in included trials, inconsistency between trials in outcome estimates, indirectness of the evidence, imprecision of estimates, and likelihood of publication bias. Conclusions: Structured critical appraisal using GRADE methods to assess risk of bias and other threats to the internal and external validity of RCTs and systematic reviews and meta-analyses of their data facilitates transparency and consistency in using evidence to inform policy and practice.

1.
McGuire
W
,
Halliday
HL
.
The research cycle: improving care and outcomes for newborn infants
.
Neonatology
.
2018
;
114
(
1
):
2
6
. .
2.
Soll
RF
,
McGuire
W
.
Evidence-based practice: improving the quality of perinatal care
.
Neonatology
.
2019
;
116
(
3
):
193
8
. .
3.
Guyatt
GH
,
Oxman
AD
,
Kunz
R
,
Vist
GE
,
Falck-Ytter
Y
,
Schünemann
HJ
.
What is “quality of evidence” and why is it important to clinicians?
BMJ
.
2008
;
336
(
7651
):
995
8
. .
4.
Guyatt
GH
,
Oxman
AD
,
Vist
GE
,
Kunz
R
,
Falck-Ytter
Y
,
Alonso-Coello
P
, et al.
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
.
BMJ
.
2008
;
336
(
7650
):
924
6
. .
5.
Alonso-Coello
P
,
Schünemann
HJ
,
Moberg
J
,
Brignardello-Petersen
R
,
Akl
EA
,
Davoli
M
, et al.
GRADE evidence to decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: introduction
.
BMJ
.
2016
;
353
:
i2016
. .
6.
World Health Organization
.
Who handbook for guideline development
. 2nd ed.
Geneva
:
World Health Organization
;
2014
.
7.
Gale
C
,
McGuire
W
,
Juszczak
E
.
Randomised controlled trials for informing perinatal care
.
Neonatology
.
2020
;
117
(
1
):
8
14
. .
8.
Odgaard-Jensen
J
,
Vist
GE
,
Timmer
A
,
Kunz
R
,
Akl
EA
,
Schünemann
H
, et al.
Randomisation to protect against selection bias in healthcare trials
.
Cochrane Database Syst Rev
.
2011
;
4
(
4
):
MR000012
. .
9.
Sterne
JAC
,
Savović
J
,
Page
MJ
,
Elbers
RG
,
Blencowe
NS
,
Boutron
I
, et al.
RoB 2: a revised tool for assessing risk of bias in randomised trials
.
BMJ
.
2019
;
366
:
l4898
. .
10.
Kirpalani
H
.
Control of bias has been recognized as important for a long time: but are we finally in control?
Neonatology
.
2019
;
116
(
2
):
185
7
. .
11.
Lai
NM
,
Ong
JMJ
,
Chen
KH
,
Chaiyakunapruk
N
,
Ovelman
C
,
Soll
R
.
Are neonatal trials better conducted and reported over the last 6 decades? An analysis on their risk-of-bias status in Cochrane reviews
.
Neonatology
.
2019
;
116
(
2
):
123
31
. .
12.
Savovic
J
,
Jones
H
,
Altman
D
,
Harris
R
,
Juni
P
,
Pildal
J
, et al.
Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies
.
Health Technol Assess
.
2012
;
16
(
35
):
1
82
.
13.
DeMauro
SB
,
Bellamy
SL
,
Fernando
M
,
Hoffmann
J
,
Gratton
T
,
Schmidt
B
, et al.
Patient, family, and center-based factors associated with attrition in neonatal clinical research: a prospective study
.
Neonatology
.
2019
;
115
(
4
):
328
34
. .
14.
Tin
W
,
Fritz
S
,
Wariyar
U
,
Hey
E
.
Outcome of very preterm birth: children reviewed with ease at 2 years differ from those followed up with difficulty
.
Arch Dis Child Fetal Neonatal Ed
.
1998
;
79
(
2
):
F83
7
. .
15.
Chen
T
,
Li
C
,
Qin
R
,
Wang
Y
,
Yu
D
,
Dodd
J
, et al.
Comparison of clinical trial changes in primary outcome and reported intervention effect size between trial registration and publication
.
JAMA Netw Open
.
2019
;
2
(
7
):
e197242
. .
16.
Furukawa
TA
,
Watanabe
N
,
Omori
IM
,
Montori
VM
,
Guyatt
GH
.
Association between unreported outcomes and effect size estimates in Cochrane meta-analyses
.
JAMA
.
2007
;
297
(
5
):
468
70
. .
17.
Jones
CW
,
Keil
LG
,
Holland
WC
,
Caughey
MC
,
Platts-Mills
TF
.
Comparison of registered and published outcomes in randomized controlled trials: a systematic review
.
BMC Med
.
2015
;
13
:
282
. .
18.
Bialy
L
,
Vandermeer
B
,
Lacaze-Masmonteil
T
,
Dryden
DM
,
Hartling
L
.
A meta-epidemiological study to examine the association between bias and treatment effects in neonatal trials
.
Evid Based Child Health
.
2014
;
9
(
4
):
1052
9
. .
19.
Mustafa
RA
,
Santesso
N
,
Brozek
J
,
Akl
EA
,
Walter
SD
,
Norman
G
, et al.
The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses
.
J Clin Epidemiol
.
2013
;
66
(
7
):
736
5
. .
20.
Guyatt
GH
,
Oxman
AD
,
Kunz
R
,
Brozek
J
,
Alonso-Coello
P
,
Rind
D
, et al.
GRADE guidelines 6. Rating the quality of evidence-imprecision
.
J Clin Epidemiol
.
2011
;
64
(
12
):
1283
93
. .
21.
Xiong
T
,
Maheshwari
A
,
Neu
J
,
Ei-Saie
A
,
Pammi
M
.
An overview of systematic reviews of randomized-controlled trials for preventing necrotizing enterocolitis in preterm infants
.
Neonatology
.
2020
;
117
(
1
):
46
56
. .
22.
Thanigainathan
S
,
Abiramalatha
T
.
Early fortification of human milk versus late fortification to promote growth in preterm infants
.
Cochrane Database Syst Rev
.
2020
;
7
(
7
):
CD013392
. .
23.
Seidler
AL
,
Gyte
GML
,
Rabe
H
,
Díaz-Rossello
JL
,
Duley
L
,
Aziz
K
, et al.
International liaison committee on resuscitation neonatal life support task force. Umbilical cord management for newborns <34 weeks’ gestation: a meta-analysis
.
Pediatrics
.
2021
;
147
(
3
):
e20200576
.
24.
Guyatt
GH
,
Oxman
AD
,
Kunz
R
,
Woodcock
J
,
Brozek
J
,
Helfand
M
, et al.
GRADE Working Group. GRADE guidelines: 7. Rating the quality of evidence-inconsistency
.
J Clin Epidemiol
.
2011
;
64
(
12
):
1294
302
.
25.
Sweet
DG
,
Carnielli
V
,
Greisen
G
,
Hallman
M
,
Ozek
E
,
Te Pas
A
, et al.
European consensus guidelines on the management of respiratory distress syndrome: 2019 update
.
Neonatology
.
2019
;
115
(
4
):
432
50
. .
26.
Lui
K
,
Jones
LJ
,
Foster
JP
,
Davis
PG
,
Ching
SK
,
Oei
JL
, et al.
Lower versus higher oxygen concentrations titrated to target oxygen saturations during resuscitation of preterm infants at birth
.
Cochrane Database Syst Rev
.
2018
;
5
(
5
):
CD010239
. .
27.
Brown
JV
,
Moe-Byrne
T
,
Harden
M
,
McGuire
W
.
Lower versus higher oxygen concentration for delivery room stabilisation of preterm neonates: systematic review
.
PLoS One
.
2012
;
7
(
12
):
e52033
. .
28.
Janvier
A
,
Bourque
CJ
,
Dahan
S
,
Robson
K
,
Barrington
KJ
;
On behalf of the Partenariat Famille (PAF) Team
.
Integrating parents in neonatal and pediatric research
.
Neonatology
.
2019
;
115
(
4
):
283
91
.
29.
Lai
NM
,
Leom
DYX
,
Chow
WL
,
Chen
KH
,
Lin
PH
,
Chaiyakunapruk
N
, et al.
How often are patient-important outcomes represented in neonatal randomized controlled trials? An analysis of Cochrane neonatal reviews
.
Neonatology
.
2020
;
117
(
4
):
428
35
. .
30.
Lai
NM
,
Lee
SWH
,
Wai
SX
,
Teh
ZW
,
Chan
MY
,
Lim
YS
, et al.
The choice of population and outcomes in neonatal trials on hyperbilirubinemia: are they relevant? An analysis of Cochrane neonatal reviews
.
Neonatology
.
2020
;
117
(
6
):
687
93
. .
31.
Kaplan
M
,
Hammerman
C
.
Surrogate outcomes: an inevitable limitation of hyperbilirubinemia-driven randomized controlled trials
.
Neonatology
.
2020
;
117
(
6
):
694
6
. .
32.
Okumura
A
,
Ichimura
S
,
Hayakawa
M
,
Arai
H
,
Maruo
Y
,
Kusaka
T
, et al.
Neonatal jaundice in preterm infants with bilirubin encephalopathy
.
Neonatology
.
2021 Mar 19
. [Online ahead of print]. .
33.
SIFT Investigators Group
.
Early enteral feeding strategies for very preterm infants: current evidence from Cochrane reviews
.
Arch Dis Child Fetal Neonatal Ed
.
2013
;
98
(
6
):
F470
2
. .
34.
Dorling
J
,
Hewer
O
,
Hurd
M
,
Bari
V
,
Bosiak
B
,
Bowler
U
, et al.
Two speeds of increasing milk feeds for very preterm or very low-birthweight infants: the SIFT RCT
.
Health Technol Assess
.
2020
;
24
(
18
):
1
94
. .
35.
Van Wyk
L
,
Tooke
L
,
Dippenaar
R
,
Rhoda
N
,
Lloyd
L
,
Holgate
S
, et al.
Optimal ventilation and surfactant therapy in very-low-birth-weight infants in resource-restricted regions
.
Neonatology
.
2020
;
117
(
2
):
217
24
. .
36.
Roberts
D
,
Brown
J
,
Medley
N
,
Dalziel
SR
.
Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth
.
Cochrane Database Syst Rev
.
2017
;
3
(
3
):
CD004454
. .
37.
WHO ACTION Trials Collaborators
;
Oladapo
OT
,
Vogel
JP
,
Piaggio
G
,
Nguyen
MH
,
Althabe
F
, et al.
Antenatal dexamethasone for early preterm birth in low-resource countries
.
N Engl J Med
.
2020
;
383
:
2514
25
.
38.
Galsworthy
MJ
,
Hristovski
D
,
Lusa
L
,
Ernst
K
,
Irwin
R
,
Charlesworth
K
, et al.
Academic output of 9 years of EU investment into health research
.
Lancet
.
2012
;
380
(
9846
):
971
2
. .
39.
Walsh
V
,
Oddie
S
,
McGuire
W
.
Ethical issues in perinatal clinical research
.
Neonatology
.
2019
;
116
(
1
):
52
7
. .
40.
Hopewell
S
,
Clarke
M
,
Stewart
L
,
Tierney
J
.
Time to publication for results of clinical trials
.
Cochrane Database Syst Rev
.
2007
;
2
(
2
):
MR000011
. .
41.
McGuire
W
,
Halliday
HL
.
Stopping trials early: responsibility and risks
.
Neonatology
. Forthcoming
2021
.
42.
Guyatt
GH
,
Oxman
AD
,
Montori
V
,
Vist
G
,
Kunz
R
,
Brozek
J
, et al.
GRADE guidelines: 5. Rating the quality of evidence-publication bias
.
J Clin Epidemiol
.
2011
;
64
(
12
):
1277
82
. .
43.
Sharif
S
,
Meader
N
,
Oddie
SJ
,
Rojas-Reyes
MX
,
McGuire
W
.
Probiotics to prevent necrotising enterocolitis in very preterm or very low birth weight infants
.
Cochrane Database Syst Rev
.
2020
;
10
:
CD005496
. .
44.
Roberts
I
,
Ker
K
,
Edwards
P
,
Beecher
D
,
Manno
D
,
Sydenham
E
.
The knowledge system underpinning healthcare is not fit for purpose and must change
.
BMJ
.
2015
;
350
:
h2463
. .
45.
Alonso-Coello
P
,
Oxman
AD
,
Moberg
J
,
Brignardello-Petersen
R
,
Akl
EA
,
Davoli
M
, et al.
GRADE evidence to decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: clinical practice guidelines
.
BMJ
.
2016
;
353
:
i2089
. .
46.
Guyatt
GH
,
Oxman
AD
,
Kunz
R
,
Falck-Ytter
Y
,
Vist
GE
,
Liberati
A
, et al.
Going from evidence to recommendations
.
BMJ
.
2008
;
336
(
7652
):
1049
51
. .
Copyright / Drug Dosage / Disclaimer
Copyright: All rights reserved. No part of this publication may be translated into other languages, reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, microcopying, or by any information storage and retrieval system, without permission in writing from the publisher.
Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug.
Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.
You do not currently have access to this content.