Background and Purpose: The National Institutes of Health Stroke Scale (NIHSS) is a widely used measure of neurological function in clinical trials and patient assessment; inter-rater scoring variability could impact communications and trial power. The manner in which the rater certification test is scored yields multiple correct answers that have changed over time. We examined the range of possible total NIHSS scores from answers given in certification tests by over 7,000 individual raters who were certified. Methods: We analyzed the results of all raters who completed one of two standard multiple-patient videotaped certification examinations between 1998 and 2004. The range for the correct score, calculated using NIHSS ‘correct answers’, was determined for each patient. The distribution of scores derived from those who passed the certification test then was examined. Results: A total of 6,268 raters scored 5 patients on Test 1; 1,240 scored 6 patients on Test 2. Using a National Stroke Association (NSA) answer key, we found that correct total scores ranged from 2 correct scores to as many as 12 different correct total scores. Among raters who achieved a passing score and were therefore qualified to administer the NIHSS, score distributions were even wider, with 1 certification patient receiving 18 different correct total scores. Conclusions: Allowing multiple acceptable answers for questions on the NIHSS certification test introduces scoring variability. It seems reasonable to assume that the wider the range of acceptable answers in the certification test, the greater the variability in the performance of the test in trials and clinical practice by certified examiners. Greater consistency may be achieved by deriving a set of ‘best’ answers through expert consensus on all questions where this is possible, then teaching raters how to derive these answers using a required interactive training module.

Keywords:
Stroke, NIH Stroke Scale, reliability, Scoring the NIHSS certification test
1.
Brott T, Adams HP Jr, Olinger CP, Marler JR, Barsan WG, Biller J, Spilker J, Holleran R, Eberle R, Hertzberg V, et al: Measurements of acute cerebral infarction: a clinical examination scale. Stroke 1989;20:864–870.
2.
Muir KW, Weir CJ, Murray GD, Povey C, Lees KR: Comparison of neurological scales and scoring systems for acute stroke prognosis. Stroke 1996;27:1817–1820.
3.
Goldstein LB, Bertels C, Davis JN: Interrater reliability of the NIH Stroke Scale. Arch Neurol 1989;46:660–662.
4.
Albanese MA, Clarke WR, Adams HP Jr, Woolson RF: Ensuring reliability of outcome measures in multicenter clinical trials of treatments for acute ischemic stroke. The program developed for the trial of ORG 10172 in acute stroke treatment (TOAST). Stroke 1994;25:1746–1751.
5.
Josephson SA, Hills NK, Johnston SC: NIH Stroke Scale reliability in ratings from a large sample of clinicians. Cerebrovasc Dis 2006;22:389–395.
6.
Lyden P, Raman R, Liu L, Grotta J, Broderick J, Olson S, Shaw S, Spilker J, Meyer B, Emr M, Warren M, Marler J: NIHSS training and certification using a new digital video disk is reliable. Stroke 2005;36:2446–2449.
7.
NINDS rt-PA Stroke Study Group: Tissue plasminogen activator for acute ischemic stroke. The National Institute of Neurological Disorders and Stroke rt-PA Stroke Study Group. N Engl J Med 1995;333:1581–1587.
8.
Furlan A, Higashida R, Wechsler L, Gent M, Rowley H, Kase C, Pessin M, Ahuja A, Callahan F, Clark WM, Silver F, Rivera F: Intra-arterial pro-urokinase for acute ischemic stroke. The PROACT II study: a randomized controlled trial. Prolyse in acute cerebral thromboembolism. JAMA 1999;282:2003–2011.
9.
IMS II Trial Investigators: The Interventional Management of Stroke (IMS) II Study. Stroke 2007;38:2127–2135.
10.
Clark WM, Wissman S, Albers GW, Jhamandas JH, Madden KP, Hamilton S: Recombinant tissue-type plasminogen activator (alteplase) for ischemic stroke 3–5 hours after symptom onset. The ATLANTIS study: a randomized controlled trial. Alteplase thrombolysis for acute noninterventional therapy in ischemic stroke. JAMA 1999;282:2019–2026.
11.
Clark WM, Williams BJ, Selzer KA, Zweifler RM, Sabounjian LA, Gammans RE: A randomized efficacy trial of citicoline in patients with acute ischemic stroke. Stroke 1999;30:2592–2597.
12.
Davis SM, Lees KR, Albers GW, Diener HC, Markabi S, Karlsson G, Norris J: Selfotel in acute ischemic stroke: possible neurotoxic effects of an NMDA antagonist. Stroke 2000;31:347–354.
13.
Del Zoppo GJ, Higashida RT, Furlan AJ, Pessin MS, Rowley HA, Gent M: PROACT: a phase II randomized trial of recombinant pro-urokinase by direct arterial delivery in acute middle cerebral artery stroke. PROACT investigators. Prolyse in acute cerebral thromboembolism. Stroke 1998;29:4–11.
14.
Grotta J: Lubeluzole treatment of acute ischemic stroke. The US and Canadian Lubeluzole Ischemic Stroke Study Group. Stroke 1997;28:2338–2346.
15.
Hacke W, Kaste M, Fieschi C, Toni D, Lesaffre E, von Kummer R, Boysen G, Bluhmki E, Hoxter G, Mahagne MH, et al: Intravenous thrombolysis with recombinant tissue plasminogen activator for acute hemispheric stroke. The European Cooperative Acute Stroke Study (ECASS). JAMA 1995;274:1017–1025.
16.
Hacke W, Kaste M, Fieschi C, von Kummer R, Davalos A, Meier D, Larrue V, Bluhmki E, Davis S, Donnan G, Schneider D, Diez-Tejedor E, Trouillas P: Randomised double-blind, placebo-controlled trial of thrombolytic therapy with intravenous alteplase in acute ischaemic stroke (ECASS II). Second European-Australasian Acute Stroke Study investigators. Lancet 1998;352:1245–1251.
17.
Johnston KC, Connors AF Jr, Wagner DP, Knaus WA, Wang X, Haley EC Jr: A predictive risk model for outcomes of ischemic stroke. Stroke 2000;31:448–455.
18.
Schlegel D, Kolb SJ, Luciano JM, Tovar JM, Cucchiara BL, Liebeskind DS, Kasner SE: Utility of the NIH Stroke Scale as a predictor of hospital disposition. Stroke 2003;34:134–137.
19.
Schlegel DJ, Tanne D, Demchuk AM, Levine SR, Kasner SE: Prediction of hospital disposition after thrombolysis for acute ischemic stroke using the National Institutes of Health Stroke Scale. Arch Neurol 2004;61:1061–1064.
20.
Lyden P, Brott T, Tilley B, Welch KM, Mascha EJ, Levine S, Haley EC, Grotta J, Marler J: Improved reliability of the NIH Stroke Scale using video training. NINDS tPA Stroke Study Group. Stroke 1994;25:2220–2226.
21.
American Stroke Association: NIH Stroke Scale training online, 2007.
22.
National Stroke Association: NIH Stroke Scale, 2007.
23.
Ende J: Feedback in clinical medical education. JAMA 1983;250:777–781.
24.
Knowles M, Holton E, Swanson R: The Adult Learner. Houston/TX, Gulf Publishing, 1998.
25.
Wood BP: Feedback: a key feature of medical training. Radiology 2000;215:17–19.
26.
Kasner SE: Clinical interpretation and use of stroke scales. Lancet Neurol 2006;5:603–612.
27.
Lyden PD, Lu M, Levine SR, Brott TG, Broderick J: A modified National Institutes of Health Stroke Scale for use in stroke clinical trials: preliminary reliability and validity. Stroke 2001;32:1310–1317.
28.
Quinn TJ, Lees KR, Hardemark HG, Dawson J, Walters MR: Initial experience of a digital training resource for modified Rankin scale assessment in clinical trials. Stroke 2007;38:2257–2261.
29.
Quinn TJ, Dawson J, Walters MR, Lees KR: Variability in modified Rankin scoring across a large cohort of international observers. Stroke 2008;39:2975–2979.
© 2009 S. Karger AG, Basel
2009
Copyright / Drug Dosage / Disclaimer
Copyright: All rights reserved. No part of this publication may be translated into other languages, reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, microcopying, or by any information storage and retrieval system, without permission in writing from the publisher.
Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug.
Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.
You do not currently have access to this content.