Background: Several tools have been created to assess competence in bronchoscopy. However, educational guidelines still use an arbitrary number of performed procedures to decide when basic competency is acquired. Objectives: The purpose of this study was to define pass/fail scores for two bronchoscopy assessment tools, and investigate how these scores relate to physicians’ experience regarding the number of bronchoscopy procedures performed. Methods: We studied two assessment tools and used two standard setting methods to create cut scores: the contrasting-groups method and the extended Angoff method. In the first we compared bronchoscopy performance scores of 14 novices with the scores of 14 experienced consultants to find the score that best discriminated between the two groups. In the second we asked an expert group of 7 experienced bronchoscopists to judge how a borderline trainee would perform on each item of the test. Results: Using the contrasting-groups method we found a standard that would fail all novices and pass all consultants. A clear pass related to prior experience of 75 procedures. The consequences of using the extended Angoff method were also acceptable: all trainees who had performed less than 50 bronchoscopies failed the test and all consultants passed. A clear pass related to 80 procedures. Conclusions: Our proposed pass/fail scores for these two methods seem appropriate in terms of consequences. Prior experience with the performance of 75 and 80 bronchoscopies, respectively, seemed to ensure basic competency. In the future objective assessment tools could become an important aid in the certification of physicians performing bronchoscopies.

Keywords:
Assessment tools, Bronchoscopy, Education, Procedural training, Standard setting
1.
Epstein RM, Hundert EM: Defining and assessing professional competence. JAMA 2002;287:226–235.
2.
Reznick RK, MacRae H: Teaching surgical skills – changes in the wind. N Engl J Med 2006;355:2664–2669.
3.
Epstein RM: Assessment in medical education. N Engl J Med 2007;356:387–396.
4.
Rhodes RS, Biester TW: Certification and maintenance of certification in surgery. Surg Clin North Am 2007;87:825–836, vi.
5.
Department of Health. Trust, Assurance and Safety – The Regulation of Health Professionals in the 21st Century. London, HM Stationery Office, 2007.
6.
Fraser SA, Klassen DR, Feldman LS, et al: Evaluating laparoscopic skills: setting the pass/fail score for the MISTELS system. Surg Endosc 2003;17:964–967.
7.
Jelovsek JE, Walters MD, Korn A, et al: Establishing cutoff scores on assessments of surgical skills to determine surgical competence. Am J Obstet Gynecol 2010;203:81–86.
8.
Schindler N, Corcoran J, DaRosa D: Description and impact of using a standard-setting method for determining pass/fail scores in a surgery clerkship. Am J Surg 2007;193:252–257.
9.
Norcini J, Guille R: Combining tests and setting standards; in Norman GR, Van der Vleuten CMP, Newble DI (ed): International Handbook of Research in Medical Education. Amsterdam, Kluwer Academic Publishers, 2002.
10.
Kane MT: Conception and status of validation in setting standards; in Cizek GJ (ed): Setting Performance Standards – Concepts, Methods, and Perspectives. Mahwah, Lawrence Erlbaum, 2001, pp 53–88.
11.
Yudkowsky R, Downing SM, Tekian A: Standard setting; in Downing SM, Yudkowsky R (eds): Assessment in Health Professions Education. New York, Routledge, 2009, pp 119–148.
12.
Ernst A, Silvestri GA, Johnstone D: Interventional pulmonary procedures: guidelines from the American College of Chest Physicians. Chest 2003;123:1693–1717.
13.
British Thoracic Society guidelines on diagnostic flexible bronchoscopy. Thorax 2001;56(suppl 1):i1–i21.
14.
ACGME Program Requirements for Fellowship Education in Pulmonary Disease and Critical Care Medicine (Internal Medicine). IM – Pulmonary Critical Care, 2011.
15.
Ost D, DeRosiers A, Britt EJ, et al: Assessment of a bronchoscopy simulator. Am J Respir Crit Care Med 2001;164:2248–2255.
16.
Wahidi MM, Silvestri GA, Coakley RD, et al: A prospective multi-center study of competency metrics and educational interventions in the learning of bronchoscopy among starting pulmonary fellows. Chest 2010;137:1040–1049.
17.
Crawford SW, Colt HG: Virtual reality and written assessments are of potential value to determine knowledge and skill in flexible bronchoscopy. Respiration 2004;71:269–275.
18.
Lunn W, Garland R, Gryniuk L, et al: Reducing maintenance and repair costs in an interventional pulmonology program. Chest 2005;127:1382–1387.
19.
Rozman A, Duh S, Petrinec-Primozic M, et al: Flexible bronchoscope damage and repair costs in a bronchoscopy teaching unit. Respiration 2009;77:325–330.
20.
Konge L, Arendrup H, von Buchwald C, et al: Using Performance in Multiple Simulated Scenarios to Assess Bronchoscopy Skills. Respiration 2011;81:483–490.
21.
Konge L, Larsen KR, Clementsen P, et al: Reliable and valid assessment of clinical bronchoscopy performance. Respiration 2012; 83:53–60.
22.
Livingston S, Zieky M: Passing Scores: A Manual for Setting Standards of Performance on Educational and Occupational Tests. Princeton, Educational Testing Service, 1982.
23.
Thompson AB, Huerta G, Robbins RA, et al: The bronchitis index. A semiquantitative visual scale for the assessment of airways inflammation. Chest 1993;103:1482–1488.
24.
Angoff WH: Scales, norms, and equivalent scores; in Thorndike RL (ed): Educational Measurement. Washington, American Council on Education, 1971, pp 508–600.
25.
Hambleton RM, Plake BS: Using an extended Angoff procedure to set standards on complex performance assessments. Appl Meas Educ 1995;8:41–56.
26.
Raymond MR, Reid JB: Selecting and training participants for standard setting; in Cizek GJ (ed): Setting Performance Standards – Concepts, Methods, and Perspectives. Mahwah, Lawrence Erlbaum, 2001, pp 119–157.
27.
Sturm LP, Windsor JA, Cosman PH, et al: A systematic review of skills transfer after surgical simulation training. Ann Surg 2008;248:166–179.
28.
Kane MT, Crooks TJ, Cohen AS: Designing and evaluating standard-setting procedures for licensure and certification tests. Adv Health Sci Educ Theory Pract 1999;4:195–207.
29.
Konge L, Arendrup H, Buchwald CV, et al: Virtual reality simulation of basic pulmonary procedures. J Bronchol Intervent Pulmonol 2011;18:38–41.
© 2011 S. Karger AG, Basel
2011
Copyright / Drug Dosage / Disclaimer
Copyright: All rights reserved. No part of this publication may be translated into other languages, reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, microcopying, or by any information storage and retrieval system, without permission in writing from the publisher.
Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug.
Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.
You do not currently have access to this content.