Background: In an era of increased emphasis on patient safety and competency-based education, demonstration of significant variability in the teaching of flexible bronchoscopy has led to initiatives for new standardized curricula and assessment tools. Objectives: To evaluate the efficacy of such curricula and to assess bronchoscopic skill, 2 measuring instruments have been developed: the Bronchoscopy Skills and Tasks Assessment Tool (BSTAT) and the Bronchoscopy Step-by-Step Evaluation Tool (BSET). We studied the validity and reliability of these 2 instruments. Methods: Two independent testers simultaneously scored 22 volunteer participants at 3 levels [novice (n = 7), fellow (n = 8) and attending (n = 7)] on a virtual reality bronchoscopy simulator using the 2 instruments; each participant was tested twice, in 2 separate sessions. Intertester and test-retest reliability were analyzed with intraclass correlations (ICC); ANOVA was used to assess concurrent validity based on the subjects’ expected skill level. Results: The ICCs between the testers were 0.98 for both the BSTAT and BSET. Comparison of the scores between the sessions showed high test-retest reliability by ICC (0.86 and 0.85 for BSTAT and BSET respectively), with a small yet statistically significant learning effect. The novice group’s scores were lower than the fellows’ and attendings’ (p < 0.001) for both the BSTAT and BSET; the fellows’ scores were consistently lower than the attendings’ on both tests, yet the differences were not statistically significant. Conclusion: This validation study of 2 objective tests of bronchoscopic skill demonstrated high reliability and concurrent validity. These instruments can now be used to evaluate the effectiveness of new competency-based bronchoscopy curricula.

Keywords:
Bronchoscopy, Education, Competency, Safety, Computer simulation
1.
Pastis N, Nietert P, Silvestri G, ACCP Interventional Chest/Diagnostic Procedures Network Steering Committee: Variation in training for interventional pulmonary procedures among US pulmonary/critical care fellowships, a survey of fellowship directors. Chest 2005;127:1614–1621.
2.
Ernst A, Silvestri G, Johnstone D, ACCP Interventional Chest/Diagnostic Procedures Network Steering Committee: Interventional pulmonary procedures: guidelines from the American College of Chest Physicians. Chest 2003;123:1693–1717.
3.
Haponik EF, Russell GB, Beamis JF, et al: Bronchoscopy training: current fellows’ experiences and some concerns for the future. Chest 2000;118:572–573.
4.
Bolliger CT, Mathur PN, Beamis JF, et al, European Respiratory Society/American Thoracic Society: ERS/ATS statement on interventional pulmonology. Eur Respir J 2002;19:356–373.
5.
Kovitz KL: From the President of the American Association for Bronchology: the future of interventional pulmonology. J Bronchol 2006;13:107–108.
6.
Lund ME: Interventional pulmonology competency standards: the time is now. J Bronchol 2006;13:1–5.
7.
Prakash UBS: Guidelines for training and practice of interventional pulmonology by the numbers? J Bronchol 2003;10:169–173.
8.
Alvarez F, Burger C, Grinton S: Competencies in pulmonary procedures. Chest 2004;125:800; author reply 800–801.
9.
Prakash UBS: Bronchoscopy: a global perspective. J Bronchol 2006;13:177–178.
10.
Carraccio C, Wolfsthal SD, Englander R, et al: Shifting paradigms: from Flexner to competencies. Acad Med 2002;77:361–367.
11.
Long DM: Competency-based residency training: the next advance in graduate medical education. Acad Med 2000;75:1178–1183.
12.
Ende J, Kelley M, Sox H: The Federated Council of Internal Medicine’s resource guide for residency education: an instrument for curricular change. Ann Intern Med 1997;127:454–457.
13.
Mayor S: UK royal colleges publish competency-based curriculums. BMJ 2002;325:1378.
14.
Mandin H, Duphinee WD: Conceptual guidelines for developing and maintaining curriculum and examination objectives: the experience of the Medical Council of Canada. Acad Med 2000;75:1031–1037.
15.
Accreditation Council for Graduate medical Education: ACGME Outcome project. 1999. http://www.acgme.org/outcome/ (accessed July 10, 2007).
16.
Association of American Medical Colleges: Report 1: Learning objectives for medical student education: guidelines for medical schools – Medical School Objectives Project. 1998. http://www.aamc.org/meded/msop/ (accessed July 10, 2007).
17.
Harden RN, Crosby JR, Davis MH: AMEE Guide No 14: outcome-based education. 1. An introduction to outcome-based education. Med Teach 1999;21:7–14.
18.
Smith SR, Dollase R: AMEE Guide No 14: outcome-based education. 2. Planning, implementing and evaluating a competency-based curriculum. Med Teach 1999;21:15–22.
19.
Association for Medical Education in Europe: AMEE Education Guide No 14: outcome-based education. 1999. http://www.amee.org/index.asp?tm=43 (accessed July 10, 2007).
20.
Feldman LS, Hagarty SE, Ghitulescu G, et al: Relationship between objective assessment of technical skills and subjective in-training evaluations in surgical residents. J Am Coll Surg 2004;198:105–110.
21.
Campos-Outcalt D, Watkins A, Ulginti J, et al: Correlations of family medicine clerkship evaluations and objective structured clinical examination scores and residency director ratings. Fam Med 1999;31:90–94.
22.
Kahn MJ, Merill WW, Anderson DS, et al: Residency program director evaluations do not correlate with performance on a required 4th-year objective structured examination. Teach Learn Med 2001;13:9–12.
23.
Davis DA, Mazmanian PE, Fordis M, et al: Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 2006;296:1094–1102.
24.
Colt HG, Crawford SW, Galbraith O: Virtual reality bronchoscopy simulation: a revolution in procedural training. Chest 2001;120:1333–1339.
25.
Moorthy K, Smith S, Brown T, et al: Evaluation of virtual reality bronchoscopy as a learning and assessment tool. Respiration 2003;70:195–199.
26.
Ost D, DeRosiers A, Britt EJ, et al: Assessment of a bronchoscopy simulator. Am J Resp Crit Care 2001;164:2248–2255.
27.
Crawford SW, Colt HG: Virtual reality and written assessments are of potential value to determine knowledge and skill in flexible bronchoscopy. Respiration 2004;71:269–275.
28.
Dawson B, Trapp RG: Basic and Clinical Biostatistics, ed 4. New York, McGraw Hill Medical, 2004.
29.
Aggarwal R, Grantcharov TP, Eriksen JR, et al: An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 2006;244:310–314.
30.
Patel AD, Gallagher AG, Nicholson WJ, et al: Learning Curves and reliability measures for virtual reality simulation in the performance assessment of carotid angiography. J Am Coll Cardiol 2006;47:1796–1802.
31.
Sokal RR, Rohlf FJ: Biometry, ed 3. New York, Freeman, 1995.
32.
Streiner DL: Being inconsistent about consistency: when coefficient alpha does and doesn’t matter. J Pers Assess 2003;80:217–222.
33.
Brasel KJ, Bragg D, Simpson DE, et al: Meeting the Accreditation Council for Graduate Medical Education competencies using established residency training program assessment tools. Am J Surg 2004;188:9–12.
34.
Scott DJ, Bergen PC, Rege RV, et al: Laparoscopic training on bench models: better and more cost effective than operating room experience? J Am Coll Surg 2000;191:272–283.
35.
Haque S, Srinivasan S: A meta-analysis of the training effectiveness of virtual reality surgical simulators. IEEE Trans Inf Technol Biomed 2006;10:51–58.
36.
Issenberg SB, McGaghie WC, Petrusa ER, et al: Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach 2005;27:10–28.
37.
Seymour NE, Gallagher AG, Roman SA, et al: Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 2002;236:458–464.
38.
Schijven MP, Jakimowicz JJ, Broeders IAMJ, et al: The Eindhoven laparoscopic cholecystectomy training course – improving operating room performance using virtual reality training. Surg Endosc 2005;19:1220–1226.
39.
Park J, MacRae H, Musselman LJ, et al: Randomized controlled trial of virtual reality simulator training: transfer to live patients. Am J Surg 2007;194:205–211.
40.
Lehmann KS, Ritz JP, Maass H, et al: A prospective randomized study to test the transfer of basic psychomotor skills for virtual reality to physical reality in a comparable training setting. Ann Surg 2005;241:442–449.
41.
Grantcharov TP, Kristiansen VB, Bendix J, et al: Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg 2004;91:146–150.
42.
Sedlack RE, Kolars JC, Alexander JA: Computer simulation training enhances patient comfort during endoscopy. Clin Gastroenterol Hepatol 2004;2:348–352.
43.
Martin M, Vashisht B, Frezza E, Ferone T, Lopez B, et al: Competency-based instruction in critical invasive skills improves both resident performance and patient safety. Surgery 1998;124:313–317.
44.
Fried MP, Satava R, Weghorst S, et al: Identifying and reducing errors with surgical simulation. Qual Saf Health Care 2004;13(suppl 1):i19–i26.
45.
Ziv A, Ben-David S, Ziv M: Simulation based-medical education: an opportunity to learn from errors. Med Teach 2005;27:193–199.
46.
Mahmood T, Darzi A: The learning curve for a colonoscopy simulator in the absence of any feedback – No feedback, no learning. Surg Endosc 2004;18:1224–1230.
47.
Davoudi M, Colt HG: Bronchoscopy simulation: a brief review. Adv Health Sci Educ Theory Pract, in press. DOI: 10.1007/s10459–007–9095-x.
© 2008 S. Karger AG, Basel
2008
Copyright / Drug Dosage / Disclaimer
Copyright: All rights reserved. No part of this publication may be translated into other languages, reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, microcopying, or by any information storage and retrieval system, without permission in writing from the publisher.
Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug.
Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.
You do not currently have access to this content.