Background: Endobronchial ultrasound (EBUS) is a revolutionary diagnostic procedure. There is currently no accepted method of assessing EBUS technical skill or competency. Objectives: This study aimed to validate a computer EBUS simulator in differentiating between operators of varying clinical EBUS experience. Methods: A convenience sample (n = 22) of bronchoscopists was separated into four cohorts based on previous bronchoscopy experience: group A = novice bronchoscopists, no EBUS experience (n = 4), group B = expert bronchoscopists, no EBUS experience (n = 5), group C = basic clinical EBUS training (n = 9), group D = EBUS experts (n = 4). After a standardized introduction session on the EBUS simulator, participants performed 2 simulated cases on an EBUS simulator with performance metrics measured by the simulator. Results: Significant differences between groups were noted for total procedure time, percentage of lymph nodes identified and percentage of successful biopsies (p < 0.05, ANOVA). Group D performed significantly better than all other groups for total procedure time and percentage of lymph nodes identified (p < 0.05). Group C performed significantly better than groups A and B for total procedure time, percentage of lymph nodes identified and percentage of successful biopsies (p < 0.05, ANOVA). Conclusions: An EBUS simulator can accurately discriminate between operators with different levels of clinical EBUS experience. EBUS simulators show promise as a tool for assessing training and evaluating competency.

Assessing competency in medical procedures is vital to patient care in order to ensure that an adequate pool of physicians competent in medical procedures exists to provide these important services. The assessment of procedural skills is also important in evaluating trainees’ progress during fellowship programs, in assessing learning during continuing medical education courses and assessing procedural skills before procedural privileges are approved within institutions. Surveys of pulmonary fellows and attending physicians demonstrate significant variation in pulmonary training experience [1,2,3,4,5], which may result in some new physicians being inadequately trained in basic procedural skills. Some studies have shown decreasing trends in the number of pulmonary procedures being performed [5,6], further decreasing the opportunity for pulmonary procedural training and emphasizing the need for methods of assessing procedural competency.

Endobronchial ultrasound (EBUS) is a minimally invasive pulmonary procedure used to diagnose pulmonary diseases by allowing visualization and safe, highly accurate biopsies of intrathoracic structures during bronchoscopy [7]. The American College of Chest Physicians guidelines for interventional pulmonary procedures states: ‘EBUS requires intensified training and practical experience in interpreting sonographic images, since the anatomic structures of the mediastinum are comparatively complex’ [8]. The European Respiratory Society/American Thoracic Society Taskforce recommends that trainees should be supervised for 40 EBUS procedures and should perform 25 procedures per year to maintain competency [9], although it is important to note that these recommendations are based on expert opinion.

Simulation training has been used for decades in the aviation and aerospace industries, where technical skills are learned and competency is assessed through simulation before learners are placed in real-life high-stakes situations. The use of simulation-based training for medical and surgical procedures has been demonstrated to be effective, cost-effective, and to increase patients’ comfort and safety [10,11,12,13,14]. Procedural simulators have been validated for assessing a variety of medical and surgical procedural skills including colonoscopy and laparoscopy [15,16]. As of 2010, successful completion of courses utilizing simulation training (Fundamentals of Laparoscopic Surgery and Advanced Trauma Life Support) are now required for certification by the American Board of Surgery [17]. Demonstrating skill transfer from the actual to the simulation environment is necessary if simulators are to be used to measure procedural skill. With a valid task simulator the simulation should mimic the actual activity, such that experts should outperform novices. In addition to having been shown to be a useful tool for enhancing learning in basic bronchoscopy [18,19], the use of a bronchoscopy simulator to measure basic bronchoscopy skill of learners has also been previously validated [20]. Similar studies have not been performed using an EBUS-specific simulator. This study aimed to validate a computer EBUS simulator in differentiating between operators of varying clinical EBUS experience.

Study Design

A prospective, nonrandomized comparative study of four groups of subjects was carried out:

Group A: novice bronchoscopists – junior pulmonary medicine trainees with >25 flexible bronchoscopy procedures experience, >4 months of pulmonary fellowship training and no EBUS experience.

Group B: expert bronchoscopists but novice EBUS operators – attending pulmonary physicians with >500 flexible bronchoscopy procedures experience, >5 years clinical practice, who perform standard transbronchial needle aspiration, and no EBUS experience.

Group C: basic clinical EBUS training – senior pulmonary medicine trainees or recent graduates (within 1 year) with >75 flexible bronchoscopy procedures experience and completed 1-month elective with the interventional pulmonary medicine service with ≥15 and ≤25 EBUS procedures experience.

Group D: expert EBUS operators – interventional pulmonologists or thoracic surgeons with >200 EBUS procedures experience.

There is currently no accepted definition of an expert bronchoscopist. In a study validating a basic flexible bronchoscopy simulator [20], the expert bronchoscopists had >500 procedures experience.

Similarly, there is currently no accepted definition of an expert EBUS operator. At the University of Calgary, the interventional pulmonary medicine fellow is considered an expert EBUS operator after completing the year-long fellowship, during which they complete >200 EBUS procedures.

Subjects received a 30-min introduction session to the EBUS simulator, and one practice case to familiarize them with the equipment. Before the practice case, subjects underwent a standardized education process to ensure they knew the different lymph node stations and what is considered to be a complete EBUS lymph node examination. In order to simplify and standardize the examinations, every case was assumed to be for lung cancer staging with a left-sided tumor, and only the 11R, 4R, 7, 4L and 11L stations were to be examined in all patients. Subjects were not allowed to perform the practice case until they demonstrated a complete understanding of a correct lymph node station examination and the associated airway and vascular anatomy by correctly answering a series of standardized questions (Appendix A). All subjects were then given the same two EBUS cases to complete with EBUS performance metrics recorded by the EBUS simulator.

EBUS Simulator Description

The EBUS simulator used for this study is the AccuTouch Flexible Bronchoscopy Simulator (CAE Healthcare, Montreal, Que., Canada), equipped with an EBUS module. This simulator has been described in detail previously [20,21]. The EBUS module consists of EBUS software and an EBUS needle system, but all other hardware components of the simulator remain the same (fig. 1, 2). At the time of completion of the manuscript there were no published studies utilizing such an EBUS simulator.

Fig. 1

The EBUS simulator. A = Proxy flexible bronchoscope; B = mannequin face with port for bronchoscope insertion; C = interface device; D = dual screen display; E = simulated EBUS biopsy needle.

Fig. 1

The EBUS simulator. A = Proxy flexible bronchoscope; B = mannequin face with port for bronchoscope insertion; C = interface device; D = dual screen display; E = simulated EBUS biopsy needle.

Close modal
Fig. 2

EBUS simulator dual screen display. A = White light bronchoscopic view; B = EBUS bronchoscopic view; C = simulated patient CT scan; D = EBUS simulator proctor (not used during this study); E = simulated patient vital signs.

Fig. 2

EBUS simulator dual screen display. A = White light bronchoscopic view; B = EBUS bronchoscopic view; C = simulated patient CT scan; D = EBUS simulator proctor (not used during this study); E = simulated patient vital signs.

Close modal

Measurement Tools

Subjects were given a pre-simulation questionnaire which assessed demographic information including computer, video game and bronchoscopy experience (Appendix B).

The EBUS simulator software comes with built-in metrics that are measured during each simulated case. These include: total procedure time, time to intubation, number of intubation attempts, number of times bronchoscope collided with closed vocal cords, number of vessel punctures, number of times needle was removed while suction was still on (contamination), number of times needle passed through far side of node, number of times needle advanced in bronchoscope without the correct use of the protective sheath (bronchoscope damage), quantity of lidocaine used, quantity of sedation used and vocal cord trauma. Two additional clinically relevant measurement metrics included were the percentage of lymph nodes correctly identified during lymph node ultrasound examination and the percentage of lymph nodes successfully biopsied.

Statistical Analysis

Three performance metrics were selected a priori as most relevant to the assessment of basic EBUS skill based on the authors’ experience in performing and teaching EBUS (primary outcome measures): total procedure time, percentage of lymph nodes correctly identified on lymph node examination and percentage of successful lymph node biopsies [successful lymph node biopsies/(failed lymph node biopsies + successful lymph node biopsies)].

The results of all metrics were analyzed between groups using the averaged performance from the two EBUS cases, with ANOVA comparison of means and post hoc analysis between individual groups with appropriate corrections for multiple comparisons with the least significant difference test.

Our sample size was based on convenience and availability of bronchoscopists of each skill level to participate in the study. Nevertheless, using a standard t test and including a minimum of 4 subjects per group should allow the detection of a 25% difference in the percentage of lymph nodes correctly identified on EBUS examination between groups (α of 0.05 and power of 0.80 with a standard deviation of 10%).

The study has received approval by the Calgary Health Research Ethics Board (ethics ID No. 22715) and all subjects gave written informed consent.

Twenty-two subjects from three centers were enrolled in the study: group A (n = 4), group B (n = 5), group C (n = 9) and group D (n = 4). All subjects were able to complete the study. Table 1 lists the demographic data of the subjects.

Table 1

Demographic data of study subjects

Demographic data of study subjects
Demographic data of study subjects

The outcome measures as they relate to EBUS and bronchoscopy experience level for the combined EBUS simulator test cases are shown in table 2. Significant differences between groups were noted for the 3 primary outcome measures of total procedure time, percentage of lymph nodes correctly identified on EBUS examination and percentage of successful biopsies (p < 0.05, ANOVA; table 2; fig. 3). Mean total procedure times for groups A, B, C and D were 34.67, 28.80, 20.11 and 14.96 min, respectively. Mean percentages of lymph nodes identified for groups A, B, C and D were 47.8, 40.8, 67.1 and 91.1%, respectively. Mean percentages of successful biopsies for groups A, B, C and D were 64.2, 63.9, 91.1 and 100%, respectively. Comparisons between individual groups were performed with statistics reported in table 2.

Table 2

Main results

Main results
Main results
Fig. 3

Mean total procedure time (min), mean percentage of lymph nodes correctly identified on EBUS examination and mean percentage of successful lymph node biopsies for each group (A = novice bronchoscopists; B = expert bronchoscopists; C = basic clinical EBUS; D = expert EBUS). Error bars are ± 1 SD.

Fig. 3

Mean total procedure time (min), mean percentage of lymph nodes correctly identified on EBUS examination and mean percentage of successful lymph node biopsies for each group (A = novice bronchoscopists; B = expert bronchoscopists; C = basic clinical EBUS; D = expert EBUS). Error bars are ± 1 SD.

Close modal

Significant differences between groups were also noted in the secondary metrics of number of intubation attempts, number of collisions with closed vocal cords, number of contaminated samples, number of penetrations beyond far side of lymph node and number of completely missed lymph nodes (table 2). There was no significant difference in time to intubation between any of the groups, and no vascular punctures or bronchoscope damage in any of the groups. There were no significant differences between groups for the amount of lidocaine or sedation used and the only procedural complication was one case of vocal cord trauma by one of the novice bronchoscopists (group A).

Computer experience and video game experience were not significantly related to the outcome measures. There were no significant differences in computer experience and video game experience between groups.

This is the first study to validate the use of an EBUS simulator in the assessment of EBUS skill level. Significant differences between groups of various clinical EBUS experience were noted for all of the three primary outcome measures selected. The expert EBUS operators outperformed all other groups for all primary outcome measures, except for percentage of successful biopsies where there was no statistically significant difference when compared to the basic clinical EBUS group. The basic clinical EBUS group also outperformed the novice and expert bronchoscopist groups in all primary outcome measures. Interestingly, the only significant difference noted between the novice and expert bronchoscopists (both groups with no clinical EBUS experience) was a shorter procedure time for the expert bronchoscopists, suggesting that EBUS is a skill separate from that of basic flexible bronchoscopy. Not only can the EBUS simulator differentiate between expert EBUS operators and bronchoscopists with no EBUS experience, it can also differentiate between expert EBUS operators and those with more basic clinical EBUS experience. It would appear that the skills acquired during clinical EBUS training and practice are directly transferable and measurable by the EBUS simulator.

Prior studies of bronchoscopy simulation have demonstrated the utility of using inanimate models combined with didactic teaching, animal models and computer simulation [19,23,24,25]. A significant limitation of inanimate models is a lack of realism. The costs, ethical concerns and anatomical differences between species are significant limitations with the use of animal models. Bronchoscopy simulators have been shown to improve the establishment of fundamental bronchoscopy skills and improve airway anatomy knowledge [22,26]. Other studies have provided evidence that bronchoscopy simulator training accelerates the acquisition of basic bronchoscopy skills and that performance-based competency metrics can be used to evaluate basic bronchoscopy skills [18,20,27]. Hospital credentialing committees want standards that can be used in determining privileges, and requiring a minimum number of procedures is a frequently used method [2,6]. However, the American Board of Internal Medicine has not adopted this approach, presumably because of the lack of standardization of the procedures being performed, the significant variability in the rate of skill acquisition and the mounting evidence that the use of procedure numbers for establishing competency is inadequate [13,18,28,29]. Recommendations for numbers of proctored EBUS procedures that need to be performed to demonstrate competency have been made [8,9], but some of these relate to radial EBUS rather than the linear technique, and the recommendations were based on expert opinion only, without supporting evidence to justify their validity. Through the use of EBUS simulators we may now have a method to more objectively standardize the evaluation of competency for this technique.

There are some limitations to this study. Due to the limitations of a yearly fellowship program, there were small sample sizes in the groups. Confirmation of our findings with larger trials would be ideal. Although the metrics used were objective measurements, most of which were recorded directly by the simulator, the use of a single proctor introduces the possibility of a bias. A very specific, scripted protocol was followed for each subject during both the introduction to EBUS phase and during the evaluated cases to ensure that this potential for bias was minimized. Proctor EBUS teaching/assisting skill could have improved with time, therefore introducing bias; however, the majority of group C and D evaluations were completed before those of groups A and B. This potential bias would therefore have favored better performance in groups A and B, and complete elimination of the bias would most likely strengthen the observed difference between groups demonstrated in this study.

Although we selected the primary outcome measures used in this study (procedure time, percentage of lymph nodes correctly identified, percentage of lymph nodes successfully biopsied) based on our experience with EBUS education and believe them to be meaningful measures of EBUS performance, other metrics may be as or more important. Our primary metrics can assess the basic goal of the EBUS procedure: correctly identifying and sampling mediastinal lymph nodes. Procedural accuracy is critical to performing a quality EBUS procedure. When performing EBUS procedures for lung cancer staging, missing an important lymph node or incorrectly identifying a lymph node station has the potential to incorrectly upstage or downstage a patient, thus negatively impacting patient management. Our third primary metric of procedure time adds a measure of efficiency and safety for the procedure given that longer procedures may result in the administration of more sedation with its inherent risks, as well as increasing the likelihood that the patient will not tolerate procedure completion. While many of the additional metrics potentially representing important measures of EBUS skill were also statistically different between groups, the frequency of these events were relatively low and may not be as useful to assess an individual operator. Similarly, frank complications such as vocal cord injury and vascular puncture were very low in all groups.

While the simulator was able to differentiate between the four groups, this study does not in itself determine the minimal skill level threshold associated with the ability to safely, accurately and independently perform a clinical EBUS procedure. Further studies correlating performance on the simulator with performance during clinical cases will be required. It should also be emphasized that this study investigates the technical aspects of EBUS procedural skill and does not attempt to study the cognitive aspects of EBUS. Conclusions regarding cognitive aspects of EBUS, such as pre-procedure preparation of the patient, selection of appropriate lymph node targets and interpretation of results, cannot be made based on our data.

A number of problems were identified with the EBUS simulator, and with all types of simulation training there can be issues with realism and transfer. The white light image provided by the simulator while the EBUS bronchoscope is above the vocal cords is closer to that of a regular flexible bronchoscope, as opposed to the 30 degree angle of view of the EBUS bronchoscope. The problem is corrected once the bronchoscope passes the vocal cords, but could confuse learners when learning the EBUS intubation technique. Lymph nodes are generally more easily identified than during clinical cases, with more distinct separation between nodes and vessels. A number of other software-related discrepancies were noted which did distract the intermediate and expert groups, such as the white light image not always corresponding correctly with the EBUS image (i.e. there may be an EBUS image when there is clearly no apposition of the transducer against the airway wall on the white light image). Finally, the EBUS simulator, including the needle and bronchoscope system, is fragile, resource intensive and expensive, with a current cost of more than USD 100,000 which may limit its widespread use.

In summary, our study is the first to demonstrate that EBUS simulators can accurately discriminate between operators with different levels of clinical EBUS experience. This suggests that there is significant skill transfer from clinical EBUS to the simulator. Demonstrating skill transfer from reality to the simulated environment does not, however, imply that there is skill transfer from the simulated environment to reality, that is, that learning on a simulator translates to improved clinical skills. Given these encouraging findings, EBUS simulators may prove to be useful tools in programs where assessment of EBUS technical competency is of interest such as in fellowship training programs, for the hospital procedure credentialing process and in continuing medical education sessions.

D.R.S., P.M., C.A.H. and K.R. have no conflicts of interest to disclose. A.T. has received consulting fees from Olympus America. The University of Calgary has received grants from Olympus Canada for support of an Interventional Pulmonary Medicine Training Program and for continuing medical education events relating to EBUS.

Validation of an EBUS Simulator in Differentiating Operator Skill Level

Validation of an EBUS Simulator in Differentiating Operator Skill Level

Haponik EF, Russell GB, Beamis JF Jr, Britt EJ, Kvale P, Mathur P, Mehta A: Bronchoscopy training: current fellow’s experiences and some concerns for the future. Chest 2000;118:625–630.
Tape TG, Blank L, Wigton R: Procedural skills of practicing pulmonologists: a national survey of 1,000 members of the American College of Chest Physicians. Am J Respir Crit Care Med 1995;151:282–287.
Haponik EF, Shure D: Underutilization of transbronchial needle aspiration: experiences of current pulmonary fellows. Chest 1997;112:251–253.
Pastis NJ, Nietert PJ, Silvestri GA: Variation in training for interventional pulmonary procedures among US pulmonary/critical care fellowships: a survey of fellowship directors. Chest 2005;127:1614–1621.
Stather DR, Jarand J, Silvestri GA, Tremblay A: An evaluation of procedural training in Canadian respirology fellowship programs: program directors’ and fellow’s perspectives. Can Respir J 2009;16:55–59.
Balfe D, Mohsenifar Z: Downward trends in bronchoscopies performed between 1991 and 1997. Chest 1999;116:238–242.
Herth FJ, Eberhardt R, Vilmann P, Krasnik M, Ernst A. Herth FJ, Eberhardt R, Vilmann P, Krasnik M, Ernst A: Real-time endobronchial ultrasound guided transbronchial needle aspiration for sampling mediastinal lymph nodes. Thorax 2006;61:795–798.
Ernst A, Silvestri GA, Johnstone D: Interventional pulmonary procedures: guidelines from the American College of Chest Physicians. Chest 2003;123:1693–1717.
Bolliger CT, Mathur PN, Beamis JF, Becker HD, Cavaliere S, Colt H, Diaz-Jimenez JP, Dumon JF, Edell E, Kovitz KL, Macha HN, Mehta AC, Marel M, Noppen M, Strausz J, Sutedja TG: European Respiratory Society/American Thoracic Society. ERS/ATS statement on interventional pulmonology. European Respiratory Society/American Thoracic Society. Eur Respir J 2002;19:356–373.
Martin M, Vashisht B, Frezza E, Ferone T, Lopez B, Pahuja M, Spence RK: Competency-based instruction in critical invasive skills improves both resident performance and patient safety. Surgery 1998;124:313–317.
Scott DJ, Bergen PC, Rege RV, Laycock R, Tesfay ST, Valentine RJ, Euhus DM, Jeyarajah DR, Thompson WM, Jones DB: Laparoscopic training on bench models: better and more cost effective than operating room experience? J Am Coll Surg 2000;191:272–283.
Sedlack RE, Kolars JC, Alexander JA: Computer simulation training enhances patient comfort during endoscopy. Clin Gastroenterol Hepatol 2004;2:348–352.
Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM: Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg2002;236:458–463.
Dong Y, Suri HS, Cook DA, Kashani KB, Mullon JJ, Enders FT, Rubin O, Ziv A, Dunn WF: Simulation-based objective assessment discerns clinical proficiency in central line placement: a construct validation. Chest 2010;137:1050–1056.
Haycock AV, Bassett P, Bladen J, Thomas-Gibson S: Validation of the second-generation Olympus colonoscopy simulator for skills assessment. Endoscopy 2009;41:952–958.
Bell R, Maseelall P, Fanning J, Fenton B, Flora R: A laparoscopic simulator tool for objective measurement of resident’s laparocopic ability. JSLS 2007;11:470–473.
Buyske J: The role of simulation in certification. Surg Clin North Am 2010;90:619–621.
Wahidi MM, Silvestri GA, Coakley RD, Ferguson JS, Shepherd RW, Moses L, Conforti J, Que L, Anstrom KJ, McGuire F, Colt H, Downie GH: A prospective multi-center study of competency metrics and educational interventions in the learning of bronchoscopy among starting pulmonary fellows. Chest 2010;137:1040–1049.
Colt HG, Crawford SW, Galbraith O: Virtual reality bronchoscopy simulation: a revolution in procedural training. Chest 2001;120:1333–1339.
Ost D, DeRosiers A, Britt EJ, Fein AM, Lessier ML, Mehta AC: Assessment of a bronchoscopy simulator. Am J Respir Crit Care Med 2001;164:2248–2255.
Colt HG, Davoudi M: Bronchoscopy simulation: a brief review. Adv in Health Sci Educ 2009;14:287–296.
Blum MG, Powers TW, Sundaresan S: Bronchoscopy simulator effectively prepares junior residents to competently perform basic clinical bronchoscopy.Ann Thorac Surg 2004;78:287–291.
Wahidi MM, Rohani NZ, Colt HG: Comparative effectiveness of low- and high-fidelity bronchoscopy simulation for training in conventional transbronchial needle aspiration and user preferences. Respiration 2010;80:327–334.
Moorthy K, Smith S, Brown T, Bann S, Darzi A: Evaluation of virtual reality bronchoscopy as a learning and assessment tool. Respiration 2003;70:195–199.
Ram B, Oluwole M, Blair RL, Mountain R, Dunkley P, White PS: Surgical simulation: An animal tissue model for training in therapeutic and diagnostic bronchoscopy. J Laryngol Otol 1999;113:149–151.
Binstadt E, Donner S, Nelson J, Flottemesch T, Hegarty C: Simulator training improves fiber-optic intubation proficiency among emergency medicine residents. Acad Emerg Med 2008;15:1211–1214.
Davoudi M, Osann K, Colt HG: Validation of two instruments to assess technical bronchoscopic skill using virtual reality simulation. Respiration 2008;76:92–101.
Crawford SW, Colt HG: Virtual reality and written assessments are of potential value to determine knowledge and skill in flexible bronchoscopy. Respiration 2004;71:269–275.
Kemp SV, El Batrawy SH, Harrison RN, Skwarski K, Munavvar M, Roelli A, Cusworth K, Shah PL: Learning curves for endobronchial ultrasound using cusum analysis. Thorax 2010;65:534–538.

This work was performed out of the Tamarratt Research Bronchoscopy Suite, Calgary, Alta., Canada.

Copyright / Drug Dosage / Disclaimer
Copyright: All rights reserved. No part of this publication may be translated into other languages, reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, microcopying, or by any information storage and retrieval system, without permission in writing from the publisher.
Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug.
Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.