New WHO Reporting System for Lung Cytopathology: Reproducibility Test of the Diagnosis and Usefulness of an Online Tutorial System for the New Cytological Categorization Open Access

Histological and cytological classifications are essential for diagnosing and managing lung cancer. Cytology specimens of lung cancer are useful for morphological and genetic diagnosis even when tissue samples are unavailable. However, no universal category classification for lung cytopathology, such as histological classification, was available. Conversely, the Papanicolaou Society of Cytopathology in the USA published a category classification [1‒3], and Japan also used a separate cytological classification system [4, 5]. Thus, categorization was different in each country. Developing a common cytological classification system that could be used in articles and presentations at academic conferences was necessary. Therefore, the Japanese Lung Cancer Society and the Japanese Society of Clinical Cytology (JSCC) jointly developed the lung cytological classification system, which included negative for malignancy, atypical cells, suspicious for malignancy, and malignancy categories [6]. Yoshizawa et al. performed a follow-up study using these four categories plus benign tumors and confirmed the usefulness of this system [7]. In 2022, the WHO Reporting System for Lung Cytopathology was published. This system was based on the Japanese system and included the following cytology categories: insufficient/inadequate/nondiagnostic, benign, atypical, suspicious for malignancy, and malignant [8‒10]. Diagnostic categories in cytological reporting system must be linked to diagnostic management recommendations to improve communication with clinicians and support patient care. The WHO reporting system must be truly international to serve the needs of patients worldwide in many differently medically resourced settings.

However, the mere creation of a classification system is not practical in actual clinical practice. Thus, we evaluated the reproducibility of the classification system through a tutorial and investigated the issues and problems associated with the wide dissemination of this classification system. This study is the first reproducibility test of the WHO Reporting System for Lung Cytopathology and is the first step toward international standardization.

Forty-two cases from our previous analyses were selected for this study [6, 7]. All cases were histologically confirmed by biopsies or surgical tissues. The cases were selected through discussions among four cytopathologists and six cytotechnologists and consisted of a mixture of easily diagnosed and difficult-to-diagnose cases. Benign, inflammatory, and malignant cases were selected, including 10 benign, 12 atypical, 12 suspicious for malignancy, and 8 malignant cases. These included 26 brushing samples, 6 touch preparations of tumors, 4 lavage specimens, 3 sputum samples, 2 transbronchial aspiration samples and 1 washing forceps specimen. All specimens were fixed in 95% ethanol and stained with Papanicolaou staining.

No inadequate/inadequate/nondiagnosis case was included in this study. The correct answers for these cases were given in the previous studies [6, 7]. The cases are listed in Table 1.

Participants were members of JSCC and qualified as cytotechnologists or cytopathologists. Participants were recruited through advertisements on the JSCC website and through the mailing list of 10,921 JSCC members. Written informed consent for the collection of participant data was obtained when a request for participation was received. The following data about the participants were collected: participant name, facility name, age, facility type, years of cytology experience, job title, and specialty. Participant names and facility names were collected only to assign personal numbers for participation in the study. Personal data were anonymized by assigning a personal number.

A web-based response system using WEBCAS (WOW WORLD Inc., Tokyo, Japan) was developed. A ×100 and a ×400 image centered on the lesion to be diagnosed was included for each of the 42 cases. The images were classified based on the following: 4 categories, including 10 benign, 12 atypical, 12 suspicious for malignancy, and 8 malignant lesions; 7 suggestive pathological diagnosis meaning the histological type inferred from the morphology of the cytology specimen, including 5 benign tumors, 17 inflammatory changes, 13 adenocarcinoma (ADC), 3, squamous cell carcinoma (SQCC), 1 small-cell carcinoma (SCLC), 0 large-cell neuroendocrine carcinoma, and 3 other malignant tumors; and 4 choices for the first and second basis for the diagnosis, including cell cluster, cytoplasmic, nuclear findings, or other. All answers were collected online; the first response system was disclosed at 4 weeks. One week after the end of the first round, the correct answers and tutorial were presented for 2 weeks. The tutorial comprised a PDF file available online for 2 weeks, detailing the cytological findings and key observation points based on case images from the study and allowing participants to access and study the material at their convenience. The second round started 4 weeks after the end of the presentation of correct answers and tutorial. The cases and the questions for the second round were the same as the first round (Table 1; Fig. 1). This study schedule was designed based on the results of the Ebbinghaus forgetting curve of human memory [11].

The correct and incorrect answer rates were defined as the number of correct and incorrect answers divided by the total number of cases. The correct response rates were defined as the number of correct respondents divided by the total number of participants.

We compared the mean correct and incorrect answer rates for the 42 cases among the participants before and after the tutorial, and the mean correct response rates for the 105 participants among the cases. Cases that were difficult to match were extracted, and diagnostic points were examined. To evaluate the impact of the tutorial based on cytology experience, we analyzed changes in correct response rates after the tutorial based on years of experience (19 participants with 1–5, 15 with 6–10, 26 with >10 years, 19 with >20 years, and 26 with >30 years of experience).

McNemar’s test was used to analyze difference of number of correct response for cytological category. Paired Student t tests were used to compare mean correct and incorrect answer rates and mean correct response rates. To examine the effects of the tutorial on each category and to identify cases with difficult-to-concordant diagnoses and cases with high tutorial effects, the first and second correct response rates were compared for each of the 42 cases using McNemar’s tests. A p value of <0.05 was considered significant.

The first round consisted of 180 research applicants and 140 participants (77.8% applicants). The second round consisted of 105 participants (58.3% of applicants, 75% of first round participants) (Fig. 1). Results from 105 participants were analyzed.

Significant difference was found only between less than 5 years and 6–10 years (p = 0.03).

As shown in Table 1, the results showed that 17 of 42 cases improved significantly, especially 8 of 12 atypical cases and 6 of 12 suspicious for malignancy cases.

As shown in Table 2 and Figure 2, the mean number of correct answers for cytologic diagnostic categories increased from 16.0 of 42 cases (38.1%) in the first round to 20.3 of 42 cases (48.3%) in the second round (p < 0.001). In the case of the benign, the atypical, and the suspicious for malignancy, the mean number and rate of correct answers increased significantly between rounds 1 and 2 (p < 0.001). Conversely, for malignant cases, the mean number and rate of correct answers increased between the first and the second rounds; however, the difference was not statistically significant (p = 0.223).

The results of the incorrect answer for 42 cases by cytological diagnostic category are shown in Table 2. The mean number of cases out of 10 benign and 12 atypical cases miscategorized as malignant were significantly lower in the second round than in the first round (p < 0.001). Of the 12 suspicious for malignancy cases, the number of cases miscategorized as malignant decreased significantly lower in the second round than in the first (p = 0.002). Of the 8 malignant cases, the number of cases miscategorized as benign decreased significantly lower in the second round than in the first (p = 0.027).

As shown in Table 3, the mean number of correct answers for all participants increased significantly from 20.3 of 42 (48.3%) in the first round to 25.1 of 42 (59.8%) in the second round (p < 0.001). The mean number of correct answers to benign tumors, inflammatory changes, and all five carcinomas of suggestive pathological diagnosis combined into one carcinoma category all increased significantly in the second round (p < 0.001).

As shown in Table 4, the mean number of correct response rates increased significantly from 40.2 of 105 (38%) in the first round to 51.5 of 105 (49%) in the second round (p = 0.015). In each category, the mean number of correct response rates for the 10 benign and 8 malignant cases increased after the tutorial, but the difference was not significant. In contrast, the mean number of correct response rates for the 12 atypical cases increased significantly from 25.3 (24%) in the first round to 41.5 (39.5%) in the second round (p < 0.001). The mean number of correct response rates for the 12 suspicious for malignancy increased significantly from 25.5 (24.3%) in the first round to 38.3 (36.5%) in the second round (p < 0.001). The mean number of correct response rates for suggestive pathological diagnosis of suspicious for malignancy and malignant increased after the tutorial, although the difference was not significant.

A McNemar’s test was performed to analyze differences in the number of correct response rates for cytological categories among the 105 participants before and after the tutorial for each case. Cases with no significant difference before and after the tutorial in each category, low correct response rates, and poor tutorial effects were selected (Table 1). Case 38 (granulomatous lesion, benign) had the lowest correct response rate among the benign cases (1st: 6.7% vs. 2nd: 14.3%, p = 0.061). Case 9 (interstitial pneumonia, atypical) had same correct response rate in the first and second rounds (1st: 26.7 vs. 2nd: 26.7%, p = 1.000). The correct response rate in Case 34 (ADC, suspicious for malignancy) did not increase significantly after the tutorial (1st: 34.3% vs. 2nd: 36.2%, p = 0.871). Case 4 (mucoepidermoid carcinoma, malignant) had the highest correct response rates among the four categories, but the correct response rate did not increase after the tutorial (1st: 41.9% vs. 2nd: 47.6%, p = 0.471) (Fig. 3a–d).

A McNemar’s test was performed to analyze the difference in the number of correct responses for the five suggestive pathological diagnoses (ADC, SQCC, SCLC, large-cell neuroendocrine carcinoma, and other malignancies) for the 20 suspicious for malignancy and malignant cases. A significant tutorial effect was detected in 8 out of 20 cases (Table 1). The correct response rate for Case 4 (mucoepidermoid carcinoma, malignant) was low but significantly increased after the tutorial (1st: 21% vs. 2nd: 38.1%, p = 0.007) (Fig. 3d). In Case 24 (SQCC, suspicious for malignancy), there was also significant difference between after the tutorial (1st: 23.8% vs. 2nd: 36.2%, p = 0.031) (Fig. 4a). Case 25 (SCLC, suspicious for malignancy) had a higher percentage of correct responses (1st: 95.2% vs. 2nd: 85.7%, p = 0.024), although the percentage of correct responses decreased after the tutorial compared to before (Fig. 4b).

A McNemar’s test was performed for the cytological category of each case, and cases with a high tutorial effect (an increased number of correct responses) were selected (Table 1). The correct response rate for Case 15 (sclerosing pneumocytoma, benign) significantly increased (1st: 29.5% vs. 2nd: 56.2%, p < 0.001) (Fig. 5a). The correct response rate for Case 3 (rheumatoid lung, atypical) (Fig. 5b) significantly increased (1st: 10.5% vs. 2nd: 43.8%, p < 0.001). The correct response rate for Case 41 (inflammatory changes, atypical) (Fig. 5c) significantly increased (1st: 4.8% vs. 2nd: 24.8%, p < 0.001). The correct response rate for Case 6 (ADC, suspicious for malignancy) (Fig. 5d) significantly increased (1st: 14.3% vs. 2nd: 34.3%, p = 0.001).

Nuclear findings were the most important basis for diagnoses in 31 of 42 cases (74%) in the first round and 31 of 42 cases (74%) in the second round, either. The details of the most important basis for diagnosis are listed in Table 5.

The mean correct answer rates increased significantly after the tutorial. As shown in Table 2, the effects of the tutorial for the new reporting system for lung cytopathology were significant, especially in atypical and suspicious for malignancy cases. The tutorial also affected the correct answer rates for benign and malignant category cases, but the effects were limited. Part of the problem in reproducibility, especially for intermediate categories, is that the morphologic changes are a spectrum through which we draw artificial lines in an attempt to correlate with biological behavior [12‒14]. In other words, benign and malignancy are easy to distinguish morphologically.

The tutorial was not effective in improving diagnoses in some cases. For example, Case 38 was a granulomatous lesion and benign. However, the participants may have diagnosed the case as atypical or malignant based on the nuclear findings, cytoplasmic thickness, and squamous metaplasia. Case 9 was an atypical one that most participants diagnosed as benign. That was an interstitial pneumonitis case, with no cell size discrepancy or homogeneous chromatin, but the image exhibited dense proliferating cells. The participants were misled to believe that the case was benign, but the dense proliferating cells indicate that the case should be diagnosed as atypical. In the suspicious for malignancy Case 34, almost one-third of the participants diagnosed malignant in the first and second rounds, indicating that the tutorial had no effect. In this case, the irregular dense proliferating cells had a variety of chromatin, suggesting ADC. However, no clear differences in size or nuclear atypia were observed, and the case should have been diagnosed as suspicious for malignancy rather than malignant. Case 4 was mucoepidermoid carcinoma. The nuclear findings were similar between the two types of cells, and the participants have been confused about the difference between malignancy and suspicious for malignancy.

Meanwhile, the highest tutorial effect was indicated in Case 15, a sclerosing pneumocytoma in the benign category. Although the cells lacked nuclear atypia, the high cell density may have discouraged participants from categorizing the case as benign in the first round, and many participants categorized the image as malignant or suspicious for malignancy. However, the tutorial highlighted the lack of size disparity in the cells and the lack of nuclear atypia. Thus, the percentage of correct responses increased in the second round.

The most important diagnostic point evaluated by the participants was nuclear findings, which may have led to diagnostic confusion when the atypia of nuclei was mild. Cytological diagnoses should be based on differences in the nuclear size and chromatin distribution. Reactive cells may also have more pleomorphic, binucleated, or enlarged nuclei than malignant cells, which may lead to overdiagnosis if the focus is solely on nuclear findings [15]. However, specimens assigned to the atypical category will demonstrate lesser degrees of cytomorphologic abnormality than those assigned to the suspicious for malignancy category [16]. In this study, the percentage of correct responses for benign, atypical, and suspicious for malignancy categories increased when the tutorial highlighted the lack of chromatin increase and nuclear atypia.

The main limitation of this study was that only two images were presented instead of the entire slide. In general, cytology specimens are examined for the target cells, the background cells, necrosis, and mucin. However, this study focused on the tutorial effects of the new classification system. Thus, we wanted the participants to focus only on the target cells, and we presented only two images. In addition, since this was a cytology study involving many cytology specialists, the use of virtual slides and the large number of cases increased the time required for case observation longer, making it difficult to maintain concentration, and there was concern that this would interfere with the work of the cytology specialists, so the number of cases was limited to 42, a slightly smaller number than before [17].

We showed that web-based tutorials are particularly effective for benign, atypical, and suspicious for malignancy categories. Malignant category had a high correct answer and response rate and remained high before and after the tutorial; therefore, the tutorial should be fine as similar volume as our study. In order for this reporting system to be used effectively in many countries, it is necessary to provide and disseminate appropriate tutorials on diagnostic criteria via the internet. In particular, detailed explanations and annotations on important points should be provided to enable correct diagnosis of cancers with mild atypia, and cancers lacking characteristic morphology.

We many thank the 140 participants, especially the 105 cytotechnologists and cytopathologists who participated in both rounds of this study and JSCC secretariats. We also thank Dr. Satoru Shimizu of Tokyo Women's Medical University for giving us advice on statistical analysis of the data in this reproducibility study of the classification through tutorials.

Written informed consent was waived for this retrospective study which analyzed. Cytological specimens with limited clinical information by the Ethics Committee of Tokyo Women’s Medical University (Approval No.: 4873-R; approval date: May 19, 2021).

The authors have no conflicts of interest to declare.

The Japan Lung Cancer Society and the Japanese Society of Clinical Cytology paid the developing of the web-based response and tutorial system and fee for the English editing service.

Study concept and design: Yuko Minami, Kenzo Hiroshima, Akihiko Yoshizawa, Akemi Takenaka, Reiji Haba, Kunimitsu Kawahara, Hirokuni Kakinuma, Yasuo Shibuki, Shinji Miyake, and Yukitoshi Satoh. Reviewing the cases: Yuko Minami, Akemi Takenaka, Hirokuni Kakinuma, Yasuo Shibuki, and Shinji Miyake. Management of images: Akemi Takenaka. Analysis and interpretation of data: Yuko Minami, Kenzo Hiroshima, Akihiko Yoshizawa, and Yukitoshi Satoh. Drafting of the manuscript: Yuko Minami. Critical revision of the manuscript: Yuko Minami, Kenzo Hiroshima, Akihiko Yoshizawa, Reiji Haba, Kunimitsu Kawahara, and Yukitoshi Satoh.

All data analyzed during this study are included in this article. Further inquiries can be directed to the corresponding author.