Artificial Intelligence for the Classification of Pigmented Skin Lesions in Populations with Skin of Color: A Systematic Review

Skin cancer is the most common malignancy worldwide, with melanoma representing the deadliest form. While skin cancers are less prevalent in people with skin of color, they are more often diagnosed at a later stage and have a poorer prognosis when compared to Caucasian populations [1‒3]. Even when diagnosed at the same stage, Hispanic, Native, Asian, and African Americans have significantly shorter survival time than Caucasian Americans (p < 0.05) [4]. Skin cancers in people with skin of color often present differently from those with Caucasian skin and are often underrepresented in dermatology training [5, 6].

The use of artificial intelligence (AI) algorithms for image analysis and detection of skin cancer has the potential to decrease healthcare disparities by removing unintended clinician bias and improving accessibility and affordability [7]. Skin lesion classification by AI algorithms to date has performed equivalently to [8] and, in some cases, better than dermatologists [9]. Human-computer collaboration can increase diagnostic accuracy further [10]. However, most AI advances have used homogenous datasets [11‒15] collected from countries with predominantly European ancestry [16]. Exclusion of skin of color in training datasets poses the risk of incorrect diagnosis or missing skin cancers entirely [8] and risks widening racial disparities that already exist in dermatology [8, 17].

While multiple reviews have compared AI-based model performances for skin cancer detection [18‒20], the use of AI in populations with skin of color has not been evaluated. The objective of this study was to systematically review the current literature for AI models for classification of pigmented skin lesion images in populations with skin of color.

The systematic review follows the PRISMA guidelines [21]. A protocol was registered with PROSPERO (International Prospective Register of Systematic Reviews) and can be accessed at https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021281347.

A PubMed search in March 2021 used search terms relating to artificial intelligence, skin cancer, and skin lesions (search strings in online suppl. eTable; for all online suppl. material, see www.karger.com/doi/10.1159/000530225). No date range was applied, language was restricted to English, and only original research was included. Covidence software was used for screening administration. Search results were screened by reviewing titles/abstracts by two independent reviewers (Y.L. 100% and B.B.S. 20%) using eligibility criteria described in Table 1. Remaining articles were assessed for eligibility by reviewing methods or full text. Disagreements were resolved following discussions with a third independent reviewer (C.P.).

Data extraction was performed using a standardized form by author Y.L. and confirmation by V.K. The following parameters were recorded: reference, ethnicity/ancestry/race, lesion number, sex, age, location, skin condition, public availability of dataset, number of images, type of images, methods of confirmation, deep learning system, model output, comparison with human input, and any missing data reported. Algorithm performance measures are recorded by either accuracy, sensitivity, specificity, and/or area under the receiver operating characteristic curve. A narrative synthesis of extracted data was used to present findings as a meta-analysis was not feasible due to heterogeneity of the study design, AI systems, skin lesions, and outcomes.

Quality was assessed using the Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology (CLEAR Derm) Consensus Guidelines [22]. This 25-point checklist offers comprehensive recommendations on factors critical to the development, performance, and application of image-based AI algorithms in dermatology [22].

Author Y.L. performed the quality assessment of all included studies, and author B.B.S. assessed 20%. Inter-rater agreement rate was 87%, with disagreements resolved via a third independent reviewer (V.K.). Each criterion was evaluated to be either fully, partially, or not addressed and scored either 1, 0.5, or 0, respectively, using a scoring rubric in online supplementary eTable2.

The database search identified 993 articles, including 13 duplicates. After screening titles/abstracts, 535 records were excluded, and the remaining 445 records were screened by methods, with 63 articles reviewed by full text. Forward and backward citations search revealed no additional articles. A total of 22 studies were included in the final review (PRISMA flow diagram in online supplementary eFig. 1).

All 22 studies were performed between 2002 and 2021 [23‒32], with 11 (50%) studies published between 2020 and 2021 [33‒44]. An overview of study characteristics is displayed in Table 2. The median number of total images used in each study for all datasets combined was 5,846 (ranging from 212 to 185,192). The median dataset size for training, testing, and validation was 4,732 images (range: 247–22,608), 362 (range: 100–40,331), and 1,258 (range: 109–14,883), respectively.

The majority of studies (15/22, 68%) analyzed clinical images (i.e., wide-field or regional images), while seven studies analyzed dermoscopy images [23, 24, 27, 29, 30, 40, 42], and one study included both [44]. All but one study included both malignant and benign pigmented skin lesions, with one investigating only benign pigmented facial lesions [43].

Histopathology was used as the ground truth in 15 studies for all malignant lesions and partially in two studies [24, 26], while one study only used histopathology to resolve clinician disagreements [23]. Seven studies used histopathology as ground truth for benign lesions [23, 27, 29, 34, 35, 41, 44]. In nine studies, ground truth was established by consensus of experienced dermatologists [25, 30‒32, 38‒40, 42, 43]. Other studies used a mix of both [24, 26, 33, 36] or were not clearly defined [28, 37].

The number of pigmented skin lesion classifications used for AI model evaluation ranged from binary outcomes (e.g., benign vs. malignant) to classification of up to 419 skin conditions [39]. While most studies (19/22, 86%) evaluated lesions across all body sites, one study exclusively analyzed the lips/mouth [33], another assessed only facial skin lesions [43], and one study specifically addressed acral melanoma [29].

Homogenous datasets were collected from the Chinese/Taiwanese (n = 8, 36%) [25, 30, 32, 37, 40, 41, 43, 44], Korean (n = 5, 23%) [27‒29, 33‒35], and Japanese populations (n = 3, 14%) [31, 38, 42]. Seven studies (32%) included populations from Caucasians/Fitzpatrick skin type I–III [23, 24, 26, 28, 36, 39, 42], with at least 10% American Indian [26], Alaska Native [26], black or African American [26], Pacific Islander [26], Native American [26], or Fitzpatrick IV–VI [23, 39] in the training and/or test set (Table 2).

The majority of studies did not specify the sex distribution (n = 13, 59%) or participant age (n = 15, 68%). Seven studies included age specification, ranging from 18 to older than 85 years [23, 25, 26, 34‒36, 41].

The outcome of the classification algorithms used either a diagnostic model, risk categorical model (e.g., low, medium, or high), or a combination of both. An overview of AI model performance is described in Table 3. Majority of studies (20/22, 91%) used a diagnostic model, either with binary classification of benign or malignant [23‒27, 29, 33, 35, 37], multiclass classification of specific lesion diagnosis [28, 30, 32, 39, 42, 43], or both [31, 34, 36, 38, 40, 41, 44]. One study used categorical risk as the outcome [32]. Another study reported both diagnostic model and risk categorical model [24].

The AI models using binary classification (16/22) reported an accuracy ranging from 70% to 99.7%. Of these studies, 6/16 reported ≥90% accuracy [25‒27, 31, 38, 41], three studies reported between 80 and 90% accuracy [29, 37, 44], and one study reported <80% accuracy [40]. Twelve AI models reported sensitivity and specificity as a measure of performance, which ranged from 58 to 100% and 72 to 99%, respectively. Eight studies provided an area under the curve (AUC) with 5/8 reporting values >0.9 [24, 25, 35‒37], with the remaining three models scoring between 0.77 and 0.86 [29, 33, 34].

For the 13 studies using multiclass output (i.e., >2 diagnoses), accuracy of models ranged from 43% to 93%. Six of these studies (6/13) scored <80% accuracy [31, 34, 36, 39, 41, 44], six others scored between 80 and 90% accuracy [30, 32, 38, 40, 42, 43], and one provided sensitivity and specificity of 86% and 86%, respectively, as a measure of performance [28].

Reader studies, where the performance of AI models and clinician classification is compared, were performed in 14/22 studies, with results provided in Table 4[23, 25, 29, 31‒39, 42, 44]. Six studies compared AI outcomes to classification by experts, e.g., dermatologists [25, 32, 34, 36, 42, 44]. Eight studies compared outcomes for both experts and non-experts, e.g., dermatology residents and general practitioners [23, 29, 31, 33, 35, 37‒39].

In reader studies comparing binary classification between AI and experts (n = 11), one study reported similar diagnostic accuracy/specificity [29], three showed higher accuracy for AI models [25, 31, 38], and two reported higher accuracy in experts [42, 44]. Five studies reported specificity, sensitivity, and AUC instead of accuracy with varying outcomes [23, 32, 33, 35, 37]. For reader studies between AI and non-experts (n = 7), AI showed higher accuracy, specificity, sensitivity, and AUC in most studies [23, 29, 31, 33, 35, 37, 38]. In reader studies that compared multiclass classification between AI and expert readers (n = 5), three studies reported higher top-1 diagnosis (i.e., diagnosis with highest statistical probability) accuracy for AI [31, 34, 44], in one study readers performed better [36], and the last study had similar results between AI and readers [39]. In multiclass reader studies with non-experts, AI reported higher accuracy in both studies (n = 2) [31, 39].

One study compared categorical risk classification with experts and showed AI to have higher sensitivity across all categories [32]. Two studies additionally showed significant increase in accuracy when human experts were aided by AI outcomes [33, 36].

Studies included in the review were evaluated against the 25-point CLEAR Derm Consensus Guidelines [22], covering four domains (data, technique, technical assessment, and application). For each checklist item, each study was assessed to determine whether it fully, partially, or did not address the criteria. A summary of results is provided in Table 5, with the individual study scores available in online supplementary eTable 3. Overall, an average score of 17.4 (±2.2) out of a possible 25 points was obtained.

The data domain consists of 15 checklist items; of these, six items were fully addressed by >90% of publications, including checklist items (1) image types, (3) technical acquisition details, (4) pre-processing procedures, and (6) public images adequately referenced. No studies included synthetic images (item 5). Checklist items (2) image artifacts, (9) potential biases, and (10) dataset partitions (e.g., lesion distribution) were only fully addressed by approximately half of the studies. About a third of studies fully addressed the sample size of training, validation, and test sets (item 11) [28, 29, 34, 35, 39, 40, 43]. The most poorly addressed criteria were patient-level metadata (e.g., sex, gender, ethnicity; item 7), skin color information (item 8), using an external test dataset (item 12), class distribution and balance of the images (item 14), and out of distribution images (item 15).

Patient-level metadata (item 7) was partially reported in most studies. While geographical location, hospital location, and race were adequately reported, sex and age specifications were limited, as were anatomical location, relevant past medical history, and history of presenting illness. Skin color information was reported as Fitzpatrick’s skin type in only three studies [23, 39, 40].

All four checklist items in the technique domain were fully addressed by most publications. Image labeling method (item 16), e.g., information on ground truth, was fully addressed in 68% (n = 15) of publications. All studies used commonly accepted diagnostic labels (item 17). Histopathologic review (item 18) was fully addressed in 73% (n = 16) of studies. A detailed description of algorithm development (item 19) was provided in 64% (n = 14) of studies.

Of the four checkpoints, the most poorly addressed item was the public evaluation of the model (item 20), with only five papers fully addressing this item [29, 34‒36, 39] and the majority (n = 17, 77%) not addressing it at all. Four studies reported public availability of their AI algorithm or offered a public-facing test interface for external testing [28, 34‒36]. Performance measures model (item 21) was fully addressed by nine studies. Benchmarking, technical comparison, and novelty (item 22) compared to previously developed algorithm and human specialists were fully addressed by 15 studies, with technical bias assessment (item 23) discussed by 10 studies.

The application domain consisted of two checklist items. Use of cases and target conditions (item 24) was fully addressed by 16 (73%) studies, while potential impacts on the healthcare team and patients (item 25) were fully addressed by only three studies [38, 39, 41].

We present, to our knowledge, the first systematic review summarizing existing AI image-based algorithms for the classification of skin lesions in people with skin of color. Our review identified 22 articles where the algorithms were primarily developed on or tested on images from people with skin of color.

Within the identified AI studies involving skin of color populations, there is further under-representation of darker skin types. We found that Asian populations (Chinese, Korean, and Japanese) were the major ethnic groups included, with limited datasets involving populations from the Middle East, India, Africa, Pacific Islands, and from Hispanic and Native Americans. Only three studies specifically reported skin color categories of type IV–VI and/or black or African American. Two North American studies reported only 4.3% [26] and 6.8% [39] black African American images, and one Italian-led study [23] reported 26.5% of images with Fitzpatrick skin type IV/V.

Adding to the issue was a lack of transparency in reporting skin color in image-based AI studies. A recently published scoping review [17] of 70 studies involving image-based AI algorithms for classification of skin conditions found only 14 (20%) provided details about race and seven (10%) provided detail about skin color. Furthermore, only three studies partially included Fitzpatrick skin type IV–VI populations. The lack of diversity in studies is likely fueled by the shortage of diverse publicly available datasets. A recent systematic review of 39 open access datasets and skin atlases reported that 12 of the 14 datasets that reported country of origin were from countries of predominantly European ancestry [16]. Only one dataset from Brazil reported that 5% of their population contained Fitzpatrick skin type IV–VI images [45], and another dataset comprised images from a South Korean population [36].

AI models in this review showed reasonably high-performance classification using both dermoscopic and clinical images in populations with skin of color. Accuracy was greater than 70% for all binary models reviewed, which is similar to models developed using Caucasian datasets [46]. It has previously been suggested that instead of training algorithms on more diverse datasets, it may be beneficial to develop separate algorithms for different populations [7]. Han et al. [28] demonstrated a significant difference in performance of an AI model trained using skin images from exclusive Asian populations, which had greater accuracy for classifying basal cell carcinomas and melanomas in Asian populations (96% and 96%, respectively) than in Caucasian populations (90% and 88%, respectively).

There is some evidence that adding out-of-distribution images (i.e., those not originally represented in the training dataset) to a pre-existing dataset and re-training can improve classification accuracy among the newly added group while not affecting accuracy among the original group [22]. Another previously suggested method is to use artificially darkened lesion images in the training dataset [47].

Our review identified several critical items from the CLEAR Checklist that were insufficiently addressed by the majority of AI imaging studies in populations with skin of color. Overall, discrepancies in image artifact description, lack of skin color information and other patient-level metadata, missing rationale for dataset sizes, and inclusion of external test sets were found. Image artifacts have been previously shown to affect algorithm performance in both dermatological images [48, 49] and clinical photographs [50]. Unclear descriptions of pre-processing of images can cause incorrect diagnosis, as can subtle alteration in color balance or rotation that can result in incorrect classifying melanoma as benign nevus [49]. Furthermore, less than half of the studies reviewed reported anatomical locations of imaged lesions. The body site of a lesion can be informative; for example, skin of color populations have a high prevalence of acral melanomas which are commonly found on the palms, soles, subungual region, and mucous membranes [51]. Future studies should consider adopting established consensus on image acquisition metrics [52] or dermatology-specific guidelines [22] to standardize images and develop robust archives of skin images used for research.

A major concern is that most AI algorithms have only been tested on internal datasets, e.g., those sampled from the same/similar populations. This issue was similarly highlighted in a recent review [18]. AI algorithms are prone to overfitting, which can result in inflated accuracy measures when only tested on internal datasets and significant performance drop when tested with images from a different source [9]. The lack of standardized performance measure model, publicly available diverse datasets, test interfaces, and reader studies presents a barrier to comparability and reproducibility of AI algorithms. An alternative solution is for algorithms to be evaluated on standardized public test datasets, with transparency on algorithm performance [22]. Testing models on external datasets and/or out-of-distribution images is a better indicator of AI model performance in a real-world setting. Given public dermoscopic datasets with diverse skin colors are now available, future algorithms should include external testing as gold standard evaluation [53].

Lastly, only limited studies included in the review addressed the potential impacts on the healthcare team and patients. Future studies would benefit by considering the clinical translation of AI models during the early stages of development and evaluating clinical utility of AI models along with their performance in conjunction with their intended users [36].

Our systematic review is not without limitations. First, our search was limited to articles published in English. This is particularly of note for populations with skin of color, where English may often not be the primary language. Additionally, as many studies did not report on skin of color status, inclusion was based on assumptions using ethnicity and geographical location. There can be significant variability in skin color even for a population of the same race or geographical location [54]. Reporting bias may also influence the review, as higher performing algorithms are more likely to be reported and accepted for publication.

AI algorithms have great potential to complement skin cancer screening and diagnosis of pigmented lesions, leading to improved dermatology workflows and patient outcomes. The performance and reliability of image-based AI algorithm significantly depend on the quality of data on which it is trained and tested. While this review provides promising development of AI algorithms in skin of color for East Asian populations, there are still significant discrepancies in the number of algorithms developed in populations with skin of color, particularly in darker skin types. Without inclusion of images from populations with skin of color, the incorporation of AI models into clinical practice could lead to missed and delayed diagnoses of neoplasms in people with skin of color, further widening existing health outcome disparities [55].

An ethics statement is not applicable because this study is based exclusively on published literature.

H. Peter Soyer is a shareholder of MoleMap NZ Ltd. and e-derm-consult GmbH and undertakes regular teledermatological reporting for both companies. H. Peter Soyer is a medical consultant for Canfield Scientific Inc., MoleMap Australia Pty Ltd., and Blaze Bioscience Inc. and a medical advisor for First Derm. The remaining authors have no conflicts of interests to declare.

H. Peter Soyer holds an NHMRC MRFF Next Generation Clinical Researchers Program Practitioner Fellowship (APP1137127). Clare Primiero is supported by an Australian Government Research Training Program Scholarship.

Yuyang Liu: conceptualization (equal), data curation (lead), formal analysis (equal), methodology (lead), validation (equal), and writing – original draft (lead). Clare Primiero: conceptualization (equal), funding acquisition (equal), methodology (equal), supervision (equal), and writing – review and editing (equal). Vishnutheertha Kulkarni: data curation (supporting), formal analysis (supporting), validation (equal), and writing – review and editing (supporting). H. Peter Soyer: conceptualization (supporting), funding acquisition (equal), supervision (supporting), and writing – review and editing (supporting). Brigid Betz-Stablein: conceptualization (equal), data curation (supporting), formal analysis (supporting), methodology (equal), supervision (equal), and writing – review and editing (equal).

This review is based exclusively on published literature. No publicly available datasets were used and no data were generated. Further inquiries can be directed to the corresponding author.