Introduction: Major depression, anxiety disorders, and post-traumatic stress disorder (PTSD) are among the most prevalent comorbid mental disorders in youth addiction treatment. Hence, screening for these internalizing disorders should be part of the standard routine at intake in substance use disorder treatment. We investigated the usefulness of the Depression, Anxiety and Stress Scale (DASS-21) as a screener for this purpose. Methods: A nationally representative sample of 421 treatment-seeking youths aged 16–22 years with a primary cannabis, alcohol, cocaine, or amphetamine use disorder participated in the study. At intake, we administered the DASS-21 and the Mini International Neuropsychiatric Interview (MINI; Sheehan et al., 1998) based on the Diagnostic and Statistical Manual of Mental Disorders, fifth edition (DSM-5), as “gold standard,” as part of a broader baseline assessment of the Youth in transition study [Moska et al. BMC Psychiatry. 2021;21(1):1–11]. Results: At comparable sensitivity (0.80–0.84), based on the optimal cut-off value, specificity was higher for the DASS-21 total score detecting any DSM-5 internalizing disorder (0.62) than for the DASS subscales specifically detecting depression, anxiety, or PTSD (0.44, 0.49, and 0.51, respectively). Receiver operating characteristic curve analyses showed an area under the curve (AUC) value of 0.80 for the DASS total score to detect any internalizing disorder (“good discrimination”), compared with AUC values of 0.70–0.75 of the DASS depression and anxiety subscales to detect DSM-5 depression, anxiety, and PTSD (“fair discrimination”). The optimal DASS total score cut-off value of ≥44 for detecting any internalizing disorder resulted in 0.81 sensitivity, 0.62 specificity, 0.80 positive predictive value, and 0.64 negative predictive value. Conclusion: Given the high prevalence of comorbid internalizing disorders in youth addiction care, the need to address these comorbid disorders in treatment, and the favorable accuracy of the DASS to detect these disorders, we recommend to implement the DASS-21 as routine screener in youth addiction treatment in the Netherlands.

Many mental health disorders originate in adolescence and early adulthood. More than one-third of adolescents in the general population meet the criteria of a past-year mental health disorder, and in about half of these cases, the disorder is associated with severe impairment [1, 2]. Past-year prevalence is highest for any anxiety disorder (25%), followed by any behavior disorder (16%), mood disorder (10%), and substance use disorder (SUD) with or without dependence (8%). Compared with the other adolescent mental disorders, SUDs have a relatively late age of onset (median approximately 15 years), and show a steep increase in incidence following this median age. For example, in the USA, “lifetime” prevalence of SUD in adolescents showed a sixfold increase from 3.7% at age 13–14 to 22.3% at age 17–18 [1, 3].

The few available general population studies in adolescents with SUD suggest high – up to 60% – rates of comorbid mental disorders in this subgroup (e.g., [4]), and more recent data indicate that these comorbid mental disorders often – in 2 out of 3 cases – precede the onset of SUD in adolescents [5]. Not surprisingly, rates of comorbid mental disorders found among adolescents who seek treatment for SUD are even higher. In a review of 10 studies of adolescents in SUD treatment, Couwenbergh et al. [6] found prevalence rates of any comorbid mental disorder ranging from 61 to 88%.

These findings indicate that systematic screening of co-occurring mental disorders is warranted in all youths in SUD treatment, and should be followed by a comprehensive diagnostic assessment of the comorbid disorder(s) in case of a positive screen. However, current best clinical practice in Dutch youth addiction care is that the clinician conducts an unstructured diagnostic evaluation with the patient if (s)he suspects a comorbid non-SUD mental disorder, but systematic screening and use of a structured diagnostic interview are by no means routine in most patients or in most addiction care organizations. Consequently, comorbid mental disorders often remain undetected and, hence, untreated. Notably, the scientific literature indicates that treatment of a comorbid mental disorder does have a favorable effect on the comorbid disorder but generally does not improve the substance use outcomes of SUD treatment. Hence, treatment of both disorders is recommended, and is likely to improve the overall prognosis of patients with concurrent SUD and other mental health disorders in SUD treatment [7, 8].

The main objective of screening is to efficiently distinguish between individuals who likely have or do not have a comorbid disorder, without going through the full, often time-consuming, diagnostic process in all individuals. Given this objective, a screening test should be brief and easy to administer by nonspecialists in the relevant field. Screening is useful only when the prevalence of the disorder in the population in question is sufficiently high and when the disorder concerns a treatable condition.

In the context of youth substance abuse treatment, internalizing disorders, including major depression and anxiety disorders, are among the most prevalent comorbid mental disorders [9, 10]. Hence, these disorders should be a primary target for screening. One of the briefest and most commonly used screening tests for depression and anxiety disorders in adults is the 21-item version of the Depression, Anxiety and Stress Scale (DASS-21) [11]. In a systematic review of 48 studies on the measurement properties of the DASS-21 in – mostly nonclinical – adult populations, Lee et al. (2019) found moderate- to high-quality evidence for high internal consistency reliability for the DASS subscales and total scale. Although the original authors proposed a three-factor solution of the instrument [11], Lee et al. [12] conducted a systematic review of studies on the measurement properties of the DASS-21 and found high-quality evidence for both a common factor – representing “general distress” or “negative emotional state” – and three specific factors, i.e., depression, anxiety, and stress [12]. Concerning adult populations, only two psychometric studies of the DASS-21 have been conducted among patients in substance abuse treatment [13, 14]. Both (small-scale) studies investigated the criterion validity of the instrument in adults, and found fair to good area under the curve (AUC) values of the DASS total score for identifying depression [14], and post-traumatic stress disorder (PTSD) [13], respectively.

Concerning youths, both Moore et al. [15] and Evans et al. [16] investigated the structural validity of the DASS-21 in a community population study among adolescents. Consistent with the findings of Lee et al. [12] in adults, they found strong support for the presence of both three specific factors representing the domains of depression, anxiety, and stress in adolescents, and of a general, overarching factor of general distress which seems to be represented by the DASS-21 total score. To date, no studies have been conducted on the usefulness and psychometric properties of the DASS-21 among youth in substance abuse treatment. Against this background, the present study aims to investigate the usefulness of the DASS-21 as a screening instrument to identify comorbid past-year major depressive disorder, anxiety disorder, and PTSD in youths who enter substance abuse treatment. In addition to the performance validity of the DASS-21 subscales, we investigated the accuracy of the DASS-21 total score – as a measure of general distress – to identify any past-year internalizing disorder, i.e., depression, anxiety disorder, and/or PTSD.

Participants and Setting

Data for the present study were collected by trained research assistants, who were independent from treatment staff, as part of a broad baseline assessment of a nationally representative prospective cohort study into the long-term course of addiction, mental health problems, and social functioning in youth entering addiction treatment in the Netherlands – the Youth in transition study (YIT) [17]. Four hundred and twenty-four youths who entered addiction treatment at ten treatment organizations were recruited to participate in the study. Inclusion criteria were as follows: age 16–22 years; primary problem substance concerned cannabis, alcohol, cocaine, or amphetamines; and willingness to participate in the study (oral and written informed consent). Exclusion criteria were as follows: referral for diagnostic evaluation only; and insufficient comprehension of basic Dutch language. Three participants had a missing value pertaining to the classification of one of the three internalizing disorders, and were excluded from the analysis. Hence, data were analyzed of 421 participants (Fig. 1). Recruitment of study participants took place between January 2019 and August 2021. The YIT study was funded by The Netherlands Organization for Health Research and Development (No. 60-63600-98-317) and was approved by the Medical Ethical Board of the Leiden University Medical Center (No. NL65903.058.18; file number P18.175).

Fig. 1.

Flowchart of the Youth in transition study.

Fig. 1.

Flowchart of the Youth in transition study.

Close modal

Instruments

The DASS-21 is a self-report measure designed to assess the severity of the core symptoms of depression, anxiety, and stress, and is derived from the original 42-item questionnaire developed by Lovibond and Lovibond [11]. Each subscale contains seven items that are rated on a 4-point severity/frequency scale ranging from 0 (“did not apply to me at all”) to 3 (“applied to me very much or most of the time”). Scores for each subscale and the total score of the DASS-21 are derived by summing the scores of the relevant item-set, which are subsequently doubled to correspond with the scoring range of the original 42-item version. Hence, DASS-21 subscale scores range from 0 to 42, and the total score ranges from 0 to 126. The usual time frame of the DASS refers to the past week. We used a time frame of past 30 days (prior to treatment entry) to harmonize the time period covered by the DASS with that of other instruments administered at baseline. In addition, we administered the DASS in an interview format, to enable questions for clarification, if needed.

In earlier research, the subscales of the Dutch version of the DASS-21 showed good (Cronbach’s alpha 0.8–0.9) to excellent (alpha >0.9) internal consistency in both a nonclinical and clinical sample [18]. In the present study sample, internal consistency amounted to alpha = 0.89 for depression, 0.78 for anxiety, 0.83 for stress, and 0.92 for the total score.

To investigate the usefulness of the DASS-21 as a screening instrument for identifying comorbid internalizing disorders among youth who apply for addiction treatment, we administered the Mini International Neuropsychiatric Interview (MINI) [19] based on the Diagnostic and Statistical Manual of Mental Disorders, fifth edition (DSM-5) in its Dutch translated version (MINI 7.0.2, 2019) as “gold standard.” From the disorders covered by the MINI, we used the sections of major depressive disorder, panic disorder, agoraphobia, social phobia, generalized anxiety disorder (GAD), and PTSD for the current study. In addition to screening for each of these disorders separately, we pooled each of these disorders to screen for the presence of “any internalizing disorder” (i.e., any of these classifications). The DASS-21 and MINI were administered during the same interview session at treatment intake, which took place shortly prior to the start of addiction treatment.

Concerning the structure of the MINI, each diagnostic section starts with one to three screening questions pertaining to the key diagnostic criteria of the disorder. If the patient answers negatively to these key questions, the disorder is coded as absent and the interview proceeds with the next diagnostic section [19]. In case of a positive answer, additional symptom questions are asked, closely following the diagnostic criteria and structure of the DSM-5. As a rule, and particularly relevant for the current study, symptoms were excluded if they were clearly induced by the physiological effect of a psychoactive substance, i.e., if the symptoms occurred only during or shortly following intoxication or withdrawal.

In the MINI, priority is given to the identification of “current” disorders to keep the instrument short [19]. However, the time frame in the DSM-5 version of the MINI for “current disorder” differs substantially across the disorders, with – for the internalizing disorders – a range of (at least) “past 2 weeks” for depression; “past month” for panic disorder and PTSD; and “past 6 months” for agoraphobia, social anxiety, and GAD. In the context of our prospective YIT cohort study, with its longitudinal assessments at treatment entry (baseline) and 2 and 4 years after treatment entry, we deemed it most relevant to use a standard time frame of “past-year,” as is most common in epidemiological studies. For this reason, and to better accommodate the different time frames included in the MINI diagnoses, patients in our sample received a past-year diagnosis for a specific disorder in the MINI if they met the minimum required duration of that disorder anytime during the previous year.

Demographic background, substance use, and SUD data were derived from the baseline YIT questionnaire and the Measurements in the Addictions for Triage and Evaluation – Youth version (MATE-Y) [20]. For an extensive description of the YIT instruments and study procedures, the reader is referred to the protocol paper of the YIT [17].

Data Analysis

Differences in mean scores on the DASS-21 subscales and total score between participants with and without DSM-5 depression, anxiety, PTSD, and any internalizing disorder were tested by independent t tests. Although the Likert-type items in the DASS-21 are in essence ordinal, quite a number of experts in the measurement field have argued that, while a single Likert item may be ordinal, the sum or the average of several of these Likert items (i.e., the Likert scale) represents an approximate continuous variable, which can well be analyzed with parametric tests, like the independent t test [21‒24]. We determined the sensitivity – the proportion of disordered individuals with a positive test result (“true-positive rate”; see Table 1); the specificity – the proportion of non-disordered individuals with a negative test result (“true-negative rate”); the positive predictive value (PPV) – the probability that an individual with a positive test actually has the disorder; and the negative predictive value (NPV) – the probability that an individual with a negative test actually does not have the disorder – of the DASS-21 depression and anxiety subscales and total score. Criterion diagnoses were DSM-5 past-year major depressive disorder, anxiety disorders (i.e., panic disorder; agoraphobia; social phobia; GAD), PTSD, and any internalizing disorder.

Table 1.

DASS-21 total score test outcome and actual disorder status: any internalizing disorder (N = 421)

Cut-off ≥44Disorder presentDisorder absent
Test positive [A] True positives = 222 [B] False positives = 56 A + B = 278 
Test negative [C] False negatives = 52 [D] True negatives = 91 C + D = 143 
 A + C = 274 B + D = 147 A + B + C + D = 421 
Cut-off ≥44Disorder presentDisorder absent
Test positive [A] True positives = 222 [B] False positives = 56 A + B = 278 
Test negative [C] False negatives = 52 [D] True negatives = 91 C + D = 143 
 A + C = 274 B + D = 147 A + B + C + D = 421 

Prevalence = A + C/(A + B + C + D) = 0.65.

Sensitivity (true-positive rate) = A/(A + C) = 0.81.

Specificity (true-negative rate) = D/(B + D) = 0.62.

Positive predictive value (PPV) = A/(A + B) = 0.80.

Negative predictive value (NPV) = D/(C + D) = 0.64.

Accuracy = A + D/(A + B + C + D) = 0.74.

In addition, we conducted receiver operating characteristic (ROC) curve analyses, in which sensitivity is plotted against 1-specificity (i.e., the true-positive vs. the false-positive rate) for each cut-off value of the DASS subscales and total score, to determine the overall classification accuracy in terms of AUC. AUC values can range between 0.50, indicating chance prediction (represented in the ROC space by a 45° diagonal) and 1.00, signifying a perfect test. As a rule of thumb, and derived from Hosmer et al. [25], AUC values of 0.5–0.69 represent poor discrimination; 0.70–0.79 fair discrimination; 0.80–0.89 good discrimination; and ≥0.90 excellent discrimination.

We prioritized high sensitivity by opting for a relatively low threshold value, determined by the lowest DASS cut-off at which sensitivity reached a value of 0.80 or higher. Opting for higher sensitivity (less false negatives) generally comes at the expense of lower specificity (more false positives), but we deemed this acceptable because we wanted to minimize the number of false negatives (i.e., participants with the actual disorder not identified as positives by the screener). Data were analyzed with the IBM Statistical Package for the Social Science (SPSS), version 27.0.

The baseline characteristics of the participants at treatment entry are summarized in Table 2. Most study participants were male (68.4%) and defined their cultural identity as Dutch-western (89.3%). On average, participants were 20 years old. The majority of youth entered treatment for a primary problem with cannabis (62.2%), followed by alcohol (18.3%), and stimulants (cocaine and amphetamine pooled; 19.5%). Based on the MINI, nearly half of the participants received a past-year DSM-5 classification of depression, one-third were classified with an anxiety disorder, and one-quarter were classified with PTSD. Nearly two-thirds met the criteria of one or more internalizing disorders.

Table 2.

Baseline characteristics of the study participants (N = 421)

%/mean (SD)
Demographic background 
 Age (range 16–22), years 19.9 (1.6) 
 Gender male, % 68.6 
 Gender female, % 31.4 
 Dutch-western cultural identity, % 78.7 
Primary SUD 
 Cannabis, %; months of regular usea (mean, SD) 62.2; 41.2 (25.7) 
 Alcohol, %; months of regular usea (mean, SD) 18.3; 27.7 (19.4) 
 Cocaine, %; months of regular usea (mean, SD) 11.9; 20.7 (18.7) 
 Amphetamines, %; months of regular usea (mean, SD) 7.6; 23.9 (22.8) 
Prevalence past-year non-SUD mental disorder 
 Internalizing disorders, % 
  Depression 48.9 
  Anxiety disorder 35.2 
  PTSD 26.1 
  Any internalizing disorder 65.1 
 Externalizing disorders, % 
  Attention deficit hyperactivity disorder 34.0 
  Conduct disorder 21.9 
DASS-21 scores 
 Subscale depression (range 0–42) 18.6 (11.5) 
 Subscale anxiety (range 0–42) 15.3 (9.5) 
 Subscale stress (range 0–42) 21.9 (9.9) 
 DASS-21 total score (0–126) 55.7 (26.8) 
%/mean (SD)
Demographic background 
 Age (range 16–22), years 19.9 (1.6) 
 Gender male, % 68.6 
 Gender female, % 31.4 
 Dutch-western cultural identity, % 78.7 
Primary SUD 
 Cannabis, %; months of regular usea (mean, SD) 62.2; 41.2 (25.7) 
 Alcohol, %; months of regular usea (mean, SD) 18.3; 27.7 (19.4) 
 Cocaine, %; months of regular usea (mean, SD) 11.9; 20.7 (18.7) 
 Amphetamines, %; months of regular usea (mean, SD) 7.6; 23.9 (22.8) 
Prevalence past-year non-SUD mental disorder 
 Internalizing disorders, % 
  Depression 48.9 
  Anxiety disorder 35.2 
  PTSD 26.1 
  Any internalizing disorder 65.1 
 Externalizing disorders, % 
  Attention deficit hyperactivity disorder 34.0 
  Conduct disorder 21.9 
DASS-21 scores 
 Subscale depression (range 0–42) 18.6 (11.5) 
 Subscale anxiety (range 0–42) 15.3 (9.5) 
 Subscale stress (range 0–42) 21.9 (9.9) 
 DASS-21 total score (0–126) 55.7 (26.8) 

DASS-21, Depression, Anxiety and Stress Scale; SD, standard deviation.

aRegular use of cannabis ≥3 days a week; alcohol ≥3 days a week, more than 4 glasses (female) or 5 glasses (male) at one time; cocaine and amphetamines 1 day a week.

As indicated by the skewness of 0.142 and kurtosis of −0.724, the DASS-21 total score data were normally distributed. Mean scores on the DASS-21 depression subscale differed significantly between youths with and without DSM-5 past-year depression {M = 23.7 (standard deviation [SD] 10.8) vs. 13.9 (SD 10.3), respectively; t(419) = −10.09, p < 0.001}, and means on the anxiety subscale differed between those with and without an anxiety disorder (M = 20.4 [SD 8.9] vs. 12.5 [SD 8.6], respectively; t(419) = −7.84, p < 0.001), and between those with and without PTSD (M = 20.2 [SD 9.7] vs. 13.5 [SD 8.8], respectively; t(419) = −6.70, p < 0.001). Likewise, the DASS mean total score differed significantly between youths with and without any past-year DSM-5 internalizing disorder (M = 65.2 [SD 24.6] vs. 38.0 [SD 21.3], respectively; t(419) = −27.17, p < 0.001). In addition, mean scores on the DASS stress subscale were significantly different (all: p < 0.001) for DSM-5 depression, anxiety, PTSD, and any internalizing disorder (statistics not reported). Mean scores on the DASS-21 were significantly higher for those with the disorder than for those without the disorder on all DASS-21 subscales and on the total score.

Table 1 shows the main accuracy parameters of the DASS-21 and their calculation, using the identification of any internalizing disorder as an example, with an optimal DASS total score cut-off value of ≥44 (see below). Among youths with the target disorder present, 81% (222/274) consisted of true positives (sensitivity) on the DASS and 19% (52/274) were false negatives. Similarly, among youths with the target disorder absent, 62% (91/147) were true negatives (specificity) on the DASS and 38% (56/147) were false positives. Concerning the PPV and NPV of the DASS for detecting any internalizing disorder, 80% (222/278) of youths with a positive test consisted of true positives (PPV). Likewise, 64% (91/143) of those with a negative test were true negatives (NPV).

Figure 2 displays the ROC curve in which the true-positive rate is plotted against the false-positive rate for each cut-off of the DASS total score, again using identification of any internalizing disorder as example. Overall classification accuracy in terms of AUC was 0.80 (“good discrimination”). Separating girls (N = 132) and boys (N = 289) in the sample, again using the optimal DASS total score value of ≥44 for detecting internalizing disorder resulted in 0.86 sensitivity for girls versus 0.78 for boys, and specificity amounted to 0.53 for girls and 0.64 for boys. The AUC value was 0.79 for both girls and boys. Hence, no relevant differences in these values were found between girls and boys and the total sample.

Fig. 2.

ROC curve DASS-21 total score test outcome and actual disorder status: any internalizing disorder.

Fig. 2.

ROC curve DASS-21 total score test outcome and actual disorder status: any internalizing disorder.

Close modal

Table 3 shows the main accuracy parameters of the DASS for specifically identifying depression, anxiety disorders, and PTSD using the depression and anxiety subscales (upper panels) and for detecting any internalizing disorder using the total score (lower panel). At comparable sensitivity (0.80–0.84) based on the optimal cut-off value (defined as the lowest DASS cut-off at which sensitivity reached a value of 0.80 or higher), specificity was higher for the total score detecting any internalizing disorder (0.62) than for the subscales detecting depression, anxiety, or PTSD (0.44–0.51). This was, consequently, also the case for the AUC, with a value of 0.80 versus 0.70–0.75, respectively. Supplemental analyses indicated that AUC values of the specific subscales were not superior (0.70–0.75; “fair discrimination”) to those found when the total score was used for specifically detecting depression, anxiety, and PTSD (0.72–0.78; “fair discrimination”; not shown in Table 2). As shown in the lower panel, opting for higher sensitivity came at the expense of lower specificity and vice versa. Lastly, the PPV was higher and the NPV was lower when screening for any internalizing disorder using the DASS total score (lower panel: 0.80 and 0.64, respectively) compared with the values for each DASS subscale separately (upper panels: 0.34–0.61 and 0.73–0.87, respectively).

Table 3.

Accuracy of DASS-21 subscales and total score for detecting DSM-5 depression, anxiety, PTSD, and any internalizing disorder

Scales/cut-offSensitivitySpecificityPPVNPVOverall accuracy, %AUC
Depression subscale for detecting depression ≥14 0.80 0.51 0.61 0.73 0.65 0.75 
Anxiety subscale for detecting anxiety ≥12 0.84 0.49 0.47 0.85 0.62 0.74 
Anxiety subscale for detecting PTSD ≥12 0.81 0.44 0.34 0.87 0.54 0.70 
Total score for detecting any internalizing disorder      0.80 
 ≥40 0.85 0.54 0.77 0.65 74  
 42 0.83 0.59 0.79 0.65 74  
44 0.81 0.62 0.80 0.64 74  
 46 0.78 0.65 0.81 0.61 74  
 48 0.76 0.68 0.82 0.60 73  
Scales/cut-offSensitivitySpecificityPPVNPVOverall accuracy, %AUC
Depression subscale for detecting depression ≥14 0.80 0.51 0.61 0.73 0.65 0.75 
Anxiety subscale for detecting anxiety ≥12 0.84 0.49 0.47 0.85 0.62 0.74 
Anxiety subscale for detecting PTSD ≥12 0.81 0.44 0.34 0.87 0.54 0.70 
Total score for detecting any internalizing disorder      0.80 
 ≥40 0.85 0.54 0.77 0.65 74  
 42 0.83 0.59 0.79 0.65 74  
44 0.81 0.62 0.80 0.64 74  
 46 0.78 0.65 0.81 0.61 74  
 48 0.76 0.68 0.82 0.60 73  

PPV, positive predictive value; NPV, negative predictive value; AUC, area under the curve.

We evaluated the usefulness of the DASS-21 as a screener for past-year DSM-5 depression, anxiety disorder, or PTSD and as a screener for the presence of any internalizing disorder in a nationally representative sample of youths aged 16–22 years old with a primary cannabis, alcohol, cocaine, or amphetamine use disorder entering substance abuse treatment in the Netherlands. The optimal DASS total score cut-off value of ≥44 for detecting any internalizing disorder resulted in 0.81 sensitivity, 0.62 specificity, 0.80 PPV, and 0.64 NPV and the AUC value amounted to 0.80 (“good discrimination”). This AUC value was higher compared to the values when using the DASS subscales for detecting depression, anxiety, or PTSD (“fair discrimination”). Hence, the DASS subscales seem to have no added value for identifying separate comorbid internalizing disorders among youths in SUD treatment, and it seems most appropriate to use the DASS total score as broad-spectrum screener for the decision of whether or not subsequent diagnostic evaluation of internalizing disorders is warranted in this population. Such application of the DASS-21 as broad-spectrum screener is in line with emerging evidence for the use of the total score as a valid measure of overall psychological distress in adolescents [16]. Moreover, broad screening seems preferable because internalizing disorders often show considerable overlap among themselves [26].

In our study, high sensitivity (≥0.80) was accompanied by relatively low specificity. In all (non-perfect) screening tests, there is a tradeoff between sensitivity and specificity, in that higher sensitivity nearly always comes at the expense of lower specificity, and – ceteris paribus – higher specificity comes at the expense of lower sensitivity. Both test values are not inherently high or low (or different from each other), but their value is highly dependent upon the choices made by the researcher or clinician.

We intentionally opted for a relatively low cut-off value on the DASS-21 total score to achieve sufficiently high sensitivity. This decision was made because we considered it crucial to minimize the number of false negatives – i.e., the number of patients who actually had the disorder but were missed by the screener. Alternatively, if we would have prioritized higher specificity, by increasing the cut-off value on the DASS-21, this would not only have resulted in less false positives (i.e., the number of patients with a positive screening test who did not have the actual disorder), but also in a higher number of false negatives – the scenario that we wanted to avoid. Please see the bottom rows in Table 3, which illustrate the tradeoff between higher/lower sensitivity and specificity.

When screening in clinical practice, actual disease status is often unknown, and sensitivity and specificity are less useful measures of test validity [27]. In practice, the most prominent question for the clinician pertains to the probability of the disease being present or absent, given the outcome of screening. We found a sufficiently high PPV, with four in five true positive screens, accompanied by a relatively low NPV, with two in three true negatives, when screening on any internalizing disorder. However, an important caveat with regard to these findings is that the predictive value of a test is highly dependent on (among others) the prevalence of the disorder in the sample, with higher prevalence “inflating” the PPV, and lower prevalence inflating the NPV. Hence, the higher PPV found when screening on any internalizing disorder and higher NPV when screening on each disorder separately may at least partly be explained by this association.

Strengths of the study are that (1) this is the first study which investigates the usefulness of the DASS-21 as a screening instrument for youths in substance abuse treatment, (2) the results are based on a representative, well-defined, sample of youths seeking treatment for the most prevalent SUDs in the Netherlands (covering nearly three quarters of total youth addiction treatment demand), and (3) we used a structured DSM-5 diagnostic interview in which participants received a negative classification of a disorder if the disorder symptoms only occurred during intoxication or withdrawal – i.e., when it concerned a substance-induced disorder (e.g., substance-induced depression) only. Study limitations include, first, that we assessed the accuracy of the DASS-21 shortly after intake – a time at which acute effects of alcohol and drugs may affect DASS-21 scores, and not at a second time point, e.g., after discontinuation or substantial reduction of substance use – when acute alcohol and drug effects are likely diminished [14], and, second, that we did not compare the performance of the DASS in youth with that of other broad-spectrum screeners of general distress, like the Outcome Questionnaire (OQ-45) [28] or Brief Symptom Inventory (BSI) [29]. Nevertheless, despite possible acute drug effects during intake, the DASS performed quite well at this time point, in particular given that symptoms that were solely drug-induced resulted in a negative classification in our structured diagnostic interview. Third, we used a time frame of “past month” for the DASS-21 instead of the usual “past week” to better accommodate the different time frames of “current disorder” between the internalizing disorders in the MINI. In addition, we considered a disorder in the MINI as “present” in the past year if a patient met the minimum required duration of that disorder – 2 weeks for depression, 1 month for panic disorder and PTSD, and 6 months for agoraphobia, social anxiety, and GAD – anytime during the previous year. Given the strong positive association in earlier research between illness severity and comorbidity of mental disorders with treatment seeking [30‒32], it is likely that people seek treatment for addiction and concurrent mental health problems at a time when they are experiencing the most acute and severe problems. Hence, for most patients in our sample, their “past-year disorder” likely pertains to the weeks or months directly preceding treatment entry, rather than to the first months of the previous year.

Given the high prevalence of comorbid internalizing disorders in youth addiction care, the need to address these comorbid disorders in treatment, and the favorable accuracy of the DASS to detect these disorders, we recommend to implement the DASS-21 as routine screener in youth addiction treatment in the Netherlands.

The Youth in Transition Study is a collaboration between the Parnassia Addiction Research Centre (PARC), the Academic Medical Centre (AMC) at the University of Amsterdam, the Nijmegen Institute for Science Practitioners in Addiction (NISPA), and department of Tranzo scientific center for care and wellbeing. The authors would like to thank the project group, Renske Spijkerman, Khedy Gorissen, Ilse Haarman, Demelza Bomers, Arnt Schellekens, Dike van de Mheen, Jannet de Jonge, Floris Bary, and Margot Peeters. We also would like to thank all participating youth, professionals, and the research assistants for their help during the data collection.

The study protocol was reviewed and approved by the Medical Ethical Board of the Leiden University Medical Center (NL65903.058.18; file number P18.175). Patients provided oral and written informed consent for participation in this study. Before the start of our study according to the Netherlands Central Committee on Research Involving Human Subjects willing participants 16 years of age and older (adults) sign the consent form themselves; written consent of (both) (custodial) parents is not required.

The authors have no conflicts of interest to declare.

This study was funded by The Netherlands Organization for Health Research and Development (ZonMW) and the Dutch Addiction Association. The Volksbond Rotterdam Foundation funded part of the study’s follow-up assessments. ZonMW, the Dutch Addiction Association, and the Volksbond Rotterdam Foundation had no further role in study design, or in the collection, analysis, and interpretation of the data, or in the writing of the report.

Vincent Hendriks, Peter Blanken, and Anna E. Goudriaan were primarily responsible for the design of the study, in co-operation with the project members Dike van de Mheen, Renske Spijkerman, Arnt Schellekens, Jannet de Jonge, Floris Bary, and Wilma Vollebergh. Christina Moska was primarily responsible for data collection, data analyses, and writing of the manuscript, with active input from all co-authors. All authors contributed to the manuscript and approved its final version.

Additional Information

Trail registration: The Netherlands National Trial Register, Trial NL7928. Date of registration January 17, 2019. This study was registered before patient enrollment.

All data generated or analyzed during this study are included in this article. Further inquiries can be directed to the corresponding author.

1.
Merikangas
KR
,
He
JP
,
Burstein
M
,
Swanson
SA
,
Avenevoli
S
,
Cui
L
.
Lifetime prevalence of mental disorders in U.S. Adolescents: results from the national comorbidity survey replication--adolescent supplement (NCS-A)
.
J Am Acad Child Adolesc Psychiatry
.
2010
;
49
(
10
):
980
9
.
2.
Kessler
RC
,
Avenevoli
S
,
Costello
EJ
,
Georgiades
K
,
Green
JG
,
Gruber
MJ
.
Prevalence, persistence, and sociodemographic correlates of DSM-IV disorders in the national comorbidity survey replication adolescent supplement
.
Arch Gen Psychiatry
.
2012
;
69
(
4
):
372
80
.
3.
Substance Abuse and Mental Health Services Administration
Results from the 2012 national survey on drug use and health: summary of national findings. NSDUH Series H-46, HHS Publication No. (SMA) 13-4795. Substance Abuse and Mental Health Administration
Rockville (MD)
Health and Human Services
2013
. Available from: Results from (samhsa.gov).
4.
Armstrong
TD
,
Costello
EJ
.
Community studies on adolescent substance use, abuse, or dependence and psychiatric comorbidity
.
J Consult Clin Psychol
.
2002
;
70
(
6
):
1224
239
.
5.
Conway
KP
,
Swendsen
J
,
Husky
MM
,
He
JP
,
Merikangas
KR
.
Association of lifetime mental disorders and subsequent alcohol and illicit drug use: results from the national comorbidity survey – adolescent supplement
.
J Am Acad Child Adolesc Psychiatry
.
2016
;
55
(
4
):
280
8
.
6.
Couwenbergh
C
,
Van den Brink
W
,
Zwart
K
,
Vreugdenhil
C
,
Van Wijngaarden-Cremers
P
,
Van der Gaag
RJ
.
Comorbid psychopathology in adolescents and young adults treated for substance use disorders: a review
.
Eur Child Adolesc Psychiatry
.
2006
;
15
(
6
):
319
28
.
7.
Linford-Hughes
AR
,
Welch
S
,
Peters
L
,
Nutt
DJ
.
BAP updated guidelines: evidence-based guidelines for the pharmacological management of substance abuse, harmful use, addiction and comorbidity: recommendations from BAP
.
J Psychopharmacol
.
2012
1
54
.
8.
Dom
G
,
Moggi
F
Co-occurring addictive and psychiatric disorders. A practice-based handbook from a European perspective
Heidelberg
Springer-Verlag Berlin
2015
.
9.
Deas
D
,
Brown
ES
.
Adolescent substance abuse and psychiatric comorbidities
.
J Clin Psychiatry
.
2006
67
Suppl 7
18
23
.
10.
Dorard
G
,
Bungener
C
,
Phan
O
,
Edel
Y
,
Corcos
M
,
Berthoz
S
.
Which psychiatric comorbidities in cannabis dependence during adolescence? Comparison of outpatients and controls [Article in French]
.
Encephale
.
2018
;
44
(
1
):
2
8
.
11.
Lovibond
SH
,
Lovibond
PF
Manual for the depression anxiety stress scales
Sydney
The psychology foundation of Australia
1995
.
12.
Lee
J
,
Lee
E
,
Moon
SH
.
Systematic review of the measurement properties of the Depression Anxiety Stress Scales-21 by applying updated COSMIN methodology
.
Qual Life Res
.
2019
;
28
(
9
):
2325
339
.
13.
Kok
T
,
De Haan
H
,
Van der Meer
M
,
Najavits
LM
,
De Jong
CAJ
.
Screening of current post-traumatic stress disorder in patients with substance use disorder using the depression, anxiety and stress scale (DASS-21): a reliable and convenient measure
.
Eur Addict Res
.
2015
;
21
(
2
):
71
7
.
14.
Beaufort
IN
,
De Weert-Van Oene
GH
,
Buwalda
VAJ
,
De Leeuw
RJ
,
Goudriaan
AE
.
The Depression, Anxiety and Stress Scale (DASS-21) as a screener for depression in substance use disorder inpatients: a pilot study
.
Eur Addict Res
.
2017
;
23
(
5
):
260
8
.
15.
Moore
SA
,
Dowdy
E
,
Furlong
MJ
.
Using the depression, anxiety, stress scales-21 with U.S. Adolescents: an alternate models analysis
.
J Psychoeduc Assess
.
2017
;
35
(
6
):
581
98
. . doi.org/
16.
Evans
L
,
Haeberlein
K
,
Chang
A
,
Handal
P
.
An evaluation of the convergent validity of and preliminary cutoff scores for the DASS-21 total score as a measure of distress in adolescents
.
Curr Psychol
.
2020
;
41
(
7
):
4283
90
. . doi.org/
17.
Moska
C
,
Goudriaan
AE
,
Blanken
P
,
Van de Mheen
D
,
Spijkerman
R
,
Schellekens
A
.
Youth in Transition: Study protocol of a prospective cohort study into the long-term course of addiction, mental health problems and social functioning in youth entering addiction treatment
.
BMC Psychiatry
.
2021
;
21
(
1
):
1
11
.
18.
De Beurs
E
,
Van Dyck
R
,
Marquenie
LA
,
Lange
A
,
Blonk
RWB
.
De DASS: Een vragenlijst voor het meten van depressie, angst en stress
.
Gedragstherapie
.
2001
;
34
:
35
53
.
19.
Sheehan
DV
,
Lecrubier
Y
,
Sheehan
KH
,
Amorim
P
,
Janavs
J
,
Weiller
E
.
The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10
.
J Clin Psychiatry
.
1998
59
Suppl 20
22
33
; quiz 34–57.
20.
Schippers
GM
,
Broekman
TG
,
Buchholz
A
,
Koeter
MWJ
,
Van den Brink
W
.
Measurements in the addictions for triage and evaluation (MATE): an instrument based on the World Health Organization family of international classifications
.
Addiction
.
2010
;
105
(
5
):
862
71
.
21.
Carifio
J
,
Perla
R
.
Resolving the 50-year debate around using and misusing Likert scales
.
Med Educ
.
2008
;
42
(
12
):
1150
2
.
22.
De Winter
JFC
,
Dodou
D
.
Five-point Likert items: t test versus mann-whitney-wilcoxon
.
Practical Assess Res Eval
.
2010
;
15
(
11
):
1
16
.
23.
Norman
G
.
Likert scales, levels of measurement and the ‘‘laws’’ of statistics
.
Adv Health Sci Educ Theor Pract
.
2010
;
15
(
5
):
625
32
.
24.
Sullivan
GM
,
Artino
AR
.
Analyzing and interpreting data from Likert-type scales
.
J Grad Med Educ
.
2013
;
5
(
4
):
541
2
.
25.
Hosmer
DW
,
Lemeshow
S
,
Sturdivant
RX
Applied logistic regression
New York
John Wiley & Sons
2000
26.
Ormel
J
,
Raven
D
,
van Oort
F
,
Hartman
CA
,
Reijneveld
SA
,
Veenstra
R
.
Mental health in Dutch adolescents: a TRAILS report on prevalence, severity, age of onset, continuity and co-morbidity of DSM disorders
.
Psychol Med
.
2015
;
45
(
2
):
345
60
.
27.
Miller
DC
,
Dunn
RL
,
Wei
JT
.
Assessing the performance and validity of diagnostic test and screening programs
. In:
Penson
DF
,
Wei
JT
, editors.
Clinical research methods for surgeons
Totowa (NJ)
Humana Press
2007
. p.
157
74
.
28.
Lambert
MJ
,
Hansen
NB
,
Umphress
V
,
Lunnen
K
,
Okiishi
J
,
Burlingame
G
Administration and scoring manual for the outcome questionnaire (OQ45.2)
Wilmington
American Professional Credentialing Services
1996
.
29.
Derogatis
LR
BSI brief symptom inventory: administration, scoring, and procedures manual
5th ed.
Minneapolis
National Computer Systems
1993
.
30.
Mojtabi
R
,
Olfson
M
,
Mechanic
D
.
Perceived need and help-seeking in adults with mood, anxiety, or substance use disorders
.
Arch Gen Psychiatry
.
2020
;
59
:
77
84
.
31.
Sareen
J
,
Cox
BJ
,
Afifi
T
,
Clara
I
,
Yu
BN
.
Perceived need for mental health treatment in a nationally representative Canadian sample
.
Can J Psychiatry
.
2005
;
50
(
10
):
643
51
.
32.
Oleski
J
,
Mota
N
,
Cox
BJ
,
Sareen
J
.
Perceived need for care, help seeking, and perceived barriers to care for alcohol use disorders in a national sample
.
Psychiatr Serv
.
2010
;
61
(
12
):
1223
31
.