Screening Second Victims for Emotional Distress: Assessment of the Clinimetric Properties of the WITHSTAND-PSY Questionnaire

Patient harm is a leading cause of morbidity and mortality across the globe and places a heavy financial burden on healthcare systems [1, 2]. While patients are the obvious first victims of unsafe care, the healthcare providers involved in adverse events (AEs) or medical errors can also be greatly impacted, becoming second victims (SVs) [3]. Two recent meta-analyses [4, 5] provide a deeper understanding of the psychological burden of AEs on healthcare providers. SVs may suffer from various symptoms, including troubling memories, anxiety, depression, and sleeping difficulties. These can further lead to allostatic overload [4, 6]. To deal with their emotional distress, SVs frequently apply strategies that are task-, emotion-, and, to lesser extent, avoidance-oriented [5, 7]. The SV phenomenon represents a unique experience which cannot be accurately and comprehensively captured by existing psychometric instruments that measure, for example, depression or anxiety. Several authors examining SVs’ responses to AEs have created their own questionnaires, [8‒10] or adapted [11, 12] existing ones [13, 14]. However, these existing tools [8‒14] address the aftermath of the event, without comparing participants’ distress before and after the event. Furthermore, they focus either on SVs’ psychological distress or on their coping strategies, but not on both, and often have not been formally validated. The only exception is the Second Victim Experience and Support Tool [15‒20] which assesses SVs’ distress and support resources but has been analyzed applying a psychometric approach rather than a clinimetric one.

Clinimetrics was introduced by Alvan Feinstein in 1982 as the science of metrics useful for clinicians [21‒27]. Over time, clinimetrics has attracted growing interest in the fields of clinical psychology and psychiatry thanks to its novel perspective on clinical diagnostics and testing. Important shortcomings of the psychometric approach, such as the assumption that all symptoms weigh equally in the scale score, have been compensated for by the clinimetric model, which focuses, for instance, on symptom intensity and quality to determine the severity of a disease [24, 25]. Given that the clinical model enables different clinical weights of the items, clinimetrics does not consider internal consistency relevant as in the psychometric model which relies on Cronbach’s alpha. Moreover, the multidimensional approach of clinimetrics avoids item redundancy and favors their local independence (i.e., the score of an item should be independent from the one of another item) [23].

With these precepts in mind, we developed a clinimetric instrument assessing SV distress before and after the AE and applied coping strategies. Clinimetric properties, such as clinical utility, construct validity, concurrent validity, and clinical validity, were investigated using Rasch analysis. Additionally, a semi-structured interview was developed to deepen understanding of the individual responses to the clinimetric instrument.

The present study was approved by the Institutional Review Board of the University of Verona (nr. 03/2021). Written informed consent was obtained from participants, and anonymity was guaranteed. Questionnaire development and clinimetric testing were part of a larger project, entitled WITHSTAND-PSY – Psychological Support After Adverse Events, established at the University of Verona, Italy, in 2016 with the aim of developing, evaluating, and implementing a supporting program for SVs [28] (https://sites.google.com/view/withstandproject/home-page?authuser=0). Figure 1 outlines the methodological steps taken in this study and is described below.

The WITHSTAND-PSY Questionnaire (WS-PSY-Q) is composed of three sections (see online suppl. File 1; for all online suppl. material, see https://doi.org/10.1159/000535006). The General Information section includes questions regarding participants’ sociodemographic characteristics and professional activity, characteristics of the AE, workplace climate, and psychological support received.

The Emotional Distress section assesses 24 psychological symptoms, before and after the AE, split into two subscales: anxiety and depression. The pre- and post-measures allow assessment of two different facets of the SV phenomenon: the emotional impact of the AE (expressed by the difference between post- and pre-score for each item) and healthcare professionals’ current emotional state (expressed by the post-score for each item). This makes it possible to consider not only the current emotional state but also the emotional distress related to the AE.

In accordance with these facets and subscales, we established two different types of cutoff scores: a first pair of cutoff scores (anxiety and depression) for the facet subscale emotional impact of the AE, and a second pair for the subscales anxiety and depression, highlighting the current emotional distress.

The Coping section, including 27 items, is divided into three subscales: Task-, Emotion-, and Avoidance-oriented coping. Each subscale has two categories: mainly adaptive and mainly maladaptive. Online supplementary File 2 illustrates the structure of the Emotional Distress and Coping sections.

The additional semi-structured interview (i.e., WS-PSY-SI) seeks a detailed understanding of the responses to the WS-PSY-Q and explores temporal and contextual variables that might have affected the psychological reaction and the adaptiveness of the coping strategies (see online suppl. File 1).

The development of the WS-PSY-Q was performed by two researchers (F.M. and R.M.) (see Fig. 1). Items were created ex novo for the WS-PSY-Q starting by the findings of the research groups’ meta-analyses on SVs’ psychological and psychosomatic symptoms and applied coping strategies [4, 5]. As regards the WS-PSY-SI, the areas of interest and the questions were created ex novo by two researchers (F.M. and M.R.) and two psychotherapists (V.M. and L.B.) based on their long-standing clinical experience (i.e., expertise in public health, clinical psychology, psychotherapy, and psychiatry) and the existing literature. Cases of disagreement regarding the structure and content of the WS-PSY-Q and the WS-PSY-SI were solved with the involvement of two additional researchers (I.M.B. and M.A.M.).

The validation process of the Italian version of the WS-PSY-Q was conducted in two phases (see Fig. 1):

• Phase 1: assessment of content validity and clinical utility of the WS-PSY-Q

Content validity was assessed applying cognitive interviews. Clinical utility was examined with the CLIPROM (Clinimetric patient-reported outcome measures) approach [23], exploring sensibility (i.e., evaluation of user-friendliness) and format (e.g., questionnaire length, wording of items, response format).

• Phase 2: assessment of construct validity, concurrent validity, clinical validity, internal consistency of the WS-PSY-Q

Rasch analysis [23] was applied to data collected through an online survey administered to SVs using the platform LimeSurvey [29]. More details are provided in online supplementary File 3 [30‒37].

The phase 1 study sample included 7 interviewees (5 females; 4 doctors, 3 psychologists). The phase 2 study sample initially comprised 287 healthcare providers. Three were excluded due to incongruent answers in the pre-post format, leaving a final sample of 284 participants (229 women). Table 1 displays the sociodemographic characteristics of the survey participants and provides information on the reported AEs.

Following the CLIPROM criteria [23], we assessed the following measurement properties of the Emotional Distress section. Analyses were conducted using R (Version 4.3.1) [38].

• To investigate construct validity, we conducted a set of analyses.

We performed confirmatory factor analysis, using the R package lavaan [39] to test for 2 latent factors (i.e., anxiety, depression). We first conducted preliminary analyses for collinearity among items. We calculated polychoric correlations since the data were not normally distributed (see online suppl. File 4). Since the polychoric correlation analyses indicated a tendency to collinearity, we subsequently checked for potentially redundant items using the semiautomated method of Rasch analysis (in-plus-out-of questionnaire log likelihood – IPOQ-LL) as proposed by Wijayanto et al. [40]. Next, we examined scalability applying Mokken analysis [41, 42] based on Loevinger’s coefficient H.

After checking for the better fit between partial credit model (PCM) and rating scale model using a likelihood ratio test [40, 43], we estimated a PCM assessing item difficulty and described item estimation properties (i.e., item fit, differential item functioning (DIF), local dependency).

• To assess concurrent validity, we performed Pearson correlations between the WS-PSY-Q subscale anxiety and the STAI-Y and the WS-PSY-Q subscale depression and the BDI, respectively.

• To assess clinical validity, we calculated the diagnostic accuracy of the two subscales using receiver operating characteristic (ROC) curve analysis.

• To measure internal consistency, we calculated the Person Separation Reliability Index (PSI) applying PCMs.

We performed two different sets of analysis when assessing construct validity, concurrent validity, clinical validity, and internal consistency. The first set considered the dataset of the pre-post AE differences, focusing on the emotional impact of the AE. The second set of analysis used the dataset post-AE, focusing on the current emotional state of the healthcare provider.

Cognitive interviewees provided comprehensive feedback on the preliminary version of the WS-PSY-Q as described in online supplementary File 5. Based on the data drawn from the interviews, the WS-PSY-Q was revised, thus leading to the version to be clinimetrically assessed. With respect to the WS-PSY-SI, all interviewees expressed a positive opinion, considering it comprehensive and well-organized, and no changes to its wording or general design were suggested.

Polychoric correlations analysis for both datasets identified values ≥0.90 for one item pair (item 14-item 15), indicating collinearity (see online suppl. File 4c). However, the check for potentially redundant items suggested that the number of selected items for both datasets was ideal (online suppl. File 4f, g). Online supplementary Files 4a, b, d, e provide a description of all items and the anxiety and depression subscales for both datasets. Likelihood ratio tests comparing rating scale model and PCM indicate that PCM provides a better fit for both datasets: χ²_(dof) = 55.13₍₁₀₎p < 0.01, χ²_(dof) = 49.69₍₁₂₎p < 0.01 for the anxiety and depression subscales of the emotional impact of the AE dataset and χ²_(dof) = 70.85₍₁₀₎p < 0.01, χ²_(dof) = 87.65₍₁₂₎p < 0.01 for subscales of the current Emotional State dataset (see online suppl. File 4d, e).

Confirmatory factor analysis of both datasets suggested that the proposed bidimensional model is acceptable, with RMSEA being below 0.08 and CFI and TLI being above 0.9 (see Table 2 for details).

As regards the dataset emotional impact of the AE, Mokken analysis revealed Loevinger’s coefficients h ≥ 0.40 for the subscale anxiety and a value of 0.39 for the subscale depression, reflecting a good scalability. Items 1 (h = 0.39) and 12 (h = 0.36) showed the lowest values for the subscale anxiety and depression, respectively, but levels were still acceptable (see online suppl. File 6a).

For the current emotional state dataset, Mokken analysis yielded Loevinger’s coefficients h ≥ 0.40 for both subscales. Item-wise, only item 24 (h = 0.38) of the subscale depression showed a value <0.40 (see online suppl. File 6b).

For the emotional impact of the AE dataset, all items of the subscale anxiety showed ordered thresholds, while items 13, 17, 24 of the subscale depression showed disordered thresholds (see online suppl. File 6c, d). This means that the estimated cutoffs are ordered differently from the expected.

Similarly, for the current emotional state dataset, the subscale anxiety did not show items with disordered thresholds, but the subscale depression included items 8, 13, 16 with disordered thresholds according to the PCM [44] (see online suppl. File 6e, f). The residual analyses at the item level suggest the data are suitable to the unidimensional Rasch model. Namely, in the emotional impact of the AE dataset, all the standardized infit and outfit MNSQ values are within the critical range (z-score < ±2.5). The same goes for the current emotional state dataset, except for item 19, which displays a prominent gap between observed and expected values (see online suppl. File 6g, h). DIF analyses, based on IPOQ-LL-DIF algorithm, compared specific subgroups of responders, split by sex, work experience, time of occurrence of AE, and severity of AE to check whether the profile of SVs differ among them (see online suppl. File 6i, j).

Meaningful DIF was observed for 4 (out of 24) items: item 1 which violates the invariance property for responders with different severity of AE in both datasets, items 8, 10, and 20 were tagged for work experience, severity of AE, and sex, respectively, in the current emotional state dataset.

Any evidence of local dependency, explored by the matrices of standardized Rasch residual correlations, was not observed in the anxiety subscales. For the depression subscale, the couple of items 4 and 6 with a medium magnitude of around 0.40 in both datasets and the couple of items 10 and 12 with a value just above the cutoff (0.31) in the current emotional state dataset are an exception. Please refer to online supplementary File 6k–n for more details.

When analyzing the dataset emotional impact of the AE, moderate correlations between the two subscales of the WS-PSY-Q and the STAI-Y (r = 0.42) and the BDI (r = 0.42) were found. The correlations between these variables improved using the dataset current emotional state, with increased associations of Pearson r = 0.62 and 0.65, respectively (see online suppl. File 7).

For the emotional impact of the AE dataset, ROC calculations revealed a moderate ROC area of 0.71 and 0.70 for the subscales anxiety and depression, respectively, suggesting that the model performance is not ideal but still usable (see online suppl. File 8a).

A cutoff of ≥2 (sensitivity: 75%, specificity: 59%; accuracy index: 68%) was identified for the subscale anxiety. Namely, if there is an overall difference of two points in the participant’s pre- and post-AE scores, his/her emotional distress seems to be linked to a change in the emotional status due to the AE. The subscale depression showed a cutoff of ≥3 (sensitivity: 71%, specificity: 55%, accuracy index: 58%).

For the current emotional state dataset, areas under the curve of 0.80 and 0.86 were found for the subscale anxiety and depression, respectively. A cutoff score of ≥16 (sensitivity: 78%, specificity: 68%; accuracy index: 74%) was found for the subscale anxiety, and a cutoff score of ≥22 (sensitivity: 83%, specificity: 76%, accuracy index: 77%) was identified for the subscale depression. Online supplementary File 8b and c show the ROC curves and provide details on sensitivity and specificity for both datasets.

For the emotional impact of the AE dataset, the Person Separation Reliability Index (PSI) of the subscale anxiety was 0.70, thus acceptable internal consistency. The subscale depression, on the contrary, showed a value of 0.56. For the current emotional state dataset, PSIs were 0.82 and 0.79 for the two subscales, thus acceptable (see online suppl. File 9).

The WS-PSY-Q is the first clinimetric tool assessing SV distress before and after the AE and the adopted coping strategies. The WS-PSY-Q is supplemented by the WS-PSY-SI, a semi-structured interview that can enrich the understanding of the mental health impact of the AE on SVs and the adaptiveness of coping strategies.

We focused on two crucial psychological facets of the SV phenomenon: the emotional impact of the AE on healthcare providers and their current emotional state. The facet emotional impact of the AE demonstrated overall acceptable clinimetric properties. The subscale anxiety met the threshold values indicating the quality of the investigated clinimetric properties. The subscale depression showed only acceptable properties. The facet current emotional state demonstrated overall better clinimetric properties for both subscales. A reason for this finding might be that for the facet emotional impact of the AE, the instruments WS-PSY-Q, BDI and STAI were not fully aligned since the two latter assess a current emotional state, while the WS-PYS-Q focuses on the pre-post differences.

The new tool can be used in clinical care and research. Regarding the first use, the WS-PSY-Q can be applied for screening, allowing mental health providers to identify healthcare providers in need of specific psychological support and to determine the best course of action.

Two different types of cutoff scores can be used: a first pair of cutoff scores is proposed for the subscales anxiety and depression, considering the emotional impact of the AE expressed by the difference between pre- and post-AE; a second pair, again for the subscales anxiety and depression, highlights the current emotional distress. The mental health professionals or researchers administering the WS-PSY-Q can decide according to their clinical or scientific needs which type of screening score to use. For example, the emotional impact of the AE screening is useful when tailoring psychological support to the specific experience of having been involved in an AE. The current emotional state screening gives a broader picture of the psychological state of the healthcare provider, which might indicate the need for further actions to ensure professional well-being.

In research, the questionnaire can be used to explore the psychological burden of AEs and its link to individual and systemic variables. A deep knowledge of SVs’ reactions is also important because AEs constitute a significant burden on the healthcare system in terms of financial loss, loss of productivity, and impaired trust in healthcare institutions [45‒48].

Healthcare organizations can respond in the aftermath of an AE according to the multilevel flowchart proposed in Figure 2, which illustrates the step-by-step process of screening for healthcare providers’ emotional distress applying the WS-PSY-Q and the WS-PSY-SI. In step 1, as a rule of thumb, all healthcare providers who were involved in an AE should be screened using the WS-PSY-Q. Based on literature about peritraumatic assessment [49, 50] and our clinical experience, we suggest that the screening should be ideally conducted within a month from the AE to identify those in need of support and reduce the risk for long-term burnout and allostatic overload. If necessary, the questionnaire can be administered later, but a certain caution is needed also because recall bias could interfere with accurate results, potentially compromising the clinimetric validity of the facet emotional impact of the AE.

In step 2, for those screening negative for emotional distress, training in patient safety and communication techniques should be offered. These courses should embrace a positive safety culture, educate about the importance of transparency at all levels of the healthcare system (i.e., patient, provider, institutional level), and train medical staff on how to effectively respond to unanticipated adverse outcomes [5, 48, 51]. Healthcare providers with emotional distress, according to at least one of the four cutoff scores, should undergo the WS-PSY-SI to further explore the impact of the AE.

In step 3, healthcare providers with emotional distress confirmed by the WS-PSY-SI should receive psychological support specifically developed for SVs. These interventions (individual and/or group sessions) should enhance SVs’ resilience and foster a just culture, in other words, a culture that holds healthcare providers responsible without blame [52‒55]. Further, SVs should be equipped with communication tools for error disclosure to patients and caregivers and with problem-solving strategies for AE management. If, by contrast, the WS-PSY-SI does not show direct links between the AE and the reported distress, other mental health services (e.g., psychological counseling, psychotherapy) should be suggested.

To provide a comprehensive response after an AE, organizations should consider action not only at the provider but also at the patient and the institutional level. Regarding the patient level, an open and transparent event disclosure is highly recommended, especially if the patient has been harmed [53, 55‒59]. Healthcare institutions should guarantee resources for patients and caregivers for such purpose [52]. Given the importance of patient autonomy in healthcare and safety [51, 60, 61], another key priority should be to promote patient empowerment [62] to improve patient’s self-determination and strengthen the patient-provider relationship [63].

At the institutional level, a comprehensive program of patient safety investigation, integrating patient’s viewpoint, is needed to thoroughly analyze the interplay of the different factors that lead to the AE [57, 64]. In terms of prevention, healthcare institutions should strive to become high reliability organizations being committed to resilience [65]. Instruments to evaluate certain prerequisites of high reliability (e.g., safety culture, leadership) as well as at which point a healthcare organization is on the path to become a high reliability organization can help in this endeavor [65, 66].

Limitations need to be considered. First, we adopted a convenient sample in recruiting participants. This may introduce self-selection and nonresponse bias, thus limiting generalizability of the results. Second, since our questionnaire is self-administered and discusses sensitive issues (e.g., responsibility for the AE), social desirability bias might have affected the response to items. Some respondents may also lack introspective abilities and therefore struggle to recognize certain symptoms or coping strategies, particularly avoidance ones [5]. Lastly, as the questionnaire retrospectively assesses healthcare providers’ emotional distress and coping, it is vulnerable to recall bias. The lower clinimetric validity of the facet emotional impact of the AE compared to the facet current emotional state might be due to inaccurate or incomplete recollections reported by our participants, particularly those whose involvement in the AE happened already long ago. For this reason, we recommended to screen for emotional distress in the month following the incident, no later.

A multivariable analysis of SVs’ emotional distress and their adaptive/maladaptive coping strategies, assessing also the links to other variables, such as characteristics of the AE and organizational climate, will be conducted. Sensitivity to change of the WS-PSY-Q (i.e., improvement after a psychological intervention tailored to SVs’ needs) will be examined. Validation of the tool in different languages and cultures is recommended given that the SV is a global phenomenon. Last, the meaningful DIF observed in few items in both datasets might be further investigated to determine the equivalence of items or the presence of measurement bias.

The WS-PSY-Q is the first open access clinimetric evaluation tool of SVs’ emotional distress comparing psychological symptoms before and after the AE and assessing the adopted coping strategies. It can be used for clinical screening to help tailor psychological support for SVs and for research purposes to explore the psychological impact of AEs and their relationship with individual and systemic variables. Assessment with the WS-PSY-Q and the WS-PSY-SI could help identify healthcare providers’ needs, find the appropriate level of mental healthcare, and mitigate burnout and allostatic overload in the long term. Promoting professional psychological well-being is a top priority of the healthcare system to increase the quality and safety of the care.

The authors wish to thank Dr. Anna De Santi and Dr. Eloise Longo from the Italian National Institute of Health, Dr. Diana Pascu, manager of the operating unit “Risk Management,” local socio-health unit Scaligera, Verona, Dr. Antonella La Commare, manager of unit for professional development and training of the Local Health Unit of Trapani, as well as Dr. Roberta Chiesa, Health Director of the Local Health Unit (ASTT) of Valcamonica, and Dr. Francesca Andreoli, medical director of the same ASST facility, for their contribution and support in recruiting participants and thank all participants of this study for their valuable support.

The study complies with the guidelines for human studies and was conducted ethically in accordance with the World Medical Association Declaration of Helsinki. This study protocol was reviewed and approved by the Committee on Human Research of the University of Verona [Comitato di Approvazione della Ricerca sulla Persona-CARP, Università di Verona], Protocol number no. 03/2021. Written informed consent was obtained from participants to participate in the study.

The authors have no conflicts of interest to declare.

There are no funding sources to declare.

All authors jointly designed the study. I.M. Busch, L. Berti, V. Marinelli, and F. Moretti conducted the recruitment process, curated the data, and performed the visualization of tables and figures. M.A. Mazzi conducted the statistical analysis, F. Cosci supervised the adoption of the clinimetric approach, and A.W.Wu supervised on content of risk management and the second victim phenomenon, and M. Rimondini on the psychological content. I.M. Busch, M.A. Mazzi, and M. Rimondini drafted the initial manuscript, which was then revised by the other co-authors. All authors approved the final manuscript.

The data that support the findings of this study are not publicly available due to their containing information that could compromise the privacy of research participants, but further inquiries can be directed to the corresponding author.