Characterizing the Validity of Using VASES to Derive DIGEST-FEES Grades Open Access

Flexible endoscopic evaluation of swallowing (FEES) is an instrumental procedure used to aid in the comprehensive assessment of functional (bolus-related) swallowing outcomes, including impairments in swallowing efficiency (pharyngeal residue) and safety (penetration and aspiration) [1]. Given the widespread use of FEES in clinical and research practices [2, 3], it is important to standardize how functional swallowing outcomes are interpreted and reported during FEES. Two complimentary methods used to guide standardized assessment of functional swallowing outcomes during FEES include the Dynamic Imaging Grade of Swallowing Toxicity for FEES (DIGEST-FEES) [4] and the Visual Analysis of Swallowing Efficiency and Safety (VASES) [5, 6].

DIGEST-FEES is a rating scale originally adapted from its videofluoroscopic swallow study counterpart – DIGEST [7, 8]. DIGEST-FEES uses trial-level ratings of pharyngeal residue, penetration, and aspiration to ultimately derive protocol-level impairment grades in swallowing safety (DIGEST_Safety), swallowing efficiency (DIGEST_Efficiency), and overall pharyngeal swallow dysfunction (DIGEST_Total). One reason DIGEST-FEES may be of interest for clinicians and researchers is because it facilitates a standardized method to characterize the severity of a person’s swallow impairment and dysphagia diagnosis.

Alternatively, VASES is an interpretation framework which establishes (1) explicit boundaries for six anatomic structures and four swallowing-related temporal phases and (2) distinct methods to rate swallowing safety and efficiency. The intent of VASES is to facilitate standardized, transparent, and reliable trial-level ratings of pharyngeal residue, penetration, and aspiration. Importantly, preliminary normative data have recently been published for VASES which may further aid clinicians in accurate diagnosis of swallowing impairments [9].

Using trial-level ratings with VASES to simultaneously derive protocol-level grades from DIGEST-FEES has important clinical and research implications. First, it has the potential to increase workflow efficiency for people using both DIGEST-FEES and VASES by eliminating the need to analyze each swallow twice – once with VASES and then again with DIGEST-FEES. Second, it has the potential for people who use DIGEST-FEES to further enhance the transparency and reliability of DIGEST-FEES grades through the use of VASES explicit interpretation framework. Third, it has the potential for people who use VASES to obtain protocol-level grades to summarize a person’s swallow dysfunction. However, methodological differences exist between DIGEST-FEES and VASES which may impact the validity of using VASES to derive DIGEST-FEES grades. For example, DIGEST-FEES residue ratings are made by estimating bolus clearance through the pharynx after the initial swallow attempt, whereas VASES residue ratings are made by estimating the amount of residue filling or covering a structure at the offset after the swallow phase.

Given this, the primary aims of this study were to (1) characterize the agreement between original DIGEST-FEES grades and VASES-derived DIGEST-FEES grades and (2) characterize the relationship between VASES-derived DIGEST-FEES grades with measures of construct validity. We hypothesized that substantial agreement would be present between original DIGEST-FEES grades and VASES-derived DIGEST-FEES grades. We also hypothesized that the strength of the relationships between VASES-derived DIGEST-FEES grades and measures of construct validity would be similarly strong as the relationships observed between original DIGEST-FEES grades and the same measures of construct validity.

This was a prospective, within-subject study design comparing DIGEST-FEES grades assigned using the originally published method to grades assigned using VASES. The inclusion criterion for study participation was having been a rater in the original DIGEST-FEES publication [4]. The exclusion criterion was having knowledge of the VASES rating framework prior to DIGEST-FEES publication. The purpose of these inclusion-exclusion criteria was to prevent including raters with any direct or indirect exposure to the VASES framework which could potentially bias DIGEST-FEES results. As a result, two of the three raters from the original DIGEST-FEES publication were included in the present study. The two raters were PhD-level speech-language pathologists, one of whom had 12 years of experience performing and interpreting FEES, and the other of whom had 20 years of experience performing and interpreting FEES.

Each of the two raters had previously used DIGEST-FEES to rate the 100 FEES included in the original DIGEST-FEES publication [4] – prior to the development of VASES. Original DIGEST-FEES grades were made by assigning trial-level ratings of the Penetration-Aspiration Scale (PAS) [10] and trial-level ratings of pharyngeal residue for each FEES video. Pharyngeal residue ratings were made using a 4-point categorical scale: <10%, 10–33%, 34–66%, and >66%.

Two years after completing the original DIGEST-FEES ratings, the raters completed 5 h of VASES training led by the first author. The first author is a speech-language pathologist with board certification in swallowing and swallowing disorders and is one of the original developers of the VASES framework. Upon completion of VASES training, the last author randomly assigned 25 FEES videos to each of the two raters. A total of 50 unique FEES were assigned which represented a range of swallow impairment levels, including normal to mild impairment, moderate impairment, and severe to profound impairment. Videos were selected from FEES previously analyzed by the two raters from the original DIGEST-FEES publication. Raters were blinded to their original DIGEST-FEES ratings from 2 years prior. FEES exams utilized a standardized swallowing protocol of two trials of each of 5 mL, 10 mL, and self-selected volume cup sips of blue-dyed thin liquid; a spoonful of blue-dyed applesauce; and a self-selected volume bite of graham cracker.

Raters analyzed each FEES using VASES by first viewing each video clip in real time and then using frame-by-frame analysis as needed. VASES outcome measures included trial-level ratings of PAS and percentage-based residue ratings for the oropharynx, hypopharynx, epiglottis, laryngeal vestibule, vocal folds, and subglottis. VASES-derived DIGEST-FEES grades were then established for each FEES exam using these new VASES ratings.

To establish VASES-derived DIGEST-FEES grades, it was first necessary to develop one singular measure of “pharyngeal residue” (as is done with DIGEST-FEES) from VASES’ six residue ratings. Combinations of VASES residue ratings were developed a priori, which included (from most parsimonious to least parsimonious):

• 1.

  Maximum rating between the oropharynx and hypopharynx

• 2.

  Average rating between the oropharynx and hypopharynx

• 3.

  Weighted ratings of 33% for the oropharynx, 33% for the hypopharynx, and 33% for the maximum rating across the epiglottis, laryngeal vestibule, vocal folds, and subglottis

• 4.

  Maximum rating across the oropharynx, hypopharynx, epiglottis, laryngeal vestibule, vocal folds, and subglottis

• 5.

  Average rating across the oropharynx, hypopharynx, epiglottis, laryngeal vestibule, vocal folds, and subglottis

• 6.

  Weighted ratings of 25% for the oropharynx, 25% for the hypopharynx, 12.5% for the epiglottis, 12.5% for the laryngeal vestibule, 12.5% for the vocal folds, and 12.5% for the subglottis.

Spearman’s rank correlation coefficients (ρ) were used to assess the relationship between each combination outlined above with the pharyngeal residue ratings made using the original DIGEST-FEES method. A correlation was sufficiently stronger than another if its coefficient was higher than the upper boundary of the 95% confidence interval (CI) of the contrasting correlation. The strongest and most parsimonious correlation was used as the measure of pharyngeal residue for DIGEST-FEES. Notably, only one-third of the total dataset was randomly selected for this preliminary analysis to avoid overfitting data for the primary aims.

All data were analyzed using R version 4.2.3 [11]. Each rater blindly re-rated 20% of their assigned FEES to assess intra-rater reliability. The first author rated 20% of the FEES videos to assess inter-rater reliability. Intra- and inter-rater reliability were statistically analyzed using weighted Kappa and absolute agreement. To remain consistent with interpretation from the original DIGEST-FEES publication, weighted Kappa was interpreted as “no agreement” when κ_w ≤ 0, “none-to-slight” agreement when κ_w = 0.01–0.20, “fair agreement” when κ_w = 0.21–0.40, “moderate agreement” when κ_w = 0.41–0.60, “substantial agreement” when κ_w = 0.61–0.80, and “almost perfect” agreement when κ_w = 0.81–1.00 [4].

Concurrent validity was assessed by comparing the raters’ original and VASES-derived DIGEST-FEES grades. Concurrent validity between the DIGEST-FEES grades was analyzed using weighted Kappa (κ_w) and absolute agreement. To be consistent with the original DIGEST-FEES publication, weighted Kappas were assessed using quadratic weights and were interpreted as previously outlined. Additionally, nonparametric Wilcoxon signed rank tests were used to determine if significant differences were present between original and VASES-derived DIGEST-FEES grades, with an alpha of p < 0.05 set as the level of statistical significance.

Construct validity was assessed by comparing VASES-derived DIGEST-FEES grades to the same measures of construct validity from the original DIGEST-FEES publication: the MD Anderson Dysphagia Inventory (MDADI) total, functional, physical, and emotional scores; Functional Oral Intake Scale (FOIS) [12]; Secretion Severity Scale (SSS) [13]; Yale Pharyngeal Residue Severity Rating Scale for the Valleculae (Yale Valleculae) and Piriforms (Yale Piriforms) [14]. Spearman’s correlation coefficients were used to assess the relationship between VASES-derived DIGEST-FEES grades and the measures of construct validity. Spearman’s correlation coefficients were also used to assess the relationship between the raters’ original VASES-derived DIGEST-FEES grades and the same measures of construct validity. VASES-derived DIGEST-FEES grades were considered significantly different from the original DIGEST-FEES grades if the original DIGEST-FEES correlation coefficient was above or below the 95% CI of the VASES-derived DIGEST-FEES grades.

A total of 50 FEES were included in this study, yielding an analysis of 544 swallowing trials. Of the 50 FEES included in this study, 6 (12%) had no impairment in swallow function, 16 (32%) had mild impairment in swallow function, 16 (32%) had moderate impairment in swallow function, 11 (22%) had severe impairment in swallow function, and 1 (2%) had profound impairment in swallow function. Demographic data are outlined in Table 1.

Intra-rater reliability for VASES-derived DIGEST grades was “almost perfect” for DIGEST_Safety (κ_w = 0.914; absolute agreement = 88.9%), DIGEST_Efficiency (κ_w = 0.962; absolute agreement = 88.9%), and DIGEST_Total (κ_w = 0.903; absolute agreement = 77.8%). Inter-rater reliability was also “almost perfect” agreement for DIGEST_Safety (κ_w = 0.914; absolute agreement = 88.9%), DIGEST_Efficiency (κ_w = 0.887; absolute agreement = 66.7%), and DIGEST_Total (κ_w = 0.903; absolute agreement = 77.8%).

All six combinations of the approaches to derive a composite VASES residue rating exhibited “strong” correlational relationships [15] with the 4-point DIGEST-FEES ratings of pharyngeal residue (r = 0.75–0.77) – with no relationship sufficiently stronger than another. Therefore, the most parsimonious combination, which involved the maximal residue rating between either the oropharynx or hypopharynx, was used to establish the singular measure of pharyngeal residue for DIGEST-FEES grades (Fig. 1).

Agreement between original and VASES-derived DIGEST-FEES grades is outlined in Table 2. Weighted Kappa revealed “substantial” agreement for DIGEST_Efficiency, and “almost perfect” agreement for DIGEST_Safety and DIGEST_Total. Descriptively, 60–62% of the original and VASES-Derived DIGEST-FEES grades matched exactly. All VASES-derived DIGEST_Safety and DIGEST_Total grades were within one grade of the original, and 92% of the VASES-derived DIGEST_Efficiency grades were within one grade of the original. Of the 19 VASES-derived DIGEST_Safety grades that did not agree exactly, 10 (52%) were less severe than the original and nine (48%) were more severe than the original. Similarly, of the 20 VASES-derived DIGEST_Efficiency grades that did not agree exactly with the original, 10 (50%) were less severe than the original and 10 (50%) were more severe than the original. Of the 20 VASES-derived DIGEST_Total grades that did not agree exactly, 12 (60%) were less severe than the original and eight (40%) were more severe than the original. Lastly, no significant differences were observed between original and VASES-derived grades for DIGEST_Safety (p = 0.51, r = 0.16, 95% CI: −0.24, 0.51; Fig. 2), DIGEST_Efficiency (p = 0.34, r = 0.24 95% CI: −0.53, 0.10; Fig. 3), DIGEST_Total (p = 0.38, r = 0.20, 95% CI: −0.14, 0.50; Fig. 4).

Correlation coefficients and their 95% CIs are outlined in Table 3. The correlations between the MDADI, FOIS, SSS, Yale Valleculae, and Yale Piriforms ranged from r = 0.34–0.83 for original DIGEST-FEES grades and r = 0.34–0.78 for VASES-derived DIGEST-FEES grades. There was insufficient evidence to demonstrate that the relationships between VASES-derived DIGEST-FEES grades and measures of construct validity were significantly stronger or weaker than the relationships between the original DIGEST-FEES grades and the same measures of construct validity.

This study aimed to determine if trial-level VASES ratings could be used to simultaneously derive valid and reliable protocol-level DIGEST-FEES impairment grades. From a concurrent validity standpoint, there was a high level of agreement between original DIGEST-FEES grades and VASES-derived DIGEST-FEES grades – with no statistically significant differences between the two grading methods. While these findings support the concurrent validity of VASES-derived DIGEST-FEES, it should be noted that the lack of statistically significant differences does not imply the two grading methods are necessarily equivalent. In fact, 38–40% of the VASES-derived DIGEST-FEES grades did not match exactly with the original DIGEST-FEES grades. However, there was no observable trend of VASES-derived DIGEST-FEES grades being consistently higher or lower than original DIGEST-FEES grades. There were also no observable trends of VASES-derived DIGEST-FEES grades being consistently higher or lower than original DIGEST-FEES grades as an effect of demographics, oncological medical history, or dysphagia severity (online suppl. material; for all online suppl. material, see https://doi.org/10.1159/000538935). Therefore, the differences that were present in DIGEST-FEES grades between the two rating methods may reflect a combined effect of methodological differences and rater reliability.

From a construct validity standpoint, original DIGEST-FEES grades and VASES-derived DIGEST-FEES showed similarly strong relationships with important metrics of swallow-related function, including management of secretions, functional oral intake, and impact of dysphagia on swallowing-related quality of life. Therefore, even while VASES-derived DIGEST-FEES grades may not perfectly align with original DIGEST-FEES, using VASES to derive DIGEST-FEES grades appears to be a valid and meaningful way to characterize swallow impairment at the protocol level.

VASES is intended to increase the transparency, standardization, and reliability of FEES analysis within clinical and research practices. In the present study, both intra- and inter-rater reliability were characterized as “almost perfect” (κ_w = 0.88–0.96), with exact agreement ranging from 66.7% to 88.9%. Descriptively, the rater reliability in the present study appears higher than the reliability results which have been previously reported for DIGEST-FEES [4, 16, 17]. While reliability can be affected by factors beyond rater reliability (e.g., number of raters, number of FEES, and variability in grades used for reliability analysis) [18], the use of VASES may be one reason for descriptively higher reliability results. These findings support the notion that VASES may be one way to increase transparency, standardization, and reliability of FEES analysis at both the bolus and protocol (DIGEST-FEES) levels.

There are several limitations to this study. First, the methods used to combine VASES residue ratings into one pharyngeal residue rating were created a priori through expert panel discussion. All the methods in the study were similarly strong, and therefore, the simplest method was used for data analysis. However, it is possible that other unexplored combinations of VASES residue ratings may be more accurate. It is also possible that the method for deriving one overall rating of residue with VASES may vary across consistencies, which was unexplored in the current study. Second, all FEES included in the current study were in patients with head and neck cancer. This may limit the generalizability of results to other patient populations. Lastly, this study was not designed to compare differences in intra- and inter-rater reliability between the original and VASES-derived DIGEST-FEES grading methods. Therefore, future research should more comprehensively assess the effects of VASES on intra- and inter-rater reliability of DIGEST-FEES by including a greater number of raters, with a range of FEES interpretation experience, who rate FEES using both grading methods.

Findings from this study demonstrate that analyzing FEES at the trial level with VASES may be a valid and reliable approach to derive global ratings of swallowing safety, swallowing efficiency, and overall swallow function using DIGEST-FEES. While results indicate that DIGEST grades derived from VASES do not perfectly align with DIGEST-FEES, nearly all participants were within one grade. Future research should expand on this study in different patient populations.

All procedures were performed in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This study protocol was reviewed and approved by the Institutional Review Board of Stanford University (Approval #IRB-51439). The need for informed consent for this study was waived by the Institutional Review Board of Stanford University.

The authors have no relevant financial disclosures or conflicts of interest.

This study was supported in part by the American Speech-Language-Hearing Foundation’s 2018 Researcher-Practitioner Collaboration Grant originally awarded to Heather Starmer and Dr. Kate Hutcheson.

All authors met the minimum criteria for authorship status, as proposed by the International Committee of Medical Journal Editors. Authorship contributions are outlined below using CRediT (https://credit.niso.org/).

James Curtis: conceptualization, data curation, formal analysis, investigation, methodology, project administration, resources, software, visualization, and writing – original draft. Loni Arrese and Lauren Tabor Gray: data curation, project administration, resources, and writing – reviewing and editing. James Borders: conceptualization, investigation, methodology, validation, visualization, and writing – reviewing and editing. Heather Starmer: conceptualization, data curation, funding acquisition, investigation, methodology, resources, supervision, and writing – reviewing and editing.

The data that support the findings of this study are not publicly available due to institutional restrictions. However, data may be available upon reasonable request and completion of a data use agreement. Requests for data can be made to the senior author (H.S.).