Validation of Hospital Performance Measures of Acute Stroke Care Quality. Riksstroke, the Swedish Stroke Register

Registers are increasingly used to monitor and benchmark between-hospital or between-country stroke care performance [1,2,3,4,5,6,7,8,9]. However, the quality of data included in the registers is often insufficiently assessed. So, a fundamental question is: Is the quality of the register data good enough to permit fair assessments of stroke care quality and benchmarking? This question comes more in focus as the registers are increasingly used for reimbursements and research.

Riksstroke, the Swedish Stroke Register, is a national stroke care quality register established in 1994. Riksstroke records information in the acute phase of stroke and, by questionnaires to survivors at follow-ups during the 3rd and 12th month after stroke. In this article, we report validation studies on items recorded during in-hospital care for acute stroke. We have addressed 3 key validity issues: (i) completeness of recording in-hospital stroke events (so that the register gives an all-inclusive picture of stroke care quality), (ii) content validity (the extent to which the items recorded in Riksstroke is considered by stroke experts to be useful to measure the quality of care), and (iii) inter-hospital reliability (variations between different hospitals recording the same items).

The main features of Riksstroke have previously been published [6] and are presented in detail at the Riksstroke website www.riksstroke.org, as are registration forms.

Riksstroke was established in 1994 with the primary aim of serving as a national register monitoring the quality of stroke care and to stimulate improvement by providing health care providers with the ‘Study' component of the Plan-Do-Study-Act cycle for quality improvement [10]. Riksstroke provides online feedback to participating hospitals with several benchmarking options and produces annual reports with comprehensive assessments of stroke care quality. At the national level, Riksstroke is an instrument to follow to what extent stroke guidelines issued by the National Board of Health and Welfare (the national governmental agency for health care) are adhered to [11]. A secondary aim of Riksstroke is to provide a database for research on stroke management.

The present report summarizes observations from studies evaluating items recorded in the acute phase of stroke. The evaluations were performed in 2012-2013 except evaluation of completeness, for which data from year 2014 was used. The validation studies are summarized in figure 1.

To estimate completeness, events recorded in Riksstroke were individually matched with acute stroke diagnoses (ICD-10 codes I61, I63 and I64) at discharge from hospital in the National Patient Register (NPR) at the National Board of Health and Welfare. NPR is compulsory and comprises all admissions to hospital [12]. In the calculations of completeness, the number of events in Riksstroke was the nominator and the number of stroke events in NPR plus cases identified by Riksstroke but not by NPR was the denominator. Emergency room visits without admission were not included. Multiple hospitalization events with an acute stroke diagnosis occurring within 28 days of the index event were excluded. As previously reported, the accuracy of acute stroke diagnoses in NPR was validated by comparison with a population-based stroke research register (Northern Sweden MONICA Study) with individual assessment of each case [13].

Content validity was assessed by 2 sets of comparisons. First, we compared items recorded in Riksstroke with a core set of 22 stroke care performance measures identified by an international panel of stroke experts (stroke physicians, neurologists, public health academics and representatives from patient organisations) gathered under the auspices of European Implementation Score (EIS) project. This set of measures is intended to be used for cross-national comparisons of the quality of acute stroke care [14]. Second, we compared the Riksstroke variables with items that are recorded by other European stroke audits and registers (Belgium, England/Wales/Northern Ireland, Germany, Scotland, Spain), also within the EIS project; only the 15 variables that were used by at least 2 of the other 5 registers were included in this comparison [15].

Inter-hospital reliability was assessed by distributing a set of 5 anonymized test cases, taken from routine medical records in an abbreviated form (deleting redundant information but otherwise verbatim), to hospitals participating in Riksstroke, asking them to record the cases by their regular routines for acute stroke care (81 items). The purposes were (a) to identify variables with low inter-hospital reliability, and (b) to identify hospitals with a high proportion of deviant responses. Pre hoc, a working group of experienced stroke staff members agreed on what coding should be regarded as correct. For each item, overall reliability was thus calculated based on 5 cases in 67 hospitals (325 observations). Between-hospital variations were also calculated.

Since 1998, Riksstroke covers all hospitals in the country admitting acute stroke patients (72 hospitals in 2014). In 2014, there was a total of 27,230 events with a diagnosis of acute stroke at hospital discharge in the national administrative register NPR or identified by Riksstroke but not NPR. Of these events, 23.253 (89.5%) were recorded in Riksstroke. For first-ever stroke events, 18,440 of 20,421 (90.3%) were recorded in Riksstroke. When the NPR was validated, the proportion of hospital admissions with a false positive diagnosis of acute stroke was 6% for first-ever events and 12% for all-stroke events [13]. The completeness by hospital is shown in figure 2. Riksstroke has set 2 target levels for coverage (without adjustment for possible false positive cases in the NPR), moderate at ≥85% and high at ≥92%. The high target level was reached by 37 of the 72 hospitals (51%), whereas 24 hospitals (33%) were in the 85-91% range and 11 hospitals (15%) had coverage below 85%.

To assess content validity in an international perspective, we compared items recorded in Riksstroke with the core set of quality indicators identified by an international panel of stroke experts, as described in Methods. The panel defined a set of 22 quality indicators in 6 domains that ideally would be included in a stroke quality register to permit cross-country comparisons. In addition, 6 background variables to be used in case-mix adjustments were identified [14]. Table 1 shows to what extent Riksstroke covers this ideal set of indicators. Of the 22 variables identified by the international panel, 18 are included in Riksstroke. One item (dead or ADL dependent) is assessed at a different time point in Riksstroke (at 3 months) than proposed by the panel (30 days). Carotid surgery is recorded in the national vascular surgery register Swedevasc but not in Riksstroke. Early administration of aspirin (2 items proposed by the panel) is not recorded in Riksstroke.

In the next step to assess the content validity, we compared the Riksstroke variables with those in 5 other European stroke audits and registers included in EIS. The indicators used by the EIS registers were highly heterogeneous. There were 15 variables that were used in at least 2 of the registers. Riksstroke included 14 of them (table 1). The item not recorded in Riksstroke was early administration of aspirin. While some other European registers include assessment for depression during hospital stay, Riksstroke measures the self-assessed mood 3 months after stroke.

Of the 72 hospitals participating in Riksstroke, 67 recorded the 5 stroke test cases. Figure 3 shows the distribution of incorrect codings in the 81 variables in the Riksstroke acute care register. For a great majority of variables (67 of 81), the hospitals coded the variables correctly with only a few deviations (0-5% incorrect codings). For 4 variables, there were more than 15% incorrect codings. These were delayed from onset to arrival in hospital, test of swallowing, day of stroke onset and CT angiography or MR angiography. The proportion of incorrect codings ranged from 0 to 7% between hospitals.

Overall, these studies show that from different validation perspectives, a nationwide stroke care register like Riksstroke can be maintained with sufficient data quality to permit monitoring of the quality of care and benchmarking between hospitals. Thus, completeness and content validity are very satisfactory, and inter-hospital reliability is high.

To be able to make fair comparisons between hospitals, it is essential that the register covers all patients admitted for acute stroke. In the early years when Riksstroke was put to use, patients who died early soon after admission were often missed out in some hospitals [16,17]. This may have resulted in a spuriously low case fatality rate in these hospitals. In recent years, the coverage has been 90% when compared to the NPR. Since there is a considerable overdiagnosis of acute stroke in NPR (false positives), the actual completeness of Riksstroke is probably well above 90%. A previous validation study showed false positive stroke diagnoses among patients recorded in Riksstroke to be 1.4% [16]. Thus, fair between-hospital benchmarking of the quality of acute stroke care seems possible. Although originally not intended for epidemiological purposes, Riksstroke now also meets next-to-all criteria for an ‘ideal' stroke epidemiological study, as defined by Sudlow and Warlow [18]. Riksstroke does not, however, cover out-of-hospital events. As summarized by Hallström et al. [19], a number of regional epidemiological studies have shown that home care for acute stroke is infrequent in Sweden; an exception may be institutionalised frail elderly people with multiple chronic disorders.

When considering what variables should be recorded in a quality assessment register, a balance must be struck between simplicity and content validity covering all important aspects of stroke care quality. Extensive recordings may deter less-enthusiastic hospitals, that is, hospitals mostly in need of monitoring the quality of care, from participating in the register. Our content validity studies show that Riksstroke includes next to all core variables that are identified by experts and other European registers, the exception in Riksstroke being acute administration of aspirin. The preconditions for collecting performance indictor data may vary greatly from country to country. The present Riksstroke form includes 81 items. This seems to be within the limits of what the stroke care staff can cope with (although there are frequent complaints about the administrative burden in Swedish health care). If more variables are considered to be added, their validity should be carefully tested in advance. Ongoing national work with automatic transfer of data from computerized medical records to national quality registers is intended to alleviate the burden of recording in Riksstroke and other national quality registers.

Coding of test cases is an inexpensive but underused method to estimate register data quality. The inter-hospital reliability was generally high for the items included in the Riksstroke acute care form. For a few items, however, the reliability was unsatisfactory, indicating that the instructions in the Riksstroke manual should be improved. The fact that none of the participating hospitals deviated much from the rest suggests that local educational activities on Riksstroke recording procedures are of satisfactory quality. The number of test cases was, however, low and it is possible that more test cases could have affected the figures on reliability (in either direction). The fact that this was a test exercise may also have affected the results.

Due to a very marked heterogeneity between European stroke registers in terms of what is recorded, the part of the content validity analyses that pertains to comparisons with other registers should be interpreted with caution.

The assessments of inter-hospital reliability were based on test cases with abbreviated medical records. Compared to regular cases, the coders therefore had more limited information and they may have had difficulties to find the necessary information. This may have resulted in a lower proportion of correct answers and, thus, an overestimate of the inter-hospital variability.

Our validation studies show that several aspects of the quality of a stroke care register can be assessed to be used for further development of the register. Although Riksstroke seems to perform satisfactorily as a register in most aspects, sometimes even excellent, our validations have identified some items that should be improved or deleted. Our experiences may serve as a model for other stroke registers in their efforts to evaluate their data quality. As data quality may change over time, it seems essential that repeated validations are incorporated in the management of stroke care performance registers.

Åsa Johansson, RN, and Sari Wallin, RN, contributed to the assessments of inter-hospital variability. Maria Sukhova, MSc, provided assistance in data management.