Development of Quality Indicators to Evaluate the Quality of Care for People with Differences of Sex Development

Achieving evidence-based, high-quality medical care and avoiding undertreatment, overtreatment, or inappropriate treatment are the overall goals of healthcare quality management. Healthcare is a complex process, and therefore, its quality is a multidimensional construct that cannot be measured directly [1‒5]. To assess healthcare quality, it is first necessary to determine what quality means in the respective context, bearing in mind that it is not possible to evaluate it in all its facets [6].

Quality indicators (QIs) are not direct measures of the quality of care, but they serve as proxies to show whether good quality is achieved or not. The objective of creating QI is not to map all potential aspects of the quality of care; rather, it is to identify the most crucial parameters, which can also be readily operationalised and clearly queried. QIs have been “relevant, valid, reliable, practicable, and appropriate for the target group” [7] and thus enable to distinguish between good and poor qualities.

In many healthcare contexts, QIs are classified as structural, process, and outcome quality [8]. This can help medical centres to identify “priority areas for quality improvement” [9], and QI can “be used for monitoring, management, and policymaking within and across healthcare systems” [9].

An example for complex requirements in healthcare is the needs of people affected by differences of sex development (DSD)¹. The umbrella term DSD refers to a heterogeneous group of rare congenital conditions that have in common an incongruence between chromosomal, gonadal, or phenotypic sex [10]. With an overall incidence of less than 5 in 10,000 people affected, DSD is classified as a rare disease according to the European Union’s definition [14]. Rare conditions pose a particular challenge for medical and psychosocial care, and support: lack of expertise and standards leads to delayed diagnoses or misdiagnoses, and lack of structured and evidence-based treatment services can lead to inadequate care and poor outcomes [15, 16]. National and international guidelines recommend multidisciplinary or interdisciplinary care in specialised DSD centres, including psychosocial support, targeted laboratory diagnostics, therapeutic options, such as hormone therapy, fertility preservation, and, if necessary, surgical therapy, and early involvement of peer counselling [10, 17]. However, evidence for these guidelines is not robust and indicators for quality assessment have yet to be established.

The German healthcare system is distinguished by a high degree of complexity and a considerable number of stakeholders involved, including politicians, health insurance funds, service providers, and others. The principle of free choice of doctor and decentralised structures present challenges to targeted patient management.

Since the 2010s, there have been increasing efforts at both European and national levels to establish adequate care structures for people with rare diseases. For instance, Centres for Rare Diseases (ZSE) have been established at university hospitals, European (since 2017) and German (since 2020) Reference Networks (European Reference Networks [ERNs], DRNs) were founded, and the National Action League for People with Rare Diseases (NAMSE) is working on coordinating and pooling activities and information [18, 19].

Despite all these efforts – including those of the DSD centres themselves, which try to attempt their services through various channels, including publications, lectures, educational courses, information brochures, and websites – the access to specialist centres for DSD in Germany remains unstructured and, in many cases, depends on “lucky coincidences.” It is still not uncommon for patients to only find a specialist centre after a lengthy diversion. It is not possible to make a well-founded statement about the number of individuals with DSD in Germany who are never seen in a specialist centre, who are not diagnosed, and who may not receive appropriate care.

Recent social and legal developments in Germany, such as the introduction of the category “diverse” in the register of births, marriages, and deaths for people with DSD in December 2018 (§§ 45 b, 22 para. 3 PStG.) [20] and the legal regulation of medically not indicated interventions on the genitals of underage children in 2021 (§ 1631e BGB) [21], emphasise self-determination and human rights. This may result in an enhanced awareness among practitioners and those affected, which will in turn lead to an increasing orientation towards the treatment guideline [22] and thus increased use of the specialised DSD centres in the future. This applies, for example, to isolated proximal hypospadias, which surgeons are increasingly referring to DSD centres because of the new law, as these centres have the interdisciplinary committees required by law to prepare a statement for the family court.

The aim of the DSDCare project, funded by the Federal Ministry of Health 2020–2023, was to improve care for people with DSD over the lifespan [23]. Ten DSD centres and two self-organisations/self-advocacy groups and two scientific institutes (experts for IT and evaluation at the Universities of Frankfurt and Lübeck) were involved in this project. The centres included were those in Germany that met the criteria of the European Reference Network (ERN) for Rare Endocrine Conditions. As the DSDCare project relates directly to the implementation of the recommendations made in the German guideline “variants of sex development” [19], those self-advocacy groups involved in the creation and revision of the guideline were included (Intergeschlechtliche Menschen e.V., AGS-Eltern- und Patienteninitiative e.V.).

The tasks of the project included the development of various tools (e.g., standard operating procedures, structured internal and external case conferences) to standardise existing tools across the centres. A regular benchmarking should be carried out to assess the quality of care for people with DSD in Germany in a standardised way and to identify avenues for improvement. This initially required the development of specific QIs in DSD – which is the focus of this article.

Based on the model of Donabedian [8, 24, 25], we developed QI to assess defined relevant aspects of the quality at the levels of structural, process, and outcome quality in the care for DSD [23]. All ten clinical DSD centres included in the DSDCare project were involved in the development of the QI and the benchmarking system.

The starting point was the analysis of the existing German guideline “variants of sex development” [19] and other important recommendations frequently cited in the field [7, 15, 26‒30] in order to identify the relevant areas of DSD care that should be covered by the QI. Additionally, we conducted a systematic literature search in PubMed (search strategy shown in Fig. 1), to identify publications that addressed aspects not yet included in our assessment of quality of care.

The next step was to operationalise the recommendations from this literature, with the German guideline “variations in sex development” serving as the starting point. In addition, we involved the relevant stakeholders and people with DSD in determining what constitutes the quality of care from their perspective. We conducted a qualitative survey of medical and psychosocial professionals in the participating DSD centres, members of the self-advocacy groups, adults with DSD, and parents of children with DSD (recruitment by the participating centres). Participation in the qualitative survey was on a random and self-recruitment basis: information about the survey was distributed to staff at the participating DSD centres by the DSDCare team, to members of self-advocacy groups by those members, which participated in the DSDCare project, and to individuals with DSD or their parents by the medical staff in the consultations. All participants sent their answers to the study centre in Lübeck anonymously.

The survey consisted of open-ended questions about participants’ experiences, their evaluation of the healthcare services, and their suggestions for improvement. The results from this survey helped to specify already formulated QI, omit unnecessary QI, and add further important QI.

From the outset, we ensured that QIs fulfil the main criteria: relevance, sensitivity, specificity, reliability, feasibility, and practicability [1, 31]. An initial version of a set of QI was commented on, added to, and subsequently agreed upon by the participating centres. A revised version was used in the centres to determine the practicability of collecting the data from which the fulfilment of each QI is assessed (scale 1 to 5). In addition, the relevance of each QI was assessed on a scale of 1 to 5. We removed QI which turned out to be impractical or of little relevance.

We collected structural parameters annually through the participating centres using the online tool LimeSurvey. This was started with a baseline survey in 2019, i.e., before the start of the project. We included the process and outcome QI into the “DSDReg,” a registry based on the Open Source Registry System for Rare Diseases (OSSE) [32, 33]. This web-based registry was tailored for documentation of the whole DSD care process (including patient socio-demographic data, diagnosis, treatment, counselling, psychological assistance, etc.), and the recording of patient-reported outcome measures and patient-reported experience measurements. The registry started in May 2021. We used the participating centres’ structural and registry data for calculating QI for annual benchmarking. Following the preparation of four benchmarking reports, the QIs were re-evaluated and revised in May 2024.

In a first step, we operationalised the 37 recommendations from the German guideline “variants of sex development” which evidence level ranges from “consensus” to “strong consensus.” Twenty-seven guideline statements were suitable for definition of QI. Ten recommendations have not been operationalised, either because they are redundant (the content can be found in other recommendations) or because the recommendation describes something that is already laid down in law (e.g., “during genetic counselling, the possibilities, limitations, and possible problems of these methods should be explained”).

The literature search in PubMed yielded ten publications with high relevance (see online suppl. Table 1; for all online suppl. material, see https://doi.org/10.1159/000540596). No completely new relevant aspects emerged from these publications, which can be taken as an indication that our previous selection adequately reflected the current state of research.

However, a further step was taken to ensure that the QIs to be developed provided a comprehensive picture: seventy-seven stakeholders from the DSD field, members of self-advocacy groups, and affected people or their caregivers took part in the qualitative survey (for a description, see Table 1). The interviews revealed 41 additional aspects, of which we considered 26 to be of limited importance in an initial evaluation (e.g., “the centre carries out public relations work”/“the structures of the centre ensure the transfer of scientific findings into practice”). We have included the remaining 15 aspects pertaining quality of DSD care in the further development of the QI.

We formulated 86 preliminary QIs based on consideration of guideline statements, literature research, and qualitative interviews. Of these, 50 refer to structural, 23 to process, and 13 to outcome quality.

Structural quality includes the centre’s infrastructure and clinical care, as well as the treatment options available. Process quality includes aspects of information, medical history and diagnosis, imaging, hormone testing, genetic testing, and surgery. Outcome quality is mapped using the results on quality of life, treatment satisfaction, surgical complications, and acceptance of one’s body.

We discussed preliminary QI in detail with regard to relevance and presumed feasibility and subsequently reduced the amount of QI to 54 (see online suppl. Table 2). This set of QI was tested in all participating DSD centres. We collected data for structural QI (later used as a baseline) and tested process and outcome QI for feasibility. Finally, we decided on 38 QI to be used for data collection and annual benchmarking as part of the DSDCare project. In May 2024, the QIs were revised again based on the experience gained from the preparation of four benchmarking reports. This led to some minor wording changes of the QI and one revised operationalisation. One QI on structural quality was removed. This resulted in a final set of 37 QIs (Table 2).

We use various ways of assessing whether a QI is fulfilled: for most structural parameters in particular, a QI is fulfilled if the respective structure is present (e.g., multidisciplinary team available: yes/no). To assess other QIs, a threshold value set for a certain number (e.g., number of patients treated in the centre) marks the distinction between good and poor qualities. For most QI on process quality, based on a proportion of patients who have received a particular service, concrete threshold values for assessing when good quality is achieved have yet to be defined. For this QI, we choose the deviation from the median to compare the centres’ quality of care.

Today, quality management is an established part of the healthcare system, particularly in the area of inpatient hospital care and with regard to common diseases (e.g., diabetes, oncology). As healthcare for rare conditions is particularly challenging due to the lack of expertise, standards, and structured and evidence-based treatment services, there is a clear need to review the quality of care in this area.

The move from eminence-based to evidence-based healthcare for rare conditions requires sharing and accumulating standardised and comparable data regarding processes, diagnostics, treatment, and outcomes. In order to be able to assess the quality of care, a regular survey of defined quality characteristics is required – this was the central aim of the DSDCare project, and it was successfully realised.

Quality management in itself is a time-consuming and resource-intensive process. The development and testing of the QI took almost 2 years. Despite the time taken, some steps in the development of the QI, such as the literature review, did not reveal new aspects. Nonetheless, this step is essential to ensure that the current state of research is taken into account.

In this project, we have developed and jointly adopted a set of QI that consider a wide range of perspectives on the quality of care for people with DSD and their families: state-of-the-art research, assessments, and recommendations from medical and psychological experts from a range of disciplines, people with DSD and their families, and representatives of self-advocacy groups. A multi-stage systematic consensus process was used to develop and finalise a set of 37 QIs. The QIs developed serve to evaluate care in the field of DSD and may add to the German criteria for certification of Centres for Rare Conditions formulated by the NAMSE in this area of expertise.

The QIs are applied to the participating DSD centres for quality benchmarking. Strictly speaking, most of our QI regarding process quality would be better described as quality criteria at this stage, since a definition of thresholds above which good care can be said to exist will have to be defined later. However, comparing the centres with each other by calculating the respective deviation from the median already provides valuable information for them.

The QI turned out to be practicable and relevant, and the results of the benchmarking useful for mapping the development of care in the field of DSD and giving centres concrete indications of where their care can be improved. A major advantage of the newly implemented benchmarking system is that it permits the collection of longitudinal data. This will allow to evaluate patient-related outcome parameters such as satisfaction with care or quality of life. Monitoring the development of quality over time and evaluating it in a meaningful way requires a long-term commitment by the centres with regard to careful documentation and thus the provision of resources for this.

Medical diagnosis and treatment, as well as the sociocultural and legal environment and needs of people with DSD and their families, are constantly changing and evolving. Regular review and, if necessary, adaptation of the QI are therefore required.

The QIs developed serve to enable a structured and standardised evaluation of the quality of care in the DSD centres. The annual benchmarking process was successfully established in the course of the DSDCare project. First results of the benchmarking will be published separately.

After the funding period of the DSDCare project, the survey of structural QI, data collection in the registry, and benchmarking are continued within the framework of the newly established German reference network for DSD. Registry management and benchmarking are carried out centrally by the DRN centre in Lübeck and are financed by the statutory health insurance funds.

Even if the structures in other countries are quite different, our quality objectives (mapped through the QI) can provide a good example for other countries and systems to review the quality of care for individuals with DSD. Adaptations to local conditions (e.g., legal framework, health system requirements) are likely to be useful and necessary.

The authors thank all DSDCare study participants. Furthermore, we would thank the DSDCare Study Group for support of development of the QI and collecting data into the registry. The DSDCare Study Group comprises the following members:Lutz Wünsch (University Hospital Schleswig-Holstein, Lübeck Campus), Paul-Martin Holterhus (University Hospital Schleswig-Holstein, Kiel Campus), Martin Wabitsch (University Hospital Ulm), Sara Brucker/Katharina Rall (University Hospital Tübingen), Nicole Reisch (Clinical Centre of the Ludwig-Maximilians-University Munich), Julia Rohayem, Sabine Kliesch (University Hospital Münster), Uta Neumann (Charité Berlin), Raimund Stein (University Hospital Mannheim), Felicitas Eckoldt-Wolke (University Hospital Jena), Susanne Krege (Essen-Mitte GmbH Clinics), Holger Storf/Jens Göbel (University Hospital Frankfurt), Manuela Brösamle/Eleonore Roggemann (AGS Parents and Patient Initiative e.V.), and Ursula Rosen/Anjo Kumst (Intergeschlechtliche Menschen e.V.).

Only employees of the participating centres and self-advocacy groups were involved in the development of the QI. In the qualitative interviews, also people with DSD and their parents from the participating centres and self-advocacy groups were asked about their ideas of good care – these data were collected completely anonymously and without any personally identifying information. The DSDCare study, including the “DSDReg” registry, was approved by the Ethics Committee of the University of Lübeck (reference 20-322). Written informed consent was obtained from all participants and, in the case of minors, from their legal guardians.

The authors have no conflicts of interest to declare.

The project DSDCare was funded by the German Federal Ministry of Health (Grant No. 2519FSB503).

M.J. and M.R. conceptualised and designed the study and directed the process of QI development. O.H., A.K., and U.D. contributed to the study design. M.S. and A.H. participated in the analysis of data. J.S. developed the DSDReg and coordinated and supervised the data collection. M.J. drafted the manuscript. All authors participated in revision of the manuscript and read and approved the final manuscript.

Olaf Hiort and Ulla Döhnert are members of the ESPE.

The data on which the described development of the QI is based are included in the article. Further enquiries can be directed to the corresponding author.