Regulatory Qualification of a Cross-Disease Digital Measure: Benefits and Challenges from the Perspective of IMI Consortium IDEA-FAST

Measures of clinical features derived from digital health technology are increasingly used for remote patient assessment in clinical practice and trials to evaluate patients for inclusion in the trial, treatment efficacy, or safety. They could also be used in health technology assessments and reimbursement decisions for medicines. The potential advantages of remotely assessing a patient with a digital measure have been discussed before [1‒3] and include higher frequency of assessment, reliability and precision, and greater ecological validity. Their use in drug development could improve time to market, development cost, and success rates for new therapies [4].

IDEA-FAST (https://idea-fast.eu), a 5.5-year Innovative Medicines Initiative (IMI, https://www.imi.europa.eu) project, aims to develop novel digital measures of fatigue, sleep quality, and impact of sleep disturbances for neurodegenerative diseases (Parkinson’s disease [PD] and Huntington’s disease) and immune-mediated inflammatory diseases (rheumatoid arthritis [RA], systemic lupus erythematosus [SLE], primary Sjögren’s syndrome [PSS], and inflammatory bowel disease [IBD]) [5, 6]. Fatigue is among the largest unmet health needs in these patients. The IDEA-FAST consortium is seeking to quantify fatigue using the same digital measures and methodology across all diseases, i.e., to develop cross-disease measures. The consortium defines a “cross-disease measure” as a variable that can be used to quantify a characteristic (e.g., biological characteristic, meaningful aspect of health, or concept of interest [7]) for which there is a strong rationale (e.g., biological mechanism and/or patient experience data) that the same characteristic is present in two or more diseases. As with a standard clinical outcome assessment or biomarker, the diseases in which it is to be used should be specified within a context of use statement [8, 9].

Measures are typically initially developed for a single disease. This may be because the developers have a specific clinical focus, but it also reduces complexity. There are significant benefits; however, in a cross-disease measure, the development, validation, and regulatory approval of a digital measure for use in clinical practice or clinical trials is a long process requiring substantial resources. Completing this for one measure across many diseases in parallel could substantially reduce the time and resources for both the developer and health authority. Furthermore, novel measures are typically developed because the inadequacy of available measures is hampering clinical practice or drug development. By bringing novel measures into use faster, one could also bring value – such as a transformative new medicine – to patients faster. One challenge of using a novel measure in a health technology assessment or reimbursement decision is that since new measures are typically only initially validated in one specific population, it is not possible to make comparisons across diseases. A cross-disease measure would enable cross-disease comparisons. Based on these considerations, the IDEA-FAST consortium recommends more projects explore cross-disease digital measures. This viewpoint discusses the benefits, pitfalls, and methodologies of such an approach based on the experience gained in the IDEA-FAST project, with fatigue as an example.

When seeking to develop a cross-disease digital measure, distinct and additional considerations should be accounted for.

First, the developer must have a strong rationale for selecting the diseases of interest. There are clinical features for which it would not be appropriate to develop a cross-disease measure. Some are peculiar to one disease, such as bradykinesia in PD. Others may be experienced differently by patients with different diseases, such as pain. If the symptom measured varies in its pathophysiological origin and course across diseases, a cross-disease measure may have lower specificity. Therefore, before developing a cross-disease measure, its use should be justified.

Justification could refer to similarities in the patient experience or functioning. Fatigue, for example, has been associated with reduced physical activity [10‒12], impaired cognitive function [13‒15], ability to socialize [16, 17], and social functioning [18] in PSS, RA, SLE, IBD, and PD, suggesting that the characteristics of fatigue may be similar across diseases, supported by the common use and validation of patient-reported outcomes of fatigue (such as FACIT-F) in a wide variety of chronic diseases. Furthermore, there is no consistent relationship between disease activity or severity of these conditions and fatigue levels [19], which could suggest a contribution of disease-independent factors.

In addition, one could refer to common biological mechanisms across various diseases. Although the specific biological mechanisms of fatigue are unknown, they are associated with similar physiological effects across diseases [19], such as autonomic dysfunction [10, 20] and reduced aerobic, cardiovascular, or cardiorespiratory capacity [12, 21‒23] in PSS, RA, SLE, IBD, and PD. The similarity of predictors of fatigue among many chronic diseases indicates that shared mechanisms may exist.

The IDEA-FAST consortium aims to develop a cross-disease digital measure. Many initiatives have proposed methodologies for developing and validating digital measures [24, 25], but there are some important differences when doing so across diseases. When selecting a digital health technology, one should evaluate the feasibility of using the technology in all target disease populations, each of which may have distinct usability challenges. The IDEA-FAST consortium has taken a three-step approach (Fig. 1).

First, candidate sensor technologies that mapped to the concepts of interest were identified based on a landscaping that included web search, literature review, and evaluation of technical specifications. Technologies were selected through an iterative committee-based process which set out to establish selection criteria, resulting in 50 principal criteria that are categorized into data quality, reliability, and analytics; data access, transparency, and handling; accessibility, usability, and user experience; regulatory concerns; scalability; and track record and data availability. Second, candidate technologies were lab-tested and removed if they did not meet minimum criteria identified by the consortium, they did not meet their own advertised specifications, or they performed particularly poorly in one or many of the previously mentioned categories. Finally, the shortlisted technologies were evaluated in a 14-month feasibility study (28- to 36-day observational period) where 148 participants representing all target populations used the technologies remotely [5]. This informed the final selection of seven technologies, which was based on sensor signal quality and coverage, comparison with gold standard measurements, user acceptance, usability, and user experience.

For validation – or for algorithm development if the digital measure has not yet been fully defined – one must acquire data from the representative populations. The design of the clinical study has complexities, similar to those of basket trials [26, 27]. Each patient population may bring a distinct set of needs for the inclusion/exclusion criteria, clinical assessments, and appropriate number and duration of clinic visits. The protocol needs to account for these differences without becoming too complex for clinical sites to administer. Furthermore, representation of all disease groups is likely to lead to a large sample size. As the IDEA-FAST consortium seeks to develop novel digital measures from the seven selected technologies, a clinical observational study is being conducted to acquire data for algorithm development and validation in all of the above populations, with an overall number of 2,000 participants measured over a 6-month observational period.

A cross-disease digital measure should be appropriate for use in not only a heterogeneous population but also in any of the individual disease groups. It is important, therefore, to ensure that the measure is not biased toward validation in one group; however, due to differences in disease prevalence, the sample sizes of individual disease populations may vary. This should be accounted for in the statistical analysis. To test for the influence of a specific disease, disease type can be included as a covariate in the statistical analysis.

Another factor to consider is the trade-off between generalizability and validity compared to a reference measure. It is likely that models developed within single disease groups perform better in terms of convergent validity. Therefore, criteria must be developed for selection of the measure which considers this trade-off, as well as incorporating other factors such as model interpretability, number of digital health technologies required to generate the measure, and patient usability.

The IDEA-FAST consortium plans to develop one or more multicomponent digital measures by combining data across different sensor modalities with statistical modeling and machine learning. The total dataset from the clinical observational study will therefore be separated into an identification population for model building and an evaluation population to test reliability and validity. Selection criteria for the digital measures will be pre-specified as much as possible so that only the most promising measures are validated in the evaluation population.

A long-term objective of the IDEA-FAST consortium is regulatory qualification of the digital measures. A qualified measure is publicly available for a defined context of use without the need for the health authority to reconsider its suitability. This should expedite the development and regulatory review processes for future drug programs that use the measure. In 2022, the IDEA-FAST consortium received qualification advice from the European Medicines Agency (EMA) on the rationale for developing cross-disease digital measures and the planned investigational approach. The EMA advice has proved extremely valuable in finalizing the protocol for the clinical observational study and informing the data analysis methodology. Advice regarding the development of a cross-disease measure is provided below along with the reflections of the consortium (see Table 1 for summary of lessons learnt and Supplementary Material for the full EMA comments on the rationale for cross-disease measure development [for all online suppl. material, see https://doi.org/10.1159/000533189]).

The EMA considered a cross-disease measure an interesting concept that was arguably feasible at this stage of the development plan. The EMA highlighted, however, that heterogeneity of the chosen population- and disease-specific differences could complicate the conduct of a clinical study. The consortium agrees, and others seeking to conduct such a study should consider this early during the planning stages, with careful consideration of how the study design and clinical operations can support the participants and sites. When reporting the results of the clinical study, the IDEA-FAST consortium plans to describe the complications encountered and mitigating actions taken, to support others in the design of future studies with heterogeneous populations.

A potential risk was identified that using a heterogeneous population for model building and testing may lead to a model that does not perform adequately if in the future it is applied in a homogeneous population. For this reason, developers who build a digital measure using statistical modeling of a heterogeneous population dataset should ensure it is tested in the homogenous subgroups. The consortium expects that the similarities in the studied clinical features will ensure that a model built using a heterogeneous dataset will also be valid within homogeneous subgroups. If the model was instead built on a dataset from a homogeneous population, however, it would be less likely that it can be generalized to other populations. For these reasons, the consortium believes in the importance of building the model using a heterogeneous dataset and testing the model in both heterogeneous and homogeneous datasets.

Although the consortium presented evidence that the studied characteristics are present across all diseases to be investigated, the EMA raised the concern that there may still be disease-specific differences in how fatigue is experienced. The IDEA-FAST consortium agrees with the EMA view that developers of a cross-disease measure should provide a strong rationale that the clinical feature being measured is cross-disease. To date, there are no studies that have systematically compared the experience of fatigue across all the studied diseases, but there are similar associations across diseases between fatigue and physical activity, cognitive ability, social functioning, and physiology, as described above. The consortium acknowledges that fatigue can be experienced differently by individuals within a disease as well as between diseases; however, inclusion of this heterogeneity may aid in building a model that is more robust and generalizable both within and across disease populations. The consortium believes that fatigue should be considered a cross-disease clinical feature, but this assumption will be explored through analysis of the clinical observational study dataset.

The EMA advised the consortium on the potential influence of confounding factors that could be disease-specific. While all studies should consider the impact of confounding factors, the EMA suggested that it would be more difficult to control for these in a heterogeneous population. Furthermore, in an assessment of treatment efficacy, disease-specific factors may be confounded with treatment-specific factors, both of which affect levels of fatigue. For example, medication that induces fatigue may be used commonly in only one of the relevant diseases. The IDEA-FAST consortium agrees that controlling for confounding factors will be an important component of the analysis, and relevant confounders will be included as covariates in the statistical analysis. The consortium expects that common characteristics of fatigue will be present in the cross-disease digital measure after controlling for the effects of these confounders; however, this assumption will again be tested through analysis of the clinical observational study dataset.

The EMA concluded that it was preferable to initially validate the digital measures in a more homogenous study population and, if valid, then gradually expand the context of use. They did agree, however, that it could be an acceptable approach to explore a cross-disease measure in the identification phase of the statistical modeling and if the results did not support a cross-disease measure, then instead use a model that shows appropriate performance in a more homogeneous subset. The IDEA-FAST consortium will adopt this pragmatic approach and would recommend it to other developers of cross-disease measures as a de-risking strategy.

As described here, development of a cross-disease measure is likely to encounter many challenges, including the design and conduct of a more complex study; the development of a measure that is appropriate both within and across diseases; differences within and across diseases in how the clinical feature is experienced; and both disease-specific and cross-disease confounding factors. With fatigue used here as a specific example, the consortium has described the EMA’s advice on these issues and provided recommendations to others attempting to develop a cross-disease measure. In addition to these challenges, however, the IDEA-FAST consortium notes many benefits. The time and resources required from both the developer and health authority should be lower than completing separate development and validation projects for each disease; reducing the time it takes to develop a novel measure across diseases should support faster development of novel medicines; and a validated cross-disease measure would enable health technology assessment bodies and others to perform cross-disease comparisons of treatment effects. These benefits have motivated the IDEA-FAST consortium, and others such as the Mobilise-D consortium (https://www.mobilise-d.eu), to tackle the significant challenges and complexities of this endeavor to develop cross-disease digital measures.

As of today, there is no precedent for regulatory qualification of a cross-disease digital measure. The advice from EMA highlights the challenges that teams such as IDEA-FAST may encounter if attempting to do so. In the interest of innovation in clinical practice and accelerating the development of novel medicines, however, it is recommended that developers explore similar approaches, and the IDEA-FAST consortium looks forward to reporting the results of its efforts when the clinical observational study concludes in 2025.

The authors would like to thank all members of the IDEA-FAST consortium for their contribution to the IDEA-FAST project and the members of the Committee for Medicinal Products for Human Use at the European Medicines Agency for the advice which is discussed in this manuscript.

David Nobbs and Sebastian C. Holst are full-time employees and shareholders of F. Hoffmann-La Roche. Wojciech Piwko is a full-time employee and shareholder of Takeda Pharmaceutical Co. Ltd. Christopher Bull is a full-time employee of the University of Newcastle upon Tyne. Francesca Cormack is a full-time employee and shareholder of Cambridge Cognition Ltd. Teemu Ahmaniemi is a full-time employee of VTT Technical Research Center of Finland Ltd. Meenakshi Chatterjee and Stefan Avey are full-time employees and shareholders of Johnson & Johnson. Wan Fai Ng has provided consultation services for the following companies in the area of Sjogren’s syndrome and/or fatigue: Novartis, GlaxoSmithKline, AbbVie, BMS, Sanofi, MedImmune, argenx, Janssen, Resolve Therapeutics, Bain Capitals, and UCB.

The IDEA-FAST project has received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement No. 853981. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme and EFPIA and associated partners.

David Nobbs authored the first draft of the manuscript. David Nobbs, Wojciech Piwko, Christopher Bull, Francesca Cormack, Teemu Ahmaniemi, Sebastian C. Holst, Meenakshi Chatterjee, Walter Maetzler, Stefan Avey, and Wan Fai Ng made substantial contributions in developing the viewpoint described in this manuscript and revising the manuscript. The IDEA-FAST consortium approved the final version of this manuscript. This communication reflects the view of the IDEA-FAST consortium, and neither the IMI nor the European Union and EFPIA are liable for any use that may be made of the information contained herein.