Reverse Engineering of Digital Measures: Inviting Patients to the Conversation

The number of studies using digital measures for evidence generation continues to rise [1], and increasing collaboration across stakeholder groups [2, 4] is yielding progress toward the adoption of digital measures as trusted outcomes for decision making in clinical trials and clinical practice. In order to continue to catalyze collaboration and examine remaining barriers to the use of digital measures in clinical development and care delivery, a public meeting was hosted by the Swiss Federal Institute of Technology in Zürich (ETHZ; German: Eidgenössische Technische Hochschule Zürich), the Foundation for the National Institutes of Health (FNIH) Biomarkers Consortium, and sponsored by the Wellcome Trust, entitled “Reverse Engineering of Digital Measures” on 15–16 September 2022 in Zurich, Switzerland [5]. This meeting built on the success of a previous FNIH-sponsored meeting [3, 6, 7] by assessing progress around one of the central themes developed in those discussions: the criticality of involving patients in the development of new digital measurement tools. The conference focused on a reverse engineering approach which was fundamentally patient centric: the data and input derived from patients’ lived experience proactively drives the creation and development of novel tools and medicines [8, 9]. Sharing the experiences of four extended case studies and involving regulators, Pharma/Biotech, technology service providers, patients, and innovators, the meeting examined how the roles, contributions, and impact of people who are patients are shaping the development and validation of new digital tools and what evidence is needed for healthcare providers (HCPs), Pharma/Biotech companies, regulators, and payors to accept these tools and the respective data (Fig. 1). Over a period of 2 days, the meeting participants explored how the human-centered, holistic, and evidence-based science of digital measure development is being put into practice and examined exemplary use cases and best practices for how this is shaping regulatory and payor interactions, patient engagement, product development, and embedding of digital measures into decision making throughout care delivery.

Human-centered is a term borrowed from systems and product development where the design process focuses on the person for whom the product is designed and their needs, ultimately safeguarding usability and success [10]. In the context of this meeting, it refers to understanding how far we are from achieving meaningful engagement with the ultimate stakeholders: patients, participants, and caregivers. Nonetheless, true patient empowerment necessitates a mind-set shift from patients as testers to patients as partners and co-creators, which can be achieved by involving them in every step of the human-centered design and development process [9]. Holistic refers to seeing the patient and their health/disease experience as a whole [11]; encompassing physical and psychological health, as well as social and environmental spheres; assessing how our understanding of disease is changing; and what efforts at breaking down silos across treatment, services, and data sources are occurring. Evidence-based science for digital measure development is built on demonstrating performance at the sensor technology and algorithm level, but ultimately aims to demonstrate fit for purpose (regulators) and treatment value (insurers) to business value (pharma, startups).

The first day focused on evidence in studies and clinical development and was titled “Qualification of New Digital Evidence.” The second day focused on evidence for care delivery and the use of evidence on the individual patient level and was titled “From Efficacy to Effectiveness.”

The meeting had several overarching aims:

• Bring together regulators, industry, patients, HCPs, payors, and technology service providers to emphasize how patient centricity is essential in shaping development and validation of digital evidence generation tools.

• Examine progress and remaining barriers to widespread use of digital measures for evidence generation in clinical development.

• Explore how human-centered, holistic, evidence-based science is being put into practice.

This paper summarizes the progress, discussions, and main takeaways presented in the meeting. We recognize that patient centricity is key for future development and shared values. This echoes recent guidance from the World Health Organization, stating that we must “…place the individual at the center of trustworthy care delivered digitally. The successful uptake and use of digital technologies in health is contingent on a patient-centered approach” [12]. New stakeholders in the field (e.g., from technology and product development fields or advocacy groups) are bringing in new knowledge and skills in how to implement this human-centered, holistic, and evidence-based vision. The intention here is to continue discourse and provide a basis for dissemination and outreach to the wider community and other stakeholders.

Digital measures, an umbrella term encompassing measurement tools based on digital health technologies (DHTs) which include digital biomarkers [13, 15], electronic clinical outcome assessments, and corresponding digital endpoints [16], are becoming more mature, in particular, community (i.e., developer) understanding of regulatory requirements for individual drug assets as well as qualification of these new digital evidence generation tools for drug development [17, 24]. In 2019, qualification of stride velocity 95th centile (SV95C) represented the first major breakthrough in this new field [25]. As the first DHT-derived measure to receive qualification from the European Medicines Agency (EMA), also currently under review by the US Food and Drug Administration (FDA), SV95C quantifies the ambulation ability of ambulant patients with Duchenne muscular dystrophy in interventional studies [26, 29]. This use case, along with others, was presented and discussed at the previous meeting called “Remote Digital Monitoring for Medical Product Development” workshop [3, 7], sparking much interest in how to increase the volume and diversity of qualified digital measures. That meeting highlighted several areas that could catalyze further momentum: more regulatory feedback and guidance are needed; patient engagement must be recognized as critical to success at every step of digital tool development; precompetitive collaborations can help share risk; and we need the community to share their experience developing and validating digital measures so that relevant frameworks and policies can be further refined.

The case studies discussed at the most recent workshop show that progress has been made in all these areas, building on new regulatory documents and publications [16, 30, 31]. All case studies integrate the patient’s voice as part of their project planning and execution and, either as part of large consortia, smaller groups, or collaborations across industry, seek to work together to drive progress toward increasing the availability of patient-centric digital evidence generation tools.

A noticeable trend and central theme of recent progress in the field has been the increase in multistakeholder, transdisciplinary collaboration. Digital health, by necessity, brings together many stakeholder groups, sitting at the nexus of drug development, clinical practice, technology, software, and analytics [2]. As a new field, such collaboration is vitally important for “norm-critical innovation” [32], as it facilitates exploration of solutions outside the experience of individual stakeholder groups. What may be novel to one group may be standard practice in another. It is critical that bridges continue to be built, including to the broader “big tech” communities (Alphabet, Apple, etc.), payor communities (national, regional, and local), and points of care (clinicians, hospitals, caregiver communities, etc.). Both incentive and opportunity to participate in tool development are different across stakeholders; thus, creating an environment of shared responsibility and parity among stakeholders is of utmost importance before we can have true co-development.

As the ultimate user and therefore central stakeholder, patients have historically not been sufficiently involved in the digital measure development process and, indeed, in healthcare research in general. Funding of large public-private consortia like IMI FACILITATE [33] and IMI PREFER [34] demonstrates that there is considerable interest in rectifying this. Indeed, experience from the technology development sector has shown that the rationale for improving patient and public engagement not only has an ethical basis but is central to creating improved understanding, engagement, and mutual value [35]. For digital measures specifically, the “measures that matter” framework has recently provided guidance for how we can define potential measurements that would be rooted in mutual value [36] and advocacy portals making it easier to connect with patients on a deep level [37], yet the practical implementation of engagement still clearly has a long way to go. The inclusion of patients in clinical development is more mature with both the EMA and FDA providing guidance [38, 39]. However, there is a lack of regulatory guidance when it comes to digital health directly related to patient involvement.

The FDA patient-focused drug development (PFDD) initiative [40], through public workshops and related guidance [41], has paved the way for concrete involvement of patients into defining meaningful concepts to be measured and gaining insights into how such measures might be implemented. The central role of patient input is also acknowledged by other regulatory agencies, including the EMA [42], and became an International Council for Harmonization (ICH) strategic priority area (see “ICH Reflection Paper: Proposed ICH Guideline Work to Advance Patient-Focused Drug Development, Endorsed by the ICH Assembly on June 2, 2021” [43]), which is expected to guide creation of a future global guideline framework for incorporation of the patient’s perspective in drug development and informing regulatory decision making.

Throughout the meeting, the clear message from the use cases was that mutual value is dependent on deep integration, beyond involvement in initial planning but extending to accountability and integration into decision making and oversight structures to create an open, trustful environment and expand the capacity for learning and understanding. Specifically, these use cases highlighted that, if we want to embed the patient voice into research, we need to embed it into the governance structures of our projects. It is not sufficient to ask a small number of patients’ opinions on items such as protocols or consent forms. Indeed, such “tick the box exercises” are one of the reasons that patient involvement is often criticized [44, 46]. Instead, meaningful integration requires patients to sit alongside the decision makers of projects to ensure that they too have decision making capabilities that extend into the research questions being asked, the development of protocols, the interpretation of results, and the dissemination of work. It was also clear during the panel discussions that the use cases are not the norm when it comes to co-creating, as it was highlighted that their methods of operationalizing patient involvement were considered to be advanced in the field. Thus, these examples, and the learnings derived from them, demonstrate that there is still significant work to be completed when it comes to creating structures and planning research that actively and meaningfully develops a patient-centered approach in the creation of multistakeholder networks that are required to enhance impact in the field. Figure 1 gives a visual summary of some of the key progress made across the four use cases toward meaningful co-creation in the digital measure development process. The breadth of these examples shows that integration of the patient voice does not (and should not) follow any set process, yet they do offer a blueprint for how it can be thoughtfully approached.

The first case study was presented by the Innovative Medicine Initiative’s (IMI) Mobilise-D consortium [47, 48] and focused on the physical functioning domain, specifically real-world mobility. As the world’s population ages and chronic diseases become more common and long-lasting, it has become clear that loss of mobility due to high age or disease is a key problem that negatively impacts people’s independence and quality of life. Although we can readily see when someone has trouble moving about, until recently we could not easily measure this loss except for snapshot assessments in clinics or laboratory settings. New technologies now allow us to measure mobility accurately in people’s homes and natural environments. Wearable digital sensors have revolutionized the ability to determine if someone is losing their ability to walk, to what degree, and in what aspects of mobility, such as walking speed, volume, or variability. By frequently measuring everyday mobility with small wearable sensors, the effort can potentially improve treatment and patient follow-up, personalize healthcare, and allow the conduct of more informative clinical trials [49, 51]. The aim of the Mobilise-D initiative is to link real-world digital assessment of mobility to established clinical endpoints to support both regulatory acceptance and clinical practice, thereby establishing digital mobility outcomes as a basis for primary, surrogate, and secondary endpoints for use in several indications. The focus is on people living with multiple sclerosis [52], Parkinson’s disease [53, 54], congestive heart failure, chronic obstructive pulmonary disease, and people recovering from proximal femur fracture (hip fracture). The disease cohorts have been chosen because they represent different classes of mobility problems relating to low physical activity, different gait disturbances, and frailty, each affecting large groups of European citizens over substantial periods of time. The disease cohorts include a broad and heterogeneous range of subject characteristics with varying chronic care needs and represent different trajectories of disability.

The consortium is undertaking a wide range of scoping reviews, working on consensus and standardization/harmonization of data [15], as well as conducting studies to establish both technical and clinical validity of digital mobility outcomes [55, 56]. Technical validation is assessed with a sample of n = 120 across the above-mentioned diseases plus healthy older adults, and clinical validation is undertaken in a large longitudinal study (n = 2,400) with 2-year follow-up [56]. The patient perspective is crucial for the Mobilise-D project, and patient involvement is supported at all stages of the project. Given the range of patient cohorts across multiple countries in Europe, it was recognized that robust patient and public involvement and engagement (PPIE) structures were required to ensure that PPIE tasks were visible, that tasks were aligned to the research objectives and patient needs, and that patient representation was present in key governance committees. Consequently, the consortium established two independent but collaborative governing groups to oversee, plan, and operationalize their PPIE activities: a PPIE Board and a Patient and Public Advisory Group (PPAG; Fig. 2). The PPIE Board, consisting mostly of researchers and other stakeholder representatives from the consortium, provides oversight and guidance on the implementation of PPIE activities across the consortium. The PPAG, consisting mostly of patient advisors, was created to ensure that the needs and opinions of patient and public contributors are embedded in the work through identifying topics of importance, highlighting changes in protocols, and supporting the interpretation and dissemination of findings. The two groups work collaboratively in a continuous feedback loop to ensure consistency in planning and development of all key activities within Mobilise-D.

Outputs relating to these structures include a conceptual framework of mobility, which outlines the experience and perception of walking from the perspective of patients. This meta-ethnography identified striking similarities in relation to how people with widely different chronic health conditions experience walking, which is an integral part of a person’s sense of self that has physical, social, and mental and emotional experiences – regardless of the specific underlying condition. Furthermore, webinars, publications, qualitative research task planning, and public-facing documents have all been conducted with PPAG input and advice, as outlined on the Mobilise-D website [57].

Key learning points from the close collaboration with patients were identified and outlined. For example, it was crucial to set budgets and time for effective patient engagement early in the process as cost and time for patient collaboration must be embedded in funding. Additionally, close patient collaboration will lead to substantial time commitment for meetings and engagements, which should be kept in mind for all projects. Ownership for patient involvement should be across the consortium, and the identification of needs and expectations for the process should be undertaken early on with patient advisers. This approach has already demonstrated positive impact on the larger project and on study protocols, including articulating measurement concepts which form the basis of all study protocols and regulatory proceedings [58] and very high levels of acceptability for the digital technologies being explored in the project [59].

The Mobilise-D case study also shared experiences with implementing a framework for selecting digital measures with the highest value for implementation [60], building on “measures that matter,” and outlined key statistical properties: Discriminant ability – How do measures differ between healthy people and those with chronic conditions? Construct validity – What relationships exist between the digital mobility measures and common measures of disease severity and physical function? Prognostic value – Do measures provide useful information about how health status might change over time? Responsiveness – Are we able to observe changes in measures due to treatment? Finally, the Mobilise-D case study shared their experiences regarding the pathway toward regulatory qualification of digital mobility outcomes, highlighting the need for early interactions with EMA and FDA, as well as the explicit advice from both agencies to approach regulation on a case-by-case basis. So far, the regulatory work in Mobilise-D has resulted in two letters of support from EMA [54, 61]. The second case study, also through IMI, was from the IDEA-FAST team [62] and aimed to identify digital endpoints that provide reliable, objective and sensitive evaluation of fatigue and sleep disturbances for multiple neurodegenerative diseases: Parkinson’s disease, Huntington’s disease, and immune-mediated inflammatory diseases: rheumatoid arthritis, systemic lupus erythematosus, primary Sjögren’s syndrome, and inflammatory bowel disease. These specific indications were chosen to be studied as problems with fatigue and sleep are highly prevalent in these chronic diseases and are often rated by patients among the most disabling aspects of these conditions [63]. The broad range of conditions allows generalizable aspects of fatigue to be studied and aligns with the capabilities of the project partners. Furthermore, assessment of these symptoms is complicated by their high variability and a lack of consistently identifiable relationship with the severity of the underlying disease. Evaluation of therapeutic intervention is often reliant on standardized questionnaires which are prone to recall bias, reliability issues, social desirability, and poor sensitivity to change. The development of low-burden, reliable, and objective digital measures to complement existing measures for fatigue and sleep disturbance is thus necessary to provide a more comprehensive and patient-centric assessment of treatments for neurodegenerative diseases and immune-mediated inflammatory diseases.

The identification and development of novel digital measures for fatigue and sleep disturbances is expected to provide more reliable measures of the severity and impact of these symptoms in the person’s normal surroundings, which could prove superior to current questionnaires. Fatigue and sleep disturbances are debilitating, as they affect work and personal life, can lead to an increase in disability claims, sick leave, and medical consultations, and even result in job loss. Poor quality of life is thus closely linked to this condition. Patients suffering from fatigue show marked differences to healthy individuals: fatigue presents as unpredictable and highly variable, it correlates poorly with the underlying disease activity (i.e., it may still be present despite the disease being in remission), and persists even after rest, it is chronic, very severe, and negatively affects life. Fatigue relates to depression, anxiety, pain, HPA axis dysfunction, CNS changes, and neurotransmitter changes. However, few treatment options are available in fatigue management, which presents a high unmet need for the development of better treatment for this debilitating condition.

The unmet need outlined above was underlined by patient testimony shared as part of the meeting, where participants from IDEA-FAST studies talked about their experience living with fatigue [5]. They spoke of their challenges communicating their invisible symptoms to different stakeholders including medical professionals: “Even talking to the consultant, by phone or talking, I don’t feel like I get my point over,” their family: “it has affected my social life; I don’t make plans to go out… I just get eye-rolling, and that is from my family!”, and even themselves: “It comes on so gradually, you just end up wearing it… it becomes part of life.” Furthermore, patients talked about their willingness to use digital monitoring, especially if it helped them understand their own symptoms better and helped other patients living with similar feelings of fatigue. Patient testimony has been incorporated into the design and planning of IDEA-FAST research; for example, multiple devices are currently being considered for use in measurement of different reported aspects of fatigue to reflect the multidimensional (physical, mental, and emotional) nature of this symptom [64] and inform future research into more complex diseases affecting multiple modalities. Early results, for example, with tools assessing cognitive symptoms [65], show high levels of acceptability in patients suffering from fatigue. Furthermore, research into methods to support mutual value and long-term engagement with digital tools is also showing promise [66], which is vital to support the long-term monitoring required in chronic conditions.

Both Mobilise-D and IDEA-FAST have the ultimate aim of expanding the range of measurement tools available for use in clinical development decision making across a range of conditions. Both consortia are pursuing qualification procedures for digital measures that have a central relevance to health-related quality of life (HRQoL), i.e., gait speed and other mobility outcomes for Mobilise-D and fatigue for IDEA-FAST, and therefore a relevance to a broad range of conditions. Both consortia have been very transparent in sharing their regulatory strategy and regular updates as procedures progress [48, 61, 67]. As a result of these experiences, gaining a better understanding of the scale of investment required to qualify a digital measure, much discussion was devoted to the best way to broaden the scope of the qualification procedure, i.e., across a range of possibly related conditions rather than for a single condition at a time. If successful, this would increase return on investment and improve efficiency by enabling, if the qualification is successful, the measure to be adopted more broadly rather than having to wait for successive qualifications in different indications or the significant investment of pursuing qualification in several indications in parallel. This in turn would have benefits for both developers and regulators by encouraging investment and would make regulatory review more efficient (i.e., a single coordinated review vs. multiple related review processes). Qualification of new measures must be made more efficient and faster as qualified measures themselves may not directly benefit patients, but their use as a tool to develop new therapies and optimize their effect on improving meaningful aspects of the patient experience will. However, the recommended qualification path of Mobilise-D via individual use cases as described above shows that we still need to build trust before being able to broaden the qualification scope in the future.

As well as the regulatory aspects of pursuing broad qualification for a new measure, the case studies raised practical aspects relevant to the collection of the evidence to support qualification procedures. Endeavoring to ensure that study populations are representative and diverse is a topic well covered in the context of individual studies and the focus of recent FDA guidance [68], but what happens when cultural differences influence understanding of a measurement concept, such that different measures associated with a given meaningful aspect of health have different relevance for different cultural groups? Or when demographic differences between indications have an influence on use and usability of a technology and the resulting data quality? For example, wearable sensors may be less acceptable in the summer than the winter if they are visible, and older populations may require additional support in order to integrate technology into their care [69, 70]. These and related issues are particularly important when thinking about measures with direct relevance to HRQoL and therefore relevance across a wide range of conditions.

Against these challenges, through the patient testimony presented, and highlighted by the differences between the two qualification-focused case studies, was the clear value and relevance of objective measures to patients themselves, especially those with “non-visible” symptoms like pain or anxiety. Although these are intrinsically subjective states and development of objective digital biomarkers is more challenging than with physical characteristics, objective digital biomarkers can deliver substantial value to patients. A repeating theme in the patient testimony in the case studies was that their subjective reports of “invisible” symptoms, i.e., the impact of their health condition on their daily activities, feelings of tiredness, or “knowing” something was wrong, was not considered in as great a depth as they expected or desired which sometimes led to substantial delays in diagnosis and subsequent treatment. In their own words, objective measures would help them in their daily life, support communication of their needs, and possibly aid in earlier diagnosis and better treatment. Indeed, the two case studies, through their use of PPIE tasks, patient advisors, and patient stories, clearly emphasized that patient centricity requires projects to move away from focusing only on physical aspects of health and instead ensure that psychological, social, and environmental aspects of a person’s life are considered a part of decision making. This has clear relevance to clinical development but also in supporting care on an individual basis [71]. Figure 3 presents a conceptual model of the factors, derived from the patient themselves and contextual information from their environment, which help build a complete picture of an individual’s HRQoL. Clearly, digital measures cannot capture all these factors; they can play a role in improving our understanding of HRQoL on an individual level.

That new measurement tools have relevance far beyond clinical development and can provide value to individual patient journeys and care was emphasized in testimony from Dr. Elena Izmailova, as she shared her experience as both a scientist and a patient [72]. This testimony preceded two further case studies focused on reimbursement, real-world evidence (RWE), and care delivery.

The third case study explored a digital assessment tool developed by Roche for people living with spinal muscular atrophy (plwSMA), focusing on motor, respiratory, and bulbar functioning domains. SMA is a rare autosomal recessive genetic disorder caused by reduced levels of SMN protein throughout the body, affecting ∼1 in 10,000 babies born worldwide each year and being the most common genetic cause of death in infants [73, 74]. SMA is marked by large heterogeneity in both disease severity as well as age. Relevant, accurate, and sensitive measures are thus crucial for clinical trials and practice as timely treatment initiation is important, with the disease having a particularly narrow window of intervention. Standard health outcome assessments have highlighted survival, motor weakness in the upper and lower limb functions, fatigue and endurance, respiratory function, motor neuron and muscle properties, as highly impacted areas in the life of plwSMA. Emerging treatment options that restore SMN expression [75] create further opportunity and urgency for understanding the impact of these novel interventions on plwSMA. Novel sensitive and specific outcomes of bulbar function, fine motor symptoms, and respiration may complement or improve regular assessment of symptom severity and progression of many of these motor neuron-affected functions in individuals receiving treatment.

The Roche SMA Digital Assessment Tool (DAT) is a combination of active smartphone-based and passive wearable remote sensor-based measurements, the design of which was specifically informed by patient input. Specifically, to understand the changing needs of plwSWA and to identify their preferences for monitoring, a mixed-methods study was conducted with plwSMA and HCPs using an online questionnaire, qualitative interviews, and focus groups [76]. The results provided evidence of various meaningful aspects of health related to motor and bulbar function which may not be fully captured by current assessments in routine clinical practice. Interviewees indicated that developing in-home remote monitoring tools can provide healthcare teams with essential information about bulbar and respiratory function and contribute to better health monitoring of plwSMA. Thus, the SMA DAT app comprises multiple active tasks for the assessment of finger, hand, lung, and speech functions via tapping, tracing, squeezing, phone-turning, sustained-phonation, and speech activities, and a high-frequency accelerometer sensor patch worn on the suprasternal notch [77]. The data collected supports establishing a value framework which is key to both plwSMA and HCPs for reimbursement for new therapies [78].

The final case study was based on efforts included with the Accelerating Medicines Partnership^® Schizophrenia (AMP^® SCZ; [79]) Program, focusing on physical, physiological, and psychological, as well as social, functioning domains. Schizophrenia is one of the leading causes of disability worldwide and yet one of the least understood brain disorders [80, 81]. Schizophrenia is most often associated with distortions in thinking and behavior, including delusions and hallucinations [80]. Less recognized, however, are persistent cognitive symptoms as well as those related to social withdrawal and diminished emotional expression, which can have a profound adverse effect on patients’ ability to function across many domains of life and that are not effectively treated by current medications [82]. Early detection and intervention before psychosis develops, when individuals are at clinical high risk for psychosis, could attenuate, postpone, or even prevent the transition to psychosis and improve individuals’ clinical and functional outcomes (Fig. 4). Utilizing the open-source mindLAMP digital phenotyping app [83], one of the many aims of AMP SCZ is to develop digital measures and tools to better define early stages of risk and predict the likelihood of progression to psychosis and other related outcomes such as anxiety, depression, and substance use disorders, and as sensitive measures that can detect clinically meaningful change. The project will assess the potential of both population as well as personal digital measures of risk over 1 year of longitudinal data capture. The need for digital measures to help advance treatment options for mental illness is clear to all stakeholders; however, the relative lack of objective metrics or any biomarkers for mental illnesses makes the validation of digital measures challenging [6, 19]. This is a challenge not limited to mental illness; indeed, much research into digital measures is motivated by a paucity of available measurement tools for advancing therapies, yet without a clear predicate or reference measure (i.e., true de novo measurement), validation and regulatory advancement can be even more challenging [19]. Human-centered design, building a clear rationale around patient needs, is even more critical in this setting.

The AMP SCZ case study offered several examples of how patient centricity had influenced their approach, picking up on several themes that have been further underlined by a recent community-led PFDD meeting [84]. One theme was a need to help patients understand and communicate their own experience. AMP SCZ specifically offers patients accessible, co-created visualizations of their own data as part of the smartphone app. This not only supports engagement through value creation, but the use of such digital measures may also help decrease stigma surrounding mental health by providing objectivity that supports patients’ communication with themselves and their community [85]. Mental health suffers from a strong association with social inequalities [86]. This has been the primary motive behind making the study app freely available. The team is also further building trust with the patient community and enabling them to contribute to research; thus, the data processing pipeline has been made publicly available, allowing further adaptation and individualization of the platform [87].

Stakeholder consensus has also shaped the overall aims of the project, with a particular focus on early detection or warning of illness onset or relapse as a major priority for enabling better care [88]. Digital measures can be a critically important tool to help people connect to the right care at the right time, and there was a consensus that such care be direct human, and not digital, interaction at this point in time.

Digital measures for mental health are, however, still in the early stages of development, and hurdles remain [89], not the least of which are ethical concerns and privacy risks. Ensuring they are developed and deployed within the right environment, with diverse teams and appropriate safeguards is critical if value is to be realized [90, 93]. Research into ethical and privacy matters related to digital health is expanding but still remains underexplored [94]. Engagement has also been a challenge across all digital measures in mental health to date [95], but when technology is created with patient feedback [96] and data are used to advance patient care, engagement can actually be quite high [97]. Overcoming these challenges and continuing research on digital measures in mental health, especially longitudinal studies, holds the potential to help understand mechanisms, individual-level behaviors, and finally move the field toward prevention [98, 99].

Broader progress of digital measures and evidence generation in post-clinical development settings (e.g., care delivery, Real-World Evidence (RWE), payor reimbursement) is dependent upon further and deeper multistakeholder engagement. Multiple discussions throughout the workshop indicated that the requirements across stakeholders are not necessarily aligned, requiring further discussions and clarifications (Fig. 1). As we think about the use of digital measure evidence beyond clinical development, the landscape of critical stakeholders’ changes. We must do more to facilitate collaboration and co-design with big tech, hospital systems, treating clinicians, payors, and of course, the patients themselves. Many tools currently in use are designed with clinical trials in mind, which provide a controlled environment where patients receive a high degree of support and significant resource is dedicated to securing and interpreting the resulting data. Without these resources, it can be hard for patients to effectively use these tools and integrate them into their management, a problem further exacerbated by the explosion of app availability [100]. Even choosing the right app can be a challenge for patients and caregivers [101], yet new clinical support roles, such as “digital navigators” [102], can aid acceptance and use of digital tools in real-world, uncontrolled environments. Beyond support, co-design that facilitates shared decision making can clearly facilitate high levels of engagement even in challenging populations [33].

There was extensive discussion on the requirements needed for stakeholders – patients, clinicians, regulators, and reimbursement decision makers – to make decisions on DHT-generated evidence. The most important is the validation evidence, which begins with a robust understanding of what is relevant to the patient and how the new measures compare to the prevailing standards. The concept of “not so gold standards” was again brought up in the context of what a measure would be compared against if the traditional Clinical Outcome Assessments (COAs) were not ideal or nonexistent. The discussion continued on the topic of population- versus individual-level measurements. Recent trends have seen RWE incorporated into clinical development, driven by the possibility to reduce the time required to bring a drug to market (i.e., collecting evidence for clinical development and phase 4 in parallel) and by an increased acceptance of RWE for regulatory decision making [103, 105]. Further efficiency gains would be possible with better alignment of the requirements for validating new tools, as the evidence collected would be valid across settings (e.g., registration and reimbursement). Such integrated evidence planning would benefit all stakeholders and is receiving increasing support [106], but will require significant evolution of traditional means of evaluation of digital measures and tools. For such far-sighted thinking, there needs to be a new paradigm on what is acceptable and of value to all stakeholders including the patient, the sponsor, and the healthcare system. In particular, there needs to be significant investment in validation of effectiveness on the individual level, for example, N-of-1 designs [107, 109]. This will include developing our understanding of how to incorporate contextual information into digital measures. On a cohort level, context can be modeled and “averaged out,” but on an individual level, real-world digital measures provide the opportunity to capture a patient’s “lived experience,” but lived experience is impacted by a set of diverse factors, some not related to the patient’s disease. Mitigating confounding factors [110] and understanding implications for interpreting these data will be milestones in the use of digital measures for individual care and management decisions.

It is important that, in view of such challenges, we remain optimistic; the very fact that development, validation, and acceptance of new digital measures, evidence generation tools across clinical development, scaling and implementation, reimbursement, and care delivery have reached levels where we are even having these discussions is cause for celebration. Considerable progress has been made, and we must continue to build on that progress if digital measures are to play a full role in bringing new therapies to patients.

There remain, of course, many challenges, and there are several tractable areas where immediate-term progress can be made. The first is to continue to evolve definition and understanding of the digital medicine lexicon [111], specifically that related to digital measures [19, 20, 112]. As more and more digital measures move out from the feasibility phase [113] and toward scaling, regulatory acceptance, and qualification, it is ever more important to have clear understanding of not just the differences between, for example, COAs and biomarkers, or qualification and acceptance, but why it matters to distinguish between them.

A related challenge will be how to encourage not just continued innovation in digital measures but commitment to pursuing qualification for those new digital measures. One possible approach is to enable and pursue much broader context of use (COU), i.e., scope, to qualification. While a relatively narrow COU, e.g., as seen with acceptance proceedings where COU is limited to a specific study, benefit short- and mid-term progress because, for example, the validation population is generally well defined and risk relatively predictable, now that more teams are looking at qualification of digital measures, it seems that in order to balance the required investment (whether from pharma, tech, etc.), the qualification needs to be broadened. Predicate for how this might be achievable can be taken from the currently ongoing qualification procedure for PROMIS-PF-8c, a COA for physical function in “adult patients (>18 years) with solid tumors or hematologic malignancies,” i.e., all adult cancer patients receiving therapy [114]. If successful, the investment in advancing this new instrument stands to benefit many patients. Encouraging broader “umbrella” COUs, especially for HRQoL-relevant digital measures, would encourage innovation and long-term investment. Alternately, simplified “pivoting” from one COU to another and continuing the deepened alignment of EMA and FDA evidence requirements [115, 116] will have a similar effect on innovators, as well as reducing load on regulators.

We must also continue to enhance the acceptability and relevance of digital measures for patients. There is a pressing need to include the voice of patients at all stages in the development of digital biomarker design which allows for greater adherence and trust [9]. As our ability to extract insights from sensor data advances, to maintain trust, we must ensure that the people who provide their data are comfortable with those insights. Clearly, very broad, undefined consent is not a solution, but perhaps the digital measure field can learn from progress made around conceptually similar challenges in genomics/genetics [117, 118]. Active development of capabilities to derive insights from sensor data makes static consent limiting. Full use of the data will require development and implementation of dynamic consent and processes for returning data to the patient community itself, as is happening in the diabetes hacker community [119, 120].

It is clear that digital measures offer considerable opportunities in the area of remote clinical trial solutions and at-home disease management. It raises several challenges, for example, technical, compliance, ethical, and a key consideration with all types of digital measures lies around data use and ownership. The challenge remains how we provide fair and transparent use of data. The data generated from the digital measures often serves to answer only the sponsor’s question. The true value of real-world evidence is not understood by the population at large. A challenge that the field needs to address is actively providing insights and value from the data generated in answering questions other stakeholders, particularly patients, wish to address.

Huge strides in the use and acceptability of digital measures have been made, and more will follow. We must continue to widely engage in multistakeholder collaborations, including pharma, big tech, payors, health systems, clinicians, and patients. Continued evolution of policy and best practices will encourage further investment and innovation: pursuit of broader COUs for qualification, alignment of evidentiary standards between EMA and FDA, clinical development of RWE, and input and oversight from regulators and payors. If we view drug development as a whole, only ending at delivery to a patient, then a human-centered, holistic, and evidence-based approach to digital measure evidence generation will deliver mutual value for all stakeholders.

The authors would like to acknowledge and thank all participants who took part in the meeting for sharing their knowledge and perspectives and contributing to such a rich discussion. We would also like to thank the organization team for their significant efforts in bringing together the meeting.

I.C. is an employee of, and holds stock options in, VivoSense Inc.; is part of the Editorial Board of Karger Digital Biomarkers and the Scientific Advisory Board for IMI IDEA-FAST; and has received fees for lectures and consulting on digital health at ETH Zürich and FHNW Muttenz. N.P. is an employee of, and holds stock options in, VivoSense Inc. S.B. is a member of Board of RheumaCura; current part-time employee of BioMarin; and shareholder of Novartis and Sandoz. K.W. has received advisory and consulting fees from GSK/Haleon, Novartis, Merck Group, Takeda, and OptiChroniX. J.T.B. has received consulting fees from Verily Life Sciences, Mindstrong Health, Inc., and Healios Ltd. C.B. is a full-time employee of Newcastle University and a member of the IDEA-FAST consortium. F.C. is an employee and shareholder of Cambridge Cognition. D.E. is employed by SYSNAV. L.F. is a holder of Evidation Health shares and stock options. R.G. is an employee of SHL Medical AG. P.G. is an employee of Idorsia and holds shares and stock options. N.G. is CEO and president of SMA Europe and president of SMA Schweiz and Schweizerische Muskelgesellschaft. N.G. has received advisory and consultancy honoraria from Biogen, Clinigen, Novartis, Novartis Gene Therapies (AveXis), and F. Hoffmann-La Roche. E.I. is an employee of Koneksa Health and may own company stock. C.K. is an employee and shareholder of Biogen. L.L. is an employee of F. Hoffmann-La Roche and holds shares of the company. K.L. is an employee of, and holds stock options in, VivoSense Inc. A.M. is an employee of Novartis Pharma and holds stock of the company; he is also a member of the Mobilise-D consortium. J.N. is an employee of F. Hoffmann-La Roche and holds shares of the company. W.N. has provided consultation services for the following companies in the area of Sjogren’s syndrome and/or fatigue: Novartis, GlaxoSmithKline, AbbVie, BMS, Sanofi, MedImmune, Argenx, Janssen, Resolve Therapeutic, and UCB. D.N. and F.O. are full-time employees and shareholders of F. Hoffmann-La Roche. T.M.P. is an employee of F. Hoffmann-La Roche and holds stock or stock options. W.P. is an employee and holds shares of Takeda Pharmaceutical Co. Ltd. A.S. is an employee and shareholder of Biogen. N.T. is an employee of Cambridge Cognition and holds stock options. J.T. is scientific advisor and stockholder of Precision Mental Wellness. S.X. is an employee of Sibel Health. L.B. is the owner of Casebase Health GmbH. D.E.C., A.K., T.G.A., A.C., A.R., B.V., T.V., J.G., S.C.H. have no conflicts of interest to declare.

The meeting was sponsored by the Wellcome Trust, with participation for some participants facilitated by the ETHZ and FNIH. Participation in the meeting was free. The IDEA-FAST project has received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement No. 853981. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme, EFPIA, and associated partners. Mobilise-D has received funding from IMI 2 Joint Undertaking under grant agreement No. 820820. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme and the European Federation of Pharmaceutical Industries and Associations (EFPIA). Accelerating Medicines Partnership and AMP are registered service marks of the US Department of Health and Human Services.

I.C., L.B., D.E.C., T.V., J.G., and S.C.H. formed the Core Organizing Committee for the meeting and selected the case studies; all authors actively participated in the meeting and resulting discussions; I.C., N.P., D.E.C., S.B., A.K., and K.W. created the first draft of the manuscript; I.C., A.K., K.W., and J.T. contributed initial figures; I.C., N.P., D.E.C., S.B., A.K., K.W., T.G.A., J.B., C.B., A.C., F.C., D.E., L.F., R.G., P.G., N.G., E.I., C.K., L.L., K.L., A.M., J.N., W.N., D.N., F.O., T.P., W.P., A.R., A.S., N.T., J.T., B.V., S.X., L.B., T.V., J.G., and S.C.H. contributed to revisions and approved the final manuscript and agree to be accountable for all aspects of the final manuscript. The views expressed in this article are the personal views of the author(s) and may not be understood or quoted as being made on behalf of or reflecting the position of the European Medicines Agency or one of its committees or working parties.