Digital Measures Development: Lessons Learned from an Expert Workshop Addressing Cross-Therapeutic Area Measures of Sleep Open Access

Along with nutrition and exercise, sleep is considered the “third pillar of health” and as such, insufficient or inadequate sleep can have a major detrimental effect on quality of life. The relationships between sleep and ill-health can be complex; for example, poor sleep may cause or exacerbate certain diseases [1]; disturbed sleep may be a symptom of an underlying condition [2]; pharmaceutical or device-based therapies may lead to sleep deprivation or fragmentation [3]; and the various stressors associated with navigating a care pathway may negatively impact sleep quality of patients as well as their caregivers [4, 5]. As such, there is increasing interest in measuring sleep using digital tools that are accessible, unobtrusive, and inexpensive, in order to capture aspects of sleep and sleep disturbance that are meaningful to patients, caregivers, and clinicians [6].

The Digital Measures Development: Core Measures of Sleep project is a precompetitive collaboration hosted by the Digital Medicine Society which aims to identify global clinically and patient-relevant meaningful aspects of health (MAHs) that can be used as the focus of digital measurement of sleep and sleep disturbance across a broad range of therapeutic areas (Fig. 1). One component of this project involves completion of a systematic review of qualitative literature to elicit patient-relevant concepts of interest (COIs) and MAHs related to sleep and sleep disturbance, which will be used to develop (1) a catalog of existing digital sleep measures related to the discovered COIs; and (2) a publicly accessible resource describing the parameters of assessment and associated meta-information and which we recommend should be used when assessing the discovered sleep COIs.

Assessing the clinically and patient-relevant aspects of sleep across all therapeutic areas is not a realistic goal; instead, several broad therapeutic areas need to be selected, with results generalized to other therapeutic areas where appropriate. To address this critical first step of the precompetitive collaboration, an expert workshop, consisting of regulator, payer, industry, academic and patient contributors, was held on April 19th, 2023 to discuss the need for omni-therapeutic sleep measurement, the challenges that arise, and highlight potential therapeutic areas which could be used to fulfill this goal. The following is a reflection on the discussion held at the workshop, describing seven important themes that can be effectively implemented into future work in this field.

The workshop began with 3 patient/caregiver representatives presenting their experiences regarding how their sleep and sleep disruptions, or those of the person they care for, have impacted their activities of daily living and health-related quality of life. The points raised by these individuals experiencing idiopathic hypersomnia, attention deficit hyperactivity disorder, and providing care for a family member living with Alzheimer’s disease, will be highlighted throughout all themes discussed in this document; however, here we will focus on one revealing quote from an individual with attention deficit hyperactivity disorder.

This individual mentioned that she likes to track her sleep patterns using a popular commercial wrist-worn device; however, she reported that she is often awake and lying still at night while attempting to sleep, and on many occasions, she has noticed that her wrist-worn device captures these periods of quiet wakefulness as sleep. As a result, she lacked confidence in the reported data.

The accuracy and precision of measurement are important aspects for capturing data that are interpretable, actionable, and trusted by patients and providers. This includes making sure that the end user of a measurement product understands the limitations associated with its use. The example above highlights the need to inform end users that sleep tracking based on movement and pulse rate may not have the specificity of determining sleep from quiet wakefulness. In addition, this example highlights the fact that more in-depth sleep assessment, such as polysomnography, may be needed as a complementary or alternative strategy when the goal is to measure the nuances of sleep or events such as microsleeps that may be important to the end user or informative to a specific research question.

During the expert panel discussions which followed the presentations by patients and caregivers, the question of “what it means to be asleep” was raised. Specifically, one panelist pointed out that it could be important to measure different states of wakefulness during waking hours resulting from good sleep, in addition to capturing various aspects of the overnight sleep period. Not only is it important to define “good sleep” but being able to measure good resulting wakefulness is also likely important.

This proposal started with the idea that feeling “foggy” or drowsy could be a measurable phenomenon related to the processes underlying sleep while also being directly related to the impacts of poor sleep in some instances. However, it led to a broader discussion about the impact of sleep on all aspects of daytime performance and waking life and that measuring sleep and sleep disturbances only addresses a part of the problem. This is also an important question to consider when assessing the restorative function of sleep as a physiological process.

One key challenge for the expert panelists and all workshop attendees lies in determining when sleep measurement stops and measurement of other constructs begin. For example, physical activity could be impacted by a poor night’s sleep, but defining measurement parameters for this aspect of health is not within the scope of the Digital Measures Development: Core Measures of Sleep project, in contrast to napping behavior that is more directly related to sleep measurement. The gray area lies in aspects such as daytime sleepiness, drowsiness, fatigue, cognition, or alertness. These measures are not part of sleep but are so intrinsically linked to sleep that it is difficult to envisage a situation in which some aspect of these constructs would not be relevant. This discussion highlighted the importance of an approach which includes the patient-relevant and clinical COI and will be carried forward into the objectives of the upcoming systematic review.

During the expert panel discussion, the important point of intersectionality between stakeholder perspectives was raised. As such, when undertaking work to define meaningful parameters of sleep and sleep disturbance, it is important to consider not just what the clinician, patients, and care partners find important but also consider this in light of possible mechanisms of actions that interventions could reasonably target. For example, a measurement parameter of sleep which is important to clinicians in understanding the underlying disease may not be relevant to a patient’s experience of their sleep and health. A parameter that is important to clinicians and patients may or may not be a possible target for treatment or intervention.

There are counterpoints to these examples, however. The first is that measures that are solely clinically relevant such as sleep apnea, are still important in understanding the disease, its manifestation, and therapy advice, even in cases where symptoms are not perceived by the patient. The Digital Measures Development: Core Measures of Sleep project aims to assess both the clinically and patient-relevant aspects of sleep and sleep disturbance, find similarities between the two, but also appreciate and develop parameters that are unique to each stakeholder group. The second counterpoint is that there are unknowns. The absence of a viable mechanistic pathway for treatment should not prevent reporting on parameters that have been identified as meaningful to one or more stakeholders. As such, the current project will not restrict itself to parameters that could be directly influenced by available and developing treatments but work closely with experts to identify concepts of sleep and sleep disruption that are likely to be current targets of intervention and concepts that are goals for the future. An example of this may be periodic leg movements during sleep, a phenotype that has no clearly proven deleterious effects on health.

One point raised during the expert panel discussion was around the importance of measuring sleep in the home environment; for example:

• 1.

  Home-based measurement offers a more naturalistic measurement of sleep behavior outside of the artificial setting of the laboratory.

• 2.

  Night-to-night variability of sleep and sleep disruption can only be understood through intensive longitudinal data collection as opposed to a single snapshot of data collected during in-laboratory assessment/s.

• 3.

  Home-based sleep testing is generally more accessible in terms of both cost and geographic region.

Currently, the predominant model for sleep assessment uses a combination of laboratory-based polysomnography and patient-reported outcomes measures targeting sleep disturbance and sleep-related functional status. While these modes will continue to play a crucial role in the assessment of future interventions and management of disease, more frequent measurements allow for a more complete picture of treatment response. This has the potential advantage of detecting change as it occurs, dose response (in the context of a clinical trial), and remote monitoring of patient improvement or worsening (in the context of disease management).

However, with great potential also comes great challenges, most notable in this case being large-scale data management and analysis techniques. Traditional methods of analyzing data are not well suited to the more frequent measurement of data arising from digital health technologies. Although the Digital Measures Development: Core Measures of Sleep project scope is to define the measurement parameters for use moving forward, other work is required focused on analysis strategies and ensuring that interoperable infrastructure is in place that is compatible with clinical workflows. Without taking this approach, the community assessing digital sleep measurements, and digital health measurements more broadly, could find itself in a situation where it has powerful new tools, but without the skills and knowledge to properly utilize them.

During the expert panel discussion, two important points were raised about the ethics of data collection and dissemination: (1) increased patient attention to sleep data can lead to increased worry and anxiety, described in the literature as “orthosomnia” [7]; and, (2) data arising from well-intentioned measurement may be used in ways that were not intended or foreseen.

To the former point, it is important to ensure that the measurement parameters developed as part of the Digital Measures Development: Core Measures of Sleep project are carefully disseminated to the appropriate audience/s. Parameters intended for clinical interpretation may not be appropriate to disseminate more widely as part of a consumer-facing platform. Additionally, easily accessible online educational resources helping people interpret their own data and understand its quality and accuracy are important. However, such information could also have a negative impact if, for example, reference values are not placed into the appropriate context.

To the latter point, there are cases where good measurements have led to unintended consequences. For example, the US Centers for Medicare and Medicaid Services (CMS) requires that patients prescribed continuous positive airway pressure (CPAP) for obstructive sleep apnea reach a minimum threshold of adherence within the first 90 days of therapy, which can be measured through remote monitoring of digital data captured by CPAP devices. Most US-based payers, as well as many healthcare systems and payers internationally, have adopted the same approach [8] despite observational studies suggesting that CPAP usage below the CMS threshold is also associated with clinical benefit [9]. Therefore, although technologies developed to measure adherence are an excellent tool for monitoring treatment usage patterns and contextualizing clinical response to treatment, they can sometimes be adopted for purposes for which they may not have been intended.

The landscape of digital health technologies is already expansive: there are a plethora of proposed solutions already on the market, and already in use in clinical trials. However, to promote trust in these products, particularly when applied in a clinical care or research setting, extensive evidence generation is needed. It was acknowledged during the regulatory discussion session of the workshop that a large body of evidence must be collected over the course of multiple clinical trials. The road map is not yet clear, but there are resources available to assist in developing such plans [6, 10‒15]. The FDA and other regulatory authorities are receiving increased submissions on the topic of digital health measurement, suggesting the authorities are gaining a better understanding of their needs and familiarity with the data and evidence requirements and interpretation.

Finally, it is important to realize this burden in the context of potential treatment and trial timelines. Therapeutic areas will vary in the speed at which a treatment effect or clinical change can be observed. For example, a cardiac condition may have a shorter timeline for assessing change than a neurodegenerative condition. This means that the return on investment for the bolus of work needed will follow different timelines depending on the therapeutic area.

The Digital Measures Development: Core Measures of Sleep project aims to help accelerate the timeline for this process by preparing the groundwork for clinical trial sponsors and sleep researchers to build from. The output of this project will not be the final guidance for measurement strategy in this field but will provide a consistent, well-researched baseline from which to build.

Throughout the workshop, measures of sleep and sleep disturbance were considered irrespective of a particular clinical context. However, the concrete work on identifying the current state-of-the-art in sleep assessment should be based on specific COIs and the corresponding MAHs.

How do we define the clinical scope of which therapeutic areas to choose for the initial assessment? There are two major aspects in the selection process: (1) the unmet needs of the clinical condition and (2) the potential impact on therapeutic development and population health. Sleep, as one of the key physiological states of human life, has long been the focus of clinical research with its own professional bodies and various clinical practice guidelines and standards. In particular, the International Classification of Sleep Disorders [16] is an extensive guide to diagnosis and coding of various sleep conditions which are considered to be addressable as primary conditions even when their nature is not fully established. Since the conditions described in ICSD are being addressed in a systematic way by healthcare professionals and clinical practice, we consider this as a covered need and place it into the “Sleep Disorders per ICSD-3” of the Therapeutic Areas map (Fig. 2). Examples of these are Cheyne-Stoke breathing in congestive heart failure and sleep disturbances in neurodegenerative disease.

However, in some cases, a sleep disturbance is not classified as a primary addressable condition (“Other Conditions”). A given sleep disturbance could be a symptom of an underlying organic disease. Changes in sleep patterns in these cases can signal disease progression, possible exacerbation, clinical improvement, or a variety of other possible outcomes. This is where we believe the largest area of unmet needs is. Observing variability in sleep patterns for many clinical cases requires longitudinal data and analyses. This could be on the short-term (days), medium-term (weeks to months), or long-term (years).

Finally, it is possible that there is no identifiable disease, but, instead, there are conditions of living, environment, and occupation that interfere with normal and healthy sleep. We identify it as “Psychological and Environmental Factors,” and we consider this group of conditions as an unmet need in population health even, though it may not have an immediate clinical manifestation. Typical examples of group 3 are shift work, sleep deprivation, jet lag, or as we discussed above sleep disruption when taking care of a relative affected with Alzheimer’s disease.

With this high-level classification model in mind, we plan to include “Sleep Disorders per ICSD-3” therapeutic areas while also having a focus on the therapeutic areas belonging to “Other Conditions” such as cardiovascular and pulmonary diseases, neurological and psychiatric disorders, menopausal transition, metabolic diseases, etc. Many specific clinical cases manifest in sleep disturbances which amplify the burden of the disease and interfere with patients’ functional status, lifestyle, and wellness. For example, the following questions could be asked in search for an appropriate clinical context (for all groups):

• Is a physiological measure recorded during sleep (as a behavior) being used as a risk predictor for an underlying disease, i.e., because the phenotype becomes more apparent during a sleep state?

• Are sleep measures being used to evaluate a change in an underlying primary sleep disorder?

• Are sleep measures being used to assess a change in general sleep health (i.e., as a secondary phenomenon) following the amelioration or treatment of a primary condition that has caused a sleep disturbance?

• In the common case that a sleep disorder has emerged as a secondary condition, but persists following the amelioration of the primary condition (e.g., insomnia following menopause), which additional factors are required to measure and understand sleep health?

There are several possible metrics to consider to estimate the impact and select therapeutic areas in which monitoring of sleep can lead to much better therapies and/or outcomes:

• Population impact: the proportion of people affected by the condition. This could include a specific demographic group that is disproportionately affected, so the impact can be smaller in absolute numbers but life-changing for that particular group.

• Acuteness or severity of the impact: measuring sleep can save lives by preventing a terminal condition.

• Meaningfulness to the patients: do patients evaluate the symptoms and impact as important to their quality of life?

• Meaningfulness to clinicians: is sleep evaluation important to their assessment of treatment efficacy, safety, and the clinical plan?

With these criteria in mind, the project defined the therapeutic areas and the initial scope of the inquiry (Table 1). Selecting particular clinical contexts does not mean that the other therapeutic areas will not benefit at the conclusion of the project. It will be necessary to generalize the outcomes and investigate the impact and usefulness of the sleep measures outside of the initial clinical scope.

The authors wish to acknowledge assistance and contributions from Bohdana Ratitch, Elaina Bolinger, Benjamin Vandendriessche, Mojgan Payombar, Derek Buhl, Eric Nofzinger, and The Digital Measures Development: Core Measures of Sleep Project Team. We would like to thank all workshop attendees for their time and input to the basis of this work. The views expressed in this manuscript are those of the authors and do not necessarily represent the views of the National Heart, Lung, and Blood Institute, the National Institutes of Health, the United States Department of Health and Human Services, Bayer, Sleep Number Corporation, National Jewish Health, NextSense, Inc., Kaiser Permanente, or Eli Lilly and Company.

This work represents a report for a public workshop and a plan for the remainder of the project. Ethical approval is not required for this study in accordance with local or national guidelines.

J.S.B. is an employee of NextSense, Inc. J.L.P. reverses a salary and is a minor shareholder in Eli Lilly. E.M. receives a salary from Stanford University and acts as a consultant/PI at the following companies: Takeda Pharmaceuticals, Avadel, Harmony Biosciences, Vanda Pharmaceuticals, Axsome Therapeutics, Eisai and Jazz Pharmaceuticals. R.M. receives a freelance salary from the Digital Medicine Society. M.A. receives a salary from Sleep Number Corp and is a shareholder in Phillips, Inc. and Sleep Number Corp. M.K. receives a salary from Bayer. C.M. receives a salary from the National Institutes of Health. J.P.B. reports financial interests in Philips, Signifier Medical Technologies, Apnimed, and Koneksa Health. P.F. is an employee of SeeingTheta and contracts with The Digital Medicine Society, IQVIA, Open Health, Sarepta Pharmaceuticals, and Clarivate.

This work was funded as part of the Digital Medicine Society Digital Measures Development: Core Measures of Sleep project. As part of this funding, P.F. and J.P.B. were employed by the Digital Medicine Society to conduct the workshop leading to this work and write the manuscript. W.N. and F.H. were paid a stipend to attend the workshop as patient representatives. R.M. is a patient representative but also an employee of DiMe and therefore received no stipend to attend the workshop but was paid salary from DiMe for her contribution to the workshop and the manuscript development. M.K., E.M., M.A., J.B., D.H., J.L.P., and C.M. are Digital Measures Development: Core Measures of Sleep project partners but received no funding as part of this manuscript development.

P.F. and J.P.B. had substantial contributions to the conception or design of the work, drafting the work, reviewing it critically for important intellectual content, final approval of the version to be published, agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. M.K., E.M., M.A., J.B., F.H., D.H., J.L.P., R.M., C.M., and W.N. had substantial contributions to the acquisition of data and interpretation of data for the work, drafting the work, reviewing it critically for important intellectual content, final approval of the version to be published, agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

This work is based on the discussions at an expert and patient workshop. There are no specific source data associated with this manuscript. Instead, all data generated or analyzed during this work are included in this article. Further inquiries can be directed to the corresponding author.