The proliferation of digital technologies and the application of sophisticated data analysis techniques are increasingly viewed as having the potential to transform translational research and precision medicine. While digital technologies are rapidly applied in innovative ways to develop new diagnostics and therapies, the ultimate approval and adoption of these emerging methods presents several scientific and regulatory challenges. To better understand and address these regulatory science gaps, a working group of the Clinical and Translational Science Awards Program convened the Regulatory Science to Advance Precision Medicine Forum focused on digital health, particularly examining gaps in the use, validation, and interpretation of data from sensors that collect and tools that analyze digital biomarkers. The key findings and recommendations provided here emerged from the Forum and include the need to enhance areas related to data standards, data quality and validity, knowledge management, and building trust between all stakeholders.

Recent advances in digital technologies (e.g., sensors, software, and algorithms), and the increasingly ubiquitous presence of these technologies in everyday life, have introduced a new frontier in research, healthcare, and regulatory science [1, 2]. These novel approaches are seen in patient engagement in research, the development of new forms of scientific evidence to support regulatory determinations, and in the potential realization of earlier “big data” aspirations in areas such as post-approval monitoring of medical products [3-6]. More broadly, we may be in the early stages of a paradigm shift in the boundaries and relationships among biomedical research, healthcare, and everyday life activities. For the first time in human history, technologies are evolving that will allow real-time monitoring of health indicators from heterogeneous sources, with incredible potential for enhancement of precision medicine at both the population and individual levels. This also provides opportunities to enhance models for healthcare delivery across patients, providers, and systems. While offering tremendous potential benefits, these new technologies and applications raise regulatory science, technology, and broader policy issues. The US Food and Drug Administration (FDA), National Institutes of Health (NIH) and other agencies have taken significant steps to promote new research and adaptive regulatory frameworks to speed innovations in precision medicine utilizing digital health technologies (e.g., Precision Medicine Initiative [7], All of Us [8], PrecisionFDA [9], INFORMED [10]) with many of these initiatives incorporated in the 21st Century Cures Act [11]. FDA is piloting new approaches to regulate and advance digital health technologies to improve personalized medicine (FDA Digital Health Innovation Action Plan [12]). Identifying and addressing the key regulatory science gaps to achieve the potential for digital health remains a significant challenge.

In the fall of 2018, the Regulatory Science to Advance Precision Medicine Working Group [13] (see online suppl. material for Forum attendees; for all online suppl. material, see www.karger.com/doi/10.1159/000505289) held their second annual Fall Forum, bringing together industry and foundation leaders, academic thought leaders, and leading scientists and experts from relevant government agencies to identify and address key topics and opportunities for regulatory science to advance precision medicine in the area of digital health. For the purposes of the Forum, we utilize the FDA definition of Regulatory Science, “the science of developing new tools, standards, and approaches to assess the safety, efficacy, quality, and performance of all FDA-regulated products.” We also use the FDA’s definitions of real-world data (RWD), as “data related to patient health status and/or the delivery of healthcare routinely collected from a variety of sources,” and real-world evidence (RWE) as “clinical evidence about the usage and potential benefits or risks of a medical product derived from analysis of RWD” [14, 15]. The Forum, which is informed by the work of a working group within the Methods and Processes Domain Task Force under the NIH Clinical and Translational Science Award (CTSA) Program [13], examined areas of emerging science, regulatory considerations, ethics and privacy, data storage and use, as well as RWD and patient-reported outcomes.

On 27 September 2018, a group of experts convened in Washington, DC, for a workshop on regulatory science gaps related to digital health. The Forum included morning presentations from experts Rob Califf (Duke Health and Verily Life Sciences) and Lara Mangravite (Sage Bionetworks), who discussed challenges and opportunities that exist at the intersection of digital health technologies and regulatory decision-making and where responsibility lies in such systems. Presentations from James McClain (NIH) and Praduman Jain (Vibrent Health) highlighted examples of the NIH All of Us research program and their utilization of digital health technologies, and Linda Ricci (Center for Devices and Radiological Health, FDA) spoke about the regulatory paradigms for digital health devices. Following this, two moderated sessions focused on various aspects related to digital health, addressing issues of both user and information interfaces between individuals and technology. The workshop included 13 leaders from academia, 16 from various government agencies, 12 attendees who represented a range of associated industries, plus several attendees from related foundations. Represented disciplines ranged from machine learning and imaging to health policy and neurology. Experts in research, development, and regulation of digital health technologies in clinical research and healthcare debated and prioritized the gaps existing in the use, validation, and interpretation of digital health tools and the associated health recommendations. Below are breakout session discussion highlights, key findings, and recommendations that were generated during and following the Forum.

Data Considerations

Technological advances are rapidly changing the sources, types, and volume of health-related data that can be collected [16, 17]. These data can potentially provide valuable information regarding health status, needs, disease burden, relative value of outcomes, and results in a longitudinal view of the health of an individual and the effectiveness of treatments or interventions. In turn, such evidence and insights can contribute to many aspects of the medical product development life cycle. Therefore, such data are poised to become a significant and important source of information that helps not only measure the impact of medicines, but also improve their value through the management of care pathways. However, data generated and continuously collected in real-world situations (e.g., at home monitoring, wearable devices, and other devices connected by the internet of things) may differ significantly in quality, quantity, and type from those generated through conventional healthcare or controlled research settings. How best to use these data for decision-making requires consideration of some crucial issues discussed at the workshop.

Collection, Interoperability, Interconnectivity, Portability: Clinical and Non-Clinical

Approaches that capture, store, and analyze RWD and RWE operate across a range of systems [1]. Integrating the array of different data sources will require pre-agreed criteria to effectively operate, communicate, and interface. The development of common data standards, formats, networks, and collection methods is still required to make diverse sources of data available for integration and analysis. It will be essential to develop and use data and information technology security standards to help ensure patient and physician confidence in contributing such data for analysis [18, 19]. In terms of compatibility and interpretability, there are also differing requirements related to data provenance for tracking the data systems, inputs, and processes (e.g., analyses) to create a record of the data that moves through the system. In a way, this can be viewed as the “plumbing” that connects disparate data sets and the data that actually flow through the “pipes.” The “plumbing” needs to consider directionality, the ability to recognize and control where it is coming from and where it should go to, along with any constraints on its use. This could include anonymization, de-identification, and levels of authorization for access to certain data. The data flow itself requires appropriate contextual information to be transferred to move beyond technical compatibility to ensure functional compatibility.

The ability for an active participant or contributor to control their contribution and have data returned, or entirely opt out of the data-sharing process, is an important consideration as well. This should also be considered in the broader framework of how individuals currently opt in or opt out of sharing their data and if these are systematically different groups, confirming that representative populations are included. It is also appropriate and necessary to consider issues of data provenance, to ensure data sources are authenticated to contribute (or receive) data and associated metadata (other associated data utilized to further describe the data). Additionally, it is important to have the ability to identify if the technology is in compliance with any collaborative agreements and/or regulatory requirements, particularly as they change over time. The governing use of such data will require tracking, perhaps through blockchain methods, so that the potential for data to be withdrawn or permissions for use to be withdrawn/modified can be accomplished. Additionally, there will likely be the need to specify how many derivative levels of data use are susceptible to data withdrawal beyond the first use, especially in terms of potential impact on derivative research and/or publications.

Digital technologies and platforms that have traditionally been considered “non-medical” such as personal phones and cameras are increasingly used to inform biomedical research and health decisions; thus, the boundary between these non-health and health technologies and the resulting data are blurring. How the resulting data are classified and potentially restricted will have a significant impact on the utility of these approaches.

Data Standards: Clinical and Non-Clinical Data Types

Data standards generally govern how data are described and recorded and need to evolve for both clinical and non-clinical data. These can include unique identifiers, which can aid in appropriately capturing data attributes and workflows (e.g., the BioCompute framework [20]). Additionally, establishing requirements for collecting and managing necessary metadata should be addressed. Biomedical research and healthcare represent a complex system that includes, at least, the patients or participants, physicians, researchers, payers, providers, pharmaceutical/biotech companies, diagnostics companies, regulators, and professional societies, among others. Each tends to create their own standards for data fields/data types based on their individual perspectives and priorities, resulting in a lack of a comprehensive and all-inclusive approach with harmonized standards. The rapid development of new technologies, data sources, and types merit an effort to facilitate development of a comprehensive and objective set of standards and compatible data models.

Data Quality and Validity

Data quality can be understood as a perception or an assessment of data’s fitness to serve its purpose in a given context. Using definitions of clinical and analytical validity as defined by the FDA and NIH [21], aspects of data quality include accuracy, completeness, consistency, and reliability. Validity in data collection indicates that the data represent the phenomenon being measured with acceptable sensitivity, specificity, precision, and overall performance of data collection (but not usefulness) and applies to the design and the data collection methodology being used. In the context of digital health, minimum quality requirements for RWD are essential to ensure its conversion to RWE, utilizing a “fit-for-purpose” approach. These requirements should carry sufficient authority, such that they are widely accessible and adoptable by the various stakeholders, including companies, physicians, prescribers, regulators, payers, grant-awarding bodies and journal editors, among others. Determining what digital evidence is required for efficacy, or to predict clinical events, will be critical to demonstrate clinical utility [21].

In terms of returning data to a contributor, it is important to consider if this will involve raw data or processed data and if any analysis or quality assurance will be provided. Providing transparency on these issues, clear expectations on the timeliness of providing these data, and understanding variability in contributor needs will be critical.

Data Protection for Building Trust

Establishing and maintaining trust among all parties, particularly ensuring transparency with those contributing their data, is essential. To help achieve this, data should be collected responsibly and with adherence to the principles of necessity (only collect the data needed for the questions at hand), and proportionality (only collect as much data as needed) and subsidiarity (use least sensitive data where possible).

The lineage and flow of data, or chain of custody, should be made clear to track any modifications, and can range from steps to protect privacy to processing and analysis. This includes the ability to detect any anomalies in the data. Access to summary data and analysis (and original data where appropriate), and particularly algorithms that are qualified (i.e., base code verified and data models for context of use agreed upon) to process it, will be critical. This could potentially be addressed via a common repository (similar to FDA’s public repository of biomarkers with summary data [22, 23]) or federated models with controlled access, while addressing security concerns. There was some debate regarding the need to access original data, and this can also raise potential intellectual property issues that will need further consideration. Additionally, some data collection is intrinsic to specific devices, and this should be noted and distinguished.

Evolving Data Protection Requirements

The General Data Protection Regulation (for citizens of the European Union) and related US policies that have been implemented to protect the online data of individuals [24-26], must be considered in the development of digital health technologies in the US that have the potential to enroll/impact relevant citizens. Organizations collecting these data have a host of evolving requirements including reporting and sharing data they hold, detailing how data are used, in some cases re-consenting to use personally identifiable data and deleting a consumer’s data at their request.

Converting Data to Evidence

For most emerging digital measures (e.g., wearables and smartphone sensors for ultrasound, continuous glucose monitoring), there is currently no gold standard (or legacy standard) for these digital health methods to validate against [27, 28]. Despite this issue, it is clear that the evidence and/or insights derived from such data should be validated via a clear process to ensure that the derived evidence is accurate, reliable, and without bias. There is an increasing desire to utilize RWD to generate RWE to support approvals of new drugs or for new indications [29, 30]. Collecting continuous RWD via digital technologies can provide a rich source of data and avoid some of the inherent limitations in the well-recognized gold standard of randomized controlled trials (RCTs) that utilize carefully selected populations with studies conducted under ideal conditions. At the same time, RWD collected through these biosensor approaches have the potential of bias in data collection, algorithmic biases, and lack randomization. RCTs as well can introduce biases in the application of inclusion/exclusion criteria [31, 32]. To ensure evidence is “fit-for-purpose,” accurate, reliable, and valid, global stakeholders and policy makers should develop minimum data standards and quality requirements for the underlying data sources and methodologies.

Broader Policy-Related Issues

While not the primary focus of the Forum, a number of broader policy issues were identified that are linked to regulatory science needs and opportunities to advance digital health [33, 34]. One concern is that access to and understanding of digital health technologies can lead to disparities and issues of fairness, potentially perpetuating biases and continuing and exacerbating a “digital divide” [35-37].

Another aspect is related to nomenclature standardization to improve communication and collaboration among the various sectors (e.g., healthcare and technology) that are applying new data sources to therapeutic development. The Unified Medical Language System and other systems provide approaches to consider that are utilized for biomedical vocabularies [38]. Incentive models are also important to consider, in areas such as the alignment of regulation and reimbursement. Additionally, there can be a natural tension between the monetization or commercialization of data, balanced against considerations of broader public good.

Ensuring data protection and mitigating data misuse can be viewed as a broader societal function. This is particularly relevant in areas such as the variability of consent and in data use policies [39-41]. Areas related to levels of data control and tracking of permissions, cybersecurity, and risk of re-identification will be critical to address. To address these concerns, clear rules for ethics approval, data privacy, and individuals’ consent for access to, and research use of these data should be established and applied consistently across the world. To this end, the use of novel approaches to data storage and encryption that allow for flexible and dynamic management of consent and privacy, yet also serve as a common international framework, should be encouraged and pursued. Data privacy and related policies continue to develop across the globe, and the current patchwork of various policies will create challenges for all stakeholders to ensure compliance and to have awareness and confidence in the level of protection of their data.

The primary areas for recommendations emerging from the Forum are outlined below and listed in Table 1.

Table 1.

Areas for recommendations

Areas for recommendations
Areas for recommendations

Data Standards

The development of good digital health principles to guide common data standards, data formats, metadata requirements, and methods for gathering and analyzing RWD should be harmonized and widely adopted. This will facilitate the linking of data from multiple sources through a distributed or federated system and ultimately realize the full potential of RWD across the complete range of possible applications. The best model would be transparent and extensible, without loss of existing terminologies. It should support ongoing evolution and the ability of specific data sources and providers to ideally capture context and measurements. Such a model could support the identification of critical data elements that may not be commonly considered across all sources. One example raised during the breakout discussion was how patient blood pressure is recorded in mm Hg, but neither the method used to acquire it nor ancillary data such as proximity to mealtime, at beginning or end of patient-physician encounter, is recorded. The development of such a data model would support the defining of both standards for data collection as well as identify potentially critical parameters to be included in future studies. This is particularly critical as we further progress in the era of data science and machine learning.

This process could be accomplished in multiple stages where local measurements and data generation would continue and potentially expand data elements, but for compatibility and data sharing, adherence to the core data model would be required. This approach has been successfully applied in large-scale programs such as the Immune Tolerance Network [42], where local data collection is maintained in separate core facilities. However, for transfer to the central repository, the data must meet the specifications of the core data model, metadata standards (a requirement intended to establish a common understanding of the meaning or semantics of the data, to ensure correct and proper use and interpretation of the data by its owners and users), and provenance standards. A metadata mandate should be required to ensure a minimum amount of data is captured under appropriate standards [43].

Data Quality and Validity

Implementing good algorithm practice guidelines will aid in improved data quality and validity [44]. Capturing associated meta data and ensuring validation of data are of primary importance. Where possible, unprocessed data should be archived either locally or centrally; however, this may not be feasible in some cases due to the size or nature of the data. Data reduction or processing algorithms will continue to evolve in many areas and maintaining original data can aid in verification and re-analysis, as needed. Additionally, first-pass data validation should compare the data to established and validated measurement ranges for the equipment used to measure them. Preliminary data evaluation made locally will help identify false, duplicate, or corrupt data. When these cases occur, they should be clearly annotated. Additionally, missing data, including key metadata, and potential data conflicts should be specifically identified, annotated, and evaluated for cause. While algorithmic-based methodologies such as artificial intelligence (AI) are evolving for data evaluation and characterization, outliers from normal ranges may be correct, just as observations within normal ranges may be incorrect. Therefore, while approaches to detect data corruption are important, current technology may not be adequate to completely address the potential for corrupt data resulting from more active threats.

The genomics field provides several analogies and lessons learned that can help inform the advancement of digital health technologies, from genomic databases and genetic testing to reimbursement and data privacy [45]. In conjunction with the FDA, establishment of criteria will help support the development of multiple types of open “approved” validation datasets and benchmarks that vary in purpose, type, contexts, and conditions and also present different specific types of failure. Best practice-validated software, algorithms, analysis methodologies, and tools should be developed. Where appropriate, these could be made available as free open-access and cloud-based, developed via a public-private partnership, or commercially available. FDA engagement in designing, adapting, or supporting use of a common software-sharing platform in drug development will encourage their use by AI developers, and ultimately promote AI interconnectivity and ecosystem growth. These platforms, such as Github [46] and Jenkins [47] provide important software development (Dev) and information-technology operations (Ops), DevOps tools, to allow interested parties to collaborate and learn from each other regarding the practical application of, for example, FDA best-practices towards AI (such as FDA’s MyStudies [48]). In other cases, intellectual property and marketability concerns will impact the development, sharing, and use of tools and applications. FDA engagement and incentives to promote these areas can have an important impact.

Training data that is specifically fit for purpose can be utilized to assist with model building, along with separate challenge/test data sets to help address overfitting to an inadequate dataset. Additionally, data and information technology security standards will help ensure patient and physician confidence in contributing data for analysis. This will include the use of error detection software and the use of white hat hacking to test security and vulnerability.

Overall, development of regulatory evidentiary standards and consideration of the strength of evidence should have a fit-for-purpose mindset and occur through a transparent process. Considerations should be based on validated digital measures that are of sufficient relevance and reliability to address the specific regulatory decision being considered. These key attributes to evaluate the appropriateness of digital measures serving as RWE in regulatory decision-making include: valid, fit-for-purpose, and transparent (Fig. 1) [49, 50]. The intersection of these attributes will advance the development of RWE and inform how risk and benefit will be assessed utilizing these evolving digital health methods. Additionally, in many cases we have seen the level of evidence required will naturally evolve over time.

Fig. 1.

Key attributes for digital measures to support regulatory decisions, where the intersection of these areas will advance the development of real-world evidence.

Fig. 1.

Key attributes for digital measures to support regulatory decisions, where the intersection of these areas will advance the development of real-world evidence.

Close modal

Knowledge Management

Knowledge management across the public, physicians, researchers, payers, providers, pharmaceutical/biotech companies, diagnostics companies, regulators, professional societies, among others, is a key issue in the rapidly evolving digital health landscape. Digital health involves the creation of interdisciplinary terminology and models that should be broadly interpretable to diverse stakeholders beyond the traditional healthcare setting. Training and access to this common terminology will be essential, particularly for technology companies that are newly entering the medicine and healthcare space. It will be beneficial to have a common location to access information and best practices to determine the appropriate level of consent required, particularly in considering different levels of risks. Understanding options for permission-based policies to guide data access and available support tools to consider will also be helpful [48].

Beyond awareness of guidance documents (and applicability to the next generation of digital health tools) and FDA’s increased transparency and adaptability to this evolving area, other areas of knowledge management remain critical to supporting the development and approval of digital health tools. A historical knowledge of existing models that can be reused for better or new inferences on old data, along with an adequate pool of trained and expert data analysts, algorithm developers, and data scientists will be essential.

Building Trust

Inherent in maintaining trust, is the obligation to provide clear rules for ethics approval, data privacy, and individuals’ consent for access and use of RWD. These policies should be established and applied consistently across the world, such as through the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use, International Organization for Standardization, or European Commission [51]. An individual’s preference regarding privacy and data use should be safeguarded, while also supporting a process for the flow of data across national borders. Research to develop new technologies that enable analysis of nationally localized datasets without the data having to move should be encouraged and supported.

The use of novel approaches to data storage and encryption that allow for flexible, dynamic management of consent and privacy without the need for extensive anonymization, potentially via blockchain and cloud-based technologies, should be considered. A small-scale pilot demonstration or development of “duplication” trials [52] could provide empirical evidence and build trust in the robustness of the new approaches and methodologies.

In addition to ensuring participants have the option to receive any data collected, guidelines can outline how contributors may modify or entirely opt-out of “data donations” but possibly continue receiving therapeutic benefit. This includes uniform and clear processes for participants to choose to no longer receive data or information, particularly if possibly inflictive, while still participating. This requires transparent rights of the participant established in a well-developed informed consent process, which considers how incidental findings may be governed and recognizes individual and cultural variability in contributor interest. This is especially important in ethno-cultural groups where culture may be a critical factor in disease presentation and patient response to both diagnosis and treatment. Additionally, while participants may view themselves as the owner of collected data, in the US many digital consents from commercial sensors and applications do not consider the participants as an owner.

The discussions and recommendations from the Fall 2018 forum represented a “snapshot” moment in the rapidly evolving area of digital health. Key and enduring themes of this workshop, which are also noted and reinforced in the broader literature, include several central points. Of these, we highlight three in conclusion.

First, without a deep standardization of conventions and processes for collection, harmonization, validation, and management of data to be used for a research purpose, the grand aspirations of advancing data from heterogeneous sources to RWE are unlikely to be realized. Such data are not inherently provided in a form to directly compare, analyze, and validate. This is a challenge that will require involvement of multiple stakeholders to develop useable consensus standards, and likely governmental authority on an international scale to implement.

A second key theme is the necessity to engage issues of data use in social and ethical context. The obligation for responsible stewardship of data includes and transcends technical issues of data confidentiality. In particular, workshop participants highlighted the ethical and social requirement to respect and implement consultation and choice, at the individual and sometimes group level, regarding use of data. We have already seen the broad negative consequences of loss of public trust in organizations that collect and (mis-)use personal data; biomedical research must emerge as an exemplar of responsible and trustworthy data stewardship if a RWD-based research enterprise is to be developed and sustained.

Finally, the universe of heterogeneous data found in RWD maps to an equally heterogeneous set of stakeholders. Digital health is an inherently multi-institutional and multi-disciplinary endeavor. Developing effective approaches for multidisciplinary work, including common language and appreciation of different frames of reference and expertise will be essential for a diverse group of stakeholders including scientists, regulators, research subjects, technology entrepreneurs, and others to work together cohesively.

The CTSA-supported Regulatory Science to Advance Precision Medicine Working Group selected the topic to bring to the Fall Forum and worked to develop background documents. The authors thank PhRMA Foundation for their support of the Forum and Eileen Cannon for her participation on the planning committee, as well as Joanne Westphal for assistance with the Fall Forum.

The authors have no conflicts to disclose.

Support for this paper and Forum came from the University of Rochester CTSA award number UL1 TR002001. Additional support for the Forum was provided by the PhRMA Foundation. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health, the National Science Foundation, or Food and Drug Administration.

1.
Coravos
A
,
Goldsack
J
,
Karlin
D
,
Nebeker
C
,
Perakslis
E
,
Zimmerman
N
, et al
Digital Medicine: A Primer on Measurement
.
Digit Biomark
.
2019
;
3
(
2
):
31
71
. 2504-110X
2.
Sim
I
.
Mobile Devices and Health
.
N Engl J Med
.
2019
Sep
;
381
(
10
):
956
68
.
[PubMed]
0028-4793
3.
NEST Coordinating Center
;
An Initiative of Medical Device Innovation Consortium (MDIC)
.
2019
; Available from: https://nestcc.org/
4.
FDA’s Sentinel Initiative
.
Transforming how we monitor the safety of FDA-regulated products
.
2019
; Available from: https://www.fda.gov/safety/fdas-sentinel-initiative
5.
Wilson
H
,
Anatchkova
M
,
Gelhorn
H
. A Perspective on the 21st Century Cures Act: Patient-Focused Drug Development, in The Evidence Forum.
2017
: https://www.evidera.com/wp-content/uploads/2017/11/04-A-Perspective-on-The-21st-Century-Cures-Act-Patient-Focused-Drug-Development_2017Nov-1.pdf
6.
FDA Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices
.
2017
.
7.
Kaiser
J
.
President Obama’s 1-million-person health study kicks off with five recruitment centers
.
Science
.
2016
. 0036-8075
8.
All of Us Initiative
. accessed June 28, 2019; Available from: https://allofus.nih.gov/
9.
Precision
FD
. accessed June 28, 2019; Available from: https://precision.fda.gov
10.
Khozin
S
,
Pazdur
R
,
Shah
A
.
INFORMED: an incubator at the US FDA for driving innovations in data science and agile technology
.
Nat Rev Drug Discov
.
2018
Aug
;
17
(
8
):
529
30
.
[PubMed]
1474-1776
11.
21st Century Cures Act.
2016
; Available from: https://www.congress.gov/114/plaws/publ255/PLAW-114publ255.pdf
12.
FDA Digital Health Innovation Action Plan
. Available from: https://www.fda.gov/medical-devices/digital-health
13.
Regulatory Science to Advance Precision Medicine Working Group
. 2019 26 Nov
2019
]; Available from: https://clic-ctsa.org/groups/regulatory-science-advance-precision-medicine
15.
Framework for FDA's Real-World Evidence Program
.
2018
.
16.
Medicine in the Digital Age
.
Nat Med
.
2019
;
25
(
1
).1078-8956
17.
Topol
EJ
.
A decade of digital medicine innovation
.
Sci Transl Med
.
2019
Jun
;
11
(
498
):
1
3
.
[PubMed]
1946-6234
18.
HIPAAJournal
. October 2019 Healthcare Data Breach Report. 2019 25 Nov
2019
; Available from: https://www.hipaajournal.com/october-2019-healthcare-data-breach-report/
19.
Principles and Practices for Medical Device Cybersecurity
. in
International Medical Device Regulators Forum
.
2019
: Medical Device Cybersecurity Working Group.
20.
Alterovitz
G
,
Dean
D
,
Goble
C
,
Crusoe
MR
,
Soiland-Reyes
S
,
Bell
A
, et al
Enabling precision medicine via standard communication of HTS provenance, analysis, and results
.
PLoS Biol
.
2018
Dec
;
16
(
12
):
e3000099
.
[PubMed]
1544-9173
21.
BEST
.
(Biomarkers, EndpointS, and other Tools) Resource. FDA-NIH Biomarker Working Group
.
Bethesda (MD)
:
National Institutes of Health
;
2016
.
22.
Amur
S
,
LaVange
L
,
Zineh
I
,
Buckman-Garner
S
,
Woodcock
J
.
Biomarker Qualification: Toward a Multiple Stakeholder Framework for Biomarker Development, Regulatory Acceptance, and Utilization
.
Clin Pharmacol Ther
.
2015
Jul
;
98
(
1
):
34
46
.
[PubMed]
0009-9236
23.
FDA List of Qualified Biomarkers
. [cited 2019 13 Aug 2019]; Available from: https://www.fda.gov/drugs/cder-biomarker-qualification-program/list-qualified-biomarkers
24.
Mondschein
C
,
Monda
C
. The EU’s General Data Protection Regulation (GDPR) in a Research Context. In:
Kubben
P
,
Dumontier
M
,
Dekker
A
, editors
.
Fundamentals of Clinical Data Science
.
Cham
:
Springer
;
2019
. pp.
55
71
.
25.
Serrato
,
J.
,
Cwalina
,
C.
,
Rudawski
,
A.
,
Coughlin
T.
and
Fardelmann
,
K.
, US states pass data protection laws on the heels of the GDPR, in Data Protection Report.
2018
.
26.
Rothstein
MA
,
Tovino
SA
.
California Takes the Lead on Data Privacy Law
.
Hastings Cent Rep
.
2019
Sep
;
49
(
5
):
4
5
.
[PubMed]
0093-0334
27.
Bothwell
LE
,
Greene
JA
,
Podolsky
SH
,
Jones
DS
.
Assessing the Gold Standard—lessons from the History of RCTs
.
N Engl J Med
.
2016
Jun
;
374
(
22
):
2175
81
.
[PubMed]
0028-4793
28.
Hampson
G
,
Towse
A
,
Dreitlein
WB
,
Henshall
C
,
Pearson
SD
.
Real-world evidence for coverage decisions: opportunities and challenges
.
J Comp Eff Res
.
2018
Dec
;
7
(
12
):
1133
43
.
[PubMed]
2042-6305
29.
Bakker
JP
,
Goldsack
JC
,
Clarke
M
,
Coravos
A
,
Geoghegan
C
,
Godfrey
A
, et al
A systematic review of feasibility studies promoting the use of mobile technologies in clinical research
.
NPJ Digit Med
.
2019
Jun
;
2
(
1
):
47
.
[PubMed]
2398-6352
30.
Franklin
JM
,
Glynn
RJ
,
Martin
D
,
Schneeweiss
S
;
Evaluating the Use of Nonrandomized Real-World Data Analyses for Regulatory Decision Making
.
Evaluating the Use of Nonrandomized Real-World Data Analyses for Regulatory Decision Making
.
Clin Pharmacol Ther
.
2019
Apr
;
105
(
4
):
867
77
.
[PubMed]
0009-9236
31.
Tinè
F
,
Attanasio
M
,
Muggeo
VM
,
Crainiceanu
CM
.
Evidence of bias in randomized clinical trials of hepatitis C interferon therapies
.
Clin Trials
.
2017
Oct
;
14
(
5
):
483
8
.
[PubMed]
1740-7745
32.
Spieth
PM
,
Kubasch
AS
,
Penzlin
AI
,
Illigens
BM
,
Barlinn
K
,
Siepmann
T
.
Randomized controlled trials - a matter of design
.
Neuropsychiatr Dis Treat
.
2016
Jun
;
12
:
1341
9
.
[PubMed]
1176-6328
33.
OECD, Recommendation of the Council on OECD Legal Instruments Artificial Intelligence
. OECD/LEGAL/0449,
2019
.
34.
Ethics Guidelines For Trustworthy
AI
.
2019
,
High-Level Expert Group On Artifical Intelligence Set Up by the European Commission.
35.
Torous
J
,
Rodriguez
J
,
Powell
A
.
The New Digital Divide For Digital BioMarkers
.
Digit Biomark
.
2017
Sep
;
1
(
1
):
87
91
.
[PubMed]
2504-110X
36.
Fang
ML
,
Canham
SL
,
Battersby
L
,
Sixsmith
J
,
Wada
M
,
Sixsmith
A
.
Exploring Privilege in the Digital Divide: Implications for Theory, Policy, and Practice
.
Gerontologist
.
2019
Jan
;
59
(
1
):
e1
15
.
[PubMed]
1758-5341
37.
Susarla
,
A.
, The new digital divide is between people who opt out of algorithms and people who don’t, in The Conversation.
2019
.
38.
Zhu
X
,
Fan
JW
,
Baorto
DM
,
Weng
C
,
Cimino
JJ
.
A review of auditing methods applied to the content of controlled biomedical terminologies
.
J Biomed Inform
.
2009
Jun
;
42
(
3
):
413
25
.
[PubMed]
1532-0464
39.
Time to discuss consent in digital-data studies
.
Nature
.
2019
Aug
;
572
(
7767
):
5
.
[PubMed]
0028-0836
40.
Martinez-Martin
N
,
Insel
TR
,
Dagum
P
,
Greely
HT
,
Cho
MK
.
Data mining for health: staking out the ethical territory of digital phenotyping
.
NPJ Digit Med
.
2018
;
1
(
1
):
1
.
[PubMed]
2398-6352
41.
Alter
G
,
Gonzalez
R
.
Responsible practices for data sharing
.
Am Psychol
.
2018
Feb-Mar
;
73
(
2
):
146
56
.
[PubMed]
0003-066X
42.
Bluestone
JA
,
Auchincloss
H
,
Nepom
GT
,
Rotrosen
D
,
St Clair
EW
,
Turka
LA
.
The Immune Tolerance Network at 10 years: tolerance research at the bedside
.
Nat Rev Immunol
.
2010
Nov
;
10
(
11
):
797
803
.
[PubMed]
1474-1733
43.
Canham
S
,
Ohmann
C
.
A metadata schema for data objects in clinical research
.
Trials
.
2016
Nov
;
17
(
1
):
557
.
[PubMed]
1745-6215
44.
Ordish
J
,
Hannah
M
,
Hall
A
.
Algorithms as Medical Devices
.
University of Cambridge
;
2019
. pp.
1
60
.
45.
Tung, J.Y., R. J. Shaw, J. M. Hagenkord, M. Hackmann, M. Muller, S. H. Beachy, V. M. Pratt, S. F. Terry, A. K. Cashion, and G. S. Ginsburg, Accelerating Precision Health by Applying the Lessons Learned from Direct-to-Consumer Genomics to Digital Health Technologies
.
National Academy of Medicine Perspectives
;
2018
.
46.
Github. Available from: https://github.com/
47.
Jenkins. Available from: https://jenkins.io/
49.
Schneeweiss
S
,
Eichler
HG
,
Garcia-Altes
A
,
Chinn
C
,
Eggimann
AV
,
Garner
S
, et al
Real World Data in Adaptive Biomedical Innovation: A Framework for Generating Evidence Fit for Decision-Making
.
Clin Pharmacol Ther
.
2016
Dec
;
100
(
6
):
633
46
.
[PubMed]
0009-9236
50.
Miksad
RA
,
Abernethy
AP
.
Harnessing the Power of Real-World Evidence (RWE): A Checklist to Ensure Regulatory-Grade Data Quality
.
Clin Pharmacol Ther
.
2018
Feb
;
103
(
2
):
202
5
.
[PubMed]
0009-9236
51.
Interoperability & Standardisation
.
Connecting eHealth Services.
Available from: https://ec.europa.eu/digital-single-market/en/interoperability-standardisation-connecting-ehealth-services
52.
Randomized
CT
.
Applying Techniques of Epidemiology.
Available from: https://www.rctduplicate.org/
Open Access License / Drug Dosage / Disclaimer
This article is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND). Usage and distribution for commercial purposes as well as any distribution of modified material requires written permission. Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug. Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.