Background: There are many approaches to evaluating aging-in-place technologies. While there are standard measures for outcomes such as health and caregiver burden, which lend themselves to statistical analysis, researchers have a harder time identifying why a particular information and communication technology (ICT) intervention worked (or not). Objective: The purpose of this paper is to review a variety of methods that can help answer these deeper questions of when people will utilize an ICT for aging in place, how they use it, and most importantly why. This review is sensitive to the special context of aging in place, which necessitates an evaluation that can explore the nuances of the experiences of older adults and their caregivers with the technology in order to fully understand the potential impact of ICTs to support aging in place. Methods: The authors searched both health (PubMed) and technology (ACM Digital Library) venues, reviewing 115 relevant papers that had an emphasis on understanding the use of aging-in-place technologies. This mini-review highlights a number of popular methods used in both the health and technology fields, including qualitative methods (e.g. interviews, focus groups, contextual observations, diaries, and cultural probes) and quantitative methods (e.g. surveys, the experience sampling method, and technology logs). Results: This review highlights that a single evaluation method often is not adequate for understanding why people adopt ICTs for aging in place. The review ends with two examples of multifaceted evaluations attempting to get at these deeper issues. Conclusion: There is no proscriptive formula for evaluating the intricate nuances of technology acceptance and use in the aging-in-place context. Researchers should carefully examine a wide range of evaluation techniques to select those that will provide the richest insights for their particular project.
The aging population is predicted to outnumber children under the age of 15 by the year 2045 , suggesting older adults will soon overwhelm the number of caregivers available to assist them.To compensate, many are looking towards helping older adults stay in their home for as long as possible, also known as ‘aging in place'. In the USA, 95% of adults over the age of 75 want to stay in their homes as they age . Aging in place improves perceived quality of life  and is cost-effective. Indeed, health care costs of staying at home are a fraction of the costs of institutionalization . But aging in place must overcome many challenges, including declines in cognitive and functional abilities of older adults as well as time pressures on formal and informal caregivers who must visit older adults in their homes throughout the community.
The use of information and communication technologies (ICTs) to overcome these challenges has gained much popularity in recent years, including technologies for home rehabilitation, remote monitoring, and social well-being . Yet, how to evaluate the impact of such technologies is still in its infancy.
The field of gerontology has many methods for evaluating if a particular intervention is successful. For interventions to support aging in place, whether the intervention is technological or not, there are numerous standard measures for health outcomes and caregiver burden. What is less clear is how to evaluate the ICT beyond these standard measures. While we can measure if an ICT produces better health outcomes, lowers caregiver burden, and saves money over the life of an individual, we have a harder time identifying why a particular ICT intervention worked (or not). Understanding the why is critical if we are to make rapid progress in this field. The purpose of this paper is to review a variety of methods that can help answer these deeper questions of when people will utilize an ICT for aging in place, how they use it, and most importantly why.
While some of the factors that impact the when, how, and why of technology use by older adults are similar to that of their younger counterparts (e.g. the technology's usefulness), others are significantly different (e.g. concerns that technology will replace human contact). But with all, the special context of aging in place necessitates an evaluation that can explore the nuances of the older adults' experiences with the technology in order to fully understand the potential impact of ICTs to support aging in place.
For this mini-review, the authors searched both health (PubMed) and technology (ACM Digital Library) venues with the following keywords: aging-in-place + technology + evaluation, aging + home technologies + evaluation, aging + assistive technologies + evaluation. 1,216 papers were returned and 115 were determined to be relevant to our objective of understanding the use of aging-in-place technologies. Methods pertaining to understanding the when, how, and why of technology use were noted and references for those methods were retrieved from the papers, additional searchers, or textbooks. Here we report on general methods, but not specific measures.
Types of Evaluations
There are two basic types of evaluations that can occur for exploring the reasons behind a technology's ultimate success or failure. Studies in a lab environment bring people into a formal evaluation lab and have them undergo a series of tasks to better understand a variety of aspects of technology use, including how easy it is for a person to use the technology to perform certain tasks, what problems they may encounter when using the technology, and the participants' initial impressions of the technology after using it for a short time. These in-lab studies often focus on usability of the technology - making sure people can perform the desired task and redesigning any interfaces that users find difficult. These lab-based studies are critical during the design process to obtain and incorporate user feedback before investing the resources necessary to deploy an ICT in people's homes.
In the area of aging in place, ‘living laboratories'  have emerged as the predominant lab environment for such evaluations. Living labs allow an ICT to be presented in a home-like environment so that participants can better imagine what the technology might be like in their own home. Several people have used such spaces to present a number of ICTs to support aging in place to small groups of people . Typically, once older adults are given a tour and allowed to interact with the technology, they are asked a series of questions to elicit their opinions on the usefulness, usability, and likelihood that they would use such a system in their own homes if available. Results from such studies allow ICT designers to refine their designs  and draw design principles for ICTs to support aging in place .
The second type of evaluation takes the technology out of the lab and places it in the real world. These in situ studies allow researchers to investigate how people use a system in their homes over time. At the point of an in situ study, the design should be refined enough that all major usability issues have been resolved. In addition to measuring the overall impact of the ICT as an intervention, key findings in these real-world deployments revolve around frequency of use, context of use, technology appropriation, privacy concerns, and user perceptions of the technology - all over time. The focus is on why a particular technology is or is not used by users so that successes can be built upon and imitated in the future and failures can be avoided. The remainder of this review will focus on such in situ studies for ICTs to support aging in place, with an emphasis on methods that help explain why some technologies are successfully adopted by older adults while others are not.
Many designers of ICTs to support aging in place utilize a number of social science methods to help answer their research questions. There is growing consensus that no single evaluation method is adequate and that several different methods are needed to triangulate on a particular question, with qualitative methods being particularly suited to answering the questions of how and why . Common qualitative methods include interviews, focus groups, contextual observations, daily journals, and technology probes. In this section, we introduce each method and discuss some of the benefits and challenges of using them to evaluate aging-in-place technologies. We then briefly discuss the common types of analysis that people employ to extract meaning from their qualitative data and the typical outcomes of such analyses.
‘Interviews' are one of the most common methods employed to evaluate aging-in-place technologies . Interviews allow the researcher to engage in a conversation with individual users, eliciting their subjective opinion about a technology, its usefulness, and barriers to use. Since technologies may be deployed for several weeks, and interviews typically occur once a week or less, interviews are often enhanced by using technology logs to prompt the participant to remember specific events.
‘Focus groups' are another common method for eliciting user's opinions and experiences. Focus groups allow multiple users to have a conversation, allowing the group to build upon individual insights to reach consensus or identify differences of opinion. Focus groups, however, can be a challenge to run effectively since some participants often naturally dominate a conversation, while others prefer to remain unnoticed or are too timid to contradict a group . In addition, focus groups may not be appropriate if the technology records sensitive information that participants may feel uncomfortable discussing in front of others. Furthermore, many researchers have found that older adults rarely see themselves as ‘old' or in need of assistive technologies , even when they have had clear instances in the past where such technologies would have been helpful, such as experiencing a fall when nobody was around to help them. Older adults are even less likely to admit to needing such technologies in front of their peers, making focus groups a challenge for eliciting honest evaluations from participants with respect to their own needs.
‘Contextual observations'  is a method in which the researcher observes a participant in situ, during their normal daily activities. Contextual observations are particularly well suited for evaluating aging-in-place technologies, as they allow a researcher to collect rich contextual information that can lead to insights about technology use that might otherwise be hidden.
Since interviews are often subject to errors in participant recall, a ‘daily diary' can help participants record their experiences close to the time when they occur. The participant burden of such a journal may be prohibitively high, but if used successfully it can provide a wealth of data . Diaries are best used by giving participants specific questions to answer, so that they know exactly what the researcher wants them to comment on. It is best to encourage the participants to fill out the diary at a particular time of day in a particular place, so they can make it a routine that they do not forget.
Finally, ‘cultural probes'  can be used to collect other types of information beyond participants answering questions. For example, cameras, hand-drawn floor plans, photo albums, and other artifacts have been used by older adults to record details of daily routines and how they spend their time in their home to give rich insights into the domestic spaces of older adults .
All of these methods produce a wealth of qualitative data, in the form of researcher notes, transcripts from interviews/focus groups, participant journals, photographs, and more. While it is beyond the scope of this review to provide details of how the analysis might occur (the authors refer the interested reader to any number of textbooks on qualitative data analysis), in addition to typical social science approaches such as bottom-up coding that allow themes to emerge from the data or top-down coding to identify existing constructs in a particular data set, technology also enables other types of analysis such as semiautomated analysis of large amounts of textual data  and concept maps . Specific examples of evaluations of ICTs for aging in place that provide examples of these types of analysis are introduced later in ‘Example Evaluations'.
Finally, researchers investigating aging-in-place technologies have produced a variety of outcomes using their qualitative data. Mynatt and Rogers  presented design considerations to assist designers when coming up with new technologies for this population in the areas of remote monitoring and memory aids, such as including reciprocity in designs so that the older adults can see into the lives of their family members and not just be a passive object of monitoring. Shankar et al.  introduced a framework of how older adults conceptualize privacy in the context of aging in place, a framework that is substantively different than existing frameworks in other domains. Since privacy is an oft-cited barrier to technology adoption, this framework should assist future researchers to focus on the privacy concerns that truly matter to older adults. Bardram et al.  explored the nature of work transfer from physician to patient when homecare technology is used, and presented design implications for how to support such ‘collaborative' care where the patient takes on more responsibility. As more healthcare is offloaded to older adults and their informal caregivers, understanding how to design technologies to support this shared work is of importance to ICT design.
While qualitative analyses are particularly relevant to explore an undertheorized area  - in essence to understand why a particular phenomenon occurs - they have limitations in size and scope of their findings and are subject to researcher bias. Qualitative evaluation is time-consuming, necessitating a small sample size and limited generalizability. And while quantitative methods are often utilized in conjunction with the qualitative ones in initial trials, quantitative methods are also particularly suited to expanding these smaller cases studies to larger populations. In the next section, we will see how quantitative methods can assist both the smaller qualitative evaluation as well as be used to validate the qualitative findings on a larger scale.
Surveys and user studies can be used to capture quantitative data on human perceptions, feelings, intentions, and actual use of technology. These methods typically involve the construction of questionnaires with rating scales to produce quantitative data. In this section, we present a brief overview of some quantitative approaches related to ICTs for aging in place.
One rich source of quantitative data when studying ICTs for aging in place is the technology itself, through the recording of detailed logs. While traditional controlled trials can only evaluate outcomes based on whether the participant was part of the control or intervention group, technology logs enable a deeper analysis, further splitting the intervention group based on how much the participants actually make use of the technology. The logs can be summarized into different usage patterns (e.g. low, medium, or high use), and then analyzed with other measures, such as health outcomes and caregiver burden, providing evidence if the amount of use contributes to specific desired outcomes .
Since some technologies do not require explicit interaction (e.g. an ambient display that shows older adults the level of activity in their child's household ), technologies can be enhanced to detect probable use. For example, there exist eye-tracking technologies that can record the location of gaze as well as the number of times someone looks at a display, and proximity sensors can detect if a person is near a particular display and thus likely to have looked at it.
Finally, not only do the technology logs allow researchers to measure the dosage of the technology intervention, but the logs can also provide insights into which parts of the system older adults use (or not). Since many ICTs for aging in place are multifaceted, this allows the researchers to draw more precise conclusions about how useful and usable different components of the system are, without having to have many different intervention groups who receive different combinations of the system components.
While technology logs provide objective quantitative data, a technology-enhanced source of subjective measure is the ‘experience sampling method' , also known as ‘ecological momentary assessment', which collects user responses throughout the day as they experience the phenomena of interest. In the case of aging in place, participants can be prompted to fill out questions immediately after they interact with the system, or even when they are in the proximity of a particular part of the system, such as a display. Since participants are interrupted from their normal routine in order to answer questions, it is important to keep the number of questions short and allow for easy answering by providing a few options for the participant to quickly select. Thus, researchers must have a firm grasp of the range of answers expected (probably through earlier qualitative work), and experience sampling method data tends to be quantitative in nature. By collecting participant feedback at the moment of interaction, researchers further reduce recall bias that still exists when using a daily journal.
Another popular quantitative measure for evaluating the impact of ICTs for aging in place builds upon existing technology acceptance models first developed in the business arena. Technology acceptance models are used to predict a subject's likelihood of using a particular ICT over time based on several measures. The most popular models such as the Technology Acceptance Model (TAM), the Theory of Planned Behavior (TPB), and the Unified Theory of Acceptance and Use of Technology (UTAUT)  consider ‘intention' as the main driving factor for technology acceptance or rejection. Intentions are normally triggered by some motivational factors which influence human behavior; they are indicators of how hard people are willing to try and how much of an effort they are planning to exert in order to perform the behavior .
The most common constructs in these technology acceptance models are shown in table 1. TAM was adapted to try to take into account the specific issues of older adults, resulting in the Senior Technology Acceptance Model (STAM) . In STAM, the main ‘perceived ease of use' and ‘perceived usefulness' constructs are refined, tailored to the needs and limitations of older adults, and differentiate acceptance from adoption. Unfortunately, initial testing of STAM found that it did not adequately capture mobile phone adoption by the elderly in the South African context. Further work is needed to be able to easily use technology acceptance models with this population.
In summary, there are many quantitative sources of data when studying technology use of older adults in their homes. Some methods collect subjective data from the user (e.g. experience sampling method and standardized questionnaires such as TAM), while others collect objective data (e.g. technology logs). When possible, user studies include control groups in order to compare the intervention outcomes to a group that does not receive the intervention. However, because of the challenges in deploying technologies in people's homes, evaluations often consist of a small number of users and may not have control groups, relying more heavily on the rich, qualitative data to better understand when, how and why older adults use ICTs in their home.
In this section, we provide a brief overview of how two projects used multiple methods to answer questions about older adults' use of ICTs in their homes. We selected these projects because they represent different emphases and foci, allowing the reader to see how such an evaluation might occur in the context of different health systems (USA vs. Europe) on technologies designed for different stakeholders (older adults vs. informal caregivers) for different outcomes (a privacy framework vs. understanding how a technical support system is integrated into a larger, complex system). There are numerous other projects which use a mixed-methods approach in the literature.
The ETHOS (Ethical Technologies in the Homes of Seniors) project focused on ethical issues that may impact older adults' usage of aging-in-place technologies, such as perceptions of privacy or fears that technology will replace human contact. This 5-year project started with focus groups and interviews in a living lab , then deployed a suite of technologies in the homes of 6 older adult participants and their informal caregivers . Participants were split into three groups: a ‘technology-only' group which received the suite of technology with no ability to control the data flows, a ‘control panel' group which in addition to the suite of technology received a control panel which allowed the older adults to control who saw different data , and a ‘mobile phone' group, which did not receive any home technologies but did receive mobile phones and acted as a control group.
The technology trial lasted for 6 weeks and included before and after questionnaires to measure factors such as health status, intimacy of social networks, and perceptions of technology. Participants answered brief, daily phone questionnaires in order to collect information to follow up on during more in-depth, in-person weekly interviews. In addition, detailed technology logs were collected. Data from the logs were used to focus in on points of interest during the interviews (such as highlighting when an older adult turned off a particular technology) as well as during poststudy analysis to measure the amount and type of interaction with the technology over time.
The mixed-methods ETHOS trial resulted in a new privacy framework tailored for older adults that helps explain why an older adult would accept monitoring technologies and how they are used over time. Specifically, the framework includes such factors as the sensitivity of the activity being monitored and the nature of the relationship with the recipient of the data . For example, a parent might be fine sharing intimate data about their bathroom habits with their daughter, but not their son. The trial also found that while older adults rarely exerted control over their data flows (e.g. turned off sharing the data), they found comfort in their ability to control the data and used the ‘data transparency' feature of the control panel frequently to see what their loved one could see about them . In addition, they had findings on how the technology impacted the relationship of the older adult and their informal caregiver, including quantitative data on the amount of communication and qualitative data on the quality of communication . Finally, through the in-depth interviews, the trial uncovered unexpected uses of the technology, such as using one technology to determine when the other person was probably available to talk on the phone or using another technology to try to ‘trick' their loved one in a playful way .
ACTION  (Assisting Carers Using Telematics Interventions to Meet Older People's Needs) stemmed from an EU-funded project (1997-2000) to help empower older people with chronic conditions (Alzheimer, paralysis, etc.) and their family caregivers. While ETHOS focused on the experiences of older adults, ACTION has an emphasis on supporting family caregivers, a critical stakeholder in aging-in-place technologies. ACTION is a self- and family-care support service which promotes aging in place as older people with chronic illnesses and their family caregivers are able to access relevant and accessible information, education, and support when needed from the comfort of their own home.
To capture and evaluate the views of all key stakeholders, they utilized a multimethod evaluation that included usability questionnaires, log diaries, field notes, interviews, and focus groups with multiple stakeholders. The user study lasted 3-4 months and consisted of 249 older adults, 767 family members, and 786 formal care providers. The interview and focus group data were analyzed for content using grounded theory to identify key themes , and the usability questionnaire was used to provide quantitative data about the users' experiences with the system.
Both the qualitative and quantitative evidence revealed that after using the ACTION ICT system in their own homes for a minimum period of 3 months, family caregivers felt more competent and secure in their caregiving role: they had gained more control over their individual caring situation and had increased their self confidence in their ability to care. Families liked the ACTION ICT education program because it provided them the necessary information on how to care for older adults with diverse chronic conditions. Additionally, families responded well to ACTION's ability to connect them with other families in similar situations, as well as with professional caregivers and health practitioners during an emergency. This focus on educational, social, and emergency support went beyond a single-technology intervention towards a holistic system to meet the broader needs of the family caregivers. In particular, the rich findings showed that the success of aging-in-place technologies is not simply about installing and learning to use technology, but rather about changing the way in which older people live and their family members provide care and seek support.
While the ETHOS and ACTION projects focused on different research questions, they both used a variety of methods to explore underlying questions of when, how, and why older adults and their families would, or would not, use aging-in-place technologies. Without this multimethod approach, their conclusions would have been more limited.
This review provides an introduction to the complexities of evaluating ICTs for aging in place. In addition to the normal measures for health outcomes, there is a need to be able to measure the users' experiences with technology: when, how and why will older adults and their caregivers use technology? Such questions require a multimodal approach to evaluation to gain deeper insights into how we may design technologies that are acceptable to people. A variety of standard social science methods can be employed (e.g. interviews), as well as techniques that have gained traction in recent years and are made possible by the existence of technology (e.g. technology logs to understand usage and cultural probes to understand daily routines). While such novel methods are gaining traction, there is still work to be done to formalize and standardize these techniques to ensure that they can be adequately compared across research projects.