Despite evidence that psychotherapy has a positive impact on psychological disorders, 30% of patients fail to respond during clinical trials, and as many as 65% of patients in routine care leave treatment without a measured benefit. In addition, therapists appear to overestimate positive outcomes in their patients relative to measured outcomes and are particularly poor at identifying patients at risk for a negative outcome. These problems suggest the need for measuring and monitoring patient treatment response over the course of treatment while applying standardized methods of identifying at-risk cases. Computer-assisted methods for measuring, monitoring, identifying potential deteriorators, and providing feedback to clinicians are described along with a model that explains why feedback is likely to be beneficial to patients. The results of 12 clinical trials are summarized and suggest that deterioration rates can be substantially reduced in at-risk cases (from baseline rates of 21% down to 13%) and that recovery rates are substantially increased in this subgroup of cases (from a baseline of 20% up to 35%) when therapists are provided this information. When problem-solving methods are added to feedback, deterioration in at-risk cases is further reduced to 6% while recovery/improvement rates rise to about 50%. It is suggested that the feedback methods become a standard of practice. Such a change in patterns of care can be achieved through minimal modification to routine practice but may require discussions with patients about their clinical progress.

Fava et al. [1] suggest the clinical inadequacy of evidence-based medicine and the need for a conceptual framework based on clinical judgment to overcome the overemphasis on clinical trials and meta-analytic reviews for guiding treatment. They point out that evidence-based medicine can provide general guidelines for the average patient but that providers have to concern themselves with specific patients whose response to treatments is problematic. They suggest the need for clinicians to systematically use clinical judgment that takes into account the individual patient and variables that are not accounted for in evidence-based medicine [2]. They suggest the importance of considering negative treatment response, something that is usually missed in clinical trials and meta-analytic reviews that inform evidence-based medicine.

There is an emerging trend in psychotherapy that focuses on enhancing patient outcomes by formal measurement of treatment response, timely identification of patients at risk of treatment failure, progress feedback to clinicians during ongoing treatment, and helping providers organize their problem-solving/decision-making with such cases [3]. To my knowledge such methods have not been used in psychiatry or related medical practice in the context of medication management. Here I present the rationale and necessary components essential for reducing treatment failure beyond applying evidence-based treatments in hopes that the adoption of such practices by medical practitioners will enhance patient outcomes.

Hansen et al. [4] examined a representative sample of randomized clinical trial outcomes based on 89 treatment comparisons (mostly cognitive-behavioral therapy) and reported an average of 57-67% recovered or improved after receiving an average of 12-13 sessions of treatment. These outcomes were contrasted with those found in over 6,000 patients who participated in routine care that lasted an average of 4 sessions with patients ranging from those treated in community mental health centers to those being seen in employee assistance programs. Rates of improvement/recovery averaged 35%, and deterioration varied from a low of 3.2% to a high of 14%, with an average rate of 8%. These findings document the surprisingly low treatment dosage in routine care in the USA as well as the large degree to which patients are leaving treatments without a measured benefit. Even when an evidence-based treatment is offered to carefully screened individuals who have the same disorder and see therapists who have been carefully selected, monitored, and supervised, 30-50% of patients fail to respond to treatments that last considerably longer.

The situation for child and adolescent outcome in routine care is even more sobering than with adult populations. The small body of outcome studies in community-based usual care settings has yielded an overall mean effect size near zero [5], yet millions of youth are served each year in these systems of care. In a comparison of children being treated in community mental health (n = 936) or through managed care (n = 3,075), estimates of deterioration were 24 and 14%, respectively [6]. Furthermore, increased attention to deterioration in treatment may be warranted given the high rates of treatment dropout observed in clinical practice. It is estimated that 40-60% of children and adolescents discontinue treatment prematurely [7, 8]; many of these dropouts are likely due to a perceived lack of benefit from treatment. With regard to measuring treatment response in child and adolescent psychotherapy, Kazdin [9] noted that “such information would be enormously helpful if used to monitor and evaluate treatment in clinical practice” (p. 555). The preceding studies were aimed at examining the effects of psychotherapy and did not separately estimate the effects of psychoactive medications in routine care although as many as 90% of patients were taking psychoactive medications as well as receiving psychotherapy.

Unfortunately, the clinicians' view of their own patients' outcome is much more positive than the measured outcome using self-report scales. In their survey of clinicians, Walfish et al. [10] suggest that they estimate about 85% of their patients improve or recover, an estimate that far exceeds estimates based on measured outcomes in clinical trials and routine care. The discrepancy between clinician estimates of success and measured success suggests the need for formally measuring and monitoring treatment response. In addition, clinicians have the common impression that they are unusually successful with their own patients relative to other therapists, with 90% of therapists rating themselves in the upper quartile of all therapists and none seeing themselves as below average (50th percentile) in providing effective services. This overly positive self-assessment bias is not unique to clinical practitioners but occurs across professions, trade, and craft groups, again suggesting the need for formal appraisals based on patient self-report. The problem of judging negative treatment response appears to be especially problematic.

A serious problem in routine practice and clinical trials is the inability of clinicians to identify patients during the course of treatment who ultimately deteriorate and to consider worsening during treatment as a warning sign of ultimate deterioration and treatment failure. Hannan et al. [11], for example, found that even when therapists were provided with the base rate of deterioration in the clinic where they worked (8%), and were asked to rate each patient that they saw at the end of each session (with regard to the likelihood of eventual treatment failure and if the patient was worse off at the current session in relation to their intake level of functioning), they rated only 3 of 550 patients as likely to have a negative outcome and seriously underestimated the number of patients who were worse off at the current session in relation to their intake level of functioning. In fact, at the end of therapy 40 clients had deteriorated, and therapists identified only a single case during ongoing treatment. In this study statistical algorithms were able to correctly identify 36 of the 40 (85%) deteriorated cases before they left treatment.

Using retrospective methodology, Hatfield et al. [12] reviewed case notes of patients who had deteriorated during treatment and found infrequent mention of worsening (20-30% of patients who actually deteriorated) even when the degree of worsening during the course of treatment was dramatic. Such results are not surprising, given psychotherapist optimism about their ability to help, the complexity of persons, and a treatment context that calls for considerable commitment and determination on the part of the therapist, who actually has very little control over the patient's life circumstances and personal characteristics. The patient's measured response to treatment is, especially in the case of a worsening state, a likely place where outside feedback based on sophisticated statistical algorithms might have the greatest chance of impact. Helping the clinician become aware of negative change and discussing such progress in the therapeutic encounter are much more likely when formal feedback is provided to therapists. Such feedback helps the patient communicate and helps the clinician to become aware of the possible need to adjust treatment, alter, or address problematic aspects of the treatment as appropriate (e.g., problems in the therapeutic relationship or in the implementation of the goals of the treatment).

Conclusion

Clinicians appear to be overly optimistic about patient benefits compared to measured outcome. Failure to recognize that a patient is not responding to treatment is a serious problem in routine care and one that appears to be made worse by the clinicians' confidence in their clinical judgment and unique healing gifts.

A partial solution to the failure of clinicians to recognize the signs of treatment failure is to use mental health “lab test” data for alerting clinicians to potential treatment failure. In order to improve outcomes of patients who are responding poorly to treatment, such patients must be identified before termination and, ideally, as early as possible in the course of treatment. It proves possible to create statistically generated expected recovery curves for differing levels of pretreatment distress and use this as a basis for identifying patients who are not making expected treatment gains and are at risk of having a poor outcome. Finch et al. [13] developed and tested the accuracy of predicting deterioration in adults. These and related signal alarm systems have been evaluated in a number of empirical investigations [14, 15, 16, 17]. This research shows that 85-100% of patients who deteriorate in treatment can be accurately identified prior to departing from treatment and often after the first few sessions of treatment. The extent to which these predictive equations apply to medication treatments is unknown but the general predictive principles probably apply: end of treatment deterioration is a function of how disturbed a patient is at intake and how much they deviate from an expected treatment response once they begin treatment.

Clinicians could benefit from employing formal mental health vital sign tracking systems because of their proven accuracy in identifying treatment failure and thereby overcoming a clinician's overly optimistic estimates of their patients' treatment responses and because of their inability to predict treatment failure, specifically negative change.

Sapyta et al. [18] reviewed the effects of feedback on human performance generally and in psychotherapy, indicating that the effectiveness of feedback is likely to vary as a function of the degree of discrepancy between clinicians' views of progress and measured progress, and that the greater the discrepancy the more likely feedback will be helpful. A key element of effective feedback is bringing into the recipient's (clinician's) awareness the discrepancy between what he or she thinks and what is “reality,” thereby prompting corrective action. Riemer et al. [19] have developed a contextual feedback intervention theory to explain how feedback is interpreted and made useful by recipients. Basic tenets of this theory are that clinicians (and professionals, generally) will benefit from feedback if they are committed to the goalof improving their performance, aware of a discrepancy between the goal and reality (particularly if the goal is attractive and the clinician believes it can be accomplished), the feedback source is credible, feedback is immediate, frequent, systematic, cognitively simple (such as graphic in nature) and unambiguous, and provides clinicians with concrete suggestions of how to improve.

As feedback research suggests, the value of monitoring and systematic feedback through psychological assessments hinges on the degree to which the information provided goes beyond what a clinician can observe and understand about patient progress without such information. It is important for the information to add something to the clinician's view of patient well-being and the need for future actions.

At relatively little cost therapists can formally measure, monitor and track patient well-being and change as well as predict end of treatment response, especially with patients who worsen in the course of treatment. This means that even if evidence-based medicine were being practiced (based on clinical trials and meta-analytic reviews) we would need to identify patients who are failing to respond to this treatment, feeding this information back to clinicians who can then take appropriate actions before these patients depart from their care. The major assumptions of formally tracking patient treatment response is that we can improve upon clinical intuition to maximize outcomes by identifying poorly responding patients in a timely fashion; clinicians could then take timely actions that will benefit patients.

The OQ-Analyst is a computer-based feedback and progress tracking system designed to help increase psychotherapy treatment effectiveness. By assessing the attainment of expected progress during therapy, the tracking system provides feedback to therapists on whether patients are staying on track toward positive treatment outcomes. This information may be shared with the patient at the therapist's discretion. In addition, the OQ-Analyst can provide decision support to the therapist to maximize the likelihood of a positive outcome for the patient. Prior to therapy sessions, patients complete the Outcome Questionnaire-45 (OQ-45), a 45-item self-report, or the Youth Outcome Questionnaire-30 (Y-OQ-30), a parent or youth self-report instrument. The OQ-45 and Y-OQ-30 were created specifically to measure and assess real-time change in adult and youth psychotherapy which can then be used by therapists to identify deteriorating cases and ultimately improve patient outcome prior to treatment termination [20].

The OQ-45 is a 45-item self-report measure designed for repeated administration throughout the course of treatment and at termination. In accordance with several reviews of the literature, the OQ-45 was conceptualized and designed to assess 3 domains of patient functioning: symptoms of psychological disturbance (particularly anxiety and depression), interpersonal problems, and social role functioning. Consistent with this conceptualization of outcome, the OQ-45 provides a total score, based on all 45 items, as well as symptom distress, interpersonal relations, and social role subscale scores. Each of these subscales also contains some items related to the positive quality of life of the individual. Research has indicated that the OQ-45 is a psychometrically sound instrument, with strong internal consistency, adequate test-retest reliability and strong concurrent validity [20].

Furthermore, the items that make up the OQ-45 have been shown to be sensitive to changes in multiple patient populations over short periods of time while remaining relatively stable in untreated individuals [21]. Evidence from factor analytic studies suggests it measures an overall psychological distress factor as well as factors consistent with the 3 subscales [22]. It provides clinicians with a mental health vital sign.

The Y-OQ is a 64-item parent/guardian report measure of treatment progress for children and adolescents (ages 4-17 years) receiving mental health intervention [23]. Similar in its intent to the OQ-45, the Y-OQ is meant to track actual change in functioning as opposed to treatment planning and diagnosis. The Y-OQ is composed of 64 items that comprise 6 separate subscales designed to tap diverse elements of healthy and unhealthy youth functioning. The subscales include interpersonal distress, somatic complaints, intrapersonal relations, critical items, social problems, and behavioral dysfunction. The Y-OQ Self-Report is the equivalent self-report measure completed by youth aged 12-17. In addition, a shorter form, the Y-OQ-30, is available for parents, clinicians, and youth.

In short, the OQ and Y-OQ are brief measures of psychological disturbance that are reliable, valid, and sensitive to changes patients make during treatment. They are well suited for tracking patient status during and following treatment and provide clinicians with a mental health vital sign. The measures have been extensively reviewed elsewhere [[20], [23], www.oqmeasures.com].

Responses to the questions are entered into a computer with the OQ-Analyst via handheld devices, computer kiosk, online, or hardcopy, and a report is generated for use by the therapist. It takes about 5 min of client time to complete the OQ-45, about half a second for the software to score the measure and predict end of treatment functioning, and about 18 s for the clinician to open up the clinician report on their personal computer. The OQ- Analyst then suggests possible courses of action if the patient is identified as at risk for treatment failure. The OQ-Analyst is designed to detect treatment effectiveness regardless of treatment modality, diagnosis, or therapy type. Questions are answered in relation to the last week of the patient's life.

Conclusion

Patients can complete a brief measure of their psychological function by using standardized rating scales, and then this information can be delivered to clinicians in real time. In addition to alerting therapists to deviations from the expected treatment response, the information gathered from patients provides novel information to therapists. Collecting this information from the patient on a session-by-session basis provides the clinician with a systematic way of monitoring life functioning from the patient's point of view. A brief formal assessment can provide a summary of life functioning that is not otherwise available to the therapist, unless the therapist spends time within the treatment hour to systematically inquire about all the areas of functioning covered by the self-report scale, an activity that detracts from service delivery. Mental health self-report data cannot capture the full range of psychological functioning any more than a thermometer can detect cancer, diabetes, or heart disease. Furthermore, collecting such data cannot cure mental illness any more than sticking a thermometer in a patient's mouth can cure the flu. Feedback data themselves are not helpful unless clinicians know how to use the data to improve treatment.

In conjunction with identifying alarm status, an instrument assessment for signal cases was developed to assist clinicians to problem-solve with the patients who backslide during treatment (i.e., when a therapist receives a warning message predicting deterioration) [24]. This 40-item measure does not produce a total score, but rather provides subscale score feedback and item feedback for therapists to consider in problem-solving. For example, the first 11 items of the assessment for signal cases require the patient to reflect on the therapeutic relationship and report his or her perceptions. The assessment for signal cases is central to the clinical support tool which is composed of a problem-solving decision tree designed to systematically direct therapists' attention to subscales and items: the therapeutic alliance, social support, readiness to change, diagnostic formulation, life events, and need for medication referral.

Methods incorporated in the OQ-Analyst are clearly helpful to patients based on a meta-analysis by Shimokawa et al. [25]. Their meta/mega-analysis sums the effects of monitoring patients on a weekly basis by looking across 6 clinical trials. In these published studies each therapist was asked to practice as they routinely do with half their case load while using the OQ-Analyst information with the other half of their case load. In this meta-analytic review of the OQ system data from 6 studies were combined (n = 6,151) and re-analyzed [26, 27, 28, 29, 30, 31].

Each of the studies evaluated the effects of providing feedback about each patient's improvement through the use of progress graphs and warnings about patients who were not demonstrating expected treatment responses (signal alarm cases). The 6 studies shared many design and methodological features: (a) consecutive cases seen in routine care regardless of patient diagnosis or comorbid conditions (rather than being disorder specific); (b) random assignment of patients to experimental conditions (various feedback interventions) and treatment-as-usual conditions (no feedback) was made in 4 of the 6 studies, while reasonable measures were taken in 2 studies to ensure equivalence in experimental and control conditions before treatment; (c) psychotherapists provided a variety of theoretically guided treatments, with most adhering to cognitive-behavioral and eclectic orientations and fewer representing psychodynamic and experiential orientations; (d) a variety of therapist experience - postgraduate therapists and graduate students each accounted for about 50% of patients seen; (e) therapists saw both experimental (feedback) and no-feedback cases, thus limiting the likelihood that outcome differences between conditions could be due to therapist effects; (f) the outcome measure as well as the methodology rules/standards for identifying signal alarm patients (failing cases) remained constant; (g) the length of therapy (dosage) was determined by patient and therapist rather than by research design or arbitrary insurance limits.

A key element in psychotherapy research is operationalizing the concepts of positive and negative outcome for the individual patient. Jacobson and Truax [32] offered a methodology by which patient changes on an outcome measure can be classified in the following categories: recovered, reliably improved, no change, deteriorated. There are 2 necessary pieces of information to make these patient outcome classifications: a Reliable Change Index and a normal functioning cutoff score. Clinical and normative data were analyzed to establish a Reliable Change Index and a cutoff score for normal functioning [20]. These 2 cutoffs when combined denote clinically significant change (recovery). The meta-analysis involved both intent-to-treat and efficacy analyses on the effects of various feedback interventions relative to treatment as usual. Here only those patients who received and completed the treatments were compared to treatment as usual. Thus, the research question was: when therapists get progress feedback with alarm signals, do patients have better outcomes (relative to the outcomes they achieve without such feedback), and if so, how much better?

The first discovery suggested by the findings of our studies was that feedback was not especially helpful for all patients. Patients who make relatively steady progress continue to make steady progress even when their therapists are notified that the progress from week to week is typical and nonproblematic. This is the majority of patients in a typical clinician's case load and reflects the fact that therapy as it is routinely practiced is helping many individuals. The feedback did make a marked difference for patients who went significantly off-track (about 20-40% of treated individuals, depending on the study sample). So an important finding from our studies is that a clinician cannot expect every patient to benefit from feedback. We believe that, consistent with contextualized feedback theory, this is because the novel information that is needed by clinicians is the fact that the patient is not responding as expected (in contrast with therapist beliefs).

Effects of OQ Progress Feedback on At-Risk Patients

Results suggest that the average patient whose therapist received feedback was better off than approximately 70% of patients in the no-feedback condition (routine care). In terms of the clinical significance at termination, 9% of patients in the feedback condition deteriorated, and 38% achieved clinically significant improvement. In contrast, among patients whose therapists did not receive feedback, 20% deteriorated while 22% achieved clinically significant improvement. When the odds of deterioration and clinically significant improvement were compared, results indicated those in the feedback group had less than half the odds of experiencing deterioration while having approximately 2.6 times higher odds of experiencing at least reliable improvement.

Effects of Patient/Therapist Feedback on At-Risk Patients

The effect size of posttreatment OQ score averaged g = 0.55, r = 0.25, p < 0.001 - effects very similar to those of the therapist-only feedback group. The rates of deterioration and clinically significant improvement when both participants received written feedback were 15 and 45%, respectively. The results suggest that patients who received feedback along with their therapist had approximately 0.7 times the odds of deterioration, while having approximately 3 times higher odds of achieving clinically significant improvement These results suggest that, although the average patient who received written feedback along with his/her therapist was better off than 71% of patients in treatment as usual, there may have been moderators that facilitated outcome enhancement in some patients while failing to prevent, or possibly contributing to worsening.

Effects of Clinical Support Tools Feedback on At-Risk Patients

When the outcomes of patients whose therapist received the clinical support tool feedback were compared to the treatment-as-usual patients, the effect size for the difference in mean posttreatment OQ scores was g = 0.70, r = 0.33, p < 0.001. These results indicate that the average patients in the clinical support tool feedback group, who stayed in treatment to experience the benefit of this intervention, are better off than 76% of patients in treatment as usual. The rates of deterioration and clinically significant improvement among those receiving clinical support tools were 6 and 53%, respectively. The results suggest that patients whose therapists used clinical support tools have less than a fourth the odds of deterioration, while having approximately 3.9 times higher odds of achieving clinically significant improvement compared to treatment as usual provided by the same therapists. In the case of the off-track patients, notifying therapists that the patient was in trouble allowed them to change the future course of therapy progress and outcome - therapists and patients found a way to turn the treatment course around. A summary of clinically significant changes is presented in Table 1.

Table 1

Clinical significance classification of not-on-track patients by treatment conditions

Clinical significance classification of not-on-track patients by treatment conditions
Clinical significance classification of not-on-track patients by treatment conditions

Unfortunately, most formal monitoring of patient mental health is being imposed on clinicians by systems of care that have come to realize that such practices can enhance patient outcomes or because they believe that the data can be used to increase the quality of care offered to patients through accountability measures. This development provides clinicians with further reasons to resist monitoring - we do not like external control and management. Nevertheless, it appears that this research-based innovation (formal monitoring and problem-solving) has little downside for clinicians (it is cheap and effective) and large upsides for patients.

There is no doubt that all of the deterioration that occurs during the time a patient is in treatment cannot be causally linked to therapist activities. Certainly, a portion of patients are on a negative trajectory at the time they enter treatment, and the deteriorating course cannot be stopped. Possibly, a portion of patients are prevented from taking their own lives as a result of effective practices, even if they do not show overall progress. Just as positive psychotherapy outcomes depend largely on patient characteristics, so do the negative changes that occur in patients who are undergoing psychological treatments. At this point in time we do not know whether the same degree of benefit can be expected in medication management, so the field awaits clinical trials on this topic.

Since the meta/mega-analysis of feedback studies was published, additional studies have been completed. These studies have demonstrated that feedback effects have a broad scope that reaches across clinical samples, treatment settings, and different nations. Table 2 provides the original studies as well as newer studies.

Table 2

Published clinical trials examining the effects of progress monitoring with alarm signals and clinical support tool feedback using the OQ-45

Published clinical trials examining the effects of progress monitoring with alarm signals and clinical support tool feedback using the OQ-45
Published clinical trials examining the effects of progress monitoring with alarm signals and clinical support tool feedback using the OQ-45

Newer published studies include one by Crits-Christoph et al. [33], who studied the effects of progress feedback and clinical support tools on substance-abusing individuals in Philadelphia, New York, and Utah. In this study the patients were, for the most part, from urban clinics and had chronic problems with high levels of comorbidity. Their treatment was subsidized by government agencies. Results appeared promising for both drug and alcohol abuse.

Simon et al. [34] studied these methods with female eating disorder patients treated in an inpatient setting. These individuals had chronic courses and high levels of comorbidity. Of considerable interest for the effects of feedback was the high level of intensive treatment that all patients received over an approximately 30-day stay in the hospital. The treatment included individual therapy twice weekly, group psychotherapy twice weekly, psychoeducation, numerous and varied recreational activities, and nutrition counseling, with 90% of patients taking psychoactive medication. With such comprehensive and intense treatment, we had our doubts about the likelihood that the effects of the feedback interventions could be detected. It was also true that the clinical staff (as is typical in our research studies) saw no need for formal tracking and feedback and so were not eager adaptors of the methodology. Nevertheless, treatment effects were apparent and substantial, much to the surprise of all.

The final 3 studies in Table 2 were conducted outside the USA in multiple clinics within the Netherlands and Norway and within an inpatient psychosomatic hospital in Germany [35, 36, 37]. Data from these studies suggest the effects of progress feedback are robust enough to translate cross-culturally in both in- and outpatient settings. Most relevant for the current review were the strong effects in the German inpatient study [38] in which feedback effects could be found in patients with unexplained physical impairments. In the Dutch study [35] and that of Simon et al. [36], we came across our first evidence that feedback effects are absent if clinicians do not examine the feedback reports. Thus, there was an individual clinician effect on patient outcomes likely due to failure to use feedback that was provided in the experimental condition.

Limitations

The major limitations of feedback research are the moderate number of studies evaluating effectiveness, the limited number of researchers responsible for the findings reviewed here, and the sole reliance on self-report measures as a method of identifying patients at risk for deterioration as well as identifying final treatment status. It is likely that future research will be done across a wider range of treatment settings and patient populations, thus illuminating the limits of these procedures and clarifying the factors that maximize patient gains. The research reviewed here utilized 2 self-report measures and just 2 different methods of providing feedback and therefore a limited view of the impact of therapy and the effects of feedback. Another important limitation is the degree to which there exist unpublished studies that did not find an effect for feedback. It is not known at this time how many such studies exist. This is commonly known as the file drawer problem which reflects the tendency of authors to fail to submit articles with no significant findings and the hesitancy of journals to publish such studies if they are submitted.

It should also be pointed out that alternative feedback systems are emerging. Drapeau [39] has identified 10 such systems and more are sure to come. It cannot be said that each of these rests on a sound (or any) evidence base, thus the emphasis on the OQ-Analyst in this paper. To reiterate a point made earlier in this paper, should one choose an alternative progress monitoring and feedback system the essential components would be the measure is suitable for tracking mental health functioning on a weekly basis, one that includes published evidence of predictive accuracy for identifying likely treatment failures, and one that relies on information technology such as online administration that allows for immediate, timely feedback.

- Use real-time patient feedback to monitor patient response to treatment and satisfaction with the therapy relationship and other hindering forces. Such feedback probably improves treatment outcomes generally and certainly does so for patients at risk for deterioration or dropout.

- Employ real-time patient feedback to compensate for the therapist's limited ability to accurately detect patient worsening in treatment. Despite considerable evidence that clinicians are not alert to treatment failure [11, 12] and strong evidence that clinical judgments are usually inferior to actuarial methods in predictive accuracy, therapists' confidence in their clinical judgment stands as a barrier to implementation of monitoring and feedback systems.

- Beware of those situations in which patients feel it may be in their interest to understate (or overstate) their problems and produce inaccurate ratings on feedback systems. The systems are predicated on accurate self-reporting of levels of disturbance and corresponding changes.

- Supplement progress feedback with clinical support tools. As suggested by the general literature on feedback and the evidence presented here, problem-solving and decision enhancement tools prove helpful to clinicians and, most importantly, patients whose treatment response is in doubt.

- As yet we are uncertain of the necessity of sharing progress feedback directly with patients. The OQ system has examined therapist feedback in the absence of knowledge about how therapists share this information with patients (if at all) and direct patient feedback with standard written feedback separately with inconclusive results about additive effectiveness of formal patient feedback.

- Consider using electronic versions of feedback systems that expedite and ease practical difficulties. Fortunately, the recent software for the OQ can provide instantaneous feedback to clinicians. If the patient takes the OQ immediately prior to the scheduled session, electronic feedback is available to the therapist within seconds and prior to beginning that session.

The author has a potential conflict of interest in that the article makes reference to several measures (Outcome Questionnaire and Assessment for Signal Cases) and a software product, the OQ- Analyst, that is used to measure, monitor, and feedback patient progress information. The measures and software are owned and distributed by OQMeasures (www.oqmeasures.com), a company that the author founded and partially owns.

1.
Fava GA, Guidi J, Rafanelli C, Sonino N: The clinical inadequacy of evidence-based medicine and the need for a conceptual framework based on clinical judgment. Psychother Psychosom 2015;84:1-3.
[PubMed]
2.
Fava GA, Cosci F, Offidani E, Guidi J: Behavioral toxicity revisited: iatrogenic comorbidity in psychiatric evaluation and treatment. J Clin Psychopharmacol 2016;36:550-553.
[PubMed]
3.
Lambert MJ: Prevention of Treatment Failure: The Use of Measuring, Monitoring, and Feedback in Clinical Practice. New York, Wiley & Sons, 2010.
4.
Hansen NB, Lambert MJ, Forman EV: The psychotherapy dose-response effect and its implications for treatment delivery services. Clin Psychol Sci Pract 2002;9:329-343.
5.
Weiss B, Catron T, Harris V, Phung TM: The effectiveness of traditional child psychotherapy. J Consult Clin Psychol 1999;67:82-94.
[PubMed]
6.
Warren JS, Nelson PL, Mondragon SA, Baldwin SA, Burlingame GM: Youth psychotherapy change trajectories and outcome in usual care: community mental health versus managed care. J Consult Clin Psychol 2010;78:144-155.
[PubMed]
7.
Kazdin AE: Dropping out of child therapy: issues for research and implications for practice. Clin Child Psychol Psychiatry 1996;1:133-156.
8.
Wierzbicki M, Pekarik G: A meta-analysis of psychotherapy dropout. Profess Psychol Res Pract 1993;24:190-195.
9.
Kazdin AE: Evidence-based assessment for children and adolescents: issues in measurement development and clinical application. J Clin Child Adolesc Psychol 2005;34:548-558.
[PubMed]
10.
Walfish S, McAlister B, O'Donnell P, Lambert MJ: An investigation of self-assessment in mental-health providers. Psychol Rep 2012;110:639-644.
[PubMed]
11.
Hannan C, Lambert MJ, Harmon C, Nielsen SL, Smart DW, Shimokawa K, Sutton SW: A lab test and algorithms for identifying clients at risk for treatment failure. J Clin Psychol 2005;61:155-63.
[PubMed]
12.
Hatfield D, McCullough L, Plucinski A, Krieger K: Do we know when our clients get worse? An investigation of therapists' ability to detect negative client change. Clin Psychol Psychother 2010;17:25-32.
[PubMed]
13.
Finch AE, Lambert MJ, Schaalje BG: Psychotherapy quality control: the statistical generation of expected recovery curves for integration into an early warning system. Clin Psychol Psychother 2001;8:231-242.
14.
Ellsworth JR, Lambert MJ, Johnson J: A comparison of the Outcome Questionnaire-45 and Outcome Questionnaire-30 in classification and prediction of treatment outcome. Clin Psychol Psychother 2006;13:380-391.
15.
Lambert MJ, Whipple JL, Bishop MJ, Vermeersch DA, Gray GV, Finch AE: Comparison of empirically derived and rationally derived methods for identifying clients at risk for treatment failure. Clin Psychol Psychother 2002;9:149-164.
16.
Lutz W, Lambert MJ, Harmon SC, Stulz N, Tschitsaz A, Schürch E: The probability of treatment success, failure and duration - what can be learned from empirical data to support decision making in clinical practice? Clin Psychol Psychother 2006;13:223-232.
17.
Spielmans GI, Masters KS, Lambert MJ: A comparison of rational versus empirical methods in the prediction of psychotherapy outcome. Clin Psychol Psychother 2006;13:202-214.
18.
Sapyta J, Riemer M, Bickman L: Feedback to clinicians: theory, research, and practice. J Clin Psychol 2005;62:145-153.
[PubMed]
19.
Riemer M, Rosof-Williams J, Bickman L: Theories related to changing clinician practice. Child Adolesc Psychiatr Clin North Am 2005;14:241-254.
[PubMed]
20.
Lambert MJ, Kahler M, Harmon C, Burlingame GM, Shimokawa K, White MM: Administration and Scoring Manual: Outcome Questionnaire OQ®-45.2. Salt Lake City, OQMeasures, 2013.
21.
Vermeersch DA, Whipple JL, Lambert MJ, Hawkins EJ, Burchfield CM, Okiishi JC: Outcome Questionnaire: is it sensitive to changes in counseling center clients? J Couns Psychol 2004;51:38-49.
22.
De Jong K, Nugter MA, Polak MG, Wagenborg JEA, Spinhoven P, Heiser WJ: The Outcome Questionnaire (OQ-45) in a Dutch population: a cross-cultural validation. Clin Psychol Psychother 2007;14:288-301.
23.
Burlingame GM, Wells MG, Lambert MJ, Cox JC: Youth Outcome Questionnaire (Y-OQ); in Maruish ME (ed): The Use of Psychological Testing for Treatment Planning and Outcome Assessment, ed 3. Mahwah, Erlbaum Associates, 2004, vol 2, pp 235-274.
24.
Lambert MJ, Bailey RJ, White M, Tingey M, Stevens E: Clinical Support Tool Manual - Brief Version-40. Salt Lake City, OQMeasures, 2015.
25.
Shimokawa K, Lambert MJ, Smart DW: Enhancing treatment outcome of patients at risk of treatment failure: meta-analytic and mega-analytic review of a psychotherapy quality assurance system. J Consult Clin Psychol 2010;78:298-311.
[PubMed]
26.
Harmon SC, Lambert MJ, Smart DW, Hawkins EJ, Nielsen SL, Slade K, Lutz W: Enhancing outcome for potential treatment failures: therapist/client feedback and clinical support tools. Psychother Res 2007;17:379-392.
27.
Hawkins EJ, Lambert MJ, Vermeersch DA, Slade K, Tuttle K: The effects of providing patient progress information to therapists and patients. Psychother Res 2004;14:308-327.
28.
Lambert MJ, Whipple JL, Smart DW, Vermeersch DA, Nielsen SL: Hawkins EJ: The effects of providing therapists with feedback on client progress during psychotherapy: are outcomes enhanced? Psychother Res 2001;11:49-68.
[PubMed]
29.
Lambert MJ, Whipple JL, Vermeersch DA, Smart DW, Hawkins EJ, Nielsen SL, Goates MK: Enhancing psychotherapy outcomes via providing feedback on client progress: a replication. Clin Psychol Psychother 2002;9:91-103.
30.
Slade K, Lambert MJ, Harmon SC, Smart DW, Bailey R: Improving psychotherapy outcome: the use of immediate electronic feedback and revised clinical support tools. Clin Psychol Psychother 2008;15:287-303.
[PubMed]
31.
Whipple JL, Lambert MJ, Vermeersch DA, Smart DW, Nielsen SL, Hawkins EJ: Improving the effects of psychotherapy: the use of early identification of treatment failure and problem solving strategies in routine practice. J Couns Psychol 2003;58:59-68.
32.
Jacobson NS, Truax P: Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol 1991;59:12-19.
[PubMed]
33.
Crits-Christoph P, Ring-Kurtz S, Hamilton J, Lambert MJ, Gallop R, McClure B, Kulaga A, Rotrosen JA: Preliminary study of the effects of individual patient-level feedback in outpatient substance abuse treatment programs. J Substance Abuse Treat 2012;42:301-309.
[PubMed]
34.
Simon W, Lambert MJ, Busath G, Vazquez A, Berkeljon A, Hyer K, Granley M, Berrett M: Effects of providing patient progress feedback and clinical support tools to psychotherapists in an inpatient eating disorders treatment program: a randomized controlled study. Psychother Res 2013;23:287-300.
[PubMed]
35.
De Jong K, van Sluis P, Nugter MA, Heiser WJ, Spinhoven P: Understanding the differential impact of outcome monitoring: therapist variables that moderate feedback effects in a randomized clinical trial. Psychother Res 2013;22:464-474.
[PubMed]
36.
Simon W, Lambert MJ, Harris MW, Busath G, Vazquez A: Providing patient progress information and clinical support tools to therapists: effects on patients at risk for treatment failure. Psychother Res 2012;22:638-647.
[PubMed]
37.
Amble I, Gude T, Stubdal S, Andersen BJ, Wampold BE: The effect of implementing the Outcome Questionnaire-45.2 feedback system in Norway: a multisite randomized clinical trial in a naturalistic setting. Psychother Res 2015;25:669-677.
[PubMed]
38.
Probst T, Lambert MJ, Loew T, Dahlbender RW, Gollner R, Tritt K: Feedback on patient progress and clinical support tools for therapists: improved outcome for patients at risk of treatment failure in in-patient therapy under the conditions of routine practice. J Psychosom Res 2013;75:255-261.
[PubMed]
39.
Drapeau M: 10 tools for progress monitoring in psychotherapy. Integrating Sci Pract 2012;2:5-43.