06-2313. Changes to the National Registry of Evidence-Based Programs and Practices (NREPP)

AGENCY:

Substance Abuse and Mental Health Services Administration, HHS.

ACTION:

Notice.

SUMMARY:

The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to preventing the onset and reducing the progression of mental illness, substance abuse, and substance-related problems among all individuals, including youth. As part of this effort, SAMHSA has expanded and refined the agency's National Registry of Evidence-based Programs and Practices (NREPP) based on a systematic analysis and consideration of public comments received in response to a previous Federal Register notice (70 FR 50381, Aug. 26, 2005).

This Federal Register notice summarizes SAMHSA's redesign of NREPP as a decision support tool for promoting a greater adoption of evidence-based interventions within typical community-based settings, and provides an opportunity for interested parties to become familiar with the new system.

FOR FURTHER INFORMATION CONTACT:

Kevin D. Hennessy, Ph.D., Science to Service Coordinator/SAMHSA, 1 Choke Cherry Road, Room 8-1017, Rockville, MD 20857, [phone].

Charles G. Curie,

Administrator, SAMHSA.

Advancing Evidence-Based Practice Through Improved Decision Support Tools: Reconceptualizing NREPP

Introduction

The Substance Abuse and Mental Health Services Administration (SAMHSA) strives to provide communities with effective, high-quality, and cost-efficient prevention and treatment services for mental and substance use disorders. To meet this goal, SAMHSA recognizes the needs of a wide range of decisionmakers at the local, state, and national levels to have readily available and timely information about scientifically established interventions to prevent and/or treat these disorders.

SAMHSA, through its Science to Service Initiative, actively seeks to promote Federal collaboration (e.g., with the National Institutes of Health [NIH]) in translating research into practice. The ideal outcome of this Initiative is that individuals at risk for or directly experiencing mental and substance abuse use disorders will be more likely to receive appropriate preventive or treatment services, and that these services will be the most effective and the highest quality that the field has to offer.

This report provides a summary of activities conducted during the past year to critically evaluate SAMHSA's recent activities and future plans for the National Registry of Evidence-based Programs and Practices (NREPP). It outlines the major themes that emerged from a formal public comment process and links this feedback to new review procedures and Web-based decision support tools that will enhance access to evidence-based knowledge for multiple audiences.

The report is presented in four sections:

• Section I briefly states the background of NREPP and SAMHSA's recent request for public comments.
• Section II discusses the analysis of comments that was conducted and presents the key recommendations for NREPP based on this analysis.
• Section III describes the new approach that SAMHSA is advancing for NREPP.
• Section IV presents the specific dimensions of the NREPP system in its new framework as a decision support tool.
• Section V describes future activities at SAMHSA to support NREPP.

I. Background: The National Registry of Evidence-Based Programs and Practices

The National Registry of Evidence-based Programs and Practices was designed to represent a key component of the Science to Service Initiative. It was intended to serve as a voluntary rating and classification system to identify programs and practices with a strong scientific evidence base. An important reason for developing NREPP was to reduce the significant time lag between the generation of scientific knowledge and its application within communities.^[1] Quality treatment and prevention services depend on service providers' ability to access evidence-based scientific knowledge, standardized protocols, practice guidelines, and other practical resources.

The precursor of NREPP, the National Registry of Effective Prevention Programs, was developed by SAMHSA's Center for Substance Abuse Prevention (CSAP) as a way to help professionals in the field become better consumers of substance abuse prevention programs. Through CSAP's Model Program Initiative, over 1,100 programs were reviewed, and more than 150 were designated as Model, Effective, or Promising Programs.

Over the past 2 years, SAMHSA convened a number of scientific panels to explore the expansion of the NREPP review system to include interventions in all domains of mental health and substance abuse prevention and treatment. In addition, SAMHSA committed itself to three guiding principles—transparency, timeliness, and accuracy of information—in the development of an evidence-based registry of programs and practices.

During this process it was determined that, to provide the most transparent and accurate information to the public, evidence should be assessed at the level of outcomes targeted by an intervention, not at the more global level of interventions or programs. Based on this decision, SAMHSA's current NREPP contractor conducted a series of pilot studies to explore the validity and feasibility of applying an outcome-specific, 16-criteria evidence rating system to an expanded array of programs and practices. Through extensive dialogues with the prevention community, SAMHSA also explored ways to provide evidence-based reviews of population- and community-level interventions within NREPP.

In an effort to augment the information gained through these activities, SAMHSA solicited formal public comments through a notice posted in the Federal Register on August 26, 2005. The notice asked for responses to the agency's plans for NREPP, including (1) revisions to the scientific review process and review criteria; (2) the conveying of practical implementation information about NREPP programs and practices to those who might purchase, provide, or receive these interventions; and (3) the types of additional agency activities that may be needed to promote wider adoption of interventions on NREPP, as well as support innovative interventions seeking NREPP status. A brief summary of the public comments and key public recommendations is presented in Section II. The complete analysis of the public responses is included in the Appendix to this report.

II. Public Responses to the Federal Register Notice

Senior staff at SAMHSA engaged in a comprehensive review of comments received in response to the Federal Register notice. Particular attention was directed to comments from prominent state and Federal stakeholders, including providers and policymakers, who stand to be the most affected by whatever system is ultimately implemented. Efforts were taken to balance SAMHSA's responsiveness to public feedback with the need to adhere to rigorous standards of scientific accuracy and to develop a system that will be fair and equitable to multiple stakeholder groups.

Recommendations for NREPP

In the more than 100 comments received as part of the public comment process, a number of recurring themes and recommendations were identified. While all specific and general recommendations for modification of the NREPP review process were carefully considered by SAMHSA, the following are those that were considered most essential to the development of an accurate, efficient, and equitable system that can meet the needs of multiple stakeholders:

• Limit the system to interventions that have demonstrated behavioral change outcomes. it is inherently appealing to the funders, providers, and consumers of prevention and treatment services to know that an intervention has a measurable effect on the actual behavior of participants. As researchers at the University of Washington recommended, “the system should be reserved for policies, programs, and system-level changes that have produced changes in actual drug use or mental health outcomes.”
• Rereview all existing programs. There was near consensus among the respondents to the notice that existing programs with Model, Effective, and Promising designations from the old reviews should be rereviewed under the new system. The Committee for Children pointed out that “a ‘grandfather’ system may give the impression to users, right or wrong, that these interventions aren't as good as those that have undergone the new review process.” One individual suggested that programs and practices needed to be rated “according to a consistent set of criteria” so that “the adoption of an intervention by a provider can be made with confidence.”
• Train and utilize panels of reviewers with specific expertise related to the intervention(s) under review. Respondents to the notice noted that it would be important for the NREPP review process to utilize external reviewers with relevant scientific and practical expertise related to the intervention being assessed. In addition, the pool of available reviewers should broadly include community-level and individual-level prevention as well as treatment perspectives. In order to promote transparency of the review process, the reviewer training protocols should be available for review by the public (e.g., posted on the NREPP Web site).
• Provide more comprehensive and balanced descriptions of evidence-based practices, by emphasizing the important dimension of readiness for dissemination. The American Psychological Association (APA) Committee on Evidence-Based Practice recommended greater emphasis on the utility descriptors (i.e., those items describing materials and resources to support implementation), stating, “these are key outcomes for implementation and they are not adequately addressed in the description of NREPP provided to date. This underscores earlier concerns noted about the transition from efficacy to effectiveness.” The APA committee noted that generalizability of programs listed on NREPP will remain an issue until this “gap between efficacy and effectiveness” is explicitly addressed under a revised review system.
• Avoid limiting flexibility and innovation; implement a system that is fair and inclusive of programs and practices with limited funding, and establish policies that seek to prevent the misuse of information contained on NREPP. The National Association for Children of Alcoholics voiced this concern: “It has been intrinsically unfair that only grants [referring to NIH-funded efforts] have been able to establish ‘evidence’ while many programs appear very effective—often more effective in some circumstances than NREPP approved programs, but have not had the Federal support or other major grant support to evaluate them. The SAMHSA grant programs continue to reinforce the designation of NREPP programs in order to qualify for funding, and the states tend to strengthen this ‘stipulation’ to local programs,who then drop good (non-NREPP) work they have been doing or purchase and manipulate NREPP programs that make the grant possible. This is not always in the best interest of the client population to be served.”
• Recognize multiple “streams of evidence” (e.g., researcher, practitioner, and consumer) and the need to provide information to a variety of stakeholders in a decision support context. A number of comments suggested that NREPP should be more inclusive of the practitioner and consumer perspective on what defines evidence. For example, one commenter noted: “The narrowed interpretation of evidence-based practice by SAMHSA focuses almost solely on the research evidence to the exclusion of clinical expertise and patient values.” Several comments noted that NREPP should be consistent with the Institute of Medicine's definition of evidence-based practice, which reflects multiple “streams of evidence” that include research, clinical, and patient perspectives.
• Provide a summary rating system that reflects the continuous nature of evidence quality. There was substantial disagreement among those responding to the notice concerning whether NREPP should include multiple categories of evidence quality. While a number of individuals and organizations argued for the use of categorical evidence ratings, there were many who suggested that NREPP should provide an average, numeric scale rating on specific evidence dimensions to better reflect the “continuous nature of evidence.” This approach would allow the user of the system to determine what level of evidence strength is required for their particular application of an intervention.
• Recognize the importance of cultural diversity and provide complete descriptive information on the populations for which interventions have been developed and applied. Most comments reflected the knowledge that cultural factors can play an important role in determining the effectiveness of interventions. The Oregon Office of Mental Health and Addiction Services noted, “SAMHSA should focus considerable effort on identifying and listing practices useful and applicable for diverse populations and rural areas. Providers and stakeholders from these groups have repeatedly expressed the concern they will be left behind if no practices have been identified which fit the need of their area. We need to take particular care to ensure that their fear is not realized.”
• In addition to estimating the effect size of intervention outcomes, NREPP should include additional descriptive information about the practical impacts of programs and practices. In general, comments suggested that that effect size should not be used as an exclusionary criterion in NREPP. It was widely noted that effect size estimates for certain types of interventions (e.g., community-level or population-based) will tend to be of smaller magnitude, and that “professionals in the field have not reached consensus on how to use effect size.” Researchers at the University of Washington suggested the inclusion of information about the reach of an intervention, when available, as complementary information to effect sizes. Several comments also suggested that effect size is often confused with the clinical significance of an intervention and its impact on participants.
• Acknowledge the need to develop additional mechanisms of Federal support for technical assistance and the development of a scientific evidence base within local prevention and treatment communities. Nearly one third of the comments directly addressed the need for SAMHSA to identify and/or provide additional technical assistance resources to communities to help them adapt and implement evidence-based practices. The Oregon Office of Mental Health and Addiction Services wrote, “The adoption of new practices by any entity is necessarily a complex and long-term process. Many providers will need technical support if adoption and implementation is to be accomplished effectively. Current resources are not adequate to meet this challenge.”

In order to align NREPP with the important recommendations solicited through the public comment process, SAMHSA also recognized the importance of the following goals:

• Provide a user-friendly, searchable array of descriptive summary information as well as reviewer ratings of evidence quality.
• Provide an efficient and cost-effective system for the assessment and review of prospective programs and practices.

Section III, Streamlined Review Procedures, provides a complete description of the modified and streamlined review process that SAMHSA will adopt in conducting evidence-based evaluations of mental health and substance abuse interventions.

III. Streamlined Review Procedures

The number and range of NREPP reviews are likely to expand significantly under the new review system, requiring that SAMHSA develop an efficient and cost-effective review process. The streamlined review procedures, protocols, and training materials will be made available on the NREPP Web site for access by all interested individuals and organizations.

Reviews of interventions will be facilitated by doctoral-level Review Coordinators employed by the NREPP contractor. Each Review Coordinator will support two external reviewers who will assign numeric, criterion-based ratings on the dimensions of Strength of Evidence and Readiness for Dissemination. Review Coordinators will provide four important support and facilitative functions within the peer review process: (1) They will assess incoming applications for the thoroughness of documentation related to the intervention, including documentation of significant outcomes, and will convey summaries of this information to SAMHSA Center Directors for their use in prioritizing interventions for review; (2) they will serve as the primary liaison with the applicant to expedite the review of interventions; (3) they will collaborate with the NREPP applicant to draft the descriptive dimensions for the intervention summaries; and (4) they will provide summary materials and guidance to external reviewers to facilitate initial review and consensus discussions of intervention ratings.

Interventions Qualifying for Review

While NREPP will retain its open submission policy, the new review system emphasizes the important role of SAMHSA's Center Directors and their staff (in consultation with key stakeholders) in setting intervention review priorities that will identify the particular content areas, types of intervention approaches, populations, or even types of research designs that will qualify for review under NREPP. Under the streamlined review procedures, the sole requirement for potential inclusion in the NREPP review process is for an intervention to have demonstrated one or more significant behavioral change outcomes. Center-specific review priorities will be established and communicated to the field by posting them to the NREPP Web site at the beginning of each fiscal year.^[2]

Review of Existing NREPP Programs and Practices

It will be the prerogative of SAMHSA Center Directors to establish priorities for the review and interventions already on, and pending entry on, NREPP. As indicated above, these decisions may be linked to particular approaches, populations, or strategic objectives as identified by SAMHSA as priority areas. Until reviews of existing NREPP programs and practices are completed and posted to the new NREPP Web site, the current listing on the SAMHSA Model Programs Web site will remain intact.

Notifications to Program/Practice Developers

Upon the completion of NREPP reviews program/practice developers (or principal investigators of a research-based intervention) will be notified in writing within 2 weeks of the review results. A complete summary, highlighting information from each of the descriptive and rating dimensions, will be provided for review. Program/practice developers who disagree with the descriptive information or ratings contained in any of the dimensions will have an opportunity to discuss their concerns with the NREPP contractor during the 2-week period following receipt of the review outcome notification. These concerns must be expressed in writing to the contractor within this 2-week period. If no comments are received, the review is deemed completed, and the results may be posted to the NREPP Web site. If points of disagreement cannot be resolved by the end of this 2-week period, then written appeals for a rereview of the intervention may be considered on a case-by-case basis.

NREPP Technical Expert Panel

SAMHSA will organize one or more expert panels to perform periodic (e.g., annual assessments of the evidence review system and recommend enhancements to to the review procedures and/or standards for evidence-based science and practice. Panel membership will represent a balance of perspectives and expertise. The panels will be comprised of researchers with knowledge of evidence-based practices and initiatives, policymakers, program planners and funders, practitioners, and consumers.

The modified NREPP system embodies a commitment by SAMHSA and its Science to Service Initiative to broaden the appeal and utility of the system to multiple audiences. While maintaining the focus on the documented outcomes achieved through a program or practice, NREPP also is being developed as a user-friendly decision support tool to present information along multiple dimensions of evidence. Under the new system, interventions will not receive single, overall ratings as was the case with the previous NREPP (e.g., Model, Effective, or Promising). Instead, an array of information from multiple evidence dimensions will be provided to allow different user audiences to both identify (through Web-searchable means) and prioritize the factors that are important to them in assessing the relative strengths of different evidence-based approaches to prevention or treatment services.

Section IV presents in more detail the specific dimensions of descriptive information and ratings that NREPP will offer under this new framework.

IV. NREPP Decision Support Tool Dimensions

The NREPP system will support evidence-based decisionmaking by providing a wide array of information across multiple dimensions. Many of these are brief descriptive dimensions that will allow users to identify and search for key intervention attributes of interest. Descriptive dimensions would frequently include a brief, searchable keyword or attribute (e.g., “randomized control trial” under the Evaluation Design dimension) in addition to narrative text describing that dimension. Two dimensions, Strength of Evidence and Readiness for Dissemination, will consist of quantitative, criterion-based ratings by reviewers. These quantitative ratings will be accompanied by reviewer narratives summarizing the strengths and weaknesses or the intervention along each dimension.

Considerations for Using NREPP as a Decision Support Tool

It is essential for end-users to understand that the descriptive information and ratings provided by NREPP are only useful within a much broader context that incorporates a wide range of perspectives—including clinical, consumer, administrative, fiscal, organizational, and policy—into decisions regarding the identification, selection, and successful implementation of evidence-based services. In fact, an emerging body of literature on implementation science ^[3] suggests that a failure to carefully attend to this broader array of data and perspectives may well lead to disappointing or unsuccessful efforts to adopt evidence-based interventions. Because each NREPP user is likely to be seeking somewhat different information, and for varied purposes, it is unlikely that any single intervention included on NREPP will fulfill all of the specific requirements and unique circumstances of a given end-user. Appreciation of this basic premise of NREPP as a decision support tool to be utilized in a broader context will thus enable system users to make their own determinations regarding how best to assess and apply the information provided.

The NREPP decision support dimensions include:

• Descriptive Dimensions
• Strength of Evidence Dimension Ratings
• Readiness for Dissemination Dimension Ratings

A complete description of these dimensions is provided in the sections below.

Descriptive Dimensions

• Intervention Name and Summary: Provides a brief summary of the intervention, including title, description of conceptual or theoretical foundations, and overall goals. Hyperlinks to graphic logic model(s), when available, could be accessed from this part of the summary.
• Contract Information: Lists key contact information. Typically will include intervention developer's title(s), affiliation, mailing address, telephone and fax numbers, e-mail address, and Web site address.
• Outcome(s): A searchable listing of the behavioral outcomes that the intervention has targeted.
• Effects and Impact: Provides a description and quantification of the effects observed for each outcome. Includes information on the statistical significance of outcomes, the magnitude of changes reported including effect size and measures of clinical significance (if available), and the typical duration of behavioral changes produced by the intervention.
• Relevant Populations and Settings: Identifies the populations and sample demographics that characterize existing evaluations. The settings in which different populations have been evaluated will be characterized along a dimension that ranges from highly controlled and selective (i.e., efficacy studies), to less controlled and more representative (i.e., effectiveness studies), to adoption in the most diverse and realistic public health and clinical settings (i.e., dissemination studies).^[4]

• Costs: Provides a breakdown of intervention cost(s) per recipient/participant or annual as appropriate (including capital costs, other direct costs [travel, etc.]). Start-up costs including staff training and development. A standardized template would be provided to applicants for estimating and summarizing the implementation and maintenance costs of an intervention.
• Adverse Effects: Reported with regard to type and number, amounts of change reported, type of data collection, analyses used, intervention and comparison group, and subgroups.
• Evaluation Design: Contains both a searchable index of specific experimental and quasi-experimental designs (e.g., pre-/posttest nonequivalent groups designs, regression-discontinuity designs, interrupted time series designs, etc.) ^[5] as well as a narrative description of the design (including intervention and comparison group descriptions) used to document intervention outcomes.

• Replication(s): Coded as “None,” or will state the number of replications to date (only those that have been evaluated for outcomes). Replications will be additionally characterized as having been conducted in efficacy, effectiveness, or dissemination contexts.
• Proprietary or Public Domain Intervention: Typically will be one or the other, but proprietary components or instruments used as part of an intervention will be identified.
• Cultural Appropriateness: Coded as “Not Available” (N/A) if either no data or no implementation/training materials for particular culturally identified groups are available. When culture-specific data and/or implementation materials exist for one or more groups, the following two Yes/No questions will be provided for each group:
• Was the intervention developed with participation by members of the culturally identified group?
• Are intervention and training materials translated or adapted to members of the culturally identified group?
• Implementation History: Provides information relevant to the sustainability of interventions. Provides descriptive information on (1) the number of sites that have implemented the intervention; (2) how many of those have been evaluated for outcomes; (3) the longest continuous length of implementation (in years); (4) the average or modal length of implementation; and (5) the approximate number of individuals who have received or participated in the intervention.

Strength of Evidence Dimension Ratings

Quantitative, reviewer-based ratings on this dimension will be provided within specific categories of research/evaluation design. In this manner, users can search and select within those categories of research designs that are most relevant to their particular standards of evidence-based knowledge. The categories of research design that are accepted within the NREPP system are described below.

Research Design

Quality of evidence for an intervention depends on the strength of adequately implemented research design controls, including comparison conditions for quasi-experimental and randomized experimental designs (individual studies). Aggregation (e.g., meta-analysis and systematic research reviews) and/or replication across well-designed series of quasi-experimental and randomized control studies provide the strongest evidence. The evidence pyramid presented below represents a typical hierarchy for classifying the strength of causal inferences that can be obtained by implementing various research designs with rigor.^[6] Designs at the lowest level of evidence pyramid (i.e., observational, pilot, or case studies), while acceptable as evidence in some knowledge development contexts, would not be included in the NREPP system.

1. Reliability ^[7]

Outcome measures should have acceptable reliability to be interpretable. “Acceptable” here means reliability at a level that is conventionally accepted by experts in the field.^[8]

0 = Absence of evidence of reliability or evidence that some relevant types of reliability (e.g., test-retest, interrater, interitem) did not reach acceptable levels.

2 = All relevant types of reliability have been documented to be at acceptable levels in studies by the applicant.

4 = All relevant types of reliability have been documented to be at acceptable levels in studies by independent investigators.

2. Validity

Outcome measures should have acceptable validity to be interpretable. “Acceptable” here means validity at a level that is conventionally accepted by experts in the field.

0 = Absence of evidence measure validity, or some evidence that the measure is not valid.

2 = Measure has face validity; absence of evidence that measure is not valid.

4 = Measure has one or more acceptable forms of criterion-related validity (correlation with appropriate, validated measures or objective criteria); OR, for objective measures of response, there are procedural checks to confirm data validity; absence of evidence that measure is not valid.

3. Intervention Fidelity

The “experimental” intervention implemented in a study should have fidelity to the intervention proposed by the applicant. Instruments that have tested acceptable psychometric properties (e.g., interrater reliability, validity as shown by positive association with outcomes) provide the highest level of evidence.

0 = Absence of evidence or only narrative evidence that the applicant or provider believes the intervention was implemented with acceptable fidelity.

2 = There is evidence of acceptable fidelity in the form of judgment(s) by experts, systematic collection of data (e.g. dosage, time spent in training, adherence to guidelines or a manual), or a fidelity measure with unspecified or unknown psychometric properties.

4 = There is evidence of acceptable fidelity from a tested fidelity instrument shown to have reliability and validity.

4. Missing Data and Attrition

Study results can be biased by participant attrition and other forms of missing data. Statistical methods as supported by theory and research can be employed to control for missing data and attrition that would bias results, but studies with no attrition needing adjustment provide the strongest evidence that results are not biased.

0 = Missing data and attrition were taken into account inadequately, OR there was too much to control for bias.

2 = Missing data and attrition were taken into account by simple estimates of data and observations, or by demonstrations of similarity between remaining participants and those lost to attrition.

4 = Attrition was taken into account by more sophisticated methods that model missing data, observations, or participants; OR there was no attrition needing adjustment.

5. Potential Confounding Variables

Often variables other than the intervention may account for the reported outcomes. The degree to which confounds are accounted for affects the strength of casual inference.

0 = Confounding variables or factors were as likely to account for the outcome(s) reported as were hypothesized causes.

2 = One or more potential confounding variables or factors were not completely addressed, but the intervention appears more likely than these confounding factors to account for the outcome(s) reported.

4 = All known potential confounding variables appear to have been completely addressed in order to allow causal inference between intervention and outcome(s) reported.

6. Appropriateness of Analyses

Appropriate analysis is necessary to make an inference that an intervention caused reported outcomes.

0 = Analyses were not appropriate for inferring relationships between intervention and outcome, OR the sample size was inadequate.

2 = Some analyses may not have been appropriate for inferring relationships between intervention and outcome, OR the sample size may have been inadequate.

4 = Analyses were appropriate for inferring relationships between intervention and outcome. Sample size and power were adequate.

Readiness for Dissemination Dimension Ratings

1. Availability of Implementation Materials (e.g., Treatment Manuals, Brochures, Information for Administrators, etc.)

0 = Applicant has insufficient implementation materials.

2 = Applicant has provided a limited range of implementation materials, or a comprehensive range of materials of varying or limited quality.

4 = Applicant has provided a comrephensive range of standard implementation materials of apparent high quality.

2. Availability of Training and Support Resources

0 = Applicant has limited or no training and support resources.

2 = Applicant provides training and support resources that are partially adequate to support initial and ongoing implementation.

4 = Applicant provides training and support resources that are fully adequate to support initial and ongoing implementation (tested training curricula, mechanisms for ongoing supervision and consultation).

3. Quality Improvement (QI) Materials (e.g., Fidelity Measures, Outcome and Performance Measures, Manuals on How To Provide QI Feedback and Improve Practices)

0 = Applicant has limited or no materials.

2 = Applicant has materials that are partially adequate to support initial and ongoing implementation.

4 = Applicant provides resources that are fully adequate to support initial and ongoing implementation (tested quality fidelity and outcome measures, comprehensive and user-friendly QI materials).

Scoring the Strength of Evidence and Readiness for Dissemination Dimensions

The ratings for the decision support dimensions of Strength of Evidence and Readiness for Dissemination are calculated by averaging individual rating criteria that have been scored by reviewers according to a uniform five-point scale. For these two quantitative dimensions, the average score on each dimension (i.e., across criteria and reviewers) as well as average score for each rating criterion (across reviewers) will be provided on the Web site for each outcome targeted by the intervention.^[9]

V. Future Activities: Implementing and Sustaining a Streamlined NREPP

SAMHSA plans to initiate reviews using the new NREPP review process and procedures in summer 2006. The precise number and characteristics of new interventions that will be prioritized for the first series of reviews have yet to be determined. SAMHSA anticipates that many of the existing programs and practices currently listed on the SAMHSA Model Programs Web site will undergo an expedited set of reviews using the new system. Regardless, the current Model Programs Web site will remain intact until all relevant programs have been included in a new Web site, http://www.national registry.samhsa.gov

The identification of collaborative mechanisms for supporting the continued development and refinement of NREPP will represent a SAMHSA priority in 2006. SAMHSA will explore means for providing adequate technical assistance resources to communities seeking to initiate and/or augment evidence-based practices. In addition, appropriate technical advisors and other scientific resources will be utilized to assure the continued evolution of NREPP as a state-of-the-art decision support tool.

Appendix: Analysis of Public Comments in Response to Federal Register Notice

Background and Overview

The Substance Abuse and Mental Health Services Administration (SAMHSA), through its Science to Service initiative, develops tools and resources for providers of prevention and treatment services to facilitate evidence-based decisionmaking and practice. An important informational resource is the National Registry of Evidence-based Programs and Practices (NREPP). NREPP is a voluntary rating and classification system designed to provide the public with reliable information on the scientific basis and practicality of interventions designed to prevent and/or treat mental and addictive disorders. NREPP originated in SAMHSA's Center for Substance Abuse Prevention (CSAP) in 1997 as a way to help professionals in the field become better consumers of prevention programs. The program was expanded in 2004 to include substance abuse treatment interventions within SAMHSA's Center for Substance Abuse Treatment (CSAT) and mental health promotion and treatment interventions within the Center for Mental Health Services (CMHS).

During the past 2 years, SAMHSA reviewed existing evidence rating systems and developed and pilot-tested a revised approach to the rating of specific outcomes achieved by programs and practices. This development effort led SAMHSA to propose 16 evidence rating criteria as well as a set of proposed utility descriptors to describe the potential of a given intervention to be “transported” to real-world settings and populations.

Considering the prominence of NREPP within its Science-to-Service initiative and the potential impact of NREPP on the research and provider communities, SAMHSA announced a formal request for public comments in the Federal Register on August 26, 2005 (70 FR 165, 50381-50390) with a 60-day public comment period ending October 26, 2005. The notice outlined in some detail the proposed review system, including scientific criteria for evidence reviews, the screening and triage of NREPP applications, and the identification by SAMHSA of priority review areas. The notice invited general as well as specific comments and included 11 questions soliciting targeted feedback. By request of the SAMHSA Project Officer, MANILA Consulting Group coded and analyzed the responses received in response to the 11 questions posted in the Federal Register notice. The results of the analysts are presented below.

Method

A total of 135 respondents submitted comments via e-mail, fax, and postal mail during the comment period. Of these 135 respondents, 109 (81%) answered at least some of the 11 questions posted in the Federal Register notice.

Respondents

The 135 respondents included 53 providers, 36 researchers, 4 consumers, 21 respondents with multiple roles, and 21 with unknown roles visa-à-vis NREPP. Respondents were labeled as having one or more of the following domains of interest: substance abuse prevention (N=68), substance abuse treatment (N=48), mental health promotion (N=22); and mental health treatment (N=20). The domain of interest was unknown for 33 respondents. The respondents represented 16 national organizations, 10 state organizations, and 14 local organizations; 90 were private citizens; and 5 were individuals with unknown affiliations. Fifty-one respondents (38%) were labeled “noteworthy” at the request of the SAMHSA Project Officer. Noteworthy respondents included those representing national or state governments or national organizations, and nationally known experts in substance abuse or mental health research or policy.

Twenty-six responses were judged by the four MANILA coders and the SAMHSA Project Officer to contain no information relevant to the 11 questions in the notice. These responses, labeled “unanalyzable” for the purposes of this report, could be categorized as follows:

• Mentioned topics related to SAMHSA but made no point relevant to the questions posted in the Federal Register notice (N=10);
• Mentioned only topics unrelated to SAMHSA or incoherent text (N=7);
• Asked general questions about NREPP and the Federal Register notice (N=4);Wanted to submit a program for NREPP review (N=4); and
• Wanted to submit a program for NREPP review (N=4); and
• Responded to another Federal Register notice (N=1).

Procedure

Before coding began, responses were read to identify recurrent themes to include in the codebook (presented in Subpart A of this Appendix). Using this codebook, each submission was then assigned codes identifying respondent characteristics (name, location, domain of interest, affiliation/type of organization, functional role, and level of response) and the content or topical themes contained in the response. One pair of coders coded the respondent data, while another pair coded the content. Content coding was conducted by two doctoral-level psychologists with extensive training and experience in social science research and methodology.

Each response could be assigned multiple codes for content. Coders compared their initial code assignments for all responses, discussed reasons for their code assignments when there were discrepancies, and then decided upon final code assignments. In many cases, coders initially assigned different codes but upon discussion agreed that both coders' assignments were applicable. Coding assignments were ultimately unanimous for all text in all responses.

Results

The following discussion of key themes in the public comments is presented in order of the 11 questions from the Federal Register notice. Tables containing detailed frequencies of themes in the comments and other descriptive information are provided in Subpart B.

Comments Addressing Question 1

Question 1. “SAMHSA is seeking to establish an objective, transparent, efficient, and scientifically defensible process for identifying effective, evidence-based interventions to prevent and/or treat mental and substance use disorders. Is the proposed NREPP system—including the suggested provisions for screening and triage of applications, as well as potential appeals by applicants—likely to accomplish these goals?”

Respondents submitted a wide range of comments addressing Question 1. Highlights of these comments are presented below, organized by topic as follows:

1. Individual-Level Criteria

2. Population-, Policy-, and System-Level Criteria

3. Utility Descriptors

4. Exclusion From NREPP Due to Lack of Funding

5. Potential Impact on Minority Populations

6. Potential Impact on Innovation

7. Provider Factors

8. Other Agencies' Standards and Resources

9. Reliance on Intervention Developers To Submit Applications

10. Generalizability

11. Other Themes and Notable Comments

1. Individual-Level Criteria

Number of respondents: 24 (22%).

Recommendations made by respondents included adding cost feasibility as a 13th criterion (one respondent) and scoring all criteria equally (two respondents). Comments regarding specific criteria are presented in Subpart C.

2. Population-, Policy-, and System-Level Criteria

Number of respondents: 29 (27%).

Comments on specific criteria are presented in Subpart D. Highlights of comments on more general issues are presented below.

Differences in Evaluation Approaches for Individual-Level and Population-, Policy-, and System-Level Outcomes

Two respondents noted the proposed NREPP approach does not acknowledge key differences between evaluating individual-level outcomes and population-, policy-, and system-level outcomes. One of these respondents argued that NREPP is based on theories of change that operate only at the individual level of analysis, with the assumption that discrete causes lead to discrete effects, and therefore “many of the NREPP criteria appear to be insufficient or inappropriate for determining the validity of community-based interventions and their context-dependent effects.”

Unclear What Interventions Are of Interest to NREPP

One organization, Community Anti-Drug Coalitions of America, recommended that SAMHSA present a clear, operational definition of the types of interventions it wants to include in NREPP.

Match Scale to Individual-Level Outcomes

Twelve respondents, including the Society for Prevention Research and a group of researchers from a major university, recommended that the same scale be used for outcomes at the individual level as for the population, policy, and system levels.

Add Attrition Criterion

The same group of university researchers suggested adding attrition as a 13th criterion to the rating criteria for studies of population outcomes. They noted, “Just as attention to attrition of individuals from conditions is essential in individual-level studies, attention to attrition of groups or communities from studies is essential in group-level studies. This is necessary in order to assess attrition as a possible threat to the validity of the claim that the population-, policy-, or system-level intervention produced observed outcomes.”

Include Only Interventions That Change Behavior

It was recommended that NREPP only include interventions proven to change behavior. A group of university researchers noted:

As currently described, these outcomes refer to implementation of changes in policy or community service systems, not to changes in behavioral outcomes themselves. In fact, as currently described, the policy or system change would not be required to show any effects on behavior in order to be included in NREPP. This is a serious mistake. The NREPP system should be reserved for policies, programs, and system-level changes that have produced changes in actual drug use or mental health outcomes.

3. Utility Descriptors

Number of respondents: 15 (14%).

Only one respondent, the Committee for Children, recommended specific changes to the utility descriptors. Their comments are presented in Subpart E of this Appendix.

Seven other respondents recommended using utility descriptors in some way to score programs. The American Psychological Association (APA) Committee on Evidence-Based Practice recommended more emphasis on the utility descriptors “as these are key outcomes for implementation and they are not adequately addressed in the description of NREPP provided to date. This underscores earlier concerns noted about the transition from effectiveness to efficacy.”

4. Exclusion From NREPP Due To Lack of Funding

Number of respondents: 28 (26%).

The possibility that NREPP will exclude programs due to lack of funding was a concern voiced by several organizations, including the National Association for Children of Alcoholics, the APA Committee on Evidence-Based Practice, the National Association of State Alcohol and Drug Abuse Directors, Community Anti-Drug Coalitions of America, and the California Association of Alcohol and Drug Program Executives. The National Association for Children of Alcoholics provided the following comment:

NREPP should establish differing criteria for projects that collected data with [National Institutes of Health] grant funds and projects that collected data with no or very small amounts of funds. It has been intrinsically unfair that only grants have been able to establish “evidence” while many programs appear very effective—often more effective in some circumstances than NREPP approved programs—but have not had the Federal support or other major grant support to evaluate them. The SAMHSA grant programs continue to reinforce the designation of NREPP programs in order to qualify for funding, and the states tend to strengthen this ‘stipulation’ to local programs, who then drop good (non-NREPP) work they have been doing or purchase and manipulate NREPP programs that make the grant possible. This is not always in the best interest of the client population to be served.

Another key concern was that funding for replication research is rarely available. Several respondents suggested that SAMHSA consider funding evaluation research, and many argued that the lack of funding resources could negatively impact minority populations or inhibit treatment innovation. The latter two themes were frequent enough to be coded and analyzed separately. Results are summarized in the following sections.

5. Potential Impact on Minority Populations

Number of respondents: 13 (12%).

Thirteen respondents noted that the proposed NREPP approach could negatively impact specific populations, including minority client populations. The Federation of Families for Children's Mental Health suggested that NREPP would effectively promote certain practices “simply because the resources for promotion, training, evaluation are readily accessible * * * thus widening the expanse and disparities that currently exist.”

Another frequently noted concern was that evidence-based practices are currently too narrowly defined, and thus as more funding sources begin to require evidence-based practices as a prerequisite for funding, some ethnic or racial minority organizations may be excluded from funding. One respondent also pointed to potential validity concerns, noting that “Very little clinical trial evidence is available for how to treat substance use disorders in specific populations who may constitute most or all of those seen in particular agencies: HIV positive patients, native Americans, adolescents, Hispanics, or African Americans. Although it is unreasonable to expect all EBTs to be tested with all populations, the external validity of existing studies remains a serious concern.” For these reasons, many respondents surmised that the widespread application of interventions developed in research contexts that might tend to limit the inclusion of minority and/or underserved populations could ultimately result in decreased cultural competence among service providers.

6. Potential Impact on Innovation

Number of respondents: 21 (19%).

Twenty-one respondents cited concerns that the proposed NREPP approach could hamper innovation. CAADPE noted that its main concerns were “the focus on the premise that treatment will improve if confined to inteventions for which a certain type of research evidence is available” and “the issue of ‘branding,’ which could lead to some of our most innovative and effective small scale providers eliminated from funding considerations.”

One respondent suggested that lists of evidence-based treatments could “ossify research and practice, and thus become self-fulfilling prophecies * * * stifling innovation and the validation of existing alternatives.” Several respondents observed that the potential for stifling innovation is even greater given that SAMHSA's NREPP is not the only list of evidence-based practices used by funders.

The APA Practice Organization recommended that NREPP focus on “developing and promoting a range of more accessible and less stigmatized services that are responsive to consumers' needs and preference, and offer more extensive care opportunities.”

7. Provider Factors

Number of respondents: 22 (20%).

A number of respondents noted the proposed NREPP approach does not acknowledge provider effects on treatment outcomes. The APA Committee on Evidence-Based Practice wrote, “Relationship factors in a therapeutic process may be more important than specific interventions and may in fact be the largest determinant in psychotherapy outcome (see Lambert & Barley, 2002). How will NREPP address this concern and make this apparent to users?”

Another respondent cited the Institute of Medicine's definition of evidence-based practice as “the integration of the best research evidence with clinical expertise and client values,” noting that “The narrowed interpretation of evidence-based practice by SAMHSA focuses almost solely on the research evidence to the exclusion of clinical expertise and patient values.”

Several respondents suggested that NREPP could place too much emphasis on highly prescriptive, annualized treatments. Counselors can become bored when they are not able ti “tinker” with or adapt treatments. In addition, making minor modifications may actually make treatments more effective with different population groups.

8. Other Agencies' Standards and Resources

Number of respondents: 27 (25%).

Nineteen respondents suggested that, in developing NREPP, SAMHSA should consult other agencies' standards and resources related to evidence-based practices—for example, the standards published by the APA, American Society for Addiction Medicine, and the Society for Prevention Research. One respondent suggested consulting with National Institutes of Health scientists about approaches for aggregating evidence; another recommended including in NREPP model programs identified by other agencies. One respondent submitted a bibliography of references for assessing the rigor of qualitative research.

One respondent suggested that SAMHSA did not provide other institutions the opportunity to provide input on the development of NREPP prior to the request for public comments.

9. Reliance on Intervention Developers To Submit Applications

Number of respondents: 4 (4%).

Four respondents cited problems with NREPP's reliance on intervention developers to submit applications, and suggested that literature reviews instead be used to identify programs eligible for NREPP. One private citizen wrote, “If no one applies on behalf of a treatment method, is that one ignored? Why not simply start with the literature and identify treatment methods with adequate evidence of efficacy?”

Another respondent observed that requiring an application creates a bias toward programs with advocates “either ideologically or because of a vested interest in sales, visibility, and profits. An alternative is to select interventions for NREPP consideration solely by monitoring the peer-reviewed published literature, and including them regardless of whether or not the scientist responds or furthers the registration process.”

The Society for Prevention Research suggested that SAMHSA convene a panel to periodically review available interventions that might not be submitted to NREPP because they “lack a champion.”

10. Generalizability

Number of respondents: 48 (44%).

Many respondents discussed the issue of generalizability of evidence, especially the concern that interventions proven to work in clinical trials do not always work in real-world settings. Several respondents pointed out the potential conflict between implementing an intervention with fidelity and having a adapt it for the setting.

The APA Evidence-Based Practice Committee suggested that the proposed NREPP approach does not adequately distinguish between “efficacy” and “effectiveness,” and strongly recommended that SAMHSA look for ways to bridge the two.

The Associations of Addiction Services recommended paying more attention to how and where treatments are replicated: “The highest level of evidence should be successful replication of the approach in multiple community treatment settings. Experience with [the National Institute on Drug Abuse] Clinical Trials Network suggests that an approach that shows meaningful outcome improvements in the ‘noisy’ setting of a publicly funded community treatment program is truly an approach worth promoting.”

A few respondents suggested that NREPP score interventions according to their readiness and amenability to application in real-world settings.

11. Other Themes and Notable Comments

Distinguishing Treatment and Prevention

Number of respondents: 7 (6%).

A few respondents called or evaluating treatment and prevention approaches differently. One respondent noted that some criteria appear to be more appropriate for treatment modalities than for preventive interventions, and recommended that SAMHSA “confer with research experts in those respective fields and separate out those criteria that are more relevant to only treatment or prevention.”

Another respondent suggested that the criteria are more appropriate for prevention that treatment:

The criteria and selection for the peer review panels should be separate for prevention and treatment programs. The criteria and models are different and the panels should not be an across the board effort, but rather representative of prevention and treatment experts specific to the program being evaluated. The plan is based as the notice states on 1,100 prevention programs with little experience with treatment programs/practices.

Synthesizing Evidence

Three respondents suggested using meta-analysis to synthesize evidence for outcomes. One recommended SAMHSA consult with National Institutes of Health experts in this area.

Replications

The Teaching-Family Association recommended considering replications when evaluating evidence. The Society for Prevention Research wrote that it is unclear how replications would be used in the proposed NREPP, and suggested averaging ratings across studies.

Add Criteria

The National Student Assistance Association Scientific Advisory Board and one other respondent suggested adding a cultural competence criterion. The Society for Prevention Research recommended adding a criterion to assess the clarity of causal inference.

Range of Reviewer Perspectives

The APA Practice Association noted the importance of having a “large and broad” reviewer pool: “A small group of reviewers representing a limited range of perspectives and constituencies would have an undue impact on the entire system. We are pleased that a nominations process is envisioned.”

Cost Effectiveness

One respondent called for incorporating program cost effectiveness into NREPP. In choosing what program to implement, end users often have to decide between diverse possibilities, such as attempting to pass a tax increase on beer or implementing additional classroom prevention curricula, each with competing claims about effectiveness. A cost-effectiveness framework may be the only way to compare these choices.

Comments Addressing Question 2

Question 2. “SAMHSA's NREPP priorities are reflected in the agency's matrix of program priority areas. How might SAMHSA engage interested stakeholders on a periodic basis in helping the agency determine intervention priority areas for review by NREPP?”

Number of respondents: 16 (15%).

Respondents recommended a number of approaches to engage stakeholders:

• Conduct meetings, conferences, and seminars.
• Send and solicit information via e-mail or a Web site.
• Send informational notices via newletters.
• Survey stakeholders.
• Work with the Addiction Technology Transfer Centers (ATTCs) to administer surveys.
• Consult the National Prevention Network and the Society for Prevention Research, which “have forged a close working relationship to foster the integration of science and practice and * * * would be very helpful in answering this question.”

Comments Addressing Question 3

Question 3. “There has been considerable discussion in the scientific literature on how to use statistical significance and various measures of effect size in assessing the effectiveness of interventions based upon both single and multiple studies (Schmidt & Hunter, 1995; Rosenthal, 1996; Mason, Schott, Chapman, & Tu, 2000; Rutledge & Loh, 2004). How should SAMHSA use statistical significance and measures of effect size in NREPP? Note that SAMHSA would appreciate receiving citations for published materials elaborating upon responders' suggestions in this area.”

Statistical Significance

Number of respondents: 13 (12%).

A group of university researchers recommended that for programs to be included in NREPP, they should be required to provide statistically significant results on drug use and/or mental health outcomes using two-tailed tests of significance at p<.05. The APA Evidence-Based Practices Committee recommended further discussion and consideration by NREPP of the conceptual distinction between statistical and clinical significance.

The County of Los Angeles Department of Health Services urged SAMHSA “not to place undue preference only on programs that offer statistically significant results. Studies of innovative approaches and of emerging populations may not have sample sizes large enough to support sophisticated statistical analyses, yet may offer valuable qualitative information on effective approaches.”

Effect Size

Number of respondents: 24 (22%).

Most of the respondents discussing effect size noted that interventions aimed at achieving population change were likely to have small effect sizes, even if they are very successful. Several respondents recommended combining effect size with reach. A group of researchers from a major university noted:

Effect sizes should be reported, but they should not be used as a criterion for inclusion or exclusion from NREPP. From a public health perspective, the impact of an intervention is a function of both its efficacy and its reach (Glasgow, Vogt, & Boles, 1999). An intervention with even a very modest effect size can have a substantial impact on public health if it reaches many people. Therefore, NREPP should report effect sizes for each statistically significant outcome reported and NREPP should also include and provide an assessment of the “reach” of that intervention. Specifically, the inclusion criteria for participation and the proportion of the recruited population that participated in the intervention study should be included in describing the likely “reach” of the program.

Three respondents noted that professionals in the field have not reached consensus on how to use effect size. One noted, “Effect sizes may vary with the difficulty of the prevention goal and the methodological rigor of the analysis. Applying standards for ‘weak,' ‘moderate,' ‘strong' or other labels fails to take into account differences in results that may be attributable to differences in goals or methods.”

One respondent suggested considering other indicators of clinical effectiveness, such as use of the RCI (reliable change index; Jacobson & Truax, 1984).

Other points made regarding effect size included the following:

• Between-group effect sizes assume a standard comparison condition, which is rare in nonmedical interventions. Meta-analyses with baseline-follow-up effect sizes or a “network approach” to effect sizes are ways to overcome this problem.
• Effect size is not the equivalent of client improvement and does not assess the significance of interventions for their clients.
• Effect size alone is not sufficient to evaluate and rate programs; cost-benefit information or other practical information are also needed.

Comments Addressing Question 4

Question 4. “SAMHSA's proposal for NREPP would recognize as effective several categories of interventions, ranging from those with high-quality evidence and more replication to those with lower quality evidence and fewer replications. This would allow for the recognition of emerging as well as fully evidence-based interventions. Some view this as a desirable feature that reflects the continuous nature of evidence; provides important options for interventions recipients, providers, and funders when no or few fully evidence-based interventions are available; and helps promote continued innovation in the development of evidence-based interventions. Others have argued that several distinct categories will confuse NREPP users. Please comment on SAMHSA's proposal in this area.”

Number of respondents: 35 (32%).

Thirty-three respondents supported the use of multiple categories as outlined in Question 4; two respondents were opposed. Of those in favor of multiple categories, nine respondents wrote that this approach would reflect the process of emerging evidence and encourage knowledge sharing early in the process. The APA Evidence-Based Practice Committee argued that “Including all of these NREPP products is seen as a desirable feature that reflects the continuous nature of evidence. This may also be critical information for providing reasonable options for stakeholders when there are no or few evidence-based practices available.”

The State Associations of Addiction Services pointed out that multiple categories would lessen the likelihood of misinterpreting information in NREPP, and the California Department of Alcohol and Drug Programs added that including multiple categories of intervention would give greater flexibility to programs using the list.

Of the two respondents against multiple categories, one suggested that a clear designation of effectiveness is needed if NREPP is to be useful to the field.

Additional Comments

One respondent argued that only two categories should be used, effective and emergent: “While distinctions such as whether a program has had independent replications as opposed to developer replications may be of interest to researchers, the majority of those responsible for choosing and implementing programs may find this level of detail to be confusing rather than particularly helpful or relevant.”

A group of university researchers recommended assigning scores to several categories of evidence quality: theoretical foundation, design adequacy, measure adequacy, fidelity, and analysis adequacy.

Several other organizations suggesting adding a category for programs not yet shown to be evidence-based, but recommended for further study. One noted that categories of effectiveness should be the same for individual-level and population-, policy-, or system-level outcomes.

One respondent proposed an approach in which SAMHSA would document the strength of evidence for each approach, and allow consumers to decide what is effective:

Various authorities have established different and sometimes conflicting standards for when there is enough evidence to constitute an EBT. Part of the problem here is drawing a discrete line (EBT or not) on what is actually a continuous dimension. * * * To inform and demystify the dichotomous and somewhat arbitrary decision as to which treatments are evidence-based and which are not, it is useful to have a compilation of the strength of evidence for (or against) different approaches. * * * Why not just stick to your main emphasis on documenting the strength of evidence for each approach, and let others decide where they want to draw the line for what they regard to be “effective.”

Another respondent argued that providing information on replications and having six potential categorizations for evidence-based practices could be too technical and confusing for some. Most consumers will be most interested in whether there is some body of evidence that the program they are considering works.

One respondent, a private citizen, recommended that SAMHSA ask stakeholders what categories would be useful to them.

Comments Addressing Question 5

Question 5. “SAMHSA recognizes the importance of considering the extent to which interventions have been tested with diverse populations and in diverse settings. Therefore, the agency anticipates incorporating this information into the Web site descriptions of interventions listed on NREPP. This may allow NREPP users to learn if interventions are applicable to their specific needs and situations, and may also help to identify areas where additional studies are needed to address the effectiveness of interventions with diverse populations and in diverse locations. SAMHSA is aware that more evidence is needed on these topics. Please comment on SAMHSA's approach in this area.

Number of respondents: 27 (25%).

Most respondents affirmed the importance of the issues raised in Question 5. Two respondents suggested that SAMHSA should facilitate research aimed at developing services for minority populations. Comments regarding what and how to report are noted below.

What To Report

Regarding what to report, respondents suggested tracking and reporting demographic changes; reporting the impact of interventions on different populations; and requiring programs that use NREPP interventions to report to SAMHSA on the impact on their client populations, as well as providers' thoughts about the intervention's applicability to various client populations.

The Oregon Office of Mental Health and Addiction Services suggested that SAMHSA “focus considerable effort on identifying and listing practices useful and applicable for diverse populations and rural areas. Providers and stakeholders from these groups have repeatedly expressed the concern they will be left behind if no practices have been identified which fit the need of their area. We need to take particular care to ensure that their fear is not realized.”

The Committee for Children suggested reporting data for two separate dimensions: setting and population. Setting dimensions would include community data—size of community, community context (e.g., suburb, town), geographic location, community socioeconomic status—and agency data, which includes the type of agency (e.g., hospital, child care, school), characteristics (e.g., outpatient vs. inpatient, middle school vs. elementary school), size, and resources required for implementation. Population dimensions would include age, socioeconomic status, ethnicity, cultural identification, immigrant/acculturation status, race, and gender.

How To Report

Three respondents submitted suggestions for how to report on intervention effectiveness with diverse populations. The APA Evidence-Based Practices Committee suggested that SAMHSA develop “a comprehensive glossary that addresses definitions of different constituencies, populations, and settings.” The Family and Child Guidance Clinic and the Native American Health Center of Oakland both suggested that a panel of Native Americans be convened to decide which evidence-based programs and practices are effective for Native Americans, then submit a monograph describing these programs and practices.

Comments Addressing Question 6

Question 6. “To promote consistent, reliable, and transparent standards to the public, SAMHSA proposes that all existing programs on NREPP meet the prevailing scientific criteria described in this proposal, and that this be accomplished through required rereviews of all programs currently on NREPP. SAMHSA has considered an alternative approach that would “grandfather” all existing NREPP programs under the new system, but would provide clear communication that these existing programs have not been assessed against the new NREPP scientific standards. Please comment on which approach you believe to be in the best interests of SAMHSA stakeholders.”

Number of respondents: 32 (29%).

Twenty-seven respondents proposed rereviewing existing programs under the revised NREPP criteria. Five respondents advocated grandfathering the programs into NREPP without review. Highlights of these viewpoints are provided below.

Arguments for Rereview

The Committee for Children wrote a grandfathering system “may give the impression to NREPP users, right or wrong, that ‘grandfathered’ interventions aren't as good as those that have undergone the new review process.”

Another respondent supported a single review process to assure programs that “all programs and practices are being rated according to a consistent set of criteria, and therefore that the adoption of an intervention by a provider can be made with confidence.”

Two researchers (both SAMHSA Model Program affiliates) noted that grandfathering will “water down” the NREPP criteria, and recommended establishing a mechanism to remove programs from NREPP when the evidence warrants.

A program developer called for a gradual transition from Model Program to rereview:

I suggest that SAMHSA maintain the current Model Program designation and grant these programs status within the new NREPP for up to 3 years. During that time period the existing programs would be screened against the new review criteria and provided an opportunity to obtain additional research findings, if needed, in order to help achieve evidence-based status within the new NREPP. * * * Many current model programs have invested extensive time and financial resources to reference SAMHSA Model Program status is their informational, training, and curricula materials, under the auspices of their partnership agreements with the SAMHSA Model Program Dissemination Project. They did this in good faith. While the SAMHSA Model Program Project has been disbanded, it is reasonable to expect SAMHSA to honor their agreements with the model programs for a period of time during the transitional phase. During this transitional phase I recommend that the model program not be earmarked as not having been assessed against the new NREPP scientific standards, but rather that they have been found to be effective under the former NREP and are awaiting review under the new criteria.”

Arguments for Grandfathering

Those who argued for grandfathering previous Model Programs discussed the possible detrimental effects that not grandfathering would have. One respondent described taking away the Model Program designation as “a breaking of faith that is just not acceptable. A subjective change in criteria does not justify harming programs that previously met the grade in all good faith * * * It also makes it hard for the end user to take the list seriously, especially if they have already expended considerable resources to replace a non-evidence-based program with one currently designated evidence-based.”

Another respondent described the destabilizing effects and potential impact on credibility of programs:

Imagine if the “model” you just selected this year at the cost of thousands of dollars (and redesigned your prevention delivery system upon) is somehow diminished or lessened in “scientific” credibility. Would you not begin to wonder if you could trust the next “model” to hold credibility? * * * There is a very real need to be careful about the criteria, and planning for a smooth and gentle segue for change * * * at the grassroots level if programs are rotating on and off of the registry system. One might well ask, how could a “model” program of today not worthy of some level of inclusion tomorrow?

Yet another respondent pointed out that not grandfathering programs could pose financial problems for organizations offering model programs. Since some organizations may only receive funding for programs designated as “model programs,” they may not be able to offer the programs while awaiting rereview.

Comments Addressing Question 7

Question 7. “What types of guidance, resources, and/or specific technical assistance activities are needed to promote greater adoption of NREPP interventions, and what direct and indirect methods should SAMHSA consider in advancing this goal?”

Venue, Channel, and Format for Promoting Adoption of NREPP Interventions

Number of respondents: 7.

Proposed strategies for promotion (venue, channel, and format) include the following:

• Identify stakeholders and take the information to them (e.g., through conferences, journals, professional magazines, professional newsletters, physicians, churches, and PTAs).
• Convene program developers and state administrators for regular meetings about programs and implementation.
• Showcase NREPP programs at national, regional, and state conferences.
• Develop fact sheets about NREPP programs (in collaboration with the program developers).
• Conduct training on NREPP programs through the Addiction Technology Transfer Centers (ATTCs).
• Work with the Office of National Drug Control Policy's National Media campaign.
• On the NREPP Web site, offer downloadable information on programs as well as a way for consumers to contact the program developers for more information.

(Note: SAMHSA's Model Program Web site currently does provide program summaries and contact information for program developers).

Technical Assistance for Promoting Adoption of NREPP Interventions

Number of respondents: 30 (28%).

Many respondents noted the importance of providing technical assistance to those looking to adopt NREPP-listed interventions. The Oregon Office of Mental Health and Addiction Services wrote, “The adoption of new practices by any entity is necessarily a complex and long-term process. Many providers will need technical support if adoption and implementation is to be accomplished effectively. Current resources are not adequate to meet this challenge.”

Another respondent suggested that SAMHSA identify point people, either at the Federal level or through the CAPTs, who can “partner with developers to gain a clear understanding of their evidence-based interventions and become knowledgeable enough to accurately discuss them with community-based preventionists.”

A group of university researchers agreed that substantial training and technical assistance are required for the effective implementation of preventive interventions. They recommended using SAMHSA's Communities That Care, which has been shown to increase the adoption of tested and effective preventive interventions in communities, to increase adoption of NREPP interventions.

The National Student Assistance Association Scientific Advisory Board recommended that SAMHSA use existing effective program and practice structures, such as Student Assistance Programs, for technical assistance, resources, and guidance.

Guidance on Adopting NREPP Interventions

Number of respondents: 10 (9%).

Several respondents recommended that SAMHSA provide guidance to individuals and organizations looking to adopt NREPP interventions. The Center for Evidence-Based Interventions for Crime and Addiction wrote, “We do not believe that just providing information about model programs on the Web will result in much diffusion of the innovation. NREPP must pay attention to training, dissemination, fidelity, and sustainability.”

The Society for Prevention Research suggested that SAMHSA survey decisionmakers and practitioners to determine their perceptions of NREPP as well as about other factors influencing their decisions in order to determine how to encourage adoption of NREPP interventions.

The APA Evidence-Based Practice Committee recommended that SAMHSA “anticipate misuses of NREPP so as to insure that funding bodies do not mistakenly assume that improving treatment comes from confining treatment to a list of recommended techniques.”

Resources for Promoting NREPP Interventions

Number of respondents: 27 (25%).

Many respondents articulated ways that SAMHSA could support and promote NREPP interventions. One common suggestion was that SAMHSA should provide the funding for and/or help create the infrastructure that is required for program implementation.

For example, the California-based Coalition of Alcohol and Drug Associations wrote:

The existing treatment infrastructure cannot handle the expectation for data collection. It is currently unlikely that most community-based treatment programs could meet the standard to be listed on the registry. How can the infrastructure be strengthened? What funding streams is SAMHSA promoting to accomplish this? * * * The initiative promises technical assistance, but this is not substitute for missing infrastructure. The financial resources to support such efforts [have] always been absent, yet the expectations and demands continue to be placed upon underfunded community-based providers, driving some out of business and requiring others to reduce services.

The Coalition of Alcohol and Drug Associations also asked how SAMHSA plans to protect providers from exploitation: “Already there are examples of large sums of money being asked for training materials on interventions developed with tax dollars. Consultants representing particular practices (especially those listed on RFAs or on SAMHSA lists) are charging fees of $3,000 per day. This is not something most nonprofits can afford.”

Another respondent, a private citizen, suggested that SAMHSA fund Services to Science grants, “a category of funding which was originally designed by SAMHSA but [is] rarely utilized.”

The State Associations of Addiction Services suggested that SAMHSA “consider new mechanisms for funding the development of the organizational capacity needed by providers to implement and sustain evidence-based practices. Such mechanisms might require new legislative authority and/or new funding.”

Comments Addressing Question 8

Question 8. “SAMHSA is committed to consumer, family, and other nonscientist involvement in the NREPP process. The panels convened by SAMHSA and described earlier in this notice suggested that these stakeholders be included specifically to address issues of intervention utility and practicality. Please comment on how consumer, family, and other nonscientist stakeholders could be involved in NREPP.”

Development of NREPP Process

Number of responses: 22 (20%).

A number of respondents discussed the need to involve nonscientist stakeholder (primarily providers) in developing the NREPP process. Seven respondents said consumers should be involved in NREPP development. The Pennsylvania Department of Health pointed out that “the use of such approaches depends heavily on local, state, and national networks of community-based providers who need to be in a position to be an active participant in discussions related to the evaluation of interventions, practices, and programs.”

The Oregon Office of Mental Health and Addiction Services argued that “Practices that are not readily acceptable by consumers and families may have limited usefulness, regardless of the evidence of technical adequacy. Consumers and families should be involved in advising SAMHSA at every level of design, development and implementation of NREPP. SAMHSA may wish to establish a specific consumer and family advisory group to provide advice on NREPP issues.”

Community Anti-Drug Coalitions of America suggested that nonscientists should review publications and recommendations to ensure they are clear to nonresearchers.

Role in NREPP Reviews

Number of respondents: 21 (19%)

Suggestions for NREPP reviews included the following:

• Involve consumers and practitioners in reviewing programs.
• Have practitioners assess the degree to which a program is implementable.
• Have consumer groups rate programs' utility.
• Have clinicians review materials for clarity.

Comments Addressing Question 9

Question 9. “SAMHSA has identified NREPP as one source of evidence-based interventions for selection by potential agency grantees in meeting the requirements related to some of SAMHSA's discretionary grants. What guidance, if any, should SAMHSA provide related to NREPP as a source of evidence-based interventions for use under the agency's substance abuse and mental health block grants?”

Technical Assistance

Number of respondents: 11 (10%).

A number of respondents suggested that SAMHSA provide training to users on the NREPP review process, as well as guidance on the appropriate use of NREPP and how to avoid misuse. For example, Student Assistance Programs (SAPs) and CAPTs could be used as technical assistance resources. One respondent wrote, “SAMHSA needs to make it clear that the NREPP ratings are established as recommendations for the field, rather than as demands upon agencies and programs—that it discourages thinking of NREPP-approved programs or practices as a finite list and encourages efforts that further refine and extend these programs and practices to new populations and settings.”

Another respondent noted that government agencies responsible for block grant allocation may need protection fro mandates about using NREPP interventions that may not be affordable or appropriate for their client populations.

Regulation

A number of respondents provided recommendations related to regulation and funding priority tied to NREPP. Twelve respondents said block grant funds should not be restricted based on NREPP status. The Society for Prevention Research and several other organizations recommended giving priority to NREPP programs, while reserving some funds specifically for innovation. One respondent suggested that block grant funding should give priority to NREPP interventions. The Maryland Alcohol and Drug Abuse Administration argued that state authority should supersede Federal authority in block grant allocation. Another respondent recommended giving funding priority to systems that implement practices known to be effective, except where evidence-based practices have not yet been identified: “Although it is clear that funding cannot entirely be limited to existing evidence-based programs because of the chilling effect on innovation that such a stance would have, nevertheless, it might be appropriate to require that a certain percentage of block grant dollars be committed to the dissemination and use of block grant monies, or to establish additional incentives for the adoption of such programs.”

One respondent warned of the potential danger of unfunded mandates: “The worst case scenario is that best of practices could cost the most money but by law or regulation become an unfunded mandate for a government-funded or not-for-profit program.”

The APA Practice Association noted that as NREPP is voluntary, “applicants should not be penalized for studying programs or interventions that are not on the NREPP.”

Two organizations, the State Associations of Addiction Services and California Alcohol and Drug Programs, considered the revised NREPP approach to be too new to use as a block grant requirement.

Comments Addressing Question 10

Question 10. “SAMHSA believes that NREPP should serve as an important, but not exclusive source, of evidence-based interventions to prevent and/or treat mental and substance use disorders. What steps should SAMHSA take to promote consideration of other sources (e.g., clinical expertise, consumer or recipient values) in stakeholders' decisions regarding the selection, delivery and financing of mental health and substance abuse prevention and treatment services?”

Number of respondents: 25 (23%).

The following suggestions were noted:

• Develop a directory of other sources of evidence-based practices. Some suggested providing links to these sources on the NREPP Web site.
• Use an external advisory committee to identify other sources of evidence-based practices.
• Include a disclaimer page that includes an introduction consistent with the issues raised in Question 10. Advertising or other promotional material created around NREPP could also include this information.
• List other sources of evaluation research such as the Collaborative for Academic, Social, and Emotional Learning, the U.S. Department of Education, the Office of Juvenile Justice and Delinquency Prevention, and the National Institute of Mental Health.

The National Association of State Alcohol/Drug Abuse directors wrote that its Exemplary Awards Program should “serve as an ‘incubator’ for programs that may wish to consider submitting into the NREPP process.”

Comments Addressing Question 11

Question 11. “SAMHSA anticipates that once NREPP is in operation, various stakeholders will make suggestions for improving the system. To consider this input in a respectful, deliberate, and orderly manner, SAMHSA anticipates annually reviewing these suggestions. These reviews would be conducted by a group of scientist and nonscientist stakeholders knowledgeable about evidence in behavioral health and the social sciences. Please comment on SAMHSA's proposal in this area.”

Number of respondents: 35 (32%).

Many of the 35 responses stated that annual review of suggestions from stakeholders is important. Four respondents noted that feedback should be reviewed more frequently than once per year. Other themes included the following:

• Use the annual review process as a mechanism for fostering innovation.
• Use marketing strategies to encourage participation in the annual review process.
• Solicit annual feedback from NREPP applicants whose programs have been labeled effective, as well as those whose programs have not been labeled effective.
• Compare NREPP results to those in other similar systems.
• Include a mechanism in NREPP for programs to be dropped from, or improve their status on, the registry (possible through the annual review).
• Periodically conduct a meta-analysis of evaluation results (possible through the annual review).
• To ensure the stability of NREPP, the criteria should be maintained without changes for a set period of time (e.g., 5 years).

Comments Beyond the 11 Posted Questions

Twenty-two respondents (20%) submitted comments on issues that were relevant but not specifically within the parameters of the 11 posted questions. These are summarized below.

Programs Versus Practices

Fourteen respondents (13%) objected to using the terms “programs” and “practices” as if they were interchangeable. One private citizen who submitted comments wrote:

It is important to distinguish between the value of rating practices and the value of rating programs. although it makes sense for reviewers to rate the quality/strength of evidence regarding a treatment practice, it is a much different proposition to rate the effectiveness of a program. The effectiveness of a treatment program is a function, among other things, of the treatment practices it employs, the ancillary services (e.g., employment counseling) it provides, the qualities and behaviors of its treatment providers * * * One could imagine a very ineffective program using evidence-based practices (e.g., one having disengaged or poorly trained counselors), and a very effective program that used other than evidence-based practices (e.g., one with committed, empathic counselors using practices that had not yet been subjected to research. Furthermore, given the multiple elements that contribute to a program's overall effectiveness, its effectiveness could change rapidly (e.g., when a charismatic program leader leaves, when there is significant counselor turnover, when funding source/amount changes, etc.). Thus, it makes much less sense to rate the effectiveness of individual programs than it does to rate the strength of evidence supporting specific treatment practices.

Terminology

The APA Evidence-Based Practices Committee suggested using a site glossary to define diagnostic terminology and client populations and communities.

Standard Outcomes

One respondent recommended including a standard set of outcomes to be evaluated.

Effect of Including Mental Health Interventions

One national organization expressed a concern that included mental health interventions will detract from the focus on substance abuse:

The proposed expansion of NREPP to include substance abuse treatment and mental health will dramatically dilute the focus of substance abuse prevention. The resources NREPP require will necessarily be diluted across a broader range of issues and inevitably detract from a focused mission of supporting efforts to prevent substance abuse.

Reporting the Date of Reviews

One respondent recommended that SAMHSA document and report the date on which a review was conducted. This will allow users to know how much time has passed since the review and prompt them to search for more recent evidence if needed.

Rationale for Revising NREPP

One respondent questioned if SAMHSA had sufficiently evaluated the existing system before deciding to revise it.

Subpart A.—Federal Register Notice Comment Codebook

Comment ID Number:

Coded by:

Date coded:

Coded by: (each item is coded by two individual coders)

Date coded:

Entered by:

Date entered:

1. Respondent Category

1.1 Commenter Name

1.1.1 First

1.1.2 MI

1.1.3 Last

1.2 Location

1.2.1 City

1.2.2 State

1.2.3 ZIP code

1.2.4 Unknown

1.3 Domain Interest

1.3.1 SAP

1.3.2 SAT

1.3.3 MHP

1.3.5 Unknown

1.4 Affiliation

1.4.1 Private

1.4.2 Organization

1.4.2.1 National

1.4.2.2 State

1.4.2.3 Local

1.4.2.4 Unknown

1.5 Functional Role

1.5.1 Provider

1.5.2 Researcher

1.5.3 Consumer

1.5.4 Multiple

1.5.5 Unknown

1.6 Response Level

1.6.1 Nonresponsive

1.6.2 Routine

1.6.3 Noteworthy (responder or comment content)

2. Topical Themes

2.1 Will the proposed NREPP system identify effective interventions

2.1.1 General, not criteria specific

2.1.2 Individual-level outcome criteria

2.1.3 Population/policy/system-level outcome criteria

2.1.4 Utility descriptors

2.1.5 Exclusion due to lack of funding

2.1.6 Negative impact on minority populations

2.1.7 Negative impact on program innovation

2.1.8 Lack of acknowledgment of provider factors

2.1.9 Use of other agencies' standards and resources

2.1.10 Reliance on developers for submitting applications

2.1.11 Generalizability issues

2.2 How can stakeholders be engaged to identify priority review areas

2.2.1 Identification (of priority areas)

2.2.2 Engagement (of stakeholders)

2.3 How should statistical significance and effect size be used to judge effectiveness

2.3.1 Statistical significance

2.3.2 Effect size

2.3.3 General, NEC

2.4 Should NREPP use multiple categories of effectiveness

2.4.1 General, not outcome specific

2.4.1 Pro

2.4.2 Con

2.4.2 Individual-level outcome rating categories

2.4.2.1 Pro

2.4.2.2 Con

2.4.3 Population/policy/system-level outcome rating categories

2.4.3.1 Pro

2.4.3.2 Con

2.5 How can NREPP best provide information on population-specific needs and situations

2.5.1 General comment

2.5.2 Venue (e.g., organized events/meetings, national or regional organizations)

2.5.3 Channel (distribution mechanisms, e.g., listservs, clearinghouses, etc.)

2.5.4 Format (media type, document type, e.g., fact sheets, white papers, policy publications, etc.)

2.6 Should current NREPP programs be “grandfathered” or rereviewed

2.6.1 Grandfathered

2.6.2 Rereviewed

2.6.3 General, NEC

2.7 How should SAMHSA promote greater adoption of NREPP interventions

2.7.1 General comment

2.7.2 Venue

2.7.3 Channel

2.7.4 Format

2.7.5 Technical assistance

2.7.6 Guidance

2.7.7 Resources

2.8 How should nonscientist stakeholders be involved in the NREPP process

2.8.1 General comment

2.8.2 Venue, channel, format

2.8.3 Potential stakeholders

2.8.4 Involvement in the development of the NREPP process

2.8.5 Involvement in program reviews

2.9 What relationship should exist between NREPP and SAMHSA block grants

2.9.1 Technical assistance provision

2.9.2 Funding support

2.9.3 Regulatory (required to use)

2.10 What additional sources of information should be considered regarding SAMHSA services

2.10.1 Steps SAMHSA should take

2.10.2 Source

2.11 How should an annual review of NREPP procedures and practices be conducted

2.12 Other issues

2.12.1 Program vs. practice

Subpart B.—Comments on SAMHSA's Federal Register Notice: Frequencies and Percentages

Table 1.—Characteristics of Respondents

[N=135]

	n	Percent
Domain interest (not mutually exclusive)
Substance abuse prevention	68	50.4
Substance abuse treatment	48	35.6
Mental health promotion	22	16.3
Mental health treatment	20	14.8
Unknown	33	24.4
Affiliation
Private	90	66.7
National organization	16	11.9
State organization	10	7.4
Local organization	14	10.4
Unknown organization	5	3.7
Functional role
Provider	53	39.3
Researcher	36	26.7
Consumer	4	3.0
Multiple roles	21	15.6
Unknown	21	15.6
Respondent clout
Noteworthy	51	37.8
Responsive	58	43.0
Unanalyzable	26	19.3
Current program status
Affiliated with a current program	10	7.4
No known affiliation with a current program	125	92.6

Table 2.—Comments Regarding the Proposed NREPP System Accomplishing Its Goals

[Question 1]

	National org.		State org.		Local org.		Unknown org.		Private
	n	%1	n	%	n	%	n	%	n	%
“Noteworthy” respondents
General, not criteria specific 2	11	78.6	4	50.0	2	100	2	66.7	16	84.2
Individual-level outcome criteria	1	7.1	1	12.5	0	0.0	1	33.3	14	73.7
Population-, policy-, or system-level outcome criteria	2	14.3	4	50.0	1	50.0	1	33.3	14	73.7
Utility descriptors	4	28.6	1	12.5	0	0.0	0	0.0	3	15.8
Funding	7	50.0	3	37.5	1	50.0	0	0.0	3	15.8
Minority populatons	1	7.1	0	0.0	1	50.0	0	0.0	2	10.5
Program innovation	4	28.6	4	50.0	2	100	0	0.0	2	10.5
Provider factors	4	28.6	4	50.0	1	50.0	1	33.3	4	21.1
Use of other agencies' standards and resources	4	28.6	2	25.0	0	0.0	0	0.0	12	63.2
Developers submitting applications	1	7.1	0	0.0	0	0.0	0	0.0	2	10.5
Generalizability	7	50.0	5	62.5	2	100	0	0.0	5	26.3
“Responsive” respondents
General, not criteria specific 2	0	0.0	0	0.0	4	40.0	2	100	18	43.9
Individual-level outcome criteria	0	0.0	0	0.0	1	10.0	0	0.0	6	14.6
Population-, policy-, or system-level outcome criteria	0	0.0	0	0.0	2	20.0	0	0.0	5	12.2
Utility descriptors	0	0.0	0	0.0	0	0.0	0	0.0	7	17.1
Funding	0	0.0	0	0.0	5	50.0	0	0.0	9	22.0
Minority populations	0	0.0	0	0.0	2	20.0	0	0.0	7	17.1
Program innovation	0	0.0	0	0.0	3	30.0	0	0.0	6	14.6
Provider factors	0	0.0	0	0.0	4	40.0	0	0.0	4	9.8
Use of other agencies' standards and resource	0	0.0	0	0.0	3	30.0	0	0.0	6	14.6
Developers submitting applicaitons	0	0.0	0	0.0	0	0.0	0	0.0	1	2.4
Generalizability	0	0.0	0	0.0	7	70.0	1	50.0	21	51.2
1 All percentages are calculated based on those providing comments.
2 These categories are not mutually exclusive.

Table 3.—Comments Regarding How SAMHSA Might Engage Interested Stakeholders To Determine Intervention Priority Areas for Review

[Question 2]

	National org.		State org.		Local org.		Unknown org.		Private
	n	% 1	n	%	n	%	n	%	n	%
“Noteworthy” respondents
Identification of priority areas 2	3	42.9	0	0.0	0	0.0	0	0.0	2	100
Engagement of stakeholders	5	71.4	1	100	1	100	0	0.0	1	50.0
“Responsive” respondents
Identification of priority areas 2	0	0.0	0	0.0	1	50.0	0	0.0	1	33.3
Engagement of stakeholders	0	0.0	0	0.0	2	100	0	0.0	3	100
1 All percentages are calculated based on those providing comments.
2 These categories are not mutually exclusive.

Table 4.—Comments Regarding Statistical Significance and Effect Size

[Question 3]

	National org.		State org.		Local org.		Unknown org.		Private
	n	% 1	n	%	n	%	n	%	n	%
“Noteworthy” respondents
Statistical significance 2	1	25.0	0	0.0	1	50.0	0	0.0	11	84.6
Effect size	2	50.0	3	100	1	50.0	1	100	13	100
General	2	50.0	0	0.0	0	0.0	0	0.0	2	15.4
“Responsive” respondents
Statistical significance 2	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Effect size	0	0.0	0	0.0	3	100	0	0.0	6	85.7
General	0	0.0	0	0.0	0	0.0	0	0.0	1	14.3
1 All percentages are calculated based on those providing comments.
2 These categories are not mutually exclusive.

Table 4.—Comments Regarding Statistical Significance and Effect Size

[Question 3]

	National org.		State org.		Local org.		Unknown org.		Private
	n	% 1	n	%	n	%	n	%	n	%
“Noteworthy” respondents
General, not outcome specific:
General comment 2	2	20.0	0	0.0	0	0.0	0	0.0	3	20.0
Pro	10	100	3	100	1	100	0	0.0	12	80.0
Con	0	0.0	0	0.0	0	0.0	0	0.0	1	6.7
Individual-level outcome rating categories:
General comment	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Pro	1	10.0	0	0.0	0	0.0	0	0.0	0	0.0
Con	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Population-, policy-, or system-level outcome rating categories:
General comment	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Pro	0	0.0	1	33.3	0	0.0	0	0.0	0	0.0
Con	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
“Responsive” respondents
General, not outcome specific:
General comment 2	0	0.0	0	0.0	1	50.0	0	0.0	3	37.5
Pro	0	0.0	0	0.0	1	50.0	1	100	6	75.0
Con	0	0.0	0	0.0	1	50.0	0	0.0	0	0.0
Individual-level outcome rating categories:
General comment	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Pro	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Con	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Population-, policy-, or system-level outcome rating categories:
General comment	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Pro	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Con	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
1 All percentages are calculated based on those providing comments.
2 These categories are not mutually exclusive.

Table 6.—Comments Regarding SAMHSA's Approach for Incorporating Information on the Extent to Which Interventions Have Been Tested With Diverse Populations and in Diverse Settings

[Question 5]

	National org.		State org.		Local org.		Unknown org.		Private
	n	% 1	n	%	n	%	n	%	n	%
“Noteworthy” respondents
General comment 2	6	100	2	100	1	100	0	0.0	12	100
Venue	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Channel	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Format	1	16.7	0	0.0	0	0.0	0	0.0	0	0.0
“Responsive” respondents
“Responsive” respondents
General comment 2	0	0.0	0	0.0	1	100	0	0.0	4	80.0
Venue	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Channel	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Format	0	0.0	0	0.0	0	0.0	0	0.0	1	20.0
1 All percentages are calculated based on those providing comments.
2 These categories are not mutually exclusive.

Table 7.—Comments Regarding Whether All Existing Programs on NREPP Should Be Rereviewed or “Grandfathered”

[Question 6]

	Noteworthy		Responsive
	n	Percent of those providing comments	n	Percent of those providing comments
Comments from individuals affiliated with an existing NREPP program
(8 individuals [3 Noteworthy, 5 Responsive] provided comments on this question)
Rereview*	2	66.7	1	20.0
Grandfather	1	33.3	3	60.0
General comment	1	33.3	2	40.0
Comments from individuals not known to be affiliated with an existing NREPP program
(29 individuals [21 Noteworthy, 8 Responsive] provided comments on this question)
Rereview	19	90.5	5	62.5
Grandfather	0	0.0	1	12.5
General comment	2	9.5	2	25.0
*Note: These categories are not mutually exclusive. There were instances of individuals who both commented specifically on whether to rereview or grandfather a program and also provided a general comment with regard to this question.

Table 8.—Comments Regarding Guidance, Resources, and/or Technical Assistance To Promote Greater Adoption of NREPP Interventions

[Question 7]

	National org.		State org.		Local org.		Unknown org.		Private
	n	% 1	n	%	n	%	n	%	n	%
“Noteworthy” respondents
General comment 2	3	30.0	2	25.0	0	0.0	0	0.0	2	11.8
Venue	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0
Channel	2	20.0	0	0.0	0	0.0	0	0.0	1	5.9
Format	1	10.0	0	0.0	0	0.0	0	0.0	0	0.0
Technical assistance	5	50.0	5	62.5	1	100	0	0.0	11	64.7
Guidance	4	40.0	0	0.0	0	0.0	0	0.0	1	5.9
Resources	6	60.0	5	62.5	0	0.0	1	100	3	17.6
“Responsive” respondents
General comment 2	0	0.0	0	0.0	0	0.0	0	0.0	3	16.7
Venue	0	0.0	0	0.0	0	0.0	0	0.0	2	11.1
Channel	0	0.0	0	0.0	1	20.0	0	0.0	3	16.7
Format	0	0.0	0	0.0	0	0.0	0	0.0	1	5.6
Technical assistance	0	0.0	0	0.0	2	40.0	0	0.0	6	33.3
Guidance	0	0.0	0	0.0	1	20.0	0	0.0	4	22.2
Resources	0	0.0	0	0.0	3	60.0	0	0.0	9	50.0
1 All percentages are calculated based on those providing comments.
2 These categories are not mutually exclusive.

Table 9.—Comments Regarding How Consumer, Family, and Other Nonscientist Stakeholders Could Be Involved in NREPP

[Question 8]

	National org.		State org.		Local org.		Unknown org.		Private
	n	% 1	n	%	n	%	n	%	n	%
“Noteworthy” respondents
General comment 2	0	0.0	0	0.0	1	50.0	0	0.0	1	50.0
Venue, channel, format	2	20.0	0	0.0	0	0.0	0	0.0	1	50.0
Potential stakeholders	7	70.0	5	71.4	0	0.0	0	0.0	0	0.0
Involvement in the development of the NREPP process	5	50.0	4	57.1	1	50.0	0	0.0	0	0.0
Involvement in program reviews	6	60.0	5	71.4	1	50.0	0	0.0	0	0.0
“Responsive” respondents
General comment 2	0	0.0	0	0.0	1	16.7	0	0.0	1	5.6
Venue, channel, format	0	0.0	0	0.0	1	16.7	0	0.0	4	22.2
Potential stakeholders	0	0.0	0	0.0	4	66.7	1	100	14	77.8
Involvement in the development of the NREPP process	0	0.0	0	0.0	4	66.7	0	0.0	8	44.4
Involvement in program reviews	0	0.0	0	0.0	2	33.3	1	100	6	33.3
1 All percentages are calculated based on those providing comments.
2 These categories are not mutually exclusive.

Table 10.—Comments Regarding Guidance SAMHSA Should Provide for Use Under the Agency's Substance Abuse and Mental Health Block Grants

[Question 9]

	National org.		State org.		Local org.		Unknown org.		Private
	n	% 1	n	%	n	%	n	%	n	%
“Noteworthy” respondents
Technical assistance 2	1	11.1	2	50.0	0	0.0	0	0.0	1	8.3
Funding support	4	44.4	3	75.0	1	100	1	100	9	75.0
Regulatory	6	66.7	1	25.0	0	0.0	0	0.0	2	16.7
“Responsive” respondents
Technical assistance 2	0	0.0	0	0.0	1	50.0	0	0.0	2	18.2
Funding support	0	0.0	0	0.0	2	100	0	0.0	9	81.8
Regulatory	0	0.0	0	0.0	1	50.0	0	0.0	2	18.2
1 All percentages are calculated based on those providing comments.
2 These categories are not mutually exclusive.

Table 11.—Comments Regarding Steps SAMHSA Should Take To Promote Consideration of Other Sources of Evidence-Based Interventions

[Questions 10]

	National org.		State org.		Local org.		Unknown org.		Private
	n	% 1	n	%	n	%	n	%	n	%
“Noteworthy” respondents
Steps SAMHSA should take 2	4	80.0	1	100	0	0.0	0	0.0	12	100
Source	1	20.0	0	0.0	1	100	1	100	0	0.0
“Responsive” respondents
Steps SAMHSA should take 2	0	0.0	0	0.0	2	100	0	0.0	2	66.7
Source	0	0.0	0	0.0	0	0.0	0	0.0	2	66.7
1 All percentages are calculated based on those providing comments.
2 These categories are not mutually exclusive.

Table 12.—Comments Regarding Annual Reviews of Suggestions for Improving the System

[Question 11]

	National org.		State org.		Local org.		Unknown org.		Private
	n	%	n	%	n	%	n	%	n	%
“Noteworthy” respondents
General comment	8	100	3	100	1	100	0	0.0	14	100
“Responsive” respondents
General comment	0	0.0	0	0.0	2	100	0	0.0	7	100
1 All percentages are calculated based on those providing comments.

Table 13.—Additional Comments Not Classified Elsewhere

	National org.		State org.		Local org.		Unknown org.		Private
	n	%1	n	%	n	%	n	%	n	%
“Noteworthy” respondents
Other issues 2	4	66.7	1	25.0	1	50.0	0	0.0	1	100
Defining terms	5	83.3	3	75.0	1	50.0	0	0.0	1	100
“Responsive” respondents
Other issues 2	0	0.0	0	0.0	1	50.0	0	0.0	5	71.4
Defining terms	0	0.0	0	0.0	2	100	0	0.0	2	28.6
1 All percentages are calculated based on those providing comments.
2 These categories are not mutually exclusive.

Subpart C.—Comments on Specific Evidence Rating Criteria

Some of the respondents to SAMHSA's August 2005 Federal Register notice submitted comments about specific evidence rating criteria. A summary and highlights of key comments about these criteria are presented below.

Intervention Fidelity

Two respondents commented on this criterion. One noted that it is difficult to monitor or confirm how treatment is delivered and how staff are trained in programs with complex approaches, such as community reinforcement or family training.

Comparison Fidelity

Eleven respondents commented on this criterion. Ten of the respondents, a group of researchers from a major university, wrote:

The comparison fidelity evidence quality criterion assumes the implementation and fidelity monitoring of a “comparison condition.” In universal and selective prevention trials, this is not standard protocol. Rather, individuals or communities selected for comparison/control conditions receive standard prevention services available in the community. In such studies, it does not make sense to measure the “fidelity” of the comparison condition. However, as currently scored, this criterion will penalize prevention studies. I recommend the criterion and rating system be changed to reflect this difference between prevention and treatment research.

Nature of Comparison Condition

Fourteen respondents provided comments on this criterion. One respondent, a director of research and evaluation for a prevention program noted:

Many program participants are drawn from undeserved or marginalized populations, e.g. incarcerated youth, the mentally ill, linguistically isolated subgroups, or those suffering from Human Immunodeficiency Virus (HIV). For these populations, there may be no option to withhold active treatment only to the intervention group, due to legal requirements, health and safety considerations, or other ethical constraints. The American Evaluation Association (AEA) duly notes this consideration in its 2003 commentary on scientifically based evaluation methods.

Another service provider noted that studies that include the target intervention, comparison intervention, and attention control “would require funding at extremely high levels to have enough N in each group for statistical analysis. To conduct such a study in today's economic climate is probably impractical.”

A private citizen who submitted comments wrote:

This is a critical criterion and should be weighted more heavily than many, if not all, of the other criteria. With the proposed system, if one were trying to “game the system,” it would be advantageous to choose a comparison intervention that was ineffective (and thus receive a low score on this criterion), so as to increase the likelihood of a significant treatment effect. Nevertheless, the practice being evaluated could have “strong evidence” by scoring highly on other criteria.

A group of university researchers said that it is unclear how prevention practices being compared to existing prevention services would be scored using this criterion.

Assurances to Participants

One respondent questioned “whether such studies [without documented assurances to participants] should ever clear the bar for NREPP consideration. If investigators do not observe appropriate procedures to safeguard study participants' interests, it is at least questionable whether their products should receive any degree of attention and support from SAMHSA.”

Participant Expectations

Three respondents commented on this criterion. Two respondents listed potential problems with controlling expectations in school settings. For example, for an intervention to be implemented effectively by teachers, the teachers would have to be trained and therefore would be aware of the intervention they implement.

Two respondents pointed out that expectations might be an active component of the intervention. One wrote that “trying to control [expectations] might reduce generalization of the eventual findings. In addition, given current ethical guidelines and human subjects policies, it is hard to see how one could ‘mask’ study conditions in many studies. In obtaining consent, one has to tell participants about the conditions to which they might be assigned and it is likely that participants will know to which condition they have been assigned.”

Data Collector Bias

Three respondents commented on this criterion. One noted, “Changes to this criterion should recognize the critical need to ensure the fidelity of psychosocial treatment interventions. Fidelity, in these cases, can only be ensured through staff awareness of the actions required of them. Masking conditions actually inhibits psychosocial treatment fidelity.”

Selection Bias

Three respondents commented on this criterion. One suggested that approaches other than random assignment, such as blocking variables of interest, should qualify for the highest score on this item. Another pointed out that random assignment to psychosocial interventions might not be possible due to ethical problems with nondisclosure. He suggested rewording the item to clarify that random assignment does not refer only to “blinding” participants to their treatment condition.

Attrition

Two respondents commented on this criterion. One pointed out that the criterion is unclear, and that “attrition needing adjustment” is not defined, nor is the difference between “crude” and “sophisticated” methods of adjusting for attrition. This respondent also pointed out that “sophisticated” does not necessarily mean better than “crude” (this comment also applied to the Missing Data criterion).

Theory-Driven Method Selection

Eleven respondents commented on this criterion. A group of university researchers wrote:

This is an important criterion. However, this criterion should recognize that a number of preventive interventions seek to address and reduce risk factors or enhance protective factors that research has shown are common shared predictors of a range of drug use, mental health, and other outcomes. It is important to explicitly recognize this fact in formulating and describing this criterion * * * Not all reviewers, especially those from treatment backgrounds, will be familiar with the concept of addressing shared predictors of broader outcomes in preventive trials in order to affect wide-ranging outcomes. This criterion needs to educate reviewers about this in the same way that the criterion currently warns against “dredging” for current significant results.

Subpart D.—Criterion-Specific Themes for Population-, Policy-, and System-Level Outcomes

Logic-Driven Selection of Measures

A group of researchers from a major university suggested that this item and the parallel item for individual-level outcomes, Theory-Driven Measure Selection, should have the same label.

Intervention Fidelity

The seven respondents who commented on this criterion observed that interventions must be adapted for individual communities to be effective. The criterion as written does not account for this.

Nature of Comparison Condition

One respondent stated that there is not consensus among evaluation researchers on this topic, and until there is, “we should reserve judgment on how best to define the nature of comparison conditions within community level interventions.” She also pointed out, “Since the collective behaviors of members in each community will vary * * * how can they possibly be compared to each other in a valid and reliable way.”

Data Collector Bias

A group of university researchers pointed out that the item assumes archival data are unbiased, while they may be biased by institutional practices. They suggested that the highest rating “be reserved for studies in which data collectors were masked to the population's condition.”

Another respondent, a national organization, wrote:

The very nature of coalition work requires coalition members to be involved in its evaluation and research efforts. It is culturally detrimental and unethical to work with coalitions in such a way that they are not involved in the evaluation process. Expecting the data collectors to be blind to the efforts of the community means that the researchers are outside the community and would have no understanding of the context in which the coalition works. Many evaluators and researchers view this as the absolute wrong way to work with coalitions. Criterion Seven [Data Collector Bias] runs counter to participatory research which is the standard in working with coalitions.

Population Studied

Eleven respondents commented on this criterion. One respondent stated that quasi-experimental time-series designs might be as internally valid as randomized control designs, and felt this should be reflected in the criterion.

A group of university researchers advocated excluding single-group pre-/posttest design studies from NREPP. They wrote, “A group randomized design with adequate numbers of groups in each condition holds the greatest potential for ruling out threats to internal validity in community-level studies. This criterion should be expanded to provide a rating of four for group randomized studies with adequate Ns.”

Subpart E.—Comment for Children's Suggestions for Utility Descriptors

1. Implementation Support

Regarding the ease of acquiring materials is there centralized ordering for all materials? What implementation support materials are included in initial program cost, and are they adequate? Are basic program updates and replacement parts all easily available? Regarding start-up support, research suggests that there are several features that are important to the effectiveness and sustainability of programs. These include an active steering committee, administrator support, engagement of family members, and wholeschool implementation (for school-based programs). Do the basic program materials provided supply adequate guidance for effectively gaining these sources of support? On the other hand, some clients are not in the position to achieve all of these goals. Is it possible to effectively implement the program without them? Are needs assessment tools offered? This is important for determining whether implementation should take place at all. What is the nature of the start-up implementation support? What is the nature of the ongoing implementation support? Is client support differentiated for new and experienced clients? Do client support personnel have adequate training to answer sophisticated questions from the most highly experienced program implementers? Is there implementation support through a variety of media? What support is there for transfer of learning? For example, practice beyond specific lessons, opportunities for population served to demonstrate, and be reinforced for skills beyond specific lessons, support for staff awareness of skills, how to recognize skills, how to reinforce skills, examples typical in the daily setting, materials for engaging family members of the population served, materials for engaging staff outside the implementers of the program (e.g., residential housekeeping staff, school playground monitors), support for engaging community members outside the implementation setting, what training is required, what training is available beyond that which is required?

2. Quality Monitoring

Are the tools supplied for quality monitoring user-friendly and inexpensive? How well are they adapted specifically to the program? What are their psychometric characteristics?

3. Unintended or Adverse Events

No further comments.

4. Population Coverage

Are the materials appropriate to the population to be served in regard to, for example: length of lessons, vocabulary, concepts and behavioral expectations, teaching strategies.

5. Cultural Relevance and Cultural Competence

To what extent was cultural relevance addressed during the development of the program? Is there a theoretical basis to the program that addresses cultural relevance? Were stakeholders from a variety of relevant backgrounds engaged in the development process? How early in the development process were they involved? In what ways were they involved? Were professionals with multicultural expertise involved in the development process? How early in the development process were they involved? In what ways were they involved?

6. Staffing

Since FTEs are often difficult to estimate and estimates many therefore be unreliable, the required time should be estimated for the following: Required training time, on-site start-up activities, implementer preparation time per week, lesson length × number of lessons per implementer, time required for other activities.

7. Cost

No further comments on this descriptor except to reiterate that cost considerations play into several of the other descriptors.

8. Motivational Issues Affecting Implementation

We suggest that consideration be given to examining what further motivational issues may impact whether the programs are implemented and sustained with fidelity. These include: appeal of materials and activities for the population to be served, appeal of materials and activities for the staff who will implement the programs, support of the program for the preexisting goals and programs of the site (e.g., school-based programs that support academics), how well the program otherwise integrates with existing goals, programs, and activities of the site (e.g., teachers are expected to direct student discussions, but not therapy), support offered for adapting the program to specific local populations, fit of materials to the typical structures of the setting (e.g., short enough lessons to fit within a class period, necessary equipment is usually available in the setting).

Footnotes

1.  As cited by the Institute of Medicine (2001), studies have suggested it takes an average of 17 years for research evidence to diffuse to clinical practice. Source: Balas, E.A., & Boren, S.A. (2000). Managing clinical knowledge for health care improvement. In: J. Bemmel & A.T. McCray (Eds.), Yearbook of medical informatics 2000: Patient-centered systems. Stuttgart, Germany: Schattauer.

2.  Except for FY06 when priorities will be established and posted when the new system Web site is launched (i.e., within the third FY quarter).

3.  Fixsen, D.L., Naoom, S.F., Blase, K.A., Friedman, R.M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, Florida: University of South Florida, Louis de la Parte Florida mental Health Institute, The National Implementation Network (FMHI Publication #231).

Rogers (1995). Diffusion of innovaations (5th Ed.) New York: The Free Press.

4.  For more description of these types of studies and their role in supporting evidence-based services, see the report: Bridging science and service: A report by the National Advisory mental Health Council's Clinical Treatment and Services Research Workgroup (http://www.nimh.nih.gov/​publicat/​nimhbridge.pdf).

5.  Campbell, D.T., & Stanley, J.C. (1966). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.

6.  Biglan, A., Mrazek, P., Carnine, D.W., & Flay, B. R. (2003). The integration of research and practice in the prevention of youth problem behaviors. American Psychologist, 58, 433-440.

Chambless, D. L., & Hollon, S. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7-18.

Gray, J. A. (1997), Evidence-based healthcare: How to make health policy and management decisions. New York: Churchill Livingstone.

7.  Each criterion would be rated on an ordinal scale ranging from 0 to 4. The endpoints and midpoints of the scale would be anchored to a narrative description of that rating. The remaining integer points of the scale (i.e., 1 and 3) would not be explicitly anchored, but could be used by reviewers to assign intermediate ratings at their discretion.

8.  Marshall, M., Lockwood, A., Bradley, C., Adams, C., Joy, C., & Fenton, M. (2000). Unpublished rating scales: A major source of bias in randomised controlled trials of treatments for schizophrenia. British Journal of Psychiatry, 176, 249-252.

9.  Note that it is unlikely that the Readiness for Dissemination dimension will vary by targeted outcome(s), insofar as the materials and resources are usually program specific as opposed to outcome specific.