2023-19272. Request for Information; Potential Changes to its Evidence-Based Practices Resource Center

AGENCY:

Substance Abuse and Mental Health Services Administration (SAMHSA), Department of Health and Human Services (HHS).

ACTION:

Notice of request for information (RFI).

SUMMARY:

SAMHSA is issuing this RFI to gather expert and public feedback to determine how to best satisfy Sec. 7002, subsections (b), (c), or (d) of the 21st Century Cures Act, which are explained below. Input from the public and experts will help SAMHSA identify the optimal ways to identify, evaluate, and disseminate programs and practices, based on their intended audiences, the nature of the evidence supporting the program or practice, and the type of product deemed best suited to the content. SAMHSA seeks input from members of the public on potential changes to its Evidence-Based Practices Resource Center (EBPRC), specifically regarding the possible introduction of three new domains for the EBPRC website ( https://www.samhsa.gov/​resource-search/​ebp). In addition to addressing four general questions about the EBPRC overall, SAMHSA encourages members of the public to comment on several questions pertaining to each of the domains described. SAMHSA believes that public and expert input on the new domains will help make the EBPRC more responsive to the needs of the public and the behavioral health field.

DATES:

Comments on this notice must be received by October 13th, 2023.

ADDRESSES:

Please submit all responses via email to [e-mail] as a Word document, Portable Document Format (PDF) file, or in the body of the email message. Please include “Request for Information: Changes to SAMHSA's EBPRC” in the subject line of the message.

FOR FURTHER INFORMATION CONTACT:

Carter A Roeber, Telephone: [phone], Email: [e-mail], or [e-mail].

SUPPLEMENTARY INFORMATION:

The EBPRC was established in 2018 to fulfill the statutory requirements of the 21st Century Cures Act (Pub. L. 144–255). Specifically, section 7002 of the 21st Century Cures Act requires that, as appropriate, SAMHSA shall “improve access to reliable and valid information on evidence-based programs and practices, including information on the strength of evidence associated with such programs and practices, related to mental and substance use disorders for States, local communities, nonprofit entities, and other stakeholders, by posting on the internet website of the Administration information on evidence-based programs and practices that have been reviewed by the Assistant Secretary in accordance with the requirements of this section.” SAMHSA has designated the EBPRC, managed by the agency's National Mental Health and Substance Use Policy Laboratory (NMHSUPL), to fulfill this charge.

With the directive to publish information on evidence-based programs and practices (EBPs), the EBPRC relies on SAMHSA's relationships with key behavioral health stakeholders, including researchers, clinical and public health service providers, program administrators, and people with lived experience to inform the content it distributes. Further, SAMHSA's repository of EBPRC materials is organized by topic area and made searchable to maximize navigability, utility, and relevancy of content to those poised to implement EBPs. In this way, the EBPRC aims to broaden the scale of EBP implementation and provide support to improve behavioral health outcomes nationwide.

Recognizing the ever-changing nature of the evidence base, the EBPRC seeks to take a dynamic and responsive approach to its curation of resources. To date, however, the EBPRC has primarily posted federally developed materials on practices. The programs and practices mentioned in the posted materials are vetted through a review of the evidence. However, unlike Crimesolutions.gov or the What Works Clearinghouse (WWC), the EBPRC does not currently incorporate and publish a systematic process for identifying, evaluating and rating specific programs and practices across the behavioral health field and related subspecialty fields for inclusion, as envisioned by section 7002, subsections (b), (c), or (d) of the 21st Century Cures Act. The inclusion of reviews and ratings of particular programs would allow users to search for and learn about specific programs that might meet their population's needs.

To ensure that the EBPRC fulfills its roles in the analysis, synthesis, and dissemination of behavioral health evidence, SAMHSA requests that members of the public respond to the following questions, the answers to which will help frame the agency's efforts to improve the EBPRC's utility to the public.

Framing Questions

• Question A: How can SAMHSA improve the EBPRC to better meet the needs of the behavioral health field?
• Question B: What strategies should the EBPRC use to ensure its content is high-quality and supported by strong evidence?
• Question C: How can SAMHSA expand the reach of the EBPRC?
• Question D: How can SAMHSA solicit feedback on the use of its resources and information?

In addition, SAMHSA requests that commenters consider the following three domains of the EBPRC, upon which it seeks specific feedback, as enumerated below.

Domain 1. Adding a program review and rating component to the EBPRC.

SAMHSA's EBPRC provides communities, clinicians, policymakers, and others with the information and tools needed to improve the quality of care, by incorporating EBPs into their communities or clinical settings. Evidence-based practices and programs (EBPs) are defined as interventions that are guided by the best research evidence with practice-based expertise, cultural competence, and the values of the persons receiving the services that promote individual or population-level outcomes. The distinction between programs and practice is important, however.

A Program is a specific set of activities carried out according to guidelines to achieve a defined purpose. To determine whether individual programs are effective, we rely on studies using randomized experimental ( i.e., controlled) evaluation designs and quasi-experimental evaluation designs. Program profiles tell us whether a specific program was found to achieve its goals when it was carefully evaluated. The results apply to the exact set of activities and procedures used for the program as implemented at the time of evaluation. Thus, the program profile tells us that a program is likely to produce the observed result if it is implemented in exactly the same way. A program profile can answer: Did the ABC mentoring program in Anytown, USA, achieve its goals?

A Practice is a general category of programs, strategies or procedures that share similar characteristics with regard to the issues they address and how they address them. We rely on meta-analyses instead of evaluations of individual programs. Practice profiles tell us about the average results from multiple evaluations of similar programs, strategies, or procedures. The programs, strategies or procedures within a practice are similar because they share certain defining characteristics that are described for each practice profile. Thus, practice profiles tell us about the average result across multiple evaluations. A practice profile can answer: Does mentoring usually achieve its goals?

It is well established that individuals, for-profit and non-profit businesses, universities, and other groups develop and market proprietary or copywritten interventions/programs, both as a source of revenue, and as a means of improving the quality of behavioral health services. Similarly, the development of a new pharmaceutical product for a substance use or mental health condition is driven by both market forces and health care needs. In the fields of substance use prevention, mental health promotion, as well as treatment and recovery for behavioral health conditions, organizations may charge fees for the specific manualized program they developed or for additional consulting services. These programs ( i.e., manualized interventions) may or may not be effective.

In its current form, the EBPRC posts relevant materials developed by SAMHSA staff or on behalf of SAMHSA by contractors. These products cover a wide range of topics and are generally in the format of a comprehensive evidence-based guide, practical guide, or advisory. Evidence-based guides are developed for topics on which the evidence is well established, providing background information, a review of the evidence, guidance on implementation and process improvement, and case studies on specific evidence-based programs and practices. These guides are targeted to the general public and various behavioral health stakeholders across disciplines, including providers, researchers, and administrators. Practical guides are developed when a topic is urgent and for which there is an established evidence base, but uptake has been slow due to various barriers. Practical guides address those barriers through an environmental scan of up-to-date literature, curated through the lens of implementation, with the aim of informing end users. Using a similar methodology as practical guides, advisories provide brief guidance on a given topic for a broad audience, focusing on actionable steps to promote effective behavioral health practice and mitigate immediate behavioral health risks.

SAMHSA also posts treatment improvement protocols, toolkits, fact sheets, and systematic evidence reviews. Organizations and providers can download these products at no cost. These products are one way of disseminating information about EBPs in summarized form. However, there may be other ways of disseminating information about EBPs.

The EBPRC does not currently seek applications for inclusion in its website as envisioned by section 7002 (b) of the 21st Century Cures Act and similarly has not established a review and rating system as outlined in section 7002(c). Section 7002(b) suggests that the Assistant Secretary for Mental Health and Substance Use Services (AS) “may establish a period for the submission of applications for evidence-based programs and practices to be posted publicly. . ..” and section 7002(c) states that the AS “may establish minimum requirements for the applications related to submission of research and evaluation.” In summary, the EBPRC may request that programs are submitted for review and rating, requiring that minimum criteria be met for inclusion on the EBPRC website. As such, this RFI seeks input on the best ways to solicit program submissions as well as to review and rate submitted programs.

Evidence-based practice registries, like CrimeSolutions.gov or What Works in Education ( https://ies.ed.gov/​ncee/​wwc/​), provide ratings of individual programs. Similar to consumer group websites, these registries' ratings provide useful information about a single program's effectiveness, without going into detail about how the program should be implemented, how much it may cost, or whether it is appropriate for a particular organization or provider. This method has the advantage of sharing critical information quickly. However, it may not provide enough information to help an organization or provider make a final decision regarding which program to select.

Relatedly, the impact of program selection on behavioral health outcomes raises the question of how the EBPRC should consider and select programs for review. The EBPRC has limited resources for conducting program reviews and rating each submission.

Therefore, SAMHSA is interested in the following questions:

• Question 1a: Please describe the extent to which a new EBPRC component that reviews the evidence for a manualized intervention/program and publicly posts the results would be of use to the behavioral health field.
• Question 1b: If SAMHSA chooses to add the program review component, how should it decide which programs get reviewed and which do not?
• Question 1c: Please describe the extent to which an evaluation of a program's dissemination materials, costs, and other issues associated with implementation would be of use to the behavioral health field.

Domain 2. Including implementation science, process improvement, capacity building and program evaluation resources.

Implementation science is defined as “the scientific study of methods to promote the systematic uptake of clinical research findings and other evidence-based practices (EBPs) into routine practice and hence improve the quality and effectiveness of health care.” ^[i] The new field of study emerged with increased emphasis on EBPs and the barriers that organizations faced in adopting them.

Improvement science draws heavily on process improvement models.^[ii] Underlying most of these models is the Plan, Do, Study, Act (PDSA) cycle, which involves systematically gaining learning and knowledge for the continual improvement of a product or process through an iterative process of planning, making small changes, and monitoring and responding to results.^[iii] Like implementation science, improvement science aims to improve population outcomes and acknowledges the gap between current and ideal practice.^[iv]

For population outcomes to improve, the individuals, organizations, systems, and communities implementing these processes must have capacity to do so successfully. As such, in achieving population impact goals, capacity building can be described as “the ways to the means,” and “the process through which individuals, groups of people, and organizations obtain, strengthen, and maintain the capabilities they need to set and advance goals.” ^[v]

Also, essential to improving population health is program evaluation,^[vi] which can be defined as “an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency.” ^[vii]

• Question 2a: What types of implementation science, process improvement, and capacity building resources should SAMHSA include in the EBPRC to facilitate the adoption of EBPs?
• Question 2b: What additional resources on program evaluation would be helpful to provide through the EBPRC?

Domain 3. Culturally informed and community-driven programs and practices.

SAMHSA's mission covers the entire continuum of care, from substance use prevention and mental health promotion to treatment and services fostering recovery.^[viii] Furthermore, as a federal agency, SAMHSA serves all Americans, with a special emphasis on equity for under-resourced communities. For the purpose of this RFI, under-resourced communities are defined as “population groups that experience greater obstacles to health, based on characteristics such as, but not limited to, race, ethnicity, religion, income, geography, gender identity, sexual orientation, and disability.” ^[ix] Steps toward fulfilling that mission include listening to and collaborating with under-resourced communities to address the barriers that limit access to behavioral health services and to support community development through culturally informed interventions. SAMHSA has a long history of promoting and disseminating community-based and culturally based programs that are developed in collaboration with under-resourced populations. Yet, for several reasons, many community- and culturally-based programs are excluded from evidence-based registries or clearinghouses. The way these programs are developed and implemented means that they cannot ethically or logistically be evaluated using randomized controlled trial or quasi-experimental designs that registries require for consideration. The programs can also be small in scale and geographically specific, making it even more difficult to randomly select participants or develop matching control groups.

Taking these factors into account, please answer the following questions:

• Question 3a: In what ways, if any, would an EBPRC component that assesses and shares findings from research on community-based and/or culturally driven behavioral health programs be of use to the behavioral health field?
• Question 3b: What methods should SAMHSA use to assess community- or culturally-based behavioral health programs and present them on the EBPRC?
• Question 3c: If SAMHSA convenes a technical expert panel, what areas of expertise should be included?

Responses from the public will inform SAMHSA's efforts to improve the EBPRC and better disseminate programs and practices to our partners and collaborators. Responses from the public are one of the best ways for SAMHSA to hear from people and organizations who are directly affected by SAMHSA's work. Thank you for your consideration.

How To Submit a Response

Responses will be accepted through October 13th, 2023. Responses must be emailed to [e-mail]. Please include “Request for Information: Changes to SAMHSA's EBPRC” in the subject line.

Responders are free to address any or all the questions listed above. Please identify the question or question(s) to which you are responding. Responses also may address concerns or issues not identified above.

The submitted information will be reviewed by SAMHSA and HHS staff. However, individual comments may not be acknowledged by SAMHSA due to the volume of comments received.

Responses to this RFI are entirely voluntary and may be submitted anonymously. Please do not include any personally identifiable information or any information that you do not wish to make public. Proprietary, classified, confidential, or sensitive information should not be included in your response.

SAMHSA will use the information submitted in response to this RFI at its discretion. SAMHSA reserves the right to use any submitted information on public websites, in reports, in summaries of the state of the science, in any possible resultant solicitation(s), grant(s), contract(s) or cooperative agreement(s), or in the development of future funding opportunity announcements.

This RFI is for informational and planning purposes only and is not a solicitation for applications or an obligation on the part of the Government to provide support for any ideas identified in response to it. Please note that the Government will not pay for the preparation of any information submitted or for use of that information.

Dated: August 31, 2023.

Carlos Castillo,

Committee Management Officer.

Footnotes

i.  Wiltsey Stirman, S., Baumann, A. A., & Miller, C. J. (2019). The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science, 14, 1–10, https://doi.org/​10.1186/​s13012-019-0898-y.

ii.  Pflager, D. (2022, August 2). Getting it right: Using implementation research to improve outcomes in early care and education. Foundation For Child Development. https://www.fcd-us.org/​getting-it-right-using-implementation-research-to-improve-outcomes-in-early-care-and-education/​.

iii.   Consumer assessment of healthcare providers and Systems (CAHPS). AHRQ. (n.d.). https://www.ahrq.gov/​cahps/​.

iv.  Nilsen, P., Thor, J., Bender, M., Leeman, J., Andersson-Gäre, B., & Sevdalis, N. (2022). Bridging the silos: a comparative analysis of implementation science and improvement science. Frontiers in Health Services, 1, 18.

v.   Early childhood systems building resource guide. Why Capacity Building? | Child Care Technical Assistance Network. (n.d.). https://childcareta.acf.hhs.gov/​systemsbuilding/​systems-guides/​capacity-building-self-assessment-tool/​why-capacity-building.

vi.  Edwards, B., Stickney, B., Milat, A., Campbell, D., & Thackway, S. (2016). Building research and evaluation capacity in population health: the NSW Health approach. Health promotion journal of Australia: official journal of Australian Association of Health Promotion Professionals, 27 (3), 264–267. https://doi.org/​10.1071/​HE16045.

vii.   Gao-21–404SP, program evaluation: Key terms and concepts. Government Accountability Office. (2021, March). https://www.gao.gov/​assets/​gao-21-404sp.pdf.

viii.  White House (2023, May 18). Biden-Harris Administration Announces New Actions to Tackle Nation's Mental Health Crisis [Fact sheet]. https://www.whitehouse.gov/​briefing-room/​statements-releases/​2023/​05/​18/​fact-sheet-biden-harris-administration-announces-new-actions-to-tackle-nations-mental-health-crisis/​.

ix.  Substance Abuse and Mental Health Services Administration (SAMHSA): Adapting Evidence-Based Practices for Under-Resourced Populations. SAMHSA Publication No. PEP22–06–02–004. Rockville, MD: National Mental Health and Substance Use Policy Laboratory. Substance Abuse and Mental Health Services Administration, 2022.