2020-07664. Agency Information Collection Activities: Proposed Collection; Comment Request

AGENCY:

Agency for Healthcare Research and Quality, HHS.

ACTION:

Notice.

SUMMARY:

This notice announces the intention of the Agency for Healthcare Research and Quality (AHRQ) to request that the Office of Management and Budget (OMB) approve the proposed information collection project “Evaluating the Implementation of Products by AHRQ's Learning Health Systems to Inform and Encourage Use of AHRQ Evidence Reports.” This proposed information collection was previously published in the Federal Register on February 4, 2020 and allowed 60 days for public comment. AHRQ did not receive comments from members of the public. The purpose of this notice is to allow an additional 30 days for public comment.

DATES:

Comments on this notice must be received by 30 days after date of publication of this notice.

ADDRESSES:

Written comments and recommendations for the proposed information collection should be sent within 30 days of publication of this notice to www.reginfo.gov/​public/​do/​PRAMain. Find this particular information collection by selecting “Currently under 30-day Review—Open for Public Comments” or by using the search function.

FOR FURTHER INFORMATION CONTACT:

Doris Lefkowitz, AHRQ Reports Clearance Officer, [phone], or by email at [e-mail].

SUPPLEMENTARY INFORMATION:

Proposed Project

Evaluating the Implementation of Products by Learning Health Systems To Inform and Encourage Use of AHRQ Evidence Reports

AHRQ's Evidence-based Practice Center (EPC) Program has 20 years of experience in synthesizing research to inform evidence-based health care practice, delivery, policies, and research. The AHRQ EPC program is committed to partnering with organizations to make sure its evidence reports can be used in practice. Historically, most of its evidence reports have been used by clinical professional organizations to support the development of clinical practice guidelines or Federal agencies to inform their program planning and research priorities. To improve the uptake and relevance of the AHRQ EPC's evidence reports, specifically for health systems, AHRQ has contracted with the American Institutes for Research (AIR) to obtain feedback from learning health systems (LHSs) to assist the AHRQ EPC program in developing and disseminating evidence reports that can be used to improve the quality and effectiveness of patient care.

Even if an EPC evidence report topic addresses LHS-specific evidence needs, the density of the information in an evidence report may preclude its easy review by busy LHS leaders and decisionmakers. AHRQ understands that to facilitate use by LHSs, complex evidence reports must be translated into a format that promotes LHS evidence-based decision making and can be contextualized within each LHS' own system-generated evidence. Such translational products, for the purposes of this notice, are referred to simply as “products.”

The purpose of this information collection is to support a process evaluation of use and implementation of two such products into LHS decisionmaking processes, workflows, and clinical care. The evaluation has the following goals:

1. Document how LHSs prioritize filling evidence gaps, make decisions about using evidence, and implement tools to support and promote evidence use in clinical care.

2. Assess the contextual factors that may influence implementation success; associated implementation resources, barriers and facilitators; and satisfaction of LHS leaders and clinical staff.

3. Provide the AHRQ EPC program with necessary insights about the perspectives, needs, and preferences of LHS leaders and clinical staff as related to decisions and implementation of products into practice.

This study is being conducted by AHRQ through its contractor, the American Institutes for Research (AIR), pursuant to AHRQ's statutory authority to conduct and support research on, and disseminate information on, health care and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services. 42 U.S.C. 299a(a)(1).

Method of Collection

To achieve the goals of this project, the following data collection activities will be implemented:

1. Key informant interviews with health system leaders, clinicians and staff; and

2. compilation and coding of notes from “implementation support” meetings (“check-ins”) between an implementation facilitator and site champions who are implementing the products.

Brief Background on the Products To Be Implemented by LHSs in This Study

AHRQ is funding the development of two products that are specifically intended to make the findings from EPC evidence reports more accessible and usable by health systems. These are the products that will be offered to LHSs for potential implementation during this project. They include a “triage tool” and a “data visualization tool” that have been designed to support LHS use of AHRQ evidence reports. The LHS triage tool presents high-level results of evidence reports that enable leaders within LHSs to quickly understand the relevance of the reports to their organization, share high-level information with key stakeholders (e.g., healthcare executives), and link to more granular data from the report. The data visualization tool presents data from the evidence review and individual studies in a dynamic, interactive website. The evaluation will capture the anticipated variation in how the LHS might use the products and the unique experience of LHSs.

Key Informant Interviews

There will be two rounds of key informant interviews: (1) In-person preliminary interviews will be conducted early in the implementation period (months 1-3) with LHS leaders and clinicians and will focus on health systems' rationale for selecting each product and early experiences with its roll-out into practice; (2) remote follow-up interviews will be conducted via telephone later in the implementation period (months 10-11) with two sets of stakeholders: (a) LHS leaders and (b) clinicians/staff (hereafter, “clinical staff”) actively implementing the product. These follow-up interviews will focus on health systems' experiences implementing their selected product(s). All interviews (preliminary and follow-up) will be 60-minutes in duration, recorded with permission of the key informants, and transcribed for analysis. Up to 88 total interviews will be conducted across the two rounds of key informant interviews. Assuming the same LHS leaders participate in the preliminary and follow-up interviews, the key informant interviews will involve 4-5 LHS leaders and clinical staff from each of the eleven LHSs implementing the study. Additional detail about the information collection components is provided below.

1. In-person preliminary interviews. The preliminary interviews will include 2-3 LHS leaders/decisionmakers at each of eleven implementation sites for a maximum of 33 interviews in the first round of data collection. The interviews will be conducted during implementation site visits that are occurring early in the project to support the health systems' testing and/or roll out of the products into clinical workflows. Specific topics explored in the preliminary interviews include LHSs' decision to participate in implementation, decision considerations for the selected product, experiences leading the implementation, and early experiences and perceptions of the selected product(s). To limit respondent burden, we will use the implementation site visits as an opportunity for conducting the preliminary interviews, thereby limiting the need to schedule additional time with respondents for a phone interview. If a respondent has limited availability during the site visit, however, we may need to do the preliminary interview remotely or substitute the respondent with another qualified staff member who is available during the implementation site visit.

2. Remote follow-up interviews. The follow-up interviews will include the 2-3 LHS leaders/decisionmakers from the preliminary interviews (maximim n = 33), along with 2 additional clinical staff (n = 22) at each of eleven implementation sites for a maximum of 55 follow-up interviews. Specific topics explored in the follow-up interviews include LHS leaders' and clinical staff's experiences with each product as well as their perceptions of the relative advantage, acceptability/compatibility, appropriateness, and feasibility of using the product; implementation fidelity (i.e., if the implementation went as planned), reach, barriers and facilitators, and associated costs; any outcomes of implementing the product (e.g., achieved any intended systemic changes); and likely sustainability of continuing to use the product in practice.

The two sets of in-depth qualitative interviews will allow for a nuanced exploration of both what LHSs value about the products and what it takes to successfully implement such tools into practice. The research on implementation and uptake of products to promote use of evidence in LHS settings is sparse, thus it is important to use a data collection strategy for the evaluation that will yield rich information about the experience of health systems, LHS decisionmakers, and the staff implementing the tools into practice. A quantitative survey would not yield the depth of individual feedback that is needed to capture the experience of implementing these tools and the unique contexts of the health systems. Thus, interviews are the preferred method of systematically collecting this data.

Implementation Support Meetings/“Check-Ins”

In addition to key informant interviews, which will be conducted only at the beginning and end of implementation, AHRQ will gather information throughout the implementation period by using monthly implementation support meetings between implementation facilitators and site champions as an ongoing opportunity to ask key questions about implementation progress. Although the primary goal of these check-in meetings is to provide technical assistance with implementation and recommendations for handling emergent challenges in the implementation process, they will also be a source of rich information for the evaluation. Because these meetings occur in real time as the implementation unfolds, they will reduce the potential biases (e.g., selective memory, recency effects, forgetting details about key events and their sequence) associated with only collecting data at the beginning or end of the implementation period.

These check-in meetings will occur by telephone and are intended to monitor implementation progress, provide support to health systems, and discuss next steps. AIR implementation facilitators for each site will schedule telephone conference calls with site champions (N = 11), during which structured notes will be taken. These notes will be supplemented with relevant information from other touchpoints between the facilitators and champions (e.g., ad hoc calls, email exchanges, and voluntary participation in monthly shared learning events) as they naturally occur. Notetakers will capture and document information related to key implementation domains as these topics arise in check-in meetings and other facilitator/champion encounters throughout implementation.

Estimated Annual Respondent Burden

Exhibit 1 shows the total estimated annualized burden of 214.5 hours for the two rounds of key informant interviews and implementation “check-ins” combined. For the key informant interviews (totaling 154 hours), burden is included for: (1) LHS leaders/decisionmakers participating in the preliminary interviews (a maximum of 33 hours), (2) LHS leaders/decisionmakers participating in the follow-up interviews (a maximum of 33 hours), (3) clinical staff participating in the follow-up interviews (a maximum of 22 hours), (4) interviewee review of materials, consent forms, and logistics in advance of their respective interviews (i.e., 16.5 + 5.5 = 22 hours) and (5) time for designated LHS staff (e.g., the LHS member, a designated site liaison, selected interviewees) to recommend key informants, coordinate implementation support, and help with scheduling of in-person preliminary interviews and remote follow-up interviews (44 hours). Also included in Exhibit 1 is the estimated annualized burden hours for monthly check-ins between implementation facilitators and LHS champions for informal technical assistance support and the quick status probes on implementation progress (a maximum of 60.5 hours). These annualized burden estimates for the key informant interviews and the coaching sessions are further explained below.

Key Informant Interviews: Expanded Detail on Burden Estimates

We estimate 1 hour for each key informant interview for: (1) LHS leaders/decisionmakers participating in the preliminary interviews (a maximum of 33 hours), (2) LHS leaders/decisionmakers participating in the follow-up interviews (a maximum of 33 hours), (3) clinical staff participating in the follow-up interviews (a maximum of 22 hours), (Total interview burden = 1.00 hour × maximum of 88 interviews = 88 hours). We estimate an additional 15 minutes (0.25 hours) will be needed for key informants to prepare for their respective interview(s) (Total interview preparation burden = 0.25 hours × maximum of 88 interviews = 22 hours; of which 16.5 hours is for leaders/decisionmakers to prepare for both preliminary and follow-up interviews and 5.5 is for clinical staff to prepare for their participation in the follow-up interviews only). Finally we estimate time for LHS leaders and staff to identify interview candidates, facilitate recruitment, coordinate implementation support, and assist with interview scheduling (4.00 hours per each of 11 LHSs; Total staff assistance burden = 4.00 hours × 11 sites = 44 hours). The “staff assistance” burden involves the following:

• In each of the eleven LHS organizations implementing the product(s), the LHS member (and/or site liaison/champion) will identify prospective key informants (i.e., other LHS leaders/decisionmakers and appropriate clinical staff), with additional key informants subsequently identified through snowball sampling.
• Designated LHS staff (i.e., LHS member, designee and/or site liaison/champion) will provide needed contact information to the AIR evaluation team for outreach and recruitment of the prospective key informant interview candidates, assist with interview scheduling, and coordinate implementation support with the AIR team.

We will develop standardized email messages to reach out to interview candidates and a written overview of the project, the evaluation, and the purpose of the interview. We will coordinate scheduling of both the implementation support check-ins and the 60-minute interviews at the most convenient time, considering the needs of the LHS leadership and staff. For the preliminary interviews, if prospective interviewees are not available during our site visit, we will ask for suggestions of other LHS staff who meet our recruitment criteria or arrange a telephone interview, if needed.

Implementation Support Meetings/Check-Ins: Expanded Detail on Burden Estimates

We estimate 60.5 hours for the monthly check-ins between implementation facilitators and LHS champions. This includes an average of 30 minutes of implementation support/check-in meetings per each of the 11 LHSs for each month of implementation (11 months). (11 months × 0.5 hours = 5.5 hours). Across LHSs, the estimated burden associated with check-ins is approximately 61 hours across the implementation period (5.5 hours × 11 LHSs = 60.5 hours).

Exhibit 1—Estimated Annualized Burden Hours

Form name	Number of respondents *	Number of responses per respondent	Hours per response	Total burden hours
In-person preliminary interviews with LHS leaders/decisionmakers	** 33	1	1.00	33
Remote follow-up interviews with LHS leaders/decisionmakers	** 33	1	1.00	33
Remote follow-up interviews with clinical staff	22	1	1.00	22
Review of materials prior to BOTH preliminary and follow-up interviews—LHS leaders/decisionmakers	33	2	0.25	16.5
Review of materials prior to interviews—clinical staff	22	1	0.25	5.5
Interview scheduling and other staff assistance	11	1	4.00	44
Implementation check-ins: Brief monthly implementation progress checks, documented for the evaluation as structured notes on implementation topics naturally occurring in coach/champion encounters	11	11	0.5	60.5
Total	165			*** 214.5
* The numbers in this column give the maximum number of respondents for each listed activity based on a range in the number of recruits per site (e.g., “2-3 LHS leaders/decisionmakers”). The balance may shift some between LHS leaders/decisionmakers and clinical staff depending on implementation team and leadership composition at each site. In any case, 88 interviews (33 + 33 + 22 = 88) is a maximum possible in the event each of the 11 sites contributes 3 “LHS leaders/decisionmakers” (likely the same people for preliminary and follow-up interviews) and 2 additional clinical staff (for follow-up interviews only) as key informants. It is more likely that the total number of interviews will be around 80.
** These are likely to be the same 33 respondents in both preliminary and follow-up interviews.
*** Total maximum burdened hours estimate based on maximum of 88 interviews.

Costs associated with the estimated annualized burden hours are provided in Exhibit 2.

Exhibit 2—Estimated Annualized Cost Burden

Form name	Number of respondents *	Total burden hours	Average hourly wage rate **	Total cost burden
In-person preliminary interviews with leaders/decisionmakers	33	33	a $94.47	$3,117.51
Remote follow-up interviews with leaders/decisionmakers	33	33	a 94.47	3,117.51
Remote follow-up interviews with clinical staff	22	22	b 52.13	1,146.86
Review of materials prior to BOTH preliminary and follow-up interviews—LHS leaders/decisionmakers	33	16.5	a 94.47	1,558.76
Review of materials prior to interviews—clinical staff	22	5.5	b 52.13	286.72
Interview scheduling and other staff assistance c	11	44	c 20.34	894.96
Implementation check-ins (documented for the evaluation as structured notes on implementation progress)	11	60.5	a 94.47	5,715.44
Total	165			15,837.76
* The numbers in this column give the maximum number of respondents for each listed activity based on a range in the number of recruits per site (e.g., “2-3 LHS leaders/decisionmakers”). As noted in the comment to Exhibit 1, the balance may shift some between LHS leaders/decisionmakers and clinical staff depending on implementation team and leadership composition at each site. In any case, 88 interviews (33 + 33 + 22 = 88) is a maximum possible.
** National Compensation Survey: Occupational wages in the United States May 2018 “U.S. Department of Labor, Bureau of Labor Statistics.”
a Based on the mean wages for Internists, General. 29-1063; annual salary of $196,490.
b Based on the mean wages for Physician Assistants, 29-1071; annual salary of $108,430.
c Based on the mean wages for Secretaries and Administrative Assistants, 43-6010; annual salary of $42,320.

Request for Comments

In accordance with the Paperwork Reduction Act, comments on AHRQ's information collection are requested with regard to any of the following: (a) Whether the proposed collection of information is necessary for the proper performance of AHRQ's health care research and health care information dissemination functions, including whether the information will have practical utility; (b) the accuracy of AHRQ's estimate of burden (including hours and costs) of the proposed collection(s) of information; (c) ways to enhance the quality, utility and clarity of the information to be collected; and (d) ways to minimize the burden of the collection of information upon the respondents, including the use of automated collection techniques or other forms of information technology.

Comments submitted in response to this notice will be summarized and included in the Agency's subsequent request for OMB approval of the proposed information collection. All comments will become a matter of public record.

Dated: April 7, 2020.

Virginia L. Mackay-Smith,

Associate Director.