2024-11199. Agency Information Collection Activities: Proposed Collection; Comment Request

AGENCY:

Agency for Healthcare Research and Quality, HHS.

ACTION:

Notice.

SUMMARY:

This notice announces the intention of the Agency for Healthcare Research and Quality (AHRQ) to request that the Office of Management and Budget (OMB) approve the proposed information collection project “Implementation and Testing of Diagnostic Safety Resources.” This proposed information collection was previously published in the Federal Register on March 7th, 2024 and allowed 60 days for public comment. AHRQ received no substantive comments from members of the public. The purpose of this notice is to allow an additional 30 days for public comment.

DATES:

Comments on this notice must be received by June 21, 2024.

ADDRESSES:

Written comments and recommendations for the proposed information collection should be sent within 30 days of publication of this notice to www.reginfo.gov/​public/​do/​PRAMain. Find this particular information collection by selecting “Currently under 30-day Review—Open for Public Comments” or by using the search function. Copies of the proposed collection plans, data collection instruments, and specific details on the estimated burden can be obtained from the AHRQ Reports Clearance Officer.

FOR FURTHER INFORMATION CONTACT:

Doris Lefkowitz, AHRQ Reports Clearance Officer, [phone], or by email [e-mail].

SUPPLEMENTARY INFORMATION:

Proposed Project

Implementation and Testing of Diagnostic Safety Resources

Patient safety is a pillar of the Agency for Healthcare Research and Quality's (AHRQ's) mission to support the highest quality healthcare. While progress has been made in many areas of patient safety, the field of diagnostic safety has emerged as a particular area of concern. It is estimated that every person in the United States will experience a diagnostic error in their lifetime (Institute of Medicine, 2015) which can lead to inappropriate, delayed, or withheld treatment and ultimately poor health outcomes, distress, and increased costs. Diagnostic errors can occur for many reasons: lack of meaningful engagement between clinicians, patients, and families; a fragmented healthcare system not designed to account for an increasingly complex diagnostic process; minimal (if any) feedback to clinicians about their diagnostic performance; and a culture that does not always support transparent disclosure of diagnostic errors (Institute of Medicine, 2015). Leaders in diagnostic excellence suggest that multi-pronged efforts are needed to address this complex problem and go beyond individual behaviors to system-level changes and empowering patients to engage in their care (Institute of Medicine, 2015; Henriksen, et al., 2017).

Improving diagnostic safety and quality is an AHRQ priority. In recognition of the multifaceted approach needed to effectively advance diagnostic safety, AHRQ recently supported the development of three tools to prevent diagnostic errors and have prioritized these tools for implementation and testing. These resources vary in the types of stakeholders they target, a critical advancement in our approach to diagnostic excellence.

• Calibrate Dx. This tool, targeted to individual clinicians, invites users to select a topic or condition, review diagnostic performance on a sample of cases for insights and learning opportunities, and debrief with a peer.This resource will be tested in all settings where clinicians are involved in the diagnostic process, including both inpatient and ambulatory settings.
• Measure Dx. This tool supports healthcare organizations in building sustainable teams for improving diagnostic excellence, identifying current capacity gaps, engaging in measurement strategies as part of a systematic approach to reviewing available data, and translating findings into learning opportunities.This resource will be tested in both inpatient and ambulatory settings; it is expected to be implemented more commonly in inpatient settings.
• Toolkit for Engaging Patients to Improve Diagnostic Safety (Patient Toolkit). This tool prepares patients, families, and health professionals to work together as partners to improve diagnostic safety; encourages patients to prepare for visits; and encourages providers to listen for 60 seconds before interrupting the patient.This resource will be tested in ambulatory settings only.

The goal of this research is to implement and test these three diagnostic safety resources to identify specific ways in which each resource can be used to maximize its value. For each resource the following will be examined:

(1) Feasibility of implementation—barriers, facilitators, success factors, and time needed for implementation

(2) Level of adoption—number and type of clinicians aware of and/or using the resource, number of organizational leaders endorsing the resource

(3) Effectiveness of the resource—number of diagnostic safety events (Measure Dx and Patient Toolkit), clinician self-efficacy for diagnostic decision-making (Calibrate Dx)

(4) Maintenance and sustainability—the number and type of patient safety processes in place, barriers and facilitators to maintenance and sustainability

This project will implement and test these three diagnostic safety resources across a minimum of 150 sites to up to 219 sites ( i.e., 50 to 73 sites per resource). An Implementation and Testing period for each resource will last 12 months, with Calibrate Dx starting implementation first and Measure Dx and the Toolkit for Engaging Patients starting implementation six months later. This timing allows for staggered recruitment to ensure adequate sample size and to pilot implementation processes with a single diagnostic safety resource first, transferring lessons learned about implementation and testing to the implementation of the two other resources. A Sustainability period will begin as soon as the 12-month Implementation and Testing Period is complete and will continue for 14 additional months for each resource.

To achieve the goals of this project the following data collections will be implemented:

1. Site Interest Form—A short form completed once by up to 1060 sites interested in participating in the project. Used to indicate interest in the project and by AHRQ to evaluate whether the site meets the minimum participation criteria.

2. Site Information Form—Completed once by site leaders at 265 sites that begin the project enrollment process, this form collects additional contact information, data on patient mix, and information on the organization's diagnostic safety teams, resource commitments, and capacity for implementing the resources.

3. Safer Dx Checklist—Completed once by 219 sites who fully complete enrollment activities and begin implementation of one of the three resources (82.6% of the 265 sites who begin enrollment activities). The Safer Dx Checklist is a tool that allows healthcare organizations to understand the current state of their diagnostic practices, identify areas to improve, and track progress toward diagnostic excellence over time. This will be completed prior to actual implementation of the resource.

4. Exit Interviews Protocol—Completed once by an estimated 69 sites (30% of those implementing one of the three resources) that withdraw from the project, this telephone interview will collect information on why the site could not sustain their efforts or participation.

5. A baseline assessment of patient safety culture will be conducted once for each of the 219 sites that begin participation. Completed once by site leads depending on the setting:

a. SOPS® Medical Office Survey with Diagnostic Safety Supplemental Item Set—Completed once by the site lead for 109 ambulatory clinics.

b. SOPS® Hospital Survey with Diagnostic Safety Supplemental Item Set—Completed once by the site lead for 110 inpatient sites.

6. Post-training Evaluation Form—Completed once by 1,350 clinicians and managers (90% of the 1,500 participants) attending the project's training sessions. The data will be used to track the perceived value of the training provided to enrolled sites.

7. Post-technical Assistance Evaluation Form—Administered up to 3 times to 1,500 clinicians and managers participating in the project's Learning Collaborative sessions; an estimated 90% response rate to this collection with a total of 4,050 forms completed. The data will be used to track the perceived value of the technical assistance provided to enrolled sites.

8. Clinical Sustainability Assessment Tool (CSAT)—Completed by 219 site leaders once between months 9 to 12 in advance of the 14-month sustainability period. The CSAT is a self-assessment to evaluate sustainability capacity of a clinical practice.

9. Implementation Interviews Protocol—A qualitative, semi-structured interview will be conducted with 438 site leads and/or frontline staff (up to 2 individuals from each site) at two points in time during implementation ( e.g., 6- and 18-months). The protocol is designed to elicit participant perspectives on implementation of the resource, capture lessons learned and best practices, and when possible, to provide support for adjustment to the implementation.

In addition to those noted above, the project will implement the following data collections specific to the individual resources.

For Measure Dx, the following data collections will be implemented:

10. Measure Dx Organizational Self-Assessment —This is one of the main components of the Measure Dx resource and is designed to gauge the organization's readiness to engage with Measure Dx. This checklist will be completed once by up to 73 Measure Dx sites during the project onboarding process.

11. Measure Dx Declaration of Measurement Strategy —The 73 Measure Dx sites will complete this form once to indicate their selection of measurement strategy to be implemented and provide confirmation of minimum necessary capabilities.

12. Diagnostic Safety Event Report —These reports will provide aggregate information on diagnostic safety events identified during a 12-month reporting period. The report will be completed by each participating site 3 times over the course of the testing and sustainability period at 3-, 12-, and 24-months; a total of 219 reports will be completed over the course of the project. Note that the contractor is not attempting to collect these reports at Month 0. Since part of the Measure Dx resource's goal is to support implementation of a measurement strategy, Month 3 will serve as the baseline.

13. Additional information on site safety culture, including use of diagnostic safety event data, activities to improve the quality of care, and the work environment will be collected through a survey at 3-, 12-, and 24-months during the implementation/sustainment. Five members of the Measure Dx team at each site will be surveyed; the expected response rate is 85% at each of the three administration periods. Depending on the setting, the following survey will be fielded:

a. Omnibus Safety and Culture Survey_Medical Offices —Completed by clinicians at 36 ambulatory clinics.

b. Omnibus Safety and Culture Survey_Hospitals —Completed by clinicians at 37 inpatient sites.

For Calibrate Dx, the following data collections will be implemented:

14. Calibrate Dx Survey —This survey collects clinicians' reflections on their diagnostic performance for 3-5 cases, with additional metrics around time to complete the review and the number of cases reviewed. This will be completed quarterly (following the Calibrate Dx guidance for implementation) during the implementation and testing period by up to 5 clinicians per site; with an estimated a 90% response rate to this collection.

15. Clinician Self-Efficacy Survey —The survey assesses clinician self-efficacy with diagnostic safety case review and improvement. Up to 5 clinicians per site will be asked to complete this survey two times, after training and again at the end of the testing phase, with an estimated 90% response rate to this collection.

For Patient Toolkit, the following data collections will be implemented:

16. Provider Characteristics Form —This form will be completed once by up to 15 providers at each of the 73 enrolled sites. This form collects information on practitioner type, years in practice, specialty, subspecialty, and percent of time spent in clinical practice.

17. Patient Toolkit Survey—Provider —This survey assesses provider-perceived skills and quality of communication. It will be administered to up to 15 providers at each site at five timepoints (Baseline, 3-, 6-, 9-, and 12-months), with a 90% anticipated response rate.

18. Provider Interview Protocol —A total of 50 qualitative, semi-structured interviews with site clinicians will be conducted during implementation. The interview protocol collects information related to diagnostic safety events; patient safety culture; feasibility, acceptability, utility, adoption, and spread of the Patient Toolkit; and insights into clinician experience.

19. Patient Toolkit Survey—Patient —The survey assesses patient-perceived experience and quality of communication, and collects basic patient demographics ( e.g., age, gender, education, race, ethnicity). This will be administered to site patients over a 1-week period at five timepoints (Baseline, 3-, 6-, 9-, and 12-months). The survey will be provided to patients upon check-out from a healthcare visit. A total of 12,500 surveys will be completed during each 1-week period.

20. Patient Interview Protocol —A total of 50 qualitative, semi-structured interviews will be completed with site patients during implementation. The interview protocol collects information on reason for visit, provider communication, and other insights into patient experience.

This study is being conducted by AHRQ through its contractor, RAND, pursuant to AHRQ's statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness, and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).

Method of Collection

The data collection methods for this evaluation were selected to reduce participant burden and, where possible, to allow participants a choice of response mode. In addition, technology is used for data capture and qualitative coding and analysis.

Several forms and data collection instruments will be administered using a web mode. Site leads and participants will receive a link allowing them to complete the form online. The Site Interest Form will also be accepted as a hardcopy should organizations prefer to mail or fax these forms. All other forms will be administered either by a fillable form that can be returned via email, mail, or fax depending on the site or participant preference.

Interviews will be conducted by phone or video call ( e.g., Microsoft Teams, Zoom) with interviewers using a hardcopy version of the protocol. Interviews will be audio-recorded and transcribed, following verbal consent from participants. Qualitative software will be used for data coding and analysis of interviews.

The patient surveys will be provided to patients upon check-out from a healthcare visit and they will be encouraged to complete the survey before leaving the office. The survey will include a QR code to allow patients to access a web version of the form. Alternatively, the patient can complete the paper survey and it will be collected at the site, minimizing the need for patients to return the paper survey by mail. The paper surveys will be formatted for data scanning, and data from all paper surveys returned to the contractor will be scanned into an electronic datafile.

Estimated Annual Respondent Burden

This section summarizes the total burden hours for this information collection effort in addition to the cost associated with those hours.

Exhibit 1 contains estimated response burdens for each subject population participating in the evaluation's data collection activities.

1. Site Interest Form —A physician or manager at an interested site will complete the form only once to indicate interest in participating. The form will be completed by 1,060 respondents and requires 6 minutes to complete.

2. Site Information Form —A physician or manager at an interested site will complete the form only once to provide additional contact information, data on patient mix, and information on the organization's diagnostic safety teams, resource commitments, and capacity for implementing the resources. The form will be completed by 265 respondents and requires 20 minutes to complete.

3. Safer Dx Checklist —A physician or manager at participating sites will complete the form only once to allow the participating site to understand the current state of their diagnostic practices, identify areas to improve, and track progress toward diagnostic excellence over time. The form will be completed by 219 respondents and requires 15 minutes to complete.

4. Exit Interviews Protocol —A physician or manager at sites that withdraw from the project will complete the form once to provide information on why the site could not sustain their efforts or participation. The form will be completed by 69 respondents and requires 10 minutes to complete.

5a. SOPS® Medical Office Survey with Diagnostic Safety Supplemental Item Set —A physician or manager at participating ambulatory sites will complete the form to provide a baseline assessment of patient safety culture. The form will be completed by 109 respondents and requires 15 minutes to complete.

5b. SOPS® Hospital Survey with Diagnostic Safety Supplemental Item Set —A physician or manager at participating hospital sites will complete the form to provide a baseline assessment of patient safety culture. The form will be completed by 110 respondents and requires 15 minutes to complete.

6. Post-Training Evaluation Form —A physician, nurse practitioner, physician assistant, or manager will complete the form once to indicate the perceived value of the training provided to participating sites. The form will be completed by 1350 respondents and requires 3 minutes to complete.

7. Post-Technical Assistance Evaluation Form —A physician, nurse practitioner, physician assistant, or manager will complete the form up to three times to indicate the perceived value of the technical assistance provided to participating sites. The form will be completed by 1350 respondents, three times, and requires 2 minutes to complete.

8. Clinical Sustainability Assessment Tool (CSAT) —A physician or manager at participating sites will complete the form to evaluate the sustainability capacity of a clinical practice. The form will be completed by 219 respondents and requires 15 minutes to complete.

9. Implementation Interviews Protocol —A physician, nurse practitioner, physician assistant, or manager will participate in an interview two times to provide their perspectives at different stages of the implementation. The interview will be completed by up to 438 respondents, two times, and requires 1 hour to complete.

10. Measure Dx Organizational Self-Assessment —A physician, nurse practitioner, physician assistant, or manager will complete the form only once to gauge the organization's readiness to engage with Measure Dx. The form will be completed by 73 respondents and requires 30 minutes to complete.

11. Measure Dx Declaration of Measurement Strategy —A physician, nurse practitioner, physician assistant, or manager will complete the form only once to indicate their selection of measurement strategy to be implemented and provide confirmation of minimum necessary capabilities. The form will be completed by 73 respondents and requires 5 minutes to complete.

12. Diagnostic Safety Event Report —A physician, nurse practitioner, physician assistant, or manager will complete the form three times to provide aggregate information on diagnostic safety events. The form will be completed by 73 respondents, three times, and requires 1 hour to complete.

13a. Omnibus Safety and Culture Survey_Medical Offices —A physician, nurse practitioner, physician assistant, or manager will complete the form three times to provide information on safety culture at ambulatory sites. The form will be completed by 162 respondents, three times, and requires 20 minutes to complete.

13b. Omnibus Safety and Culture Survey_Hospitals —A physician, nurse practitioner, physician assistant, or manager will complete the form three times to provide information on safety culture at inpatient sites. The form will be completed by 167 respondents, three times, and requires 20 minutes to complete.

14. Calibrate Dx Survey —A physician, nurse practitioner, or physician assistant will complete the form four times to provide reflections on their diagnostic performance for 3-5 cases, with additional metrics around time to complete the review and the number of cases reviewed. The form will be completed by 329 respondents, four times, and requires 30 minutes to complete.

15. Clinician Self-Efficacy Survey —A physician, nurse practitioner, or physician assistant will complete the form two times to provide information on their self-efficacy with diagnostic safety case review and improvement. The form will be completed by 329 respondents, two times, and requires 3 minutes to complete.

16. Provider Characteristics Form —A physician, nurse practitioner, or physician assistant will complete the form once to provide information on practitioner type, years in practice, specialty, subspecialty, and percent of time spent in clinical practice. The form will be completed by 986 respondents and requires 1 minute to complete.

17. Patient Toolkit Survey—Provider —A physician, nurse practitioner, or physician assistant will complete the form five times to provide information on provider-perceived skills and quality of communication. The form will be completed by 986 respondents, five times, and requires 2 minutes to complete.

18. Provider Interview Protocol —A physician, nurse practitioner, or physician assistant will participate in an interview once to provide information related to diagnostic safety events; patient safety culture; feasibility, acceptability, utility, adoption, and spread of the Patient Toolkit; and insights into clinician experience. The interview will be completed by up to 50 respondents and requires 45 minutes to complete.

19. Patient Toolkit Survey—Patient —Patients will complete the form only once to provide information on their experience and quality of communication, and demographics information. The form will be completed by 62,500 respondents and requires 5 minutes to complete.

20. Patient Interview Protocol —Patients will participate in an interview once to provide information on reason for visit, provider communication, and other insights into patient experience. The interview will be completed by up to 50 respondents and requires 45 minutes to complete.

For the three-year clearance period, the estimated annualized burden hours for the data collection activities are 8,195.

Exhibit 1—Estimated Annualized Burden Hours

Form name	Number of respondents	Number of responses per respondent	Hours per response	Total burden hours
1: Site Interest Form	1,060	1	6/60	106
2: Site Information Form	265	1	20/60	88
3: Safer Dx Checklist	219	1	15/60	55
4: Exit Interviews Protocol	69	1	10/60	12
5a: SOPS® Medical Office Survey with Diagnostic Safety Supplemental Item Set	109	1	15/60	27
5b: SOPS® Hospital Survey with Diagnostic Safety Supplemental Item Set	110	1	15/60	28
6: Post-training Evaluation Form	1,350	1	3/60	68
7: Post-technical Assistance Evaluation Form	1,350	3	2/60	135
8: Clinical Sustainability Assessment Tool (CSAT)	219	1	15/60	55
9: Implementation Interviews Protocol	438	2	1	876
10: Measure Dx Organizational Self-Assessment	73	1	30/60	37
11: Measure Dx Declaration of Measurement Strategy	73	1	5/60	6
12: Diagnostic Safety Event Report	73	3	1	219
13a: Omnibus Safety and Culture Survey_Medical Offices	162	3	20/60	162
13b: Omnibus Safety and Culture Survey_Hospitals	167	3	20/60	167
14: Calibrate Dx Survey	329	4	30/60	657
15: Clinician Self-Efficacy Survey	329	2	3/60	33
16: Provider Characteristics Form	986	1	1/60	16
17: Patient Toolkit Survey—Provider	986	5	2/60	164
18: Provider Interview Protocol	50	1	45/60	38
19: Patient Toolkit Survey—Patient	62,500	1	5/60	5,208
20: Patient Interview Protocol	50	1	45/60	38
Total				8,195

Exhibit 2 shows the estimated annualized cost burden based on the respondents' time to complete the data collection forms. The total cost burden is estimated to be $457,432.

Exhibit 2—Estimated Annualized Cost Burden

Form name	Number of respondents	Total burden hours	Average hourly wage rate *	Total cost burden
1: Site Interest Form	1,060	106	a  $97.30	$10,314
2: Site Information Form	265	88	a  97.30	8,562
3: Safer Dx Checklist	219	55	a  97.30	5,352
4: Exit Interviews Protocol	69	12	a  97.30	1,168
5a: SOPS® Medical Office Survey with Diagnostic Safety Supplemental Item Set	109	27	a  97.30	2,627
5b: SOPS® Hospital Survey with Diagnostic Safety Supplemental Item Set	110	28	a  97.30	2,724
6: Post-training Evaluation Form	1,350	68	b  102.90	6,997
7: Post-technical Assistance Evaluation Form	1,350	135	b  102.90	13,892
8: Clinical Sustainability Assessment Tool (CSAT)	219	55	a  97.30	5,352
9: Implementation Interviews Protocol	438	876	b  102.90	90,140
10: Measure Dx Organizational Self-Assessment	73	37	b  102.90	3,807
11: Measure Dx Declaration of Measurement Strategy	73	6	b  102.90	617
12: Diagnostic Safety Event Report	73	219	b  102.90	22,535
13a: Omnibus Safety and Culture Survey_Medical Offices	162	162	b  102.90	16,670
13b: Omnibus Safety and Culture Survey_Hospitals	167	167	b  102.90	17,184
14: Calibrate Dx Survey	329	657	c  102.83	67,559
15: Clinician Self-Efficacy Survey	329	33	c  102.83	3,393
16: Provider Characteristics Form	986	16	c  102.83	1,645
17: Patient Toolkit Survey—Provider	986	164	c  102.83	16,864
18: Provider Interview Protocol	50	38	c  102.83	3,908
19: Patient Toolkit Survey—Patient	62,500	5,208	d  29.76	154,990
20: Patient Interview Protocol	50	38	d  29.76	1,131
Total				457,432
* National Compensation Survey: Occupational wages in the United States May 2022, “U.S. Department of Labor, Bureau of Labor Statistics.”
a  Based on the weighted mean hourly wage for physicians (broad) ($121.15; occupation code 29-1210; 60%) and Medical and Health Services Managers ($61.53; Code 11-9111; 40%).
b  Based on the weighted mean hourly wage for physicians (broad) ($121.15; occupation code 29-1210; 70%); nurse practitioners (broad) ($59.94; occupation code 29-1170; 15%); physician assistants (broad) ($60.23; occupation code 29-1070; 10%); and medical and health services managers (broad) ($61.53; Code 11-9111; 5%).
c  Based on the weighted mean hourly wage for physicians (broad) ($121.15; occupation code 29-1210; 70%); nurse practitioners (broad) ($59.94; occupation code 29-1170; 15%); and physician assistants (broad) ($60.23; occupation code 29-1070; 15%).
d  Based on the mean wages for All Occupations (Code 00-0000).

Request for Comments

In accordance with the Paperwork Reduction Act, 44 U.S.C. 3501-3520, comments on AHRQ's information collection are requested with regard to any of the following: (a) whether the proposed collection of information is necessary for the proper performance of AHRQ's health care research and health care information dissemination functions, including whether the information will have practical utility; (b) the accuracy of AHRQ's estimate of burden (including hours and costs) of the proposed collection(s) of information; (c) ways to enhance the quality, utility and clarity of the information to be collected; and (d) ways to minimize the burden of the collection of information upon the respondents, including the use of automated collection techniques or other forms of information technology.

Comments submitted in response to this notice will be summarized and included in the Agency's subsequent request for OMB approval of the proposed information collection. All comments will become a matter of public record.

Comments submitted in response to this notice will be summarized and included in the Agency's subsequent request for OMB approval of the proposed information collection. All comments will become a matter of public record.

Dated: May 16, 2024.

Mamatha Pancholi,

Deputy Director.