2021-01273. Adoption of Recommendations

AGENCY:

Administrative Conference of the United States.

ACTION:

Notice.

SUMMARY:

The Administrative Conference of the United States adopted six recommendations and one official statement at its virtual Seventy-third Plenary Session. The appended recommendations address: (a) Rules on Rulemakings; (b) Protected Materials in Public Rulemaking Dockets; (c) Agency Appellate Systems; (d) Government Contract Bid Protests Before Agencies; (e) Publication of Policies Governing Agency Adjudicators; and (f) Agency Litigation Webpages. The official statement addresses Agency use of Artificial Intelligence.

FOR FURTHER INFORMATION CONTACT:

For Recommendations 2020-1 and 2020-2, Todd Rubin; for Recommendation 2020-3, Gavin Young; for Recommendations 2020-4 and 2020-6, and Statement #20, Mark Thomson; and for Recommendation 2020-5, Leigh Anne Schriever. For each of these actions the address and telephone number are: Administrative Conference of the United States, Suite 706 South, 1120 20th Street NW, Washington, DC 20036; Telephone [phone].

SUPPLEMENTARY INFORMATION:

The Administrative Conference Act, 5 U.S.C. 591-596, established the Administrative Conference of the United States. The Conference studies the efficiency, adequacy, and fairness of the administrative procedures used by Federal agencies and makes recommendations to agencies, the President, Congress, and the Judicial Conference of the United States for procedural improvements (5 U.S.C. 594(1)). For further information about the Conference and its activities, see www.acus.gov. At its virtual Seventy-third Plenary Session on December 16-17, 2020, the Assembly of the Conference adopted six recommendations and one official statement.

Recommendation 2020-1, Rules on Rulemakings. This recommendation encourages agencies to consider issuing rules governing their rulemaking procedures. It identifies subjects that agencies should consider addressing in their rules on rulemakings—without prescribing any particular procedures—and it urges agencies to solicit public input on these rules and make them publicly available.

Recommendation 2020-2, Protected Materials in Public Rulemaking Dockets. This recommendation offers agencies best practices for protecting sensitive personal and confidential commercial information in public rulemaking dockets. It identifies, in particular, best practices for agencies to use when redacting, summarizing, and aggregating comments that contain such information. It also encourages agencies to provide public notices that discourage commenters from submitting such information in the first place.

Recommendation 2020-3, Agency Appellate Systems. This recommendation offers agencies best practices to improve administrative review of hearing-level adjudicative decisions with respect to case selection, decision-making process and procedures, management oversight, and public disclosure and transparency. In doing so, it encourages agencies to identify the objectives of such review and structure their appellate systems to serve those objectives.

Recommendation 2020-4, Government Contract Bid Protests Before Agencies. This recommendation suggests improvements to the procedures governing agency-level procurement contract disputes—commonly called bid protests—under the Federal Acquisition Regulation and agency-specific regulations to make those procedures more simple, transparent, and predictable. It urges agencies to clarify what types of decisions can be the subjects of agency-level bid protests, what processes and deadlines will govern such protests, and who in the agency will decide such protests; make it easier for protesters to get information about the decisions they protest; and publish more data on agency-level protests.

Recommendation 2020-5, Publication of Policies Governing Agency Adjudicators. This recommendation encourages agencies to disclose policies governing the appointment and oversight of adjudicators that bear on their impartiality and constitutional status. It offers best practices on how to provide descriptions of, and access to, such policies on agency websites.

Recommendation 2020-6, Agency Litigation Webpages. This recommendation offers agencies best practices for making their federal court filings and relevant court opinions available to the public on their websites, with particular emphasis on materials from litigation dealing with agency regulatory programs. It provides guidance on the types of litigation materials that will be of greatest interest to the public and on how agencies can disseminate the materials in a way that makes them easy to find.

Statement #20, Agency Use of Artificial Intelligence. This statement identifies issues agencies should consider when adopting, revamping, establishing policies and practices governing, and regularly monitoring artificial intelligence systems. Among the topics it addresses are transparency, harmful biases, technical capacity, procurement, privacy, security, decisional authority, and oversight.

The Appendix below sets forth the full texts of these six recommendations and the official statement. The Conference will transmit the recommendations and statement to affected agencies, Congress, and the Judicial Conference of the United States, as appropriate. The recommendations and statement are not binding, so the entities to which they are addressed will make decisions on their implementation.

The Conference based these recommendations and the statement on research reports that are posted at: https://www.acus.gov/​meetings-and-events/​plenary-meeting/​73rd-plenary-session. Committee-proposed drafts of the recommendations and statement, and public comments received in advance of the plenary session, are also available using the same link.

Dated: January 14, 2021.

Shawne C. McGibbon,

General Counsel.

Appendix—Recommendations and Statement of the Administrative Conference of the United States

Administrative Conference Recommendation 2020-1

Rules on Rulemakings

Adopted December 16, 2020

Numerous agencies have promulgated rules setting forth the policies and procedures they will follow when conducting informal rulemakings under 5 U.S.C. 553.^[1] The rules can cover a variety of practices, including processes for initiating and seeking public input on new rules, coordinating with the Office of Management and Budget and other agencies as a rule is being formulated, and obtaining approval from agency leadership before a proposed rule is issued or finalized. Agencies refer to these rules by different names. This Recommendation calls them “rules on rulemakings.”

Rules on rulemakings vary—in terms of the particular matters they address, their scope and comprehensiveness, and other characteristics—but they share several common features. First, they authoritatively reflect the agency's position as to what procedures it will observe when adopting new rules. By “authoritative,” this Recommendation means that a rule on rulemakings sets forth the procedures that agency officials responsible for drafting and finalizing new rules will follow in at least most cases within the rule on rulemakings' scope, though it may contemplate the possibility that agency leadership could authorize an alternative set of procedures.^[2]

Second, rules on rulemakings do more than simply summarize or explain rulemaking requirements of the Administrative Procedure Act and other statutes, although they often serve an explanatory function at the same time that they set forth the procedures the agencies will follow in conducting rulemakings. Rules on rulemakings set forth additional commitments by an agency concerning how it will conduct rulemakings. And third, agencies disseminate rules on rulemakings publicly rather than only internally. They appear on agency websites and are often published not only in the Federal Register but also in the Code of Federal Regulations (CFR).

Rules on rulemakings can serve at least four important objectives. First, they promote efficiency by ensuring that both agency officials and those outside the agency know where to go to find the agency's rulemaking policies. Second, they promote predictability by informing the public that the agency will follow particular procedures, thereby allowing the public to plan their participation in the rulemaking process accordingly. Third, they promote accountability by ensuring that agency leadership has approved the policies and procedures the agency will follow. And they can also provide accountability in connection with individual rulemakings by creating an internal approval process by which agency leadership reviews proposed and final rules. Finally, they promote transparency by affording the public access to the agency's internal procedures pertaining to its rulemaking process.

In promulgating a rule on rulemakings, an agency may wish to solicit public input to inform the rule's development, even if such a rule is subject to 5 U.S.C. 553's exemption from notice-and-comment procedures as a rule of procedure, general statement of policy, or otherwise. In soliciting public input, agencies may wish to use mechanisms that facilitate more robust participation, including by underrepresented communities.^[3] As the Administrative Conference has acknowledged in past recommendations, public comment can both provide valuable input from the public and enhance public acceptance of an agency's rules.^[4]

An agency may also wish to publish its rule on rulemakings in the CFR. Doing so can enhance transparency and facilitate accountability. Importantly, publishing a rule on rulemakings in the CFR does not, by itself, make the rule on rulemakings judicially enforceable.^[5]

This Recommendation does not seek to resolve whether, when, or on what legal bases a court might enforce a rule on rulemakings against an agency.^[6]

Recommendation

1. Agencies should consider promulgating rules on rulemakings setting forth the policies and procedures they will follow in informal rulemaking under 5 U.S.C. 553.

2. In issuing rules on rulemakings, agencies should consider including provisions addressing the following topics (which reflect topics frequently covered in existing rules on rulemakings):

a. Procedures prior to the issuance of a notice of proposed rulemaking;

b. Procedures connected with the notice-and-comment process;

c. Procedures connected with the presidential review process, if applicable;

d. Procedures for handling post-comment period communications;

e. Internal approval procedures for issuing and finalizing rules; and

f. Procedures for reassessing existing rules.

The appendix gives examples of particular subtopics agencies may wish to consider under each of these topics.

3. Agencies should make rules on rulemakings available in a prominent, easy-to-find place on the portion of their websites dealing with rulemaking matters. Additionally, agencies should consider publishing them in the Federal Register and the Code of Federal Regulations. When posting rules on rulemakings on their websites, agencies should use techniques like linked tabs, pull-down menus, indexing, tagging, and sorting tables to ensure that relevant documents are easily findable. Agencies should also design their search engines to allow people to easily identify relevant documents.

4. In addition to issuing rules on rulemakings, agencies should consider explaining in accessible language how the rulemaking process works in order to educate the public. Such explanations might be integrated within a rule on rulemakings or might be contained in separate explanatory documents (e.g., documents identifying frequently asked questions). When providing such explanations, an agency should, to the extent practicable, distinguish between procedures it intends to follow and material provided purely by way of background.

5. Agencies should consider a broad range of means of seeking public input on rules on rulemakings, even if the Administrative Procedure Act does not require it.

6. Agencies should consider the extent to which procedures required by a rule on rulemakings should be made internally waivable and, if so, by whom. For example, they might consider drafting a rule on rulemakings in a way that allows high-level agency officials to permit other officials to use alternative procedures.

Appendix

Non-Exhaustive List of Topics for Agencies To Consider Including Within Their Rules on Rulemakings

(a) Procedures Prior to the Issuance of a Notice of Proposed Rulemaking

Subtopic Examples:

(1) Regulatory planning; ^[7]

(2) Issuing advance notices of proposed rulemaking and obtaining feedback from members of the public using means other than the notice-and-comment process, such as requests for information and focus groups; ^[8]

(3) Accepting, reviewing, and responding to petitions for rulemaking; ^[9]

(4) Considering options besides rulemaking;

(5) Performing ex ante regulatory analyses (e.g., benefit-cost analysis and regulatory flexibility analysis); ^[10]

(6) Using plain language in regulatory drafting; ^[11]

(7) Preparing for potential judicial review of rulemakings, including deciding whether to make any of the provisions of a rule severable; ^[12]

(8) Conducting negotiated rulemaking; ^[13] and

(9) Establishing an effective date for rules.

(b) Procedures Connected With the Notice-and-Comment Process

Subtopic Examples:

(1) Materials to be published on Regulations.gov with the notice; ^[14]

(2) Minimum comment periods to be allowed; ^[15]

(3) Policies on ex parte contacts; ^[16]

(4) Handling external merits communications not filed as comments;

(5) Incorporating standards by reference; ^[17]

(6) Using social media to engage the public in rulemaking; ^[18]

(7) Obtaining feedback from American Indian tribes, other historically underrepresented or under-resourced groups, and state and local governments; ^[19]

(8) Posting, analyzing, and responding to public comments, including comments that may contain confidential commercial information, protected personal information, or other kinds of sensitive submissions; ^[20]

(9) Waiving or invoking of Administrative Procedure Act exemptions to notice and comment; ^[21] and

(10) Using interim final rules or direct final rules.^[22]

(c) Procedures Connected With the Presidential Review Process, if Applicable

Subtopic Examples:

(1) Interacting with the Office of Information and Regulatory Affairs, the Office of the Federal Register, the Regulatory Information Service Center, the Small Business Administration's Office of Advocacy, and other offices with government-wide rulemaking responsibilities;

(2) Participating in the interagency review process; and

(3) Procedures related to international regulatory cooperation.^[23]

(d) Procedures for Handling Post-Comment Period Communications

Subtopic Examples:

(1) Provisions pertaining to reply comments ^[24] and

(2) Handling late-filed comments.^[25]

(e) Internal Approval Procedures for Issuing and Finalizing Rules

Subtopic Examples:

(1) Procedures for submitting rules to offices with legal, economic, and other responsibilities within the agency for review ^[26] and

(2) Procedures for submitting rules to the relevant agency official for final approval.

(f) Procedures for Reassessing Existing Rules

Subtopic Examples:

(1) Issuing regulatory waivers and exemptions; ^[27]

(2) Engaging in retrospective review of rules; ^[28]

(3) Maintaining and preserving rulemaking records, including transparency of such records and the handling of confidential commercial information, protected personal information, or other kinds of sensitive information contained therein; ^[29] and

(4) Handling rules that have been vacated or remanded without vacatur.^[30]

Administrative Conference Recommendation 2020-2

Protected Materials in Public Rulemaking Dockets

Adopted December 16, 2020

As part of the rulemaking process, agencies create public rulemaking dockets, which consist of all rulemaking materials agencies have: (1) Proactively published online or (2) made available for public inspection in a reading room. Public rulemaking dockets include materials agencies generate themselves and comments agencies receive from the public. Their purpose is to provide the public with the information that informed agencies' rulemakings.^[1]

The Administrative Conference has issued several recommendations to help agencies balance the competing considerations of transparency and confidentiality in managing their public rulemaking dockets.^[2] This project builds on these recommendations.

The scope of the Recommendation is limited to personal information and confidential commercial information that agencies have decided to withhold from their public rulemaking dockets, which this Recommendation calls “protected material.” The Recommendation specifies how agencies should consider handling protected material. For purposes of this Recommendation, personal information is information about an individual including his or her education, financial transactions, medical history, criminal or employment history, or similarly sensitive information, and that contains his or her name, or the identifying number, symbol, or other identifying particular assigned to the individual.^[3] Confidential commercial information is commercial information that is customarily kept private, or at least closely held, by the person or business providing it.^[4] Other types of information, such as national security information and copyrighted materials, are beyond the Recommendation's scope. The Recommendation is also limited to addressing procedures for protecting materials that agencies decide warrant protection. It is not intended to define the universe of protected materials. In particular, the Recommendation does not address any issue that may arise if agencies choose to rely on protected material in explaining their rulemakings, whether in notices of proposed rulemaking, regulatory impact analyses, or otherwise.

Agencies accept public comments for their public rulemaking dockets primarily through Regulations.gov, their own websites, and email. Regulations.gov and many agency websites that accept comments expressly notify the public that agencies may publish the information submitted in public comments.^[5] When people submit comments to agencies, however, agencies typically do not immediately publish the comments. Instead, agencies generally take time to screen comments before publishing them. Most agencies perform at least some kind of screening during this period.

For all agencies, whether to withhold or disclose protected material is governed by various laws: Some mandate disclosure, some mandate withholding, and some leave agencies with substantial discretion in deciding whether to disclose. Although a full description of those laws is beyond the scope of this Recommendation, a brief overview of at least some of this body of law helps to identify the issues agencies face.

The Administrative Procedure Act requires agencies to “give interested persons an opportunity to participate in rulemaking through submission of written data, views, or arguments.” ^[6] The United States Court of Appeals for the D.C. Circuit has interpreted this provision to ordinarily require that agencies make publicly available the critical information—including studies, data, and methodologies—underlying proposed rules.^[7]

The Privacy Act and the Trade Secrets Act place limits on the disclosure norm discussed above. Generally, the Privacy Act prevents agencies from disclosing any information about a person, such as medical records, educational background, and employment history, contained in agencies' systems of records, without that person's written consent.^[8] The Trade Secrets Act generally prevents agencies from disclosing trade secrets and other kinds of confidential commercial information, such as corporate losses and profits.^[9]

Both the Privacy Act and the Trade Secrets Act have exceptions. For the Privacy Act, the main exception relevant to this Recommendation is for information required to be released under the Freedom of Information Act (FOIA).^[10] The Trade Secrets Act only has one exception, which covers any materials authorized to be disclosed by statute (including FOIA) or regulation.^[11] Whether a particular piece of personal or confidential commercial information meets one of these exceptions often involves a complex determination that depends upon the exact type of information at issue and its contemplated use, and agencies must determine the applicability of the exceptions on a case-by-case basis. For example, whether FOIA authorizes disclosure of confidential commercial information may turn in part on whether agencies in receipt of the information assured submitters that the information would be withheld from the public.^[12] If agencies offer assurances that they will not disclose confidential commercial information, agencies and submitters may rely on those assurances as a defense against compelled disclosure under FOIA. In many cases, agencies assure companies that they will not disclose such information in order to encourage companies to submit it.

Particular cases are governed by specific requirements of law, not broad categorical labels. But agencies often consider certain categories of personal information and confidential commercial information to be protected material (e.g., trade secrets, social security numbers, bank account numbers, passport numbers, addresses, email addresses, medical information, and information concerning a person's finances).

There are many ways protected material may arrive at the agency in a rulemaking. A person might submit his or her own information, intentionally or unintentionally, and then ask the agency not to disclose it. A third party might submit another person's information, with or without that person's knowledge. A company might submit a document containing its own confidential commercial information, intentionally or unintentionally, with or without the agency's prior assurance of protection. Or a company might submit another company's or person's information. Depending on the information in question and the manner in which it was submitted, there may be issues of waiver of statutory protection. Such questions, like all questions regarding the substance of the laws governing protected material, are beyond this Recommendation's scope, but they illustrate the various considerations that agencies and the public often face in the submission and handling of such material.

This Recommendation proposes steps agencies can take to withhold protected materials from their public rulemaking dockets while still providing the public with the information upon which agencies relied in formulating proposed rules.^[13]

Recommendation

Recommendations for All Agencies

1. To reduce the risk that agencies will inadvertently disclose protected material, agencies should describe what kinds of personal and confidential commercial information qualify as protected material and should clearly notify the public about their treatment of protected material. An agency's notifications should:

a. Inform members of the public that comments are generally subject to public disclosure, except when disclosure is limited by law;

b. Inform members of the public whether the agency offers assurances of protection from disclosure for their confidential commercial information and, if so, how to identify such information for the agency;

c. Provide guidance to the public concerning the submission of protected material that pertains to third parties, including instructions that the disclosure of some protected material may be prohibited by law;

d. Advise members of the public to review their comments for the material identified above in (c) and, if they find such material, to remove any such material that is not essential to the comment;

e. Inform members of the public that they may request, during the period between when a comment is received and when it is made public, that protected material they inadvertently submitted be withheld from the public rulemaking docket;

f. Inform members of the public that they may request, after the agency has published any comment, that protected material pertaining to themselves or to their dependents within the comment be removed from the public rulemaking docket; and

g. Inform members of the public that the agency reserves the right to redact or aggregate any part of a comment if the agency determines that it constitutes protected material, or may withhold a comment in its entirety if it determines that redaction or aggregation would insufficiently prevent the disclosure of this material.

2. Agencies should include the notifications described in Paragraph 1, or a link to those notifications, in at least the following places:

a. Within the rulemaking documents on which agencies request comments, such as a notice of proposed rulemaking or an advance notice of proposed rulemaking;

b. On agencies' own comment submission forms, if agencies have them;

c. Within any automatic emails that agencies send acknowledging receipt of a comment;

d. On any part of agencies' websites that describe their rulemaking process or within any rules on rulemakings they may have, as described in Recommendation 2020-1, Rules on Rulemakings; and

e. Within any notices of public meetings pertaining to a rule.

3. The General Services Administration's eRulemaking Program Management Office should work with agencies that participate in Regulations.gov to include or refer to the notifications described in Paragraph 1 within any automated emails Regulations.gov sends acknowledging receipt of a comment.

4. If a submitter notifies an agency that the submitter inadvertently included protected material in the submitter's comment, the agency should act as promptly as possible to determine whether such material warrants withholding from the public rulemaking docket and, if so, withhold it from the public rulemaking docket, or, if already disclosed, remove it from the public rulemaking docket. If an agency determines that such material does not qualify as protected, it should promptly notify the submitter of this finding with a brief statement of reasons.

5. Agencies should allow third parties to request that protected material pertaining to themselves or a dependent be removed from the public rulemaking docket. Agencies should review such requests and, upon determining that the material subject to the request qualifies as protected material, should remove it from the public rulemaking docket as promptly as possible. If an agency determines that the material does not qualify as protected, it should promptly notify the requestor of this finding with a brief statement of reasons.

Recommendations for Agencies That Screen Comments for Protected Material Before Publication in the Public Rulemaking Docket

6. Agencies that screen comments for protected material before publication in the public rulemaking docket, either as required by law or as a matter of discretion, should redact the protected material and publish the rest of the comment. Redaction should be thorough enough to prevent the public from discerning the redacted material, but not so broad as to prevent the public from viewing non-protected material.

7. If redaction is not feasible within a comment, agencies should consider presenting the data in a summarized form.

8. If redaction is not feasible across multiple, similar comments, agencies should consider presenting any related information in an aggregated form. Agencies should work with data science experts and others in relevant disciplines to ensure that aggregation is thorough enough to prevent someone from disaggregating the information.

9. If the approaches identified in Paragraphs 6-8 would still permit a member of the public to identify protected material, agencies should withhold the comment in its entirety. When doing so, they should describe the withheld material for the public in as much detail as possible without compromising its confidentiality.

10. When deciding whether and how to redact, aggregate, or withhold protected material, agencies should explore using artificial intelligence-based tools to aid in identifying protected material. Agencies should consult with private sector experts and technology-focused agencies, such as the General Services Administration's Technology Transformation Service and the Office of Management and Budget's United States Digital Service, to determine which tools are most appropriate and how they can best be deployed given the agencies' resources.

Recommendations for Agencies That Offer Assurances of Protection From Disclosure of Confidential Commercial Information

11. Agencies that offer assurances of protection from disclosure of confidential commercial information should decide how they will offer such assurances. Agencies can choose to inform submitters, directly upon submission, that they will withhold confidential commercial information from the public rulemaking docket; post a general notice informing submitters that confidential commercial information will be withheld from the public rulemaking docket; or both.

12. Such agencies should adopt policies to help them identify such information. Agencies should consider including the following, either in tandem or as alternatives, as part of their policies, including within any rules on rulemakings they may have, as described in Recommendation 2020-1, Rules on Rulemakings:

a. Instructing submitters to identify clearly that the document contains confidential commercial information;

b. Instructing submitters to flag the particular text within the document that constitutes confidential commercial information; and

c. Instructing submitters to submit both redacted and unredacted versions of a comment that contains confidential commercial information.

Administrative Conference Statement #20

Agency Use of Artificial Intelligence

Adopted December 16, 2020

Artificial intelligence (AI) techniques are changing how government agencies do their work.^[1] Advances in AI hold out the promise of lowering the cost of completing government tasks and improving the quality, consistency, and predictability of agencies' decisions. But agencies' uses of AI also raise concerns about the full or partial displacement of human decision making and discretion.

Consistent with its statutory mission to promote efficiency, participation, and fairness in administrative processes,^[2] the Administrative Conference offers this Statement to identify issues agencies should consider when adopting or modifying AI systems and developing practices and procedures for their use and regular monitoring. The Statement draws on a pair of reports commissioned by the Administrative Conference,^[3] as well as the input of AI experts from government, academia, and the private sector (some ACUS members) provided at meetings of the ad hoc committee of the Administrative Conference that proposed this Statement.

The issues addressed in this Statement implicate matters involving law, policy, finances, human resources, and technology. To minimize the risk of unforeseen problems involving an AI system, agencies should, throughout an AI system's lifespan, solicit input about the system from the offices that oversee these matters. Agencies should also keep in mind the need for public trust in their practices and procedures for use and regular monitoring of AI technologies.

1. Transparency

Agencies' efforts to ensure transparency in connection with their AI systems can serve many valuable goals. When agencies set up processes to ensure transparency in their AI systems, they should consider publicly identifying the processes' goals and the rationales behind them. For example, agencies might prioritize transparency in the service of legitimizing its AI systems, facilitating internal or external review of its AI-based decision making, or coordinating its AI-based activities. Different AI systems are likely to satisfy some transparency goals more than others. When possible, agencies should use metrics to measure the performance of their AI-transparency processes.

In setting transparency goals, agencies should consider to whom they should be transparent. For instance, depending on the nature of their operations, agencies might prioritize transparency to the public, courts, Congress, or their own officials.

The appropriate level or nature of transparency and interpretability in agencies' AI systems will also depend on context. In some contexts, such as adjudication, reason-giving requirements may call for a higher degree of transparency and interpretability from agencies regarding how their AI systems function. In other contexts, such as enforcement, agencies' legitimate interests in preventing gaming or adversarial learning by regulated parties could militate against providing too much information (or specific types of information) to the public about AI systems' processes. In every context, agencies should consider whether particular laws or policies governing disclosure of information apply.

In selecting and using AI techniques, agencies should be cognizant of the degree to which a particular AI system can be made transparent to appropriate people and entities, including the general public. There may be tradeoffs between explainability and accuracy in AI systems, so that transparency and interpretability might sometimes weigh in favor of choosing simpler AI models. The appropriate balance between explainability and accuracy will depend on the specific context, including agencies' circumstances and priorities.

The proprietary nature of some AI systems may also affect the extent to which they can be made transparent. When agencies' AI systems rely on proprietary technologies or algorithms the agencies do not own, the agencies and the public may have limited access to the information about the AI techniques. Agencies should strive to anticipate such circumstances and address them appropriately, such as by working with outside providers to ensure they will be able to share sufficient information about such a system. Agencies should not enter into contracts to use proprietary AI systems unless they are confident that actors both internal and external to the agencies will have adequate access to information about the systems.

2. Harmful Bias

At their best, AI systems can help agencies identify and reduce the impact of harmful biases.^[4] Yet they can also unintentionally create or exacerbate those biases by encoding and deploying them at scale. In deciding whether and how to deploy an AI system, agencies should carefully evaluate the harmful biases that might result from the use of the AI system as well as the biases that might result from alternative systems (such as an incumbent system that the AI system would augment or replace). Because different types of bias pose different types of harms, the outcome of the evaluation will depend on agencies' unique circumstances and priorities and the consequences posed by those harms in those contexts.

AI systems can be biased because of their reliance on data reflecting historical human biases or because of their designs. Biases in AI systems can increase over time through feedback. That can occur, for example, if the use of a biased AI system leads to systematic errors in categorizations, which are then reflected in the data set or data environment the system uses to make future predictions. Agencies should be mindful of the interdependence of the models, metrics, and data that underpin AI systems.

Identifying harmful biases in AI systems can pose challenges. To identify and mitigate biases, agencies should, to the extent practical, consider whether other data or methods are available. Agencies should periodically examine and refresh AI algorithms and other protocols to ensure that they remain sufficiently current and reflect new information and circumstances relevant to the functions they perform.

Data science techniques for identifying and mitigating harmful biases in AI systems are developing. Agencies should stay up to date on developments in the field of AI, particularly on algorithmic fairness; establish processes to ensure that personnel that reflect various disciplines and relevant perspectives are able to inspect AI systems and their decisions for indications of harmful bias; test AI systems in environments resembling the ones in which they will be used; and make use of internal and external processes for evaluating the risks of harmful bias in AI systems and for identifying such bias.

3. Technical Capacity

AI systems can help agencies conserve resources, but they can also require substantial investments of human and financial capital. Agencies should carefully evaluate the short- and long-term costs and benefits of an AI system before committing significant resources to it. Agencies should also ensure they have access to the technical expertise required to make informed decisions about the type of AI systems they require; how to integrate those systems into their operations; and how to oversee, maintain, and update those systems.

Given the data science field's ongoing and rapid development, agencies should consider cultivating an AI-ready workforce, including through recruitment and training efforts that emphasize AI skills. When agency personnel lack the skills to develop, procure, or maintain AI systems that meet agencies' needs, agencies should consider other means of expanding their technical expertise, including by relying on tools such as the Intergovernmental Personnel Act,^[5] prize competitions, cooperative research and development agreements with private institutions or universities, and consultation with external technical advisors and subject-matter experts.

4. Obtaining AI Systems

Decisions about whether to obtain an AI system can involve important trade-offs. Obtaining AI systems from external sources might allow agencies to acquire more sophisticated tools than they could design on their own, access those tools sooner, and save some of the up-front costs associated with developing the technical capacity needed to design AI systems.^[6] Creating AI tools within agencies, by contrast, might yield tools that are better tailored to the agencies' particular tasks and policy goals. Creating AI systems within agencies can also facilitate development of internal technical capability, which can yield benefits over the lifetime of the AI systems and in other technological tasks the agencies may confront.

Certain government offices are available to help agencies with decisions and actions related to technology.^[7] Agencies should make appropriate use of these resources when obtaining an AI system. Agencies should also consider the cost and availability of the technical support necessary to ensure that an AI system can be maintained and updated in a manner consistent with its expected life cycle and service mission.

5. Data

AI systems require data, often in vast quantities. Agencies should consider whether they have, or can obtain, data that appropriately reflect conditions similar to the ones the agencies' AI systems will address in practice; whether the agencies have the resources to render the data into a format that can be used by the agencies' AI systems; and how the agencies will maintain the data and link them to their AI systems without compromising security or privacy. Agencies should also review and consider statutes and regulations that impact their uses of AI as a potential collector and consumer of data.^[8]

6. Privacy

Agencies have a responsibility to protect privacy with respect to personally identifiable information in AI systems. In a narrow sense, this responsibility demands that agencies comply with requirements related to, for instance, transparency, due process, accountability, and information quality and integrity established by the Privacy Act of 1974, Section 208 of the E-Government Act of 2002, and other applicable laws and policies.^[9] More broadly, agencies should recognize and appropriately manage privacy risks posed by an AI system. Agencies should consider privacy risks throughout the entire life cycle of an AI system from development to retirement and assess those risks, as well as associated controls, on an ongoing basis. In designing and deploying AI systems, agencies should consider using relevant privacy risk management frameworks developed through open, multi-stakeholder processes.^[10]

7. Security

Agencies should consider the possibility that AI systems might be hacked, manipulated, fooled, evaded, or misled, including through manipulation of training data and exploitation of model sensitivities. Agencies must ensure not only that their data are secure, but also that their AI systems are trained on those data in a secure manner, make forecasts based on those data in a secure way, and otherwise operate in a secure manner. Agencies should regularly consider and evaluate the safety and security of AI systems, including resilience to vulnerabilities, manipulation, and other malicious exploitation. In designing and deploying AI systems, agencies should consider using relevant government guidance or voluntary consensus standards and frameworks developed through open, multi-stakeholder processes.^[11]

8. Decisional Authority

Agencies should be mindful that most AI systems will involve human beings in a range of capacities—as operators, customers, overseers, policymakers, or interested members of the public. Human factors may sometimes undercut the value of using AI systems to make certain determinations. There is a risk, for example, that human operators will devolve too much responsibility to AI systems and fail to detect cases in which the AI systems yield inaccurate or unreliable determinations. That risk may be acceptable in some settings—such as when the AI system has recently been shown to perform significantly better than alternatives—but unacceptable in others.

Similarly, if agency personnel come to rely reflexively on algorithmic results in exercising discretionary powers, use of an AI system could have the practical effect of curbing the exercise of agencies' discretion or shifting it from the person who is supposed to be exercising it to the system's designer. Agencies should beware of such potential shifts of practical authority and take steps to ensure that appropriate officials have the knowledge and power to be accountable for decisions made or aided by AI techniques.

Finally, there may be some circumstances in which, for reasons wholly apart from decisional accuracy, agencies may wish to have decisions be made without reliance on AI techniques, even if the law does not require it. In some contexts, accuracy and fairness may not be the only relevant values at stake. In making decisions about their AI systems, agencies may wish to consider whether people will perceive the systems as unfair, inhumane, or otherwise unsatisfactory.^[12]

9. Oversight

It is essential that agencies' AI systems be subject to appropriate and regular oversight throughout their lifespans. There are two general categories of oversight: External and internal. Agencies' mechanisms of internal oversight will be shaped by the demands of external oversight. Agencies should be cognizant of both forms of oversight in making decisions about their AI systems.

External oversight of agencies' uses of AI systems can come from a variety of government sources, including inspectors general, externally facing ombuds, the Government Accountability Office, and Congress. In addition, because agencies' uses of AI systems might lead to litigation in a number of circumstances, courts can also play an important role in external oversight. Those affected by an agency's use of an AI system might, for example, allege that use of the system violates their right to procedural due process.^[13] Or they might allege that the AI system's determination violated the Administrative Procedure Act (APA) because it was arbitrary and capricious.^[14] When an AI system narrows the discretion of agency personnel, or fixes or alters the legal rights and obligations of people subject to the agency's action, affected people or entities might also sue on the ground that the AI system is a legislative rule adopted in violation of the APA's requirement that legislative rules go through the notice-and-comment process.^[15] Agencies should consider these different forms of potential external oversight as they are making and documenting decisions and the underlying processes for these AI systems.

Agencies should also develop their own internal evaluation and oversight mechanisms for their AI systems, both for initial approval of an AI system and for regular oversight of the system, taking into account their system-level risk management, authorization to operate, regular monitoring responsibilities, and their broader enterprise risk management responsibilities.^[16] Successful internal oversight requires advance and ongoing planning and consultation with the various offices in an agency that will be affected by the agency's use of an AI system, including its legal, policy, financial, human resources, internally-facing ombuds, and technology offices. Agencies' oversight plans should address how the agencies will pay for their oversight mechanisms and how they will respond to what they learn from their oversight.

Agencies should establish a protocol for regularly evaluating AI systems throughout the systems' lifespans. That is particularly true if a system or the circumstances in which it is deployed are liable to change over time. In these instances, review and explanation of the system's functioning at one stage of development or use may become outdated due to changes in the system's underlying models. To enable that type of oversight, agencies should monitor and keep track of the data being used by their AI systems, as well as how the systems use those data. Agencies may also wish to secure input from members of the public or private evaluators to improve the likelihood that they will identify defects in their AI systems.

To make their oversight systems more effective, agencies should clearly define goals for their AI systems. The relevant question for oversight purposes will often be whether the AI system outperforms alternatives, which may require agencies to benchmark their systems against the status quo or some hypothetical state of affairs.

Finally, AI systems can affect how agencies' staffs do their jobs, particularly as agency personnel grow to trust and rely on the systems. In addition to evaluating and overseeing their AI systems, agencies should pay close attention to how agency personnel interact with those systems.

Administrative Conference Recommendation 2020-3

Agency Appellate Systems

Adopted December 16, 2020

In Recommendation 2016-4,^[1] the Administrative Conference offered best practices for evidentiary hearings in administrative adjudications. Paragraph 26 recommended that agencies provide for “higher-level review” (or “agency appellate review”) of the decisions of hearing-level adjudicators.^[2] This Recommendation offers best practices for such review. The Administrative Conference intends this Recommendation to cover appellate review of decisions resulting from (1) hearings governed by the formal hearing provisions of the Administrative Procedure Act (APA) and (2) evidentiary hearings that are not governed by those provisions but are required by statute, regulation, or executive order. Agencies may also decide to apply this Recommendation to appellate review of decisions arising from other hearings, depending on their level of formality.

Appellate review of hearing-level decisions can be structured in numerous ways. Two structures are most common. In the first, litigants appeal directly to the agency head, which may be a multi-member board or commission. In the second, litigants appeal to an appellate adjudicator or group of adjudicators—often styled as a board or council—sitting below the agency head. The appellate decision may be the agency's final action or may be subject to further appeal within the agency (usually to the agency head).

The Administrative Conference has twice before addressed agency appellate review. In Recommendations 68-6 and 83-3, it provided guidance to agencies when establishing new, and reviewing existing, organizational structures of appellate review.^[3] Both recommendations focused on the selection of “delegates”—individual adjudicators, review boards composed of multiple adjudicators, or panels composed of members of a multi-member agency—to exercise appellate review authority vested in agency heads (including boards and commissions). Recommendation 83-3 also addressed when agencies should consider providing appellate review as a matter of right and when as a matter of discretion, and, in the case of the latter, under what criteria.

With the exception of the appropriate standard for granting review, this Recommendation's focus lies elsewhere. It addresses, and offers best practices with respect to, the following subjects: First, an agency's identification of the purpose or objective served by its appellate review; second, its selection of cases for appellate review, when review is not required by statute; third, its procedures for review; fourth, its appellate decision-making processes; fifth, its management, administration, and bureaucratic oversight of its appellate system; and sixth, its public disclosure of information about its appellate system.^[4]

Most importantly, this Recommendation begins by suggesting that agencies identify, and publicly disclose, the purpose(s) or objective(s) of their appellate systems. Appellate systems may have different purposes, and any given appellate system may have multiple purposes. Purposes or objectives can include the correction of errors, inter-decisional consistency of decisions, policymaking, political accountability, management of the hearing-level adjudicative system, organizational effectiveness and systemic awareness, and the reduction of litigation in federal courts. The identification of purpose is important both because it dictates (or should dictate) how an agency administers its appellate system—including what cases it hears and under what standards of review it decides them—and provides a standard against which an agency's performance can be evaluated.

This Recommendation proceeds from the recognition that agency appellate systems vary enormously—as to their purposes or objectives, governing substantive law, size, and resources—and that what may be a best practice for one system may not always be the best practice for another. In offering the best practices that follow, moreover, the Administrative Conference recognizes that (1) an agency's procedural choices may sometimes be constrained by statute and (2) available resources and personnel policies may dictate an agency's decision as to whether and how to implement the best practices that follow. The Administrative Conference makes this Recommendation subject to these important qualifications.

Recommendation

Objectives of Appellate Review

1. Agencies should identify the objective(s) of appellate review; disclose those objectives in procedural regulations; and design rules and processes, especially for scope and standard of review, to serve them.

Procedures for Appellate Review

2. Agencies should promulgate and publish procedural regulations governing agency appellate review in the Federal Register and codify them in the Code of Federal Regulations. These regulations should cover all significant procedural matters pertaining to agency appellate review, including but not limited to the following:

a. The objectives of the agency's appellate review system;

b. The timing and procedures for initiating review, including any available interlocutory review;

c. The standards for granting review, if review is discretionary;

d. The standards for permitting participation by interested persons and amici;

e. The standard of review;

f. The allowable and required submissions by litigants and their required form and contents;

g. The procedures and criteria for designating decisions as precedential and the legal effect of such designations;

h. The record on review and the opportunity, if any, to submit new evidence;

i. The availability of oral argument or other form of oral presentation;

j. The standards of and procedures for reconsideration and reopening, if available;

k. Any administrative or issue exhaustion requirements that must be satisfied before seeking agency appellate or judicial review, including whether agency appellate review is a mandatory prerequisite to judicial review;

l. Openness of proceedings to the public and availability of video or audio streaming or recording;

m. In the case of multi-member appellate boards, councils, and similar entities, the authority to assign decision-making authority to fewer than all members (e.g., panels); and

n. Whether seeking agency appellate review automatically stays the effectiveness of the appealed agency action until the appeal is resolved (which may be necessary for appellate review to be mandatory, see 5 U.S.C. 704), and, if not, how a party seeking agency appellate review may request such a stay and the standards for deciding whether to grant it.

3. Agencies should include in the procedural regulations governing their appellate programs: (a) A brief statement or explanation of each program's review authority, structure, and decision-making components; and (b) for each provision based on a statutory source, an accompanying citation to that source.

4. When revising existing or adopting new appellate rules, agencies should consider the appellate rules (Rules 400-450) in the Administrative Conference's Model Rules of Agency Adjudication (rev. 2018).

5. When materially revising existing or adopting new appellate rules, agencies should use notice-and-comment procedures or other mechanisms for soliciting public input, notwithstanding the procedural rules exemption of 5 U.S.C. 553(b)(A), unless the costs clearly outweigh the benefits of doing so.

Case Selection for Appellate Review

6. Based on the agency-specific objectives of appellate review, agencies should decide whether the granting of review should be mandatory or discretionary (assuming they have statutory authority to decide); if discretionary, the criteria for granting review should track the objectives of the appellate system, and they should be published in the procedural regulations.

7. Agencies should consider implementing procedures for sua sponte appellate review of non-appealed hearing-level decisions, as well as for the referral of cases or issues by hearing-level adjudicators to the appellate entity for interlocutory review.

Appellate Decision-Making Processes and Decisions

8. Whenever possible, agencies should consider maintaining electronic case management systems that ensure that hearing records are easily accessible to appellate adjudicators. Such systems may include the capability for electronic filing.

9. Although the randomized assignment of cases to appellate adjudicators is typically an appropriate docketing method for an agency appellate system, agencies should consider the potential benefits of sorting and grouping appeals on the appellate docket, such as reduced case processing times and more efficient use of adjudicators', staff attorneys', and law clerks' skills and time. Criteria for sorting and grouping cases may include the size of a case's record, complexity of a case's issues, subject matter of a case, and similarity of a case's legal issues to those of other pending cases.

10. Consistent with the objectives of the agency's appellate system and in light of the costs of time and resources, agencies should consider adopting an appellate model of judicial review in which the standard of review is not de novo with respect to findings of fact and application of law to facts. For similar reasons, many agencies should consider limiting the introduction of new evidence on appeal that is not already in the administrative record from the hearing-level adjudication.

11. Taking agency resources into account, agencies should emphasize concision, readability, and plain language in their appellate decisions and explore the use of decision templates, summary dispositions, and other quality-improving measures.

12. Agencies should establish clear criteria and processes for identifying and selecting appellate decisions as precedential, especially for appellate systems with objectives of policymaking or inter-decisional consistency.

13. Agencies should assess the value of oral argument and amicus participation in their appellate system based on the agencies' identified objectives for appellate review and should establish rules governing both. Criteria that may favor oral argument and amicus participation include issues of high public interest; issues of concern beyond the parties to the case; specialized or technical matters; and a novel or substantial question of law, policy, or discretion.

Administration, Management, and Bureaucratic Oversight

14. Agency appellate systems should promptly transmit their precedential decisions to all appellate program adjudicators and, directly or through hearing-level programs, to hearing-level adjudicators (as appropriate). Appellate programs should include in their transmittals, when feasible, brief summaries of the decision.

15. Agencies should notify their adjudicators of significant federal court decisions reviewing the agencies' decisions and, when providing notice, explain the significance of those decisions to the program. As appropriate, agencies should notify adjudicators if the agency will not acquiesce in a particular decision of the federal courts of appeals.

16. Agencies in which decision making relies extensively on their own precedential decisions should consider preparing or having prepared indexes and digests—with annotations and comments, as appropriate—to identify those decisions and their significance.

17. As appropriate, agency appellate systems should communicate with agency rule-writers and other agency policymakers—and institutionalize communication mechanisms—to address whether recurring issues in their decisions should be addressed by rule rather than precedential case-by-case adjudication.

18. The Office of the Chairman of the Administrative Conference should provide for, as authorized by 5 U.S.C. 594(2), the “interchange among administrative agencies of information potentially useful in improving” agency appellate systems. The subjects of interchange might include electronic case management systems, procedural innovations, quality-assurance reviews, and common management problems.

Public Disclosure and Transparency

19. Agencies should disclose on their websites any rules (sometimes styled as “orders”), and statutes authorizing such rules, by which an agency head has delegated review authority to appellate adjudicators.

20. Regardless of whether the Government in the Sunshine Act (5 U.S.C. 552b) governs their appellate review system, agencies should consider announcing, livestreaming, and maintaining video recordings on their websites of appellate proceedings (including oral argument) that present significant legal and policy issues likely to be of interest to regulated parties and other members of the public. Brief explanations of the issues to be addressed by oral argument may usefully be included in website notices of oral argument.

21. Agencies should include on their websites brief and accessibly written explanations as to how their internal decision-making processes work and, as appropriate, include links to explanatory documents appropriate for public disclosure. Specific subjects that agencies should consider addressing include: The process of assigning cases to adjudicators (when fewer than all of the programs' adjudicators participate in a case), the role of staff, and the order in which cases are decided.

22. When posting decisions on their websites, agencies should distinguish between precedential and non-precedential decisions. Agencies should also include a brief explanation of the difference.

23. When posting decisions on their websites, agencies should consider including, as much as practicable, brief summaries of precedential decisions and, for precedential decisions at least, citations to court decisions reviewing them.

24. Agencies should include on their websites any digests and indexes of decisions they maintain. It may be appropriate to remove material exempt from disclosure under the Freedom of Information Act or other laws.

25. Agencies should affirmatively solicit feedback concerning the functioning of their appellate systems and provide a means for doing so on their websites.

Administrative Conference Recommendation 2020-4

Government Contract Bid Protests Before Agencies

Adopted December 17, 2020

Federal law establishes policies and procedures governing how federal executive agencies procure supplies and services.^[1] The primary source of these policies and procedures is the Federal Acquisition Regulation (FAR),^[2] which applies to all executive-agency acquisitions of supplies and services with appropriated funds by and for the use of the federal government, unless expressly excluded. Other relevant policies and procedures are found in federal statutes and agencies' own procurement rules.

If a vendor believes a federal executive agency has not complied with the law or the terms of a solicitation, it may file what is called a bid protest—that is, a written objection to a government agency's conduct in acquiring supplies and services for its direct use or benefit.^[3] Responding to bid protests can require agencies to reevaluate their procurement processes and, sometimes, make improvements. That, in turn, results in more competitive, fairer, and more transparent procurement processes, benefiting vendors, agencies, and ultimately the public.

To file a bid protest, an actual or prospective vendor must show that it is an “interested party”—meaning that its direct economic interest would be adversely affected by the award of, or failure to award, the contract in question ^[4] —and that it suffered prejudice because of an error in the procurement process. Ordinarily, vendors who meet those requirements may file bid protests in any of three forums: (1) The procuring agency,^[5] (2) the Government Accountability Office (GAO),^[6] or (3) the United States Court of Federal Claims (COFC),^[7] and depending on where the protest is initiated, may be able to file protests in series. For example, a protest may be filed first at the agency, then (if unsuccessful at the agency) at GAO, and then (if again unsuccessful) at COFC.^[8] The procedural tools available in a given forum, along with other strategic and cost considerations, typically drive vendors' decisions about where to file their bid protests.

Bid protests filed with procuring agencies are commonly referred to as agency-level protests. Agency-level protests have important benefits for the public, contractors, procuring agencies, and COFC and GAO. By “provid[ing] for inexpensive, informal, procedurally simple, and expeditious resolution of protests,” ^[9] agency-level protest mechanisms allow small businesses (among other vendors) to affordably contest agencies' procurement decisions. They also give procuring agencies the chance to review and improve their own procurement practices. And they funnel some protests away from COFC and GAO, reducing the likelihood that the number of protests will overwhelm those institutions.

Vendors, however, seldom file agency-level protests. Although there is little data on the number of agency-level protests filed each year, available evidence suggests that substantially more protests are filed with COFC and GAO each year than with procuring agencies.^[10] There are several reasons why vendors may forego agency-level protests. Those reasons implicate the themes of transparency, predictability, and accountability.

First, some vendors report shying away from agency-level protests because they perceive the agency as unlikely to change its decision.^[11] Sometimes, for instance, the official responsible for soliciting or awarding a procurement contract is also responsible for handling any agency-level protests that are filed regarding the procurement. This perception of a pre-judgment by the agency may cause some vendors to file their protests at GAO or COFC, rather than at the agency level.

Second, some vendors report that they view agency-level protest processes as opaque.^[12] Agencies do not publish or provide comprehensive data on their bid protest decisions. And the FAR and agency-specific bid protest rules establish few hard-and-fast requirements for the process. For example, although the FAR states that “[a]gencies shall make their best efforts to resolve agency protests within 35 days after [an agency-level protest] is filed,” ^[13] that language is hortatory and does not establish any binding deadlines for agency decisions. Nothing in the FAR does. The failure to provide for any binding deadlines distinguishes the FAR from other federal procurement statutes, such as the Contract Disputes Act,^[14] which sets or requires contracting officers to set firm deadlines for deciding most claims ^[15] and provides that the passage of the deadline for a claim means the claim is deemed denied.^[16]

Third, some vendors report being dissuaded by their inability to compel production of the procurement record as part of an agency-level protest.^[17] The FAR gives disappointed offerors the right to an agency debriefing—a procedure whereby contracting personnel provide offerors with an explanation of the agency's evaluation process and an assessment of the offerors' proposals. But nothing in the FAR guarantees vendors the right to view the procurement record itself. The FAR provides only that agencies “may exchange relevant information” with agency-level protesters.^[18] By contrast, vendors who file bid protests at GAO may demand to see the entire record of the procurement, and procuring agencies must respond to such requests within twenty-five days and produce the responsive documents within thirty days (unless they are withheld for a valid reason).^[19]

Finally, some vendors deem agency-level protests to be too risky.^[20] In many cases, vendors who do not obtain relief through an agency-level protest will seek relief from GAO by pursuing their protest in that forum. But GAO's deadline for filing such “follow-on protests” often begins to run as soon as the vendor has actual or constructive notice of some “adverse agency action,” which can occur before a protester receives the decision in its agency-level protest.^[21] In this way, delayed notification about an agency's decision in a bid protest can seriously prejudice protesters' rights at GAO.^[22] This causes some vendors to forego agency-level protests altogether.^[23]

The perception that agency-level protests lack transparency, predictability, and accountability also makes it more likely that protesters who do file at the agency level and whose protests are denied will file follow-on protests with GAO or COFC. Such follow-on protests not only tax the limited resources of GAO and COFC, but also can disrupt activities at procuring agencies. For instance, just as the filing of an agency-level protest automatically prohibits the contract from being awarded or performed until the agency denies or dismisses the protest and takes some adverse action,^[24] a follow-on protest at GAO may automatically prevent the contract from being awarded or performed (if the requisite filing deadlines are met) until GAO denies or dismisses the protest.^[25] Thus, when an agency-level protest is followed by another protest at GAO, delays in procurements can be substantial.

Protesters, agencies, and the public would all benefit from an improved agency-level protest system. Protesters would benefit because agency-level protests are typically the least formal and least costly types of bid protest procedures. Agencies would benefit from an improved agency-level protest system because greater use of agency-level protests means more agency control over the timing and conduct of protests and more opportunities for agencies to superintend their own procurement processes. And the public would benefit from more competitive, fairer, and more transparent agency procurements.

Because an improved agency-level protest system is of significant value to contractors, agencies, and the public, this Recommendation identifies changes to make it more likely vendors will avail themselves of agency-level protest procedures. The recommended changes reflect three overarching principles—transparency, simplicity, and predictability—meant to address contractors' principal concerns about agency-level protest systems.

Recommendation

Identification of Decisions Subject to Agency-Level Protests

1. Agencies should clearly identify which categories of procurement decisions may or may not be made the subjects of agency-level protests.

Transparency for the Process and Personnel for Agency-Level Protests

2. Agencies should formalize and compile in a document that is publicly available online the procedures they apply in adjudicating agency-level protests. In so doing, they should be guided by the principles set out in Recommendation 2018-5, Public Availability of Adjudication Rules.

3. Agencies should clearly identify who within the agency will adjudicate an agency-level protest. They should consider designating at least one Agency Protest Official (APO)—a person who specializes in handling agency-level protests—to oversee and coordinate agency-level protests and hear protests brought to a level above the contracting officer. Agencies lacking the resources to designate their own APO might consider sharing an APO with other agencies.

Notice of the Timeline for Agency-Level Protests

4. Agencies should consider adopting presumptive timelines for agency-level protests, similar to the ones under the Contract Disputes Act. Agencies should also make best efforts to notify protesters of the timelines applicable to their agency-level protests.

5. Agencies should clearly and immediately provide written notice to protesters of any adverse agency action affecting the rights of the protester under the challenged procurement. Agency rules should provide that protests are deemed denied after a specified number of days without a decision and that agencies may grant case-specific extensions based on identified criteria.

Compiling the Record and Making It Available

6. Agencies should make available to protesters as much of the procurement record as is feasible. To address confidential information in the record, agencies should consider using tools such as enhanced debriefings.

7. Agencies should consider adopting a thirty-day deadline, running from the date a protest is filed, for providing protesters with as much of the procurement record as is feasible.

Protecting Against Adverse Consequences

8. Although the Federal Acquisition Regulation (FAR) prohibits the award of a contract or continued performance under an awarded contract during an agency-level protest, agencies should provide for a short extension of the stay after a final decision in an agency-level bid protest as permitted by the FAR. The short extension should be of sufficient duration (e.g., five days) to give the protester time to bring a follow-on protest at the Government Accountability Office (GAO) or the United States Court of Federal Claims after the agency's decision.

9. Congress should provide that, if a protester promptly files a GAO protest after an adverse decision in an agency-level protest, the agency shall not award the contract or commence performance under the contract during the pendency of the GAO protest, subject to potential override in urgent and compelling circumstances.

10. GAO should amend its bid protest procedures to ensure that follow-on protests at GAO are handled on an expedited basis, to the extent feasible.

Publishing Data on Agency-Level Protests

11. Agencies should collect and annually publish data about the bid protests they adjudicate. To the extent feasible, the data should at least include what the GAO currently provides in its annual reports about the bid protests it adjudicates (e.g., the number of bid protests filed with the agency; the effectiveness rate of agency-level bid protests (the ratio of protests sustained or in which corrective action is afforded versus total agency-level protests filed); the number of merits decisions by the agency; the number of decisions sustaining the protest; the number of decisions denying the protest; and the time required for bid protests to be resolved).

Administrative Conference Recommendation 2020-5

Publication of Policies Governing Agency Adjudicators

Adopted December 17, 2020

[Note: Appendix B referenced in this Recommendation has been omitted from this notice because of the inaccessible images it contains. The full appendix may be found online at https://www.acus.gov/​recommendation/​publication-policies-governing-agency-adjudicators.]

Federal agency officials throughout the country preside over hundreds of thousands of adjudications each year.^[1] As the Administrative Conference has previously observed, litigants, their lawyers, and other members of the public benefit from having ready online access to procedural rules, decisions, and other key materials associated with adjudications.^[2] They also benefit from having ready online access to the policies and practices by which agencies appoint and oversee administrative law judges and other adjudicators. The availability of these policies and practices helps inform the public about, among other things, any actions agencies have taken to ensure the impartiality of administrative adjudicators ^[3] and promotes an understanding of adjudicators' constitutional status under the Appointments Clause and other constitutional provisions. The Administrative Conference acknowledges ongoing litigation regarding the constitutional status of many agency adjudicators and the continuing validity of the means and circumstances of their appointment and removal.^[4]

Agencies may benefit from disclosures about agency adjudicators because it allows them to compare their own policies with those made publicly available by other agencies. Agencies' proactive disclosures, which may sometimes already be required under the Freedom of Information Act and the E-Government Act, may also be more cost-effective than agencies' responding to individual requests for information.^[5]

Like other recent recommendations regarding adjudicators,^[6] this Recommendation pertains to officials who preside over (1) hearings governed by the formal hearing provisions of the Administrative Procedure Act (APA) ^[7] and (2) hearings that are not governed by those provisions but are required by statute, regulation, or executive order. It also covers officials (agency heads excluded) who review hearing-level adjudicators' decisions on appeal. For ease of reference, this Recommendation refers to the covered adjudicators as either “administrative law judges” (ALJs) or “administrative judges” (AJs).^[8] Agencies may decide to include on their websites the disclosures identified in this Recommendation for other adjudicators, depending on the level of formality of the proceedings over which they preside and whether they serve as full-time adjudicators. Agencies may also decide to make similar disclosures with respect to agency heads if their websites do not already provide sufficient information.

This Recommendation focuses on policies and practices relating to adjudicators that agencies should disclose, including those addressing appointment and qualifications; compensation (including salaries, bonuses, and performance incentives); duties and responsibilities; supervision and assignment of work; position within agencies' organizational hierarchies; methods of evaluating performance; limitations on ex parte communications and other policies ensuring separation between adjudicative and enforcement functions; recusal and disqualification; the process for review of adjudications; and discipline and removal.

Many of the policies and practices applicable to ALJs governing these matters are already publicly available because they are in the APA, Office of Personnel Management rules, or other legal authorities.^[9] Nevertheless, agencies that employ ALJs can take steps to improve the public's access to this information.

ALJs, in any case, make up a small portion of federal adjudicators. There are many more AJs than ALJs.^[10] AJs are regulated by a complex mix of statutory provisions, including civil service laws, agency rules codified in the Code of Federal Regulations, and agency-specific policies that take a variety of forms. Many types of information about AJs reside in these sources, but they may be difficult to find.^[11] Some relevant sources may not be publicly available, including internal administrative and personnel manuals, position descriptions, and labor agreements. This is particularly true with respect to certain kinds of policies, such as those relating to compensation and performance incentives.^[12] Of course, the Administrative Conference recognizes that some of these agency policies and practices may qualify for an exemption under the Freedom of Information Act,^[13] Privacy Act,^[14] or other laws and executive-branch policies.

Agency websites are the most helpful location for agencies to make relevant policies and practices publicly available. Individuals most naturally seek information about administrative policies and practices on agencies' websites. Agencies can situate information about their adjudicators in a logical and easily identifiable place on their websites and structure their websites to synthesize policies in plain language and link to information from many different sources.^[15]

This Recommendation encourages agencies to post on their websites clear and readily accessible descriptions of the policies governing the appointment and oversight of ALJs and AJs, and to include links to relevant legal documents. How, exactly, they should do so will of course depend on the specific features of their adjudicative programs and their institutional needs.

Recommendation

1. Each adjudicative agency should prominently display on its website a short, straightforward description of all generally applicable policies and practices, along with the legal authority, governing the appointment and oversight of Administrative Law Judges (ALJs) and Administrative Judges (AJs), including, as applicable, those that address:

a. Procedures for assessing, selecting, and appointing candidates for adjudicator positions and the legal authority under which such appointments are made;

b. Placement of adjudicators within agencies' organizational hierarchies;

c. Compensation structure and performance incentives, such as bonuses, nonmonetary awards, and promotions;

d. Procedures for assigning cases;

e. Assignment, if any, of nonadjudicative duties to adjudicators;

f. Limitations on ex parte communications, including between adjudicators and other agency officials, related to the disposition of individual cases, as well as other policies ensuring a separation of adjudication and enforcement functions;

g. Standards for recusal by and disqualification of adjudicators;

h. Administrative review of adjudicators' decisions;

i. Supervision of adjudicators by higher-level officials;

j. Evaluation of adjudicators, including quantitative and qualitative methods for appraising adjudicators' performances, such as case-processing goals, if any; and

k. Discipline and removal of adjudicators.

Agencies may choose not to provide access to policies covered by a Freedom of Information Act exemption.

2. On the same web page as the information described in Paragraph 1 appears, each adjudicative agency should post links to key legal documents or, when links are not available, citations to such documents. These documents may include (a) federal statutes, including relevant provisions of the Administrative Procedure Act (APA) and other laws applicable to ALJs and AJs; (b) agency-promulgated rules regarding adjudicators, including Office of Personnel Management rules applicable to ALJs; (c) publicly available agency-promulgated guidance documents relating to adjudicators, including manuals, bench books, and other explanatory materials; (d) delegations of authority; and (e) position descriptions. To the extent that some policies concerning adjudicators may be a matter of custom, such as assignment of nonadjudicative duties, each adjudicative agency should consider documenting those policies to make them publicly accessible to the extent practicable.

3. The web page containing the information described in Paragraphs 1 and 2 should present the materials in a clear, logical, and comprehensive fashion. One possible method of presenting this information appears in Appendix A. The appendix gives one example for ALJs and another for AJs.

4. If an agency's mission consists exclusively or almost exclusively of conducting adjudications, the agency should provide a link to the web page containing the information described in Paragraphs 1 and 2 on the agency's homepage. If conducting adjudications is one of an agency's many functions, the agency should provide a link to these materials from a location on the website that is both dedicated to adjudicative materials and logical in terms of a user's likelihood of finding the documents in the selected location. One example would be an enforcement or adjudication page or the homepage for the component in which a particular category of adjudicators works. Citations to agency web pages that currently provide this information in a way that makes it easy for the public to locate, as well as descriptions of how to find those pages on agency websites, appear in Appendix B.

Appendix A

Sample Website Text for Administrative Law Judges

About Our Administrative Law Judges

Administrative Law Judges (ALJs) at [agency] conduct hearings and decide cases under [insert name of authorizing act]. They are part of the [agency component in which ALJs are located], which is directed by [title of office head] and has offices in [cities]. Visit [link to agency organization chart] to see how [office] relates to other offices at [agency].

[Agency] is committed to ensuring that all hearings and appeals are conducted in a fair and equitable manner. Parties are entitled to a due process hearing presided over by an impartial, qualified ALJ. ALJs resolve cases involving [kinds of cases ALJs hear] in a fair, transparent, and accessible manner. Our ALJs are appointed by [agency official], and are [describe qualifications]. ALJs are paid according to the [pay scale for ALJs with link to the scale] scale set by statute under 5 U.S.C. 5372, subject to annual pay adjustments.

Cases are assigned to ALJs [in each geographic office] in rotation so far as practicable. The ALJ assigned to your case is responsible for [job duties, like taking evidence, hearing objections, issuing decisions]. ALJs are required by statute to perform their functions impartially. 5 U.S.C. 556(b). To ensure impartiality, they do not take part in investigative or enforcement activities, nor do they report to officials in the [agency]'s investigative or enforcement components. 5 U.S.C. 554(d), 3105. The ALJ assigned to your case may not communicate privately about the facts of your case with other agency officials. [More details on [agency]'s rules about communicating with ALJs are available [location of agency-specific ex parte prohibitions]].

By law, [agency] does not reward or discipline ALJs for their decisions. A federal statute provides that [agency] may remove, or take certain other disciplinary actions, against an ALJ it employs only for good cause established and determined by the Merit Systems Protection Board on the record after opportunity for hearing before the Board. 5 U.S.C. 7521.

The agency has adopted rules of recusal [link] that allow a participant to request that the ALJ in charge of his or her case be disqualified if the participant believes the ALJ cannot fairly and impartially decide the case.

If you are dissatisfied with an ALJ's decision, you can request reconsideration from the ALJ or appeal that decision to [agency office/official]. Visit [link] for information on appealing an ALJ decision. [Agency office/official] may also review your case on [its/his or her] own initiative if there is an issue with the ALJ's decision.

For Further Information:

• Hiring process: [link]
• Pay rates: [link]
• How cases are assigned to ALJs: [link]
• Communicating with ALJs (ex parte communications): [link]
• Process for addressing allegations that an ALJ has a conflict of interest (recusal and disqualification procedures): [link]
• How to appeal an ALJ decision: [link]
• Case-processing goals: [link]
• Process for addressing allegations of ALJ misconduct: [link]

See also:

• Statutory provisions governing ALJs: 5 U.S.C. 554, 557, 3105, 4301, 5372, 7521
• OPM's regulations governing ALJs: 5 CFR 930.205-930.207, 930.211
• MSPB's regulations governing ALJs: 5 CFR 1201.127-1201.142
• [Additional legal provisions governing ALJs]
• Executive Orders pertaining to ALJs: E.O. 13,843 (giving agencies control over the hiring process of ALJs) [add other pertinent EOs]

Sample Website Text for Administrative Judges

If agencies have different kinds of adjudicators, they should consider providing a separate web page for each.

About Our [Insert Adjudicator Title]

[Adjudicator title] at [agency] [conduct hearings and decide cases/review appeals] under [name of authorizing act(s)]. They are part of the [agency component in which adjudicators are located], which is directed by [title of office head] and has offices in [cities]. Visit [link to agency organization chart] to see how [office] relates to other offices at [agency].

[Agency] is committed to ensuring that all hearings and appeals are conducted in a fair and equitable manner. Parties are entitled to a due process hearing presided over by an impartial, qualified [adjudicator title]. [Adjudicator title] resolve cases involving [kinds of cases] in a fair, transparent, and accessible manner. Our [adjudicator title] are appointed pursuant to [authorizing statute] by [agency official] [for terms of [number of years] years], and are [describe qualifications]. [Adjudicator title] are paid according to [[the pay scale for the adjudicator with link to the scale] or [the discretion of the agency head]].

Cases are [describe how cases are assigned]. The [adjudicator title] assigned to your case is responsible for [job duties, like taking evidence, hearing objections, issuing decisions]. [Description of policies (if any exist) that ensure the agency component or adjudicators remain independent from investigative or enforcement activities]. [Description of rules about ex parte communications, if any exist].

[Agency official or body] is responsible for evaluating the quality of [adjudicator title] decisions, and [agency official or body] conducts performance reviews of [adjudicator title]. [Agency official/entity from another agency] may remove the [adjudicator title] or [agency official or body/other entity] may discipline the [adjudicator title] by [kinds of discipline] when warranted.

The agency has adopted rules of recusal [link] that allow a participant to request that the [adjudicator title] in charge of his or her case be disqualified if the participant believes the [adjudicator title] cannot fairly and impartially decide the case.

If you are dissatisfied with an [adjudicator title] decision, you can request reconsideration from the [adjudicator title] or appeal that decision to [agency office/official]. Visit [link] for information on appealing an [adjudicator title] decision. [Agency office/official] may also review your case on [its/his or her] own initiative if there is an issue with the [adjudicator title]'s decision.

For Further Information:

• Hiring process: [link]
• Pay rates: [link]
• Bonuses and performance incentives: [link]
• How cases are assigned to [adjudicator title]: [link]
• Communicating with [adjudicator title] (ex parte communications): [link]
• Process for addressing allegations that an [adjudicator title] has a conflict of interest (recusal and disqualification procedures): [link]
• How to appeal an [adjudicator title] decision: [link]
• Case-processing goals: [link]
• Process for addressing allegations of [adjudicator title] misconduct: [link]

See also:

• Statutory provisions regarding [adjudicator title], including the appointment authority: [statutory citations]
• Agency regulations governing [adjudicator title]: [CFR provisions]

Appendix B

[Note: Appendix B has been omitted from this notice because of the inaccessible images it contains. The full appendix may be found online at https://www.acus.gov/​recommendation/​publication-policies-governing-agency-adjudicators.]

Administrative Conference Recommendation 2020-6

Agency Litigation Web Pages

Adopted December 17, 2020

Federal agencies and their component units ^[1] participate in thousands of court cases every year. Most such cases result in “agency litigation materials,” which this Recommendation defines as including agencies' publicly filed pleadings, briefs, and settlements, as well as court decisions, where such materials bear on agencies' regulatory or enforcement activities.

Public access to agency litigation materials is desirable for at least two reasons. First, because agency litigation materials often clarify how the federal government interprets and aims to enforce federal law, they can help people understand their legal obligations. Second, public access to agency litigation materials promotes accountable and transparent government. Those two reasons distinguish agency litigation materials from litigation filings by private parties.

However valuable public access to agency litigation materials might be, federal law does little to mandate it. When it comes to agencies' own litigation filings, only the Freedom of Information Act (FOIA) requires disclosure, and then only when members of the public specify the materials in which they are interested (and no FOIA exception applies).^[2] In the same vein, the E-Government Act of 2002 requires federal courts to make their written opinions, including opinions in cases involving federal agencies, available on websites.^[3] But that requirement has not always made judicial opinions readily accessible to the public, partly because most courts' websites lack functions and features that would allow users to easily identify cases about specific topics or agencies.

The most comprehensive source of agency litigation materials is the federal courts' Public Access to Court Electronic Records (PACER) service, which provides the public with instantaneous access to virtually every document filed in every federal court. But PACER searches often cost money, and the costs can add up quickly, especially when users are uncertain about what cases or documents they are trying to find. PACER's limited search functionality also makes it difficult to find cases involving particular agencies, statutes, regulations, or types of agency action. For example, a person interested in identifying ongoing cases to which the United States Fish and Wildlife Service (FWS) is a party would have to search for a host of terms—including “United States Fish and Wildlife Service,” “U.S. Fish and Wildlife Service,” and the names of FWS's recent directors—just to come close to identifying all such cases. Even after conducting all those searches, the person would still have to scroll through and eliminate search results involving state fish-and-wildlife agencies and private citizens with the same names as FWS's recent directors. Similarly, were a person interested in finding cases about FWS's listing of species under the Endangered Species Act (ESA), PACER would not afford that person any way to filter search results to include only cases about ESA listings. The person's only option would be to open and review documents in potentially thousands of cases.

The cost and time involved in performing this type of research limit PACER's usefulness as a tool for locating and searching agency litigation materials. And although paid legal services, such as Westlaw and Lexis, have far greater search capabilities than PACER, their costs can dissuade many individuals and researchers.

Agency litigation web pages, by contrast, can be a convenient way for the public to examine agency litigation materials. For purposes of this Recommendation, an agency litigation web page is a web page on an agency's website that systematically catalogs and links to agency litigation materials that may aid the public in understanding the agency's regulatory or enforcement activities. When agencies maintain up-to-date, search-friendly agency litigation web pages, the public can visit them and quickly find important filings in court cases concerning matters of interest. Agency litigation web pages thus make it easier for the public to learn about the law and to hold government accountable for agencies' actions.

Several federal agencies already maintain agency litigation web pages.^[4] A survey of websites for twenty-five federal agencies revealed a range of practices regarding agency litigation web pages.^[5] The survey suggests that most federal agencies do not maintain active agency litigation web pages. Among those that do, the quality of the agency litigation web pages varies appreciably. Some contain vast troves of agency litigation materials; others contain much more limited collections. Some are updated regularly; others are updated only sporadically. Some are easy to locate and search; others are not. In short, there appears to be no standard practice for publishing and maintaining agency litigation web pages, save that all the surveyed agency litigation web pages contained only the publicly filed versions of agency litigation materials, with all confidential material—such as trade secrets and personally identifiable information—redacted.

An inspection of agencies' litigation web pages suggests four general features that make an agency litigation web page useful. First, an agency's litigation web page must be easy to find. Second, it must contain a representative and up-to-date collection of agency litigation materials. Third, those materials must be easy to search and sort. And fourth, the agency's litigation web page must give visitors the information they need to understand the materials on the web page, including information about materials the agency omitted from the web page and the criteria the agency employed to determine which materials to include on the web page.

Agency litigation web pages can promote transparency and accountability. The Administrative Conference recognizes, however, that creating and maintaining a useful agency litigation web page takes time, money, and effort. An agency's decision to launch an agency litigation web page will necessarily be informed by considerations such as the agency's mission, litigation portfolio, existing technological capacity, budget, and the anticipated benefits—to the agency and the public—of creating an agency litigation web page.^[6] Further, an agency's decisions about what content to include on an agency litigation web page should be tailored to the agency's particular circumstances. An agency that litigates thousands of cases each year, for example, could choose to feature only a representative sample of agency litigation materials on its agency litigation web page.

Similarly, an agency that litigates many repetitive, fact-based cases could reasonably choose to post documents from just a few representative cases instead of posting documents from all of its cases.^[7] And an agency that litigates many different types of cases, some of obviously greater interest to the public than others, might appropriately restrict the contents of its agency litigation web page to agency litigation materials from the types of cases that are of greater public interest, particularly when the agency determines that the resources required to post more agency litigation materials can be better applied elsewhere.

Since the decision to create and maintain an agency litigation web page involves balancing factors that will differ from agency to agency, this Recommendation should not be read to suggest that agency litigation web pages be created and maintained by all agencies, especially those that litigate thousands of cases each year. Nor should this Recommendation be read as dictating the precise contents or structure of agency litigation web pages. While encouraging the creation and maintenance of agency litigation web pages, the Administrative Conference recognizes that an agency's particular circumstances might ultimately militate against creating an agency litigation web page or might support only the creation of a comparatively limited version.

At bottom, this Recommendation simply offers best practices and factors for agencies to consider in making their agency litigation materials available on their websites, should the agencies choose to do so. The Recommendation leaves the weighing and balancing of those factors to the sound discretion of individual agencies.

Recommendation

Providing Access to Agency Litigation Materials

1. Agencies should consider providing access on their websites to publicly filed pleadings, briefs, and settlements, as well as court decisions bearing on agencies' regulatory or enforcement activities (collectively “agency litigation materials”).

2. Should an agency choose to post such material, an agency with a large volume of court litigation could decide not to post documents from every case. The agency might, for instance, post examples of filings from routine litigation and all or a portion of the filings from cases raising important or unusual questions.

3. In determining whether to provide access to agency litigation materials on their websites, and in determining which types of agency litigation materials to include on their websites, among the factors agencies should consider are the following:

a. The public's interest in having ready access to certain categories of the agency's litigation materials;

b. The extent to which providing access to agency litigation materials on the agency's website will advance the agency's mission;

c. The internal benefits of maintaining a web page providing access to certain types of agency litigation materials;

d. The costs of creating and maintaining a web page providing access to the types of agency litigation materials the agency sees fit to include;

e. The nature of the agency's litigation portfolio, including the quantity of litigation materials the agency generates each year;

f. The degree to which the agency's existing technological capacity can accommodate the creation and maintenance of a web page providing access to certain types of agency litigation materials;

g. The availability and cost of other technological services that may more reliably and effectively give access to agency litigation material because of its scale or volume and the wide variety of issues and matters involved; and

h. The risk of disclosure or increased dissemination of confidential or sensitive information of private litigants.

4. In determining which agency litigation materials to include on their websites, agencies should ensure that they have implemented appropriate safeguards to protect relevant privacy or business interests implicated by the disclosure of agency litigation materials. Each agency should implement a protocol to ensure that, before a document is posted to the agency's litigation web page, the document has been reviewed and determined not to contain confidential information, such as trade secrets and personal identifying information.

5. Agencies should disclose materials in a way that gives a full and accurate picture of their litigating positions. To provide proper context, agencies should:

a. Use objective, clear, and publicly posted criteria to determine which agency litigation materials the agencies will publish on their websites;

b. Regularly review their websites to ensure the agency litigation materials posted there (especially court opinions) are complete and up-to-date, and consider including notations regarding when material on the web page was last updated;

c. Provide appropriate context for agency litigation materials, at least when failure to do so might confuse or mislead the public;

d. Explain the types of litigation in which the agency is involved and other ways to search for any additional agency litigation materials not included on the agency's litigation web page, as well as opposing counsel's litigation filings;

e. When resources permit, consider posting opposing parties' litigation filings when they are significant or important to understanding an issue;

f. Neither present litigation materials as a means of setting policy, nor use those materials to circumvent rulemaking processes;

g. Ensure that descriptions of agency litigation materials, if any, fairly reflect the litigation; and

h. Recognize that some types of agency litigation materials may be of greater significance than others.

6. Agencies that choose to post significant quantities of agency litigation materials on their websites should consider grouping together links to those materials on a single, dedicated web page (an “agency litigation web page”). If an agency is organized so that its component units have their own litigation portfolios, some or all of the component units may wish to have their own agency litigation web pages, or the agency may wish to maintain an agency litigation web page compiling litigation materials from or relating to the agency's component units.

Making It Easy To Locate Agency Litigation Web Pages

7. Agencies that post agency litigation materials on their websites should make sure that website users can easily locate those materials. Agencies can accomplish this goal by:

a. Displaying links to agency litigation web pages in readily visible locations on the homepage for the agency's website; and

b. Maintaining a search engine and a site map or index, or both, on the agency's homepage.

8. When an agency collects its component units' litigation materials on a single agency litigation web page, those component units' websites should clearly note that fact and include links to the agency's litigation web page. When an agency's component units maintain their own litigation web pages, the agency's website should clearly note that fact and include links to the component units' litigation web pages.

Making It Easy To Find Relevant Materials on Agency Litigation Web Pages

9. Agencies and their component units should have substantial flexibility in organizing materials. Agencies should consider grouping together materials from the same and related cases on their agency litigation web pages. Agencies might, for example, consider providing a separate docket page for each case, with a link to the docket page on their agency litigation web pages. Agencies should also consider linking to the grouped-together materials when issuing press releases concerning a particular litigation.

10. Agencies should consider offering general and advanced search and filtering options within their agency litigation web pages. The search and filtering options could, for instance, allow users to sort, narrow, or filter searches according to criteria such as action or case type, date, topic, case number, party name, a relevant statute or regulation, or specific words and phrases, along with any other criteria the agency decides are especially useful given its litigation activities.

Footnotes

1.  This Recommendation does not address rulemakings subject to the formal hearing requirements of the Administrative Procedure Act. See 5 U.S.C. 556-57.

2.  Cf. Admin. Conf. of the U.S., Recommendation 2019-1, Agency Guidance Through Interpretive Rules, 84 FR 38927 (Aug. 8, 2019); Admin. Conf. of the U.S., Recommendation 2017-5, Agency Guidance Through Policy Statements, 82 FR 61734 (Dec. 29, 2017).

3.  See, e.g., 2 U.S.C. 1534 (Unfunded Mandates Reform Act); 5 U.S.C. 609 (Regulatory Flexibility Act); Exec. Order No. 13,175, Consultation and Coordination with Indian Tribal Governments, 65 FR 67249 (Nov. 11, 2000).

4.  See Admin. Conf. of the U.S., Recommendation 92-1, The Procedural and Practice Rule Exemption from the APA Notice-and-Comment Rulemaking Requirements, 57 FR 30102 (July 8, 1992); see also Recommendation 2019-1, supra note 2; Recommendation 2017-5, supra note 2.

5.  See, e.g., Health Ins. Ass'n of Am. v. Shalala, 23 F.3d 412, 423 (D.C. Cir. 1994) (stating that “publication in the Code of Federal Regulations, or its absence” is only “a snippet of evidence of agency intent” that the published pronouncement be given binding effect).

6.  Some rules on rulemakings include a statement that they do not create any substantive or procedural rights or benefits. This Recommendation does not address whether such disclaimers should be included or what legal effect they may have on judicial review. These questions cannot be answered in isolation from the broader question of when a rule on rulemakings is judicially enforceable.

7.  See Admin. Conf. of the U.S., Recommendation 2015-1, Promoting Accuracy and Transparency in the Unified Agenda, 80 FR 36757 (June 26, 2015).

8.  See Admin. Conf. of the U.S., Recommendation 2018-7, Public Engagement in Rulemaking, 84 FR 2146 (Feb. 6, 2019).

9.  See Admin. Conf. of the U.S., Recommendation 2014-6, Petitions for Rulemaking, 79 FR 75117 (Dec. 17, 2014).

10.  See Admin. Conf. of the U.S., Recommendation 2012-1, Regulatory Analysis Requirements, 77 FR 47801 (Aug. 10, 2012).

11.  See Admin. Conf. of the U.S., Recommendation 2017-3, Plain Language in Regulatory Drafting, 82 FR 61728 (Dec. 29, 2017).

12.  See Admin. Conf. of the U.S., Recommendation 2018-2, Severability in Agency Rulemaking, 83 FR 30685 (June 29, 2018).

13.  See Admin. Conf. of the U.S., Recommendation 2017-2, Negotiated Rulemaking and Other Options for Public Engagement, 82 FR 31040 (July 5, 2017).

14.  See Admin. Conf. of the U.S., Recommendation 2018-6, Improving Access to Regulations.gov's Rulemaking Dockets, 84 FR 2143 (Feb. 6, 2019).

15.  See Admin. Conf. of the U.S., Recommendation 2011-2, Rulemaking Comments, 76 FR 48791 (Aug. 9, 2011).

16.  See Admin. Conf. of the U.S., Recommendation 2014-4, “Ex Parte” Communications in Informal Rulemaking, 79 FR 35993 (June 25, 2014).

17.  See Admin. Conf. of the U.S., Recommendation 2011-5, Incorporation by Reference, 77 FR 2257 (Jan. 17, 2012).

18.  See Admin. Conf. of the U.S., Recommendation 2013-5, Social Media in Rulemaking, 78 FR 76269 (Dec. 17, 2013).

19.  See Recommendation 2018-7, supra note 8.

20.  See Admin. Conf. of the U.S., Recommendation 2020-2, Protected Materials in Public Rulemaking Dockets, 86 FR (approved Dec. 16, 2020); Admin. Conf. of the U.S., Recommendation 2011-1, Legal Considerations in e-Rulemaking, 76 FR 48789 (Aug. 9, 2011).

21.  See Recommendation 92-1, supra note 4.

22.  See Admin. Conf. of the U.S., Recommendation 95-4, Procedures for Noncontroversial and Expedited Rulemakings, 60 FR 43108 (Aug. 18, 1995).

23.  See Admin. Conf. of the U.S., Recommendation 2011-6, International Regulatory Cooperation, 77 FR 2259 (Jan. 17, 2012).

24.  See Recommendation 2011-2, supra note 15.

25.  See id.

26.  See Admin. Conf. of the U.S., Recommendation 2019-5, Agency Economists, 84 FR 71349 (Dec. 27, 2019).

27.  See Admin. Conf. of the U.S., Recommendation 2017-7, Regulatory Waivers and Exemptions, 82 FR 61742 (Dec. 29, 2017).

28.  See Admin. Conf. of the U.S., Recommendation 2014-5, Retrospective Review of Agency Rules, 79 FR 75114 (Dec. 17, 2014).

29.  See Admin. Conf. of the U.S., Recommendation 2013-4, Administrative Record in Informal Rulemaking, 78 FR 41358 (July 10, 2013).

30.  See Admin. Conf. of the U.S., Recommendation 2013-6, Remand Without Vacatur, 78 FR 76272 (Dec. 17, 2013).

1.  The public rulemaking docket is distinguished from “the administrative record for judicial review,” which is intended to provide courts with a record for evaluating challenges to the rule, and the “rulemaking record,” which means all comments and materials submitted to agencies during comment periods and any other materials agencies considered during the course of the rulemaking. See Admin. Conf. of the U.S., Recommendation 2013-4, The Administrative Record in Informal Rulemaking, 78 FR 41358 (July 10, 2013).

2.  Recommendation 2011-1, Legal Considerations in e-Rulemaking, advises agencies to allow submitters to flag confidential information, including trade secrets, and advises agencies to devise procedures for reviewing and handling such information. Admin. Conf. of the U.S., Recommendation 2011-1, Legal Considerations in e-Rulemaking, ¶ 1, 76 FR 48789, 48790 (Aug. 9, 2011). Recommendation 2013-4, supra note 1, ¶ 11, advises agencies to develop guidance on managing and segregating protected information, such as confidential commercial information and sensitive personal information, while disclosing non-protected materials; see also Admin. Conf. of the U.S., Recommendation 89-7, Federal Regulation of Biotechnology, 54 FR 53494 (Dec. 29, 1988); Admin. Conf. of the U.S., Recommendation 82-1, Exemption (b)(4) of the Freedom of Information Act, 47 FR 30702 (July 15, 1982); Admin. Conf. of the U.S., Recommendation 80-6, Intragovernmental Communications in Informal Rulemaking Proceedings, 45 FR 86408 (Dec. 31, 1980).

3.  See Privacy Act of 1974 § 3, 5 U.S.C. 552a(a)(4).

4.  See Food Mktg. Inst. v. Argus Leader Media, 139 S. Ct. 2356, 2363 (2019); see also Exec. Order No. 12,600, Predisclosure Notification Procedures for Confidential Commercial Information, 52 FR 23781 (June 23, 1987).

5.  See Christopher Yoo, Protected Materials in Public Rulemaking Dockets 24 (Nov. 24, 2020) (report to the Admin. Conf. of the U.S.), https://www.acus.gov/​report/​final-report-protected-materials-public-rulemaking-dockets.

6.  5 U.S.C. 553(c).

7.  Portland Cement Ass'n v. Ruckelshaus, 486 F.2d 375, 393 (D.C. Cir. 1973). In addition to these public transparency requirements, there are a number of federal record-retention requirements of which agencies should be aware. See, e.g., 44 U.S.C. 3301.

8.  5 U.S.C. 552a(b).

9.  18 U.S.C. 1905.

10.  5 U.S.C. 552a(b)(2).

11.  See CNA Fin. Corp. v. Donovan, 830 F.2d 1132, 1137-43 (D.C. Cir. 1987).

12.  See Food Mktg. Inst., 139 S. Ct. at 2361.

13.  Permitting the submission of anonymous and pseudonymous comments is one way that some agencies attempt to reduce the privacy risks commenters face when submitting protected material. Issues regarding the submission of anonymous and pseudonymous comments are being considered in an ongoing project of the Administrative Conference titled Mass, Computer-Generated, and Fraudulent Comments and are beyond the scope of this Recommendation.

1.  There is no universally accepted definition of “artificial intelligence,” and the rapid state of evolution in the field, as well as the proliferation of use cases, makes coalescing around any such definition difficult. See, e.g., John S. McCain National Defense Authorization Act for Fiscal Year 2019, Public Law 115-232, 238(g), 132 Stat. 1636, 1697-98 (2018) (using one definition of AI); Nat'l Inst. of Standards & Tech., U.S. Leadership in AI: A Plan for Federal Engagement in Developing Technical Standards and Related Tools 7-8 (Aug. 9, 2019) (offering a different definition of AI). Generally speaking, AI systems tend to have characteristics such as the ability to learn to solve complex problems, make predictions, or undertake tasks that heretofore have relied on human decision making or intervention. There are many illustrative examples of AI that can help frame the issue for the purpose of this Statement. They include, but are not limited to, AI assistants, computer vision systems, biomedical research, unmanned vehicle systems, advanced game-playing software, and facial recognition systems as well as application of AI in both information technology and operational technology.

2.  See 5 U.S.C. 591.

3.  David Freeman Engstrom, Daniel E. Ho, Catherine M. Sharkey, & Mariano-Florentino Cuéllar, Government by Algorithm: Artificial Intelligence in Federal Administrative Agencies (Feb. 2020) (report to the Admin. Conf. of the U.S.), https://www.acus.gov/​report/​government-algorithm-artificial-intelligence-federal-administrative-agencies;​; Cary Coglianese, A Framework for Governmental Use of Machine Learning (Dec. 8, 2020) (report to the Admin. Conf. of the U.S.), https://www.acus.gov/​report/​framework-governmental-use-machine-learning-final-report.

4.  While the term bias has a technical, statistical meaning, the Administrative Conference here uses the term more generally, to refer to common or systematic errors in decision making.

5.  5 U.S.C. 3371-76.

6.  Agencies may also obtain AI systems that are embedded in commercial products. The considerations applicable to such embedded AI systems should reflect the fact that agencies may have less control over their design and development.

7.  Within the General Services Administration, for example, the office called 18F routinely partners with government agencies to help them build and buy technologies. Similarly, the United States Digital Service (which is within the Executive Office of the President) has a staff of technologists whose job is to help agencies build better technological tools. While the two entities have different approaches—18F acts more like an information intermediary and the Digital Service serves as an alternative source for information technology contracts—both could aid agencies with obtaining, developing, and using different AI techniques.

8.  See, e.g., Paperwork Reduction Act, 44 U.S.C. 3501-20.

9.  See, e.g., 5 U.S.C. 552a(e), (g), & (p); 44 U.S.C. 3501 note.

10.  See Nat'l Inst. of Standards & Tech. Special Publication SP-800-37 revision 2, Risk Management Framework for Information Systems and Organizations: A System Lifecycle Approach for Security and Privacy (Dec. 2018); Office of Mgmt. & Budget, Exec. Off. of the President, Circular A-130, Managing Information as a Strategic Resource (July 28, 2016); see also Nat'l Inst. of Standards & Tech., NIST Privacy Framework: A Tool for Improving Privacy Through Enterprise Risk Management, Version 1.0 (Jan. 16, 2020).

11.  See supra note 10; see also Office of Mgmt. & Budget, Exec. Off. of the President, M-21-06, Guidance for Regulation of Artificial Intelligence Applications (Nov. 17, 2020); Nat'l Inst. for Standards & Tech., Framework for Improving Critical Infrastructure Cybersecurity (Apr. 16, 2018).

12.  Cf. Admin. Conf. of the U.S., Recommendation 2018-3, Electronic Case Management in Federal Administrative Adjudication, 83 FR 30,686 (June 29, 2018) (suggesting, in the context of case management systems, that agencies consider implementing electronic systems only when they conclude that doing so would lead to benefits without impairing either the objective “fairness” of the proceedings or the subjective “satisfaction” of those participating in those proceedings).

13.  Courts would analyze such challenges under the three-part balancing framework from Mathews v. Eldridge, 424 U.S. 319, 335 (1976).

14.  See 5 U.S.C. 706(2)(A). Courts would likely review such challenges under the standard set forth in Motor Vehicle Manufacturers Ass'n v. State Farm Mutual Automobile Insurance Co., 463 U.S. 29, 43 (1983).

15.  See 5 U.S.C. 553(b)-(c).

16.  See Office of Mgmt. & Budget, Circular A-130, supra note 10; Office of Mgmt. & Budget, Exec. Office of the President, Circular A-123, Management's Responsibilities for Enterprise Risk Management and Internal Control (July 15, 2016).

1.  Admin. Conf. of the U.S., Recommendation 2016-4, Evidentiary Hearings Not Required by the Administrative Procedure Act, 81 FR 94,314 (Dec. 23, 2016).

2.  Recommendation 2016-4 addressed agency adjudications in which an evidentiary hearing, though not governed by the formal hearing provisions of the Administrative Procedure Act (APA) (5 U.S.C. 554, 556-57), is required by statute, regulation, or executive order. Those adjudications, which are often as formal as APA adjudications in practice, far outnumber so-called APA adjudications. Although Recommendation 2016-4 addresses only non-APA adjudications, most of its best practices are as applicable to APA adjudications as non-APA adjudications. Some such practices, in fact, are modeled on the APA's formal hearing provisions.

3.  Admin. Conf. of the U.S., Recommendation 83-3, Agency Structures for Review of Decisions of Presiding Officers Under the Administrative Procedure Act, 48 FR 57,461 (Dec. 30, 1983); Admin. Conf. of the U.S., Recommendation 68-6, Delegation of Final Decisional Authority Subject to Discretionary Review by the Agency, 38 FR 19,783 (July 23, 1973). Both recommendations concerned only the review of decisions in proceedings governed by the formal hearing provisions of the APA. Their principles, though, are not so confined.

4.  Christopher J. Walker & Matthew Lee Wiener, Agency Appellate Systems (Dec. 14, 2020) (report to the Admin. Conf. of the U.S.), https://www.acus.gov/​report/​final-report-agency-appellate-systems.

1.  See Federal Acquisition Regulation, 48 CFR ch. 1; see also Competition in Contracting Act of 1984, Public Law 98-369, div. B, tit. VII, 98 Stat. 494, 942-85 (codified, as amended, in various parts of the U.S. Code); Federal Acquisition Streamlining Act of 1994, Public Law 103-355, 108 Stat. 3243; Federal Acquisition Reform Act of 1996, Public Law 104-106, 110 Stat. 186 (later renamed the Clinger-Cohen Act of 1996); Exec. Order No. 12,979, Agency Procurement Protests, 60 FR 55,171 (Oct. 25, 1995).

2.  See 48 CFR ch. 1.

3.  See Admin. Conf. of the U.S., Recommendation 95-5, Government Contract Bid Protests, 60 FR 43,108, 43,113 (Aug. 18, 1995).

4.  See 4 CFR 21.0(a)(1) (defining “interested party” for purposes of bid protest proceedings before the Government Accountability Office); 48 CFR 33.101 (defining “interested party” for purposes of bid protest proceedings before procuring agencies); CliniComp Int'l, Inc. v. United States, 904 F.3d 1353, 1358 (Fed. Cir. 2018) (defining “interested party” for purposes of 28 U.S.C. 1491(b), which covers actions in the Court of Federal Claims). There are some instances in which Congress has restricted the ability to file a protest, regardless of whether a vendor is an “interested party.” See, e.g., 41 U.S.C. 4106(f) (limiting the ability to protest the issuance or proposed issuance of a task or delivery order); 48 CFR 16.505(a)(10) (same).

5.  See 48 CFR 33.103.

6.  See 31 U.S.C. 3552(a), 3553(a). For civilian agencies, GAO has exclusive jurisdiction over protests of task and delivery orders in excess of $10 million, unless the protest is on the grounds that the order increases the scope, period, or maximum value of the contract. See 41 U.S.C. 4106(f); 48 CFR 16.505(a)(10).

7.  See 28 U.S.C. 1491(b).

8.  See Admin. Conf. of the U.S., Info. Interchange Bull. No. 007, Agency Bid Protests (June 2020), https://www.acus.gov/​fact-sheet/​iib-007-agency-bid-protests.

9.  See Exec. Order. No. 12,979, Agency Procurement Protests, 60 FR 55,171, 55,171 (Oct. 25, 1995).

10.  See Christopher Yukins, Stepping Stones to Reform: Making Agency-Level Bid Protests Effective for Agencies and Bidders by Building on Best Practices from Across the Federal Government 12-13 (May 1, 2020) (report to the Admin. Conf. of the U.S.), https://www.acus.gov/​report/​agency-level-bid-protests-final-report.

11.  Id. at 23.

12.  Id. at 13.

13.  48 CFR 33.103(g).

14.  41 U.S.C. 7101-09.

15.  See id. § 7103(f)(1)-(2).

16.  See id. § 605(c)(5).

17.  Yukins, supra note 10, at 39.

18.  48 CFR 33.103(g) (italics added).

19.  4 CFR 21.3(c)-(d); 48 CFR 33.104(a).

20.  Yukins, supra note 10, at 31.

21.  See 4 CFR 21.0(e), 21.2.

22.  See Yukins, supra note 10, at 13-14, 18-19.

23.  See id. at 23.

24.  48 CFR 33.103(f). Under certain circumstances, the agency can override the regulatory stay for agency-level protests. See 48 CFR 33.103(f)(1), (f)(3).

25.  31 U.S.C. 3553(c)(1), (d)(3). Under certain circumstances, the agency can override the statutory stay for protests to GAO. See 31 U.S.C. 3553(c)-(d); 48 CFR 33.104(b)-(c).

1.  See Admin. Conf. of the U.S., Recommendation 2016-2, Aggregate Agency Adjudication, 81 FR 40,260, 40,260 (June 21, 2016).

2.  Admin. Conf. of the U.S., Recommendation 2018-5, Public Availability of Adjudication Rules, 84 FR 2142 (Feb. 6, 2019); Admin. Conf. of the U.S., Recommendation 2017-1, Adjudication Materials on Agency Websites, 82 FR 31,039 (July 5, 2017).

3.  Cf. Admin. Conf. of the U.S., Recommendation 2018-4, Recusal Rules for Administrative Adjudicators, 84 FR 2139 (Feb. 6, 2019).

4.  See, e.g., Lucia v. SEC, 138 S. Ct. 2044 (2018); Arthrex v. Smith & Nephew, 941 F.3d 1320 (Fed. Cir. 2019), cert. granted, __S. Ct. __(Oct. 13, 2020) (No. 19-1434).

5.  FOIA Improvement Act of 2016, Public Law 114-185, 2, 130 Stat. 538, 538 (amending 5 U.S.C. 552(a)(2)); E-Government Act of 2002, Public Law 140-347, 206, 116 Stat. 2899, 2916 (amending 44 U.S.C. 3501).

6.  See, e.g., Admin. Conf. of the U.S., Recommendation 2018-4, Recusal Rules for Administrative Adjudicators, 84 FR 2139 (Feb. 6, 2019).

7.  See 5 U.S.C. 554, 556-57.

8.  The vast majority of ALJs work at the Social Security Administration. AJs work at many different agencies under a variety of titles, including not only “Administrative Judge” but also, by way of example, “Hearing Officer,” “Immigration Judge,” “Veterans Law Judge,” “Administrative Patent Judge,” and “Administrative Appeals Judge.”

9.  5 U.S.C. 554, 557, 3105, 4301, 5372, 7521; 5 CFR pt. 930, subpt. B; Exec. Order No. 13,843, Executive Order Excepting Administrative Law Judges from the Competitive Service, 83 FR 32,755 (July 13, 2018) (issued July 10, 2018).

10.  Kent Barnett et al., Non-ALJ Adjudicators in Federal Agencies: Status, Selection, Oversight, and Removal 1 (Sept. 24, 2018) (report to the Admin. Conf. of the U.S.), https://www.acus.gov/​report/​non-alj-adjudicators-federal-agencies-status-selection-oversight-and-removal-1.

11.  Leigh Anne Schriever, Public Availability of Information About Adjudicators 10 (Nov. 23, 2020) (report to the Admin. Conf. of the U.S.), https://www.acus.gov/​report/​final-report-public-availability-information-about-agency-adjudicators.

12.  Id. at 7.

13.  5 U.S.C. 552.

14.  Id. § 552a.

15.  Cf. Admin. Conf. of the U.S., Recommendation 2017-3, Plain Language in Regulatory Drafting, 82 FR 61,728 (Dec. 29, 2017).

1.  The term “component units” encompasses an agency's sub-units, which are often identified under terms like “agency,” “bureau,” “administration,” “office,” “division,” or “service.” For example, the United States Fish and Wildlife Service is a component unit of the Department of the Interior, and the Office of Water is a component unit of the United States Environmental Protection Agency.

2.  See 5 U.S.C. 552(a)(3).

3.  See 44 U.S.C. 3502(a).

4.  See Mark Thomson, Report on Agency Litigation web pages 14-16 (Nov. 24, 2020) (report to the Admin. Conf. of the U.S.), https://www.acus.gov/​report/​report-agency-litigation-web pages.

5.  See id. at 12-19 (identifying variations in agency practices). The survey conducted for this Recommendation covered all kinds of agencies—big and small, independent and not, regulatory and benefit-oriented, and so forth—with the aim of covering a broad and at least somewhat representative cross-section of federal agencies. In particular, the survey focused on agencies that are frequently in federal court or that are parties to a significant number of high-profile cases.

6.  Most federal agencies do not have independent litigation authority but are represented in court by the Department of Justice (DOJ). In most cases, these agencies designate a DOJ liaison, who is then added as a recipient for all court filing notices, resulting in automatic access to all filings via PACER. This automatic access should enable implementation of this Recommendation by client agencies.

7.  Cf. Admin. Conf. of the U.S., Recommendation 2017-1, Adjudication Materials on Agency websites, 82 FR 31,039, 31,040 (July 5, 2017) (“Agencies that adjudicate large volumes of cases that do not vary considerably in terms of their factual contexts or the legal analyses employed in their dispositions should consider disclosing on their websites a representative sampling of actual cases and associated adjudication materials.”).