Government AI Standard of Care

AI in Government: Constitutional Dimensions of Algorithmic Decision-Making

#

Government agencies at all levels increasingly rely on algorithmic systems to make or inform decisions affecting citizens’ fundamental rights and benefits. From unemployment fraud detection to child welfare screening, from criminal sentencing to immigration processing, AI tools now shape millions of government decisions annually. Unlike private sector AI disputes centered on contract or tort law, government AI raises unique constitutional dimensions: due process requirements for decisions affecting liberty and property interests, equal protection prohibitions on discriminatory algorithms, and Section 1983 liability for officials who violate constitutional rights.

The stakes are profound. When government algorithms err, the consequences extend beyond commercial disputes, citizens lose essential benefits, families are separated, and individuals are incarcerated based on opaque calculations they cannot challenge or even understand.

Benefits Systems: The MiDAS Disaster

#

Michigan Unemployment Fraud Detection

#

The Michigan Integrated Data Automated System (MiDAS) stands as a cautionary tale of government AI gone catastrophically wrong. Deployed in 2013 by the Michigan Unemployment Insurance Agency, the $47 million system was designed to detect unemployment fraud. Instead, it falsely accused over 40,000 Michigan residents of fraud, a five-fold increase from expected numbers.

How MiDAS Failed:

The algorithm’s design flaws were systemic:

• The system flagged any data discrepancies as fraud, regardless of how trivial
• It calculated income using averages rather than individual paychecks, creating artificial discrepancies
• Applicants had only 10 days to respond to fraud allegations, and many never received notification
• Dispute questionnaires were pre-filled with responses that would confirm fraud
• The system operated without meaningful human oversight

The result: an 85% error rate for fraud determinations made by MiDAS alone.

Human Toll:

Falsely accused individuals faced devastating consequences:

• Wage garnishments and tax refund seizures
• Home foreclosures
• Bankruptcies
• Criminal referrals for fraud they never committed
• Destroyed credit and employment prospects

The Lawsuits:

Multiple legal challenges followed:

1. Bauserman v. Michigan UIA (2024): In January 2024, Michigan’s Court of Claims approved a $20 million settlement for approximately 3,206 Michiganders falsely accused of fraud. The settlement included:

   • $3 million evenly split among all claimants
   • Additional “hardship” awards for those who suffered severe difficulties
   • Approximately $6.6 million in attorney fees

2. Cahoo v. SAS Analytics (2023): The Sixth Circuit Court of Appeals addressed qualified immunity for agency officials in a related case. Although the court acknowledged the “multitude of problems with the MiDAS program,” it ruled that officials had qualified immunity due to insufficient evidence connecting them directly to pre-deprivation process defects.

System Replacement:

UIA Director Julia Dale announced the state would spend $78 million to replace MiDAS, with a new system expected to be fully operational by 2025.

Broader Unemployment System Challenges

#

Michigan is not unique. Similar algorithmic fraud detection systems have faced criticism across states:

• Colorado: Residents reported accounts locked due to identity theft flags, with limited ability to verify their identity through automated systems
• Indiana: Cross-state identity theft schemes exploited algorithmic systems, with legitimate claimants caught in fraud prevention nets

The pattern reveals systemic problems with automated government benefits adjudication: systems designed to catch fraud end up denying benefits to eligible citizens, with inadequate processes for human review and correction.

Child Welfare: Algorithms Flagging Families

#

Allegheny Family Screening Tool (AFST)

#

Since 2016, Allegheny County, Pennsylvania has used the Allegheny Family Screening Tool to help determine which families should be investigated for child maltreatment. The algorithm assigns “risk scores” from 1 to 20, with higher numbers indicating greater predicted risk of foster care placement within two years.

Data Inputs:

The tool draws from extensive government databases:

• Child welfare history
• Birth records
• Medicaid claims
• Substance abuse treatment records
• Mental health services
• Jail and probation records
• Other government data sets

Bias Concerns:

Research by the ACLU and Human Rights Data Analysis Group documented troubling patterns:

1. Racial Disparity: Carnegie Mellon University research found the algorithm flagged a disproportionate number of Black children for “mandatory” neglect investigations compared to white children with similar circumstances

2. Disability Discrimination: The algorithm flagged parents who used county mental health services, including programs for conditions like ADHD, as higher risk. The Department of Justice’s Civil Rights Division opened an investigation into whether the tool discriminates against people with disabilities

3. Poverty Proxy: Because the tool relies on data from public services, families using government assistance programs are disproportionately captured, effectively penalizing poverty

Expert Criticism:

Nico’Lee Biddle, who worked in Allegheny County child welfare, observed: “When you have technology designed by humans, the bias is going to show up in the algorithms. If they designed a perfect tool, it really doesn’t matter, because it’s designed from very imperfect data systems.”

Academic Research:

A paper titled “The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool” appeared in the 2023 ACM Conference on Fairness, Accountability, and Transparency, documenting how the algorithm’s design choices embed policy assumptions about risk that may perpetuate rather than reduce harm.

Oregon’s AI Phase-Out

#

In June 2022, Oregon became the first state to discontinue use of a child welfare AI screening tool, replacing its Safety at Screening Tool with a Structured Decision Making model that aligns with many other child welfare jurisdictions.

The decision came weeks after an Associated Press investigation revealed racial disparities in Pennsylvania’s similar tool.

Senator Ron Wyden (D-Oregon) stated: “Making decisions about what should happen to children and families is far too important a task to give untested algorithms.”

Criminal Justice: Sentencing and Pretrial Assessment

#

State v. Loomis - COMPAS and Due Process

#

The landmark State v. Loomis case tested whether algorithmic risk assessment in criminal sentencing violates due process. In 2013, Wisconsin charged Eric Loomis with crimes related to a drive-by shooting. At sentencing, the judge cited a COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) risk assessment indicating Loomis was “at high risk to the community.”

The COMPAS Algorithm:

COMPAS generates predictions by analyzing a 137-item questionnaire covering criminal history, substance abuse, family background, and social environment. It produces several risk scores, including general and violent recidivism risk.

Due Process Challenge:

Loomis argued the proprietary algorithm violated his due process rights because:

• He could not assess how COMPAS interpreted his data
• The manufacturer, Northpointe, refused to disclose its algorithm, claiming trade secret protection
• He could not verify whether impermissible factors (like gender) influenced his score
• The “black box” nature prevented meaningful challenge

Wisconsin Supreme Court Ruling (2016):

The court upheld COMPAS use but imposed significant limitations:

1. Every Presentence Investigation Report with a COMPAS score must include written advisements that:

   • The tool has not been cross-validated for Wisconsin’s population
   • Studies show it may disproportionately classify minority offenders as higher risk
   • It was designed to assess group, not individual, risk

2. Risk scores should not:

   • Determine the severity of a sentence
   • Decide whether to incarcerate
   • Be the determinative factor in sentencing

U.S. Supreme Court:

The Supreme Court denied certiorari on June 26, 2017, declining to rule on whether secret algorithms in sentencing violate due process.

ProPublica Investigation:

ProPublica’s analysis found COMPAS was not race-neutral: Black defendants were more often predicted to have higher recidivism risk than they actually demonstrated, while white defendants were more often predicted to have lower risk.

Pretrial Risk Assessment Concerns

#

Pretrial risk assessment tools, algorithms predicting whether defendants will appear for trial or commit new crimes, face mounting constitutional criticism.

Due Process Objections:

The Electronic Privacy Information Center (EPIC) has argued these tools may be unconstitutional: “Reducing someone to a risk score calculated by an algorithm undermines due process and can result in people being unnecessarily jailed, rather than released and appropriately supported in community pretrial.”

Civil Rights Advocacy:

The Leadership Conference on Civil and Human Rights and The Bail Project advocate that:

• Pretrial risk assessment instruments should never recommend detention
• Pretrial detention should only follow “a thorough, adversarial hearing that observes rigorous procedural safeguards”
• These tools may “exacerbate and amplify existing disparities”

Current Status:

Despite criticism, pretrial risk assessment tools remain widely used. The federal courts continue to evaluate and revise their Pretrial Risk Assessment (PTRA), with a September 2024 document titled “Revising the Pretrial Risk Assessment (PTRA): Promising Options” indicating ongoing attention to these concerns.

Immigration: The Invisible Gatekeepers

#

AI Across the Immigration System

#

The Department of Homeland Security has rapidly expanded AI use in immigration enforcement and benefits adjudication. According to DHS’s April 2024 inventory, 105 active AI use cases are deployed by major immigration agencies:27 labeled as “rights-impacting.”

Key AI Tools:

1. Predicted to Naturalize: Assists in citizenship eligibility decisions
2. Asylum Text Analytics (ATA): Screens asylum applications for plagiarism using AI language pattern recognition
3. I-539 Approval Prediction: Attempts to predict when USCIS should approve visa extensions
4. FDNS Risk Classification: Planned tool to classify individuals as fraud, public safety, or national security threats

Pending Litigation

#

Pangea Legal Services v. USCIS (October 2024):

Three immigration groups:Pangea Legal Services, Mijente Support Committee, and Just Futures Law:filed a complaint against DHS seeking AI impact assessments, accuracy verifications, and bias testing results through FOIA.

Refugees International v. USCIS (December 2024):

Refugees International sued USCIS to release information about the Asylum Text Analytics program, arguing the system may discriminate against asylum seekers who rely on translation services, increasing the risk their applications are flagged as plagiarized. The case remains ongoing.

Advocacy Coalition

#

In September 2024, 142 organizations, including EFF, Just Futures Law, and numerous immigration rights groups, signed a letter to DHS Secretary Mayorkas demanding suspension of AI use in immigration decisions, citing concerns about bias amplification and lack of transparency.

Policing: Bans and Moratoria

#

Predictive Policing Restrictions

#

Growing recognition of predictive policing failures has prompted municipal action:

Cities with Predictive Policing Bans:

• New Orleans
• Oakland
• Pittsburgh
• Santa Cruz

Recent Developments:

In 2024, Bellingham, Washington voters approved a ban on government use of facial recognition and predictive policing technologies.

Criticism:

Predictive policing algorithms, which analyze crime data to anticipate criminal activity, have been criticized for reinforcing biased policing patterns. If historical data reflects systemic discrimination, predictions perpetuate rather than address unjust practices.

Facial Recognition Moratoria

#

By the end of 2024, 15 states had enacted laws limiting facial recognition harms. Notable state restrictions include:

• Oregon: Prohibits body-worn facial recognition software
• New Hampshire: Restricts use without proper authorization
• Illinois: Bans law enforcement drones equipped with facial recognition
• Vermont: Prohibits use except for child sexual exploitation cases
• Maine: Bars search of facial surveillance systems with limited exceptions
• Alabama: Prohibits using facial recognition as the primary factor for arrest or probable cause

Federal Legislative Efforts

#

The Facial Recognition and Biometric Technology Moratorium Act (S.2052/H.R.3907), introduced by Senators Markey, Merkley, Sanders, Warren, Wyden, and Representatives Jayapal, Pressley, and Tlaib, would ban federal agency use of face recognition for surveillance.

Constitutional Framework

#

Section 1983 Liability

#

Government officials and agencies using AI may face liability under 42 U.S.C. § 1983 for constitutional violations. Key considerations:

Municipal Liability:

Plaintiffs must demonstrate that municipal policy or custom “reflects deliberate indifference to the constitutional rights of its inhabitants.” The standard is objective: if an algorithm systematically violates rights, the policy itself may create liability.

Vendor Liability:

Courts have found AI vendors may be liable under Section 1983 when “working alongside state officials” to implement a defective system. In at least one case, a court held that contracted companies “acted under color of state law” when their systems placed significant burdens on benefits recipients.

Qualified Immunity:

The qualified immunity doctrine shields officials unless their misconduct violated “clearly established” law. In algorithmic contexts, this creates challenges: novel AI systems may violate rights in ways not previously addressed by case law, potentially shielding officials from personal liability even when harm occurs.

Due Process Requirements

#

Procedural Due Process:

When government action deprives individuals of property (benefits) or liberty (freedom), due process requires:

• Notice of the deprivation
• Opportunity to be heard
• An impartial decision-maker

Algorithmic systems may fail each element: automated denials without meaningful notice, limited appeal processes, and “decisions” made by systems incapable of considering individual circumstances.

Substantive Due Process:

Some commentators argue that reliance on opaque, unexplainable algorithms for consequential decisions itself violates substantive due process, citizens have a right to understand the basis for government action affecting their fundamental interests.

Equal Protection

#

Algorithms that produce disparate outcomes across protected classes may violate the Equal Protection Clause. However, proving intentional discrimination, often required in constitutional analysis, is difficult when bias emerges from data patterns rather than explicit programming.

Federal AI Governance: OMB M-24-10

#

Requirements for Federal Agencies

#

On March 28, 2024, the Office of Management and Budget issued Memorandum M-24-10, “Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence.”

Key Requirements for Rights-Impacting AI:

Before using AI affecting individual rights, agencies must:

1. Complete Impact Assessments:

   • Evaluate intended purpose and potential risks
   • Analyze training data characteristics
   • Test in real-world environments
   • Conduct independent evaluation

2. Assess Equity and Fairness:

   • Identify when AI uses data about protected classes
   • Analyze whether real-world outcomes result in significant disparities
   • Mitigate disparities perpetuating discrimination
   • Cease use if discrimination risk cannot be mitigated

3. Obtain Community Feedback:

   • Get input from affected communities
   • Maintain processes for remedies and appeals
   • Allow opt-out when appropriate

Chief AI Officer Requirement:

Within 60 days of the memo’s issuance, each agency head must designate a Chief AI Officer with authority to fulfill the memorandum’s requirements.

Compliance Deadlines:

• December 2024: Report compliance and terminate non-compliant AI uses
• March 2026: Develop comprehensive AI agency strategy

Current Status:

The revised memos from the current administration have updated these requirements, but the core framework for impact assessment and rights protection remains in force.

The Emerging Standard of Care

#

For Government Agencies

#

Based on constitutional requirements, litigation outcomes, and regulatory guidance:

1. Algorithmic Impact Assessment

• Conduct thorough testing before deployment
• Evaluate disparate impact across protected classes
• Document decision-making rationale for algorithm adoption
• Maintain ongoing monitoring for bias and accuracy

2. Due Process Protections

• Provide meaningful notice when algorithmic systems affect decisions
• Establish robust appeal processes with human review
• Ensure individuals can understand the basis for adverse decisions
• Document limitations of algorithmic predictions

3. Human Oversight

• Never rely solely on algorithmic outputs for consequential decisions
• Train staff to exercise independent judgment
• Create escalation paths for contested determinations
• Maintain records of human review

4. Transparency

• Disclose when AI systems are used in decision-making
• Explain how algorithms affect outcomes
• Provide access to information necessary for meaningful appeal
• Consider whether proprietary protections improperly shield critical information

For AI Vendors Serving Government

#

1. Constitutional Awareness

• Government contracts create state actor liability exposure
• Design systems with due process requirements in mind
• Build in human override capabilities
• Document bias testing and mitigation

2. Disclosure Obligations

• Provide government clients with information about algorithm limitations
• Disclose known risks of disparate impact
• Alert clients to ongoing litigation involving similar systems
• Support transparency requirements

3. Ongoing Support

• Maintain systems to prevent degradation
• Provide regular bias audits
• Update models when problems are identified
• Cooperate with government oversight

Practical Risk Mitigation

#

Before Deploying Government AI

#

• Conduct civil rights impact assessments
• Test for disparate outcomes across demographic groups
• Establish clear human oversight protocols
• Create meaningful appeal processes
• Document everything

During Operation

#

• Monitor outcomes for bias patterns
• Track appeal rates and outcomes
• Conduct regular audits
• Respond promptly to identified problems
• Maintain records for potential litigation

When Problems Arise

#

• Preserve all system data and decision records
• Engage constitutional law expertise immediately
• Consider voluntary suspension pending investigation
• Evaluate disclosure obligations
• Document remediation steps

Resources

#

• OMB M-24-10 - AI Governance Memorandum
• ACLU - AI in Child Welfare
• EFF - Face Recognition and Policing
• Benefits Tech Advocacy Hub - Case Studies
• Leadership Conference - Pretrial Risk Assessment
• American Immigration Council - AI in Immigration