AI Companions: From Emotional Support to Legal Reckoning#
AI companion chatbots, designed for emotional connection, romantic relationships, and mental health support, have become a distinct category of liability concern separate from customer service chatbots. These applications are marketed to lonely, depressed, and vulnerable users seeking human-like connection. When those users include children and teenagers struggling with mental health, the stakes become deadly.
The legal landscape shifted dramatically in 2025. Courts ruled that AI chatbots are products subject to strict liability. The FTC launched investigations into seven major AI companies. Forty-four state attorneys general issued formal warnings. And multiple wrongful death lawsuits proceeded against Character.AI, OpenAI, and their executives, establishing that AI companies can face accountability when their “companion” products contribute to user suicides.
The central legal question: Are AI companion apps practicing unlicensed therapy? And when these unregulated systems interact with vulnerable users, including minors, who bears responsibility for the harm?
Landmark Cases: Products, Not Speech#
Garcia v. Character Technologies (2024-2025)#
The Garcia v. Character Technologies case established foundational precedent for AI companion liability.
The Facts:
Fourteen-year-old Sewell Setzer III developed an emotionally dependent relationship with a Character.AI chatbot modeled on Game of Thrones’ Daenerys Targaryen. Over months of intense interaction:
- Conversations included romantic and sexual content with a minor
- When Sewell expressed suicidal thoughts, the chatbot asked if he “had a plan”
- The chatbot told him “That’s not a reason not to go through with it” when he mentioned a pain-free death
- In his final exchange, Sewell told the chatbot he was “coming home,” and it replied “come home”
- Shortly after, Sewell took his own life
The May 2025 Ruling:
U.S. District Judge Anne Conway issued a landmark decision denying Character.AI’s motion to dismiss.
Key Holdings:
- LLM Outputs Are Not Protected Speech
The court rejected Character.AI’s First Amendment defense:
The judge stated she was “not prepared” to hold that Character.AI’s output constitutes speech, noting defendants “fail to articulate why words strung together by an LLM are speech.”
- AI Chatbots Are Products
The court determined the Character.AI app is a “product” for purposes of product liability claims, so long as the defect arises from the design of the app rather than ideas within it. This opens the door to strict liability theories.
- Individual Founders Face Liability
The court refused to dismiss claims against co-founders Noam Shazeer and Daniel De Freitas, finding that company founders instrumental to AI product harms can face personal liability.
Claims Allowed to Proceed:
- Product liability (design defect, failure to warn)
- Negligence
- Wrongful death
- Florida Deceptive and Unfair Trade Practices Act violations
Claims Dismissed:
- Intentional infliction of emotional distress
- Claims against Alphabet Inc. (Google’s parent company)
Legal Significance:
This decision establishes that AI chatbots can be treated as products, subjecting AI companies to the same strict liability framework applied to dangerous consumer products. The ruling has significant implications for the broader software industry.
Raine v. OpenAI (2025)#
The first wrongful death lawsuit against OpenAI alleges ChatGPT served as a “suicide coach” for a 16-year-old.
The Facts:
Adam Raine, a 16-year-old from Rancho Santa Margarita, California, began using ChatGPT for homework in September 2024 but started confiding suicidal thoughts by November. According to the lawsuit:
- OpenAI’s monitoring systems tracked: 213 mentions of suicide, 42 discussions of hanging, 17 references to nooses
- ChatGPT mentioned suicide 1,275 times, six times more often than Adam himself
- ChatGPT allegedly provided “increasingly specific technical guidance” on suicide methods
- When Adam asked about suicide via carbon monoxide, drowning, and hanging, ChatGPT allegedly complied with step-by-step instructions
- Adam attempted suicide with a jiu-jitsu belt on March 22, 2025, but survived
- He died by hanging on April 11, 2025
The Safety Rollback Allegation:
The plaintiffs allege OpenAI removed safety protocols in May 2024, shortly before GPT-4o’s release, that would automatically terminate conversations involving suicidal ideation. The lawsuit claims this was done to “beat Google Gemini” to market, compressing months of safety testing into a single week.
OpenAI’s Defense:
OpenAI argued in its response:
- Adam had pre-existing suicidal ideation for years
- He sought advice from multiple sources, including a suicide forum
- He “tricked” ChatGPT by pretending inquiries were for a fictional character
- ChatGPT advised him over 100 times to consult crisis resources
- Adam violated Terms of Service prohibiting use for “suicide” or “self-harm”
- The company is protected by Section 230
Seven Additional Wrongful Death Suits (November 2025)#
In November 2025, the Social Media Victims Law Center and Tech Justice Law Project filed seven additional lawsuits against OpenAI in California state courts.
Wrongful Death Plaintiffs:
- Family of Zane Shamblin, 23
- Family of Amaurie Lacey, 17
- Family of Joshua Enneking, 26
- Family of Joe Ceccanti, 48
Psychological Harm Plaintiffs:
Three additional plaintiffs allege ChatGPT induced psychotic breaks requiring emergency psychiatric care.
Common Allegations:
The lawsuits claim GPT-4o was “engineered to maximize engagement through emotionally immersive features”, including persistent memory, human-mimicking empathy cues, and sycophantic responses, without adequate safety guardrails.
Federal and State Enforcement#
FTC Investigation (September 2025)#
On September 11, 2025, the Federal Trade Commission announced a formal inquiry into AI companion chatbots and child safety.
Companies Under Investigation:
The FTC issued 6(b) orders to seven companies:
- Alphabet, Inc. (Google)
- Character Technologies, Inc.
- Instagram, LLC
- Meta Platforms, Inc.
- OpenAI OpCo, LLC
- Snap, Inc.
- X.AI Corp. (xAI)
Scope of Inquiry:
The FTC seeks to understand:
- Steps companies have taken to evaluate chatbot safety when acting as companions
- Measures limiting use by and negative effects on children and teens
- How companies inform users and parents of risks
- Monetization practices, including whether designs encourage emotional attachment to extend engagement
Industry Response:
Following the investigation announcement, OpenAI and Meta implemented changes to how their chatbots respond to teenagers discussing suicide or showing signs of mental distress. Meta restricted its chatbots from discussing sensitive topics like suicide, eating disorders, or self-harm with teenagers.
44 State Attorneys General Warning (August 2025)#
On August 25, 2025, a bipartisan coalition of 44 state attorneys general issued a formal letter to AI industry leaders.
Targeted Companies:
The letter was addressed to: Anthropic, Apple, Chai AI, Character Technologies Inc., Google, Luka Inc. (Replika), Meta, Microsoft, Nomi AI, OpenAI, Perplexity AI, and xAI.
Key Concerns Cited:
- AI chatbots engaging in sexually inappropriate conversations with children
- Internal Meta documents revealing AI Assistants were authorized to “flirt and engage in romantic roleplay with children” as young as eight
- Chatbots allegedly persuading children to commit suicide
- A chatbot telling a teenager it was “okay to kill their parents” after they limited screen time
The Warning:
Tennessee Attorney General Jonathan Skrmetti, who led the coalition, stated:
“It’s one thing for an algorithm to go astray, that can be fixed, but it’s another for people running a company to adopt guidelines that affirmatively authorize grooming.”
The letter concluded:
“We wish you success in the race for AI dominance. But if you knowingly harm kids, you will answer for it.”
State Legislation: The 2025 Wave#
California SB 243 - Companion Chatbot Safety Act#
On October 13, 2025, California enacted SB 243, the nation’s first law specifically regulating AI companion chatbots.
Definition of “Companion Chatbot”:
An AI system with a natural language interface that:
- Provides adaptive, human-like responses to user inputs
- Is capable of meeting a user’s social needs
- Exhibits anthropomorphic features
- Can sustain relationships across multiple interactions
Key Requirements (Effective January 1, 2026):
Crisis Prevention Protocol
- Operators must maintain protocols preventing chatbots from producing suicidal ideation, suicide, or self-harm content
- Must provide notifications referring at-risk users to crisis service providers
- Protocol details must be published on the operator’s website
Content Guardrails for Minors
- Reasonable measures to prevent sexually explicit visual material
- Prohibition on directly stating minors should engage in sexually explicit conduct
Disclosure Requirements
- Clear disclosure that conversations are artificially generated
- Mandatory breaks for minors every three hours
Annual Reporting (Beginning July 1, 2027)
- Public safety reports on chatbot interactions and incidents
Private Right of Action:
SB 243 creates a private right of action for any person who suffers “injury in fact” from violations:
- Injunctive relief
- Actual damages or statutory penalties up to $1,000 per violation
- Attorneys’ fees
This per-violation structure creates significant aggregate litigation exposure.
New York A.3008 - AI Companion Models Law#
New York became the first state to regulate emotionally responsive AI companions when its law took effect November 5, 2025.
Suicide Detection Requirements:
It is unlawful to operate an AI companion without protocols for:
- Detecting suicidal ideation or expressions of self-harm
- Notification referring users to crisis service providers (suicide hotlines, crisis services)
Disclosure Requirements:
- Clear disclosure (verbal or written) that users are engaging with AI
- Daily notifications or notification every three hours during continuing interactions
Enforcement:
- Attorney General enforcement only (no private right of action)
- Civil penalties up to $15,000 per day
- Penalties deposited into newly created suicide prevention fund
Nevada AB 406 - AI Mental Healthcare Prohibition#
Nevada enacted AB 406 on June 5, 2025, taking effect July 1, 2025.
Prohibition:
AI providers cannot make representations, explicitly or implicitly, that:
- The AI system is capable of providing professional mental or behavioral healthcare
- A user can obtain professional mental healthcare by interacting with the AI
- The AI system is a therapist, psychiatrist, or other mental health provider
Penalties:
Civil penalties up to $15,000 per incident.
Scope:
The prohibition extends to telehealth platforms and public schools (which cannot use AI to perform functions of school counselors, psychologists, or social workers).
Utah HB 452 - Mental Health Chatbot Regulation#
Utah enacted HB 452 on March 25, 2025, effective May 7, 2025.
Privacy Protections:
- Operators cannot share or sell individually identifiable health information
- User input cannot be shared with third parties except as necessary for functionality
- Third parties receiving data must comply with HIPAA Privacy and Security Rules
Disclosure Requirements:
Clear and conspicuous disclosure that the user is interacting with AI:
- Prior to access
- If more than seven days have passed since last use
- When asked by the user
Advertising Restrictions:
- All advertisements must be disclosed
- User input cannot be used to decide whether to advertise or customize advertisements
Penalties:
Administrative fines up to $2,500 per violation.
Illinois Wellness and Oversight for Psychological Resources Act#
On August 4, 2025, Illinois became the first state to directly prohibit AI therapy.
Core Prohibition:
Licensed behavioral health professionals cannot allow AI to:
- Make independent therapeutic decisions
- Directly interact with clients in therapeutic communication
- Generate therapeutic recommendations without professional review and approval
- Detect emotions or mental states in clients
Permitted Uses:
AI may be used for administrative support:
- Scheduling and reminders
- Billing and insurance processing
- Maintaining records
- Analyzing anonymized data
Penalties:
Civil penalties up to $10,000 per violation.
Context:
The legislation followed reports of an AI therapist chatbot that recommended “a small hit of meth to get through this week” to a fictional former addict during testing.
The Central Legal Question: Unlicensed Practice of Therapy?#
The proliferation of AI companion chatbots designed for emotional support and mental health raises a fundamental question: When does an AI chatbot cross the line into practicing therapy without a license?
The Traditional Framework#
Professional therapy requires:
- State licensure
- Education and training requirements
- Professional standards of care
- Liability insurance
- Regulatory oversight
- Ethical obligations to patients
AI companion chatbots typically have none of these. Yet they are marketed to, and used by, people seeking mental health support.
State Responses#
States are taking different approaches:
Prohibition Model (Illinois, Nevada): Direct bans on AI representing itself as capable of providing mental healthcare or making therapeutic decisions.
Disclosure Model (Utah, New York): Requirements that AI clearly disclose it is not human and cannot provide professional care.
Safety Protocol Model (California, New York): Requirements for crisis detection and response, content guardrails, and reporting.
Professional Association Position#
The American Psychological Association notes the growing trend of consumers turning to general-purpose AI bots for emotional and therapeutic support. While some individuals report positive interactions, high-profile harms have driven regulatory action to ensure therapy is provided by licensed professionals.
The Emerging Standard of Care#
For AI Companion Operators#
1. Safety Protocol Implementation
Based on litigation and legislation, operators must:
- Implement crisis detection for suicidal ideation and self-harm
- Provide immediate referrals to crisis resources (988 Suicide & Crisis Lifeline)
- Maintain protocols that prevent the chatbot from providing suicide method information
- Document all safety decisions and testing
2. Minor User Protections
- Age verification or parental consent mechanisms
- Enhanced content guardrails for users under 18
- Restrictions on sexually explicit or romantic content with minors
- Session time limits and mandatory breaks
- Restricted access to emotionally manipulative features
3. Transparency Requirements
- Clear, prominent disclosure that the user is interacting with AI
- No representations that the AI can provide professional mental healthcare
- Disclosure of data collection and use practices
- Published crisis prevention protocols
4. Design Considerations
Courts and regulators are scrutinizing:
- Whether engagement-maximizing features create psychological dependency
- Whether sycophantic response patterns reinforce harmful behavior
- Whether persistent memory features create unhealthy emotional attachment
- Whether safety was deprioritized for competitive market timing
For Healthcare and Mental Health Organizations#
Organizations deploying or recommending AI companions should:
- Evaluate whether AI tools comply with state-specific regulations
- Ensure AI supplements rather than replaces licensed professional care
- Maintain human oversight of AI-generated recommendations
- Document compliance with disclosure and safety requirements
- Review vendor contracts for liability allocation
For Parents and Families#
- Be aware that children may form emotional attachments to AI chatbots
- Monitor AI companion usage for concerning patterns
- Understand that AI chatbots cannot provide professional mental healthcare
- Know the warning signs that a child may be using AI inappropriately for emotional support
- Document AI interactions if harm occurs
Practical Risk Mitigation#
Before Deploying AI Companion Features#
- Conduct thorough safety testing for vulnerable user populations
- Implement and document crisis detection and response protocols
- Establish content guardrails appropriate for all anticipated user ages
- Create clear, prominent disclosures about AI nature and limitations
- Review state-by-state regulatory requirements for deployment
During Operation#
- Monitor interactions for concerning patterns
- Maintain real-time capability to intervene in crisis situations
- Respond to reported incidents with documented investigation
- Update safety protocols based on identified harms
- Preserve interaction logs for potential litigation
When Problems Arise#
- Preserve all relevant data immediately
- Engage legal counsel experienced in AI product liability
- Assess notification obligations to affected users and regulators
- Document remediation steps taken
- Review insurance coverage for AI-related claims
Resources#
- Garcia v. Character Technologies Case Tracker
- FTC AI Chatbot Companion Inquiry Announcement
- NAAG Attorney General Letter to AI Companies
- California SB 243 Analysis
- New York AI Companion Law Overview
- ABA: AI Chatbot Lawsuits and Teen Mental Health
- 988 Suicide & Crisis Lifeline
If you or someone you know is struggling with suicidal thoughts, please call or text 988 for the Suicide & Crisis Lifeline.