New York’s AI Companion Bill: Guardrails for Virtual Relationships
New York lawmakers are aiming for “AI companions” — AI systems (including generative AI and emotional recognition tools) built to simulate an ongoing human-like relationship with a user.
Under the bill, operators can’t offer an AI companion unless it has a built-in protocol to detect and respond to suicidal thoughts or self-harm statements. That means two things:
The AI must be able to flag when a user expresses suicidal ideation or self-harm.
It must notify the user and point them toward crisis service providers.
The bill also requires AI companion providers to disclose—at the start of each conversation and every three hours during long chats—that the user is not talking to a human.
Enforcement rests with the New York Attorney General, who could seek injunctions and civil penalties of up to $15,000 per day for violations. There’s no private right of action, so individuals can’t sue directly under the bill.
Hey there!
You’re reading Law & Learning Machines — a slow(ish) newsletter about the messy, fascinating intersection of law and artificial intelligence. Every issue digs into the policies, court cases, and big questions shaping how humans and machines interact.
GPT-5 Is Here — And It’s Smarter, Faster, and Has Fewer Hulitations.
OpenAI just introduced GPT-5, its most advanced AI yet, with major leaps in reasoning, coding, writing, and even health applications. Unlike past releases, GPT-5 is designed as a unified system that decides whether to respond quickly or take more time to “think,” depending on the complexity of your prompt.
Regulatory Impact: Smarter, more autonomous AI blurs lines around professional licensing, liability, and malpractice — especially in legal and health contexts.
Copyright at Scale: GPT-5’s ability to generate complex, polished front-end code in a single prompt raises IP questions about authorship, originality, and copyright.
Reduced Hallucinations: Fewer made-up facts (Maybe) means fewer AI-generated legal citations gone wrong, but “fewer” is not “none.” This still matters for evidentiary reliability.
In short, GPT-5 isn’t just a tech upgrade — it’s a bigger, faster-moving target for lawyers, policymakers, and anyone thinking about AI governance.
Court Issues Admonishment Over Fabricated Case Law in AI-Linked Filing
In an Eastern District of New York case, the court considered sanctions after fabricated legal citations appeared in a plaintiff’s opposition brief — allegedly the result of AI misuse.
Defense counsel flagged the fake citations in a pre-motion letter. Plaintiff’s counsel responded, voluntarily dropped certain claims, and later explained the brief had been drafted by a clerk using Google without verifying citations. Counsel attributed the oversight to the sudden death of her spouse, which she said profoundly affected her ability to focus on her practice.
Under Rule 11, attorneys certify that filings are grounded in fact and law after a reasonable inquiry. The court stressed this duty is nondelegable — lawyers must confirm the existence and validity of cited legal authority themselves. While the court found bad faith, it also acknowledged counsel’s remorse and the “unspeakable tragedy” she endured.
In the end, the court issued an admonishment but no monetary penalty.
The takeaway: Even in extraordinary personal circumstances, Rule 11’s duty to verify citations remains absolute — and fabricated case law can still trigger public admonishment.
It Ain’t All About AI
Most recent sanctions headlines involve fake case citations where courts admonish or fine attorneys for filing phantom authorities without verifying them. These cases drive home the nondelegable Rule 11 duty to confirm every citation is real. Some Rule 11 Sanctions are for bread-and-butter old-timey dishonesty.
Philip Brumley, General Counsel for the Watch Tower Bible and Tract Society of Pennsylvania (WTPA), was sanctioned for submitting an affidavit that misrepresented facts about the court’s jurisdiction. Brumley swore that WTPA had no contact with Montana congregations, didn’t set policy, and didn’t appoint church officials — all in the present tense. The court found these statements knowingly false and misleading, given WTPA’s substantial historical role.
He was fined $158,448.11 due to his misleading testimony and the resultant delays.
That is a rap, folks.