Your Claude chats are a police evidence feed and no lab will say how often

On 26 September, according to a Lee County arrest report, a 30-year-old woman in Bonita Springs, Florida typed into Claude that she would attack the sheriff’s office. Early the next morning, the report says, she sent another message claiming she had a new gun and calling it a last chance. Anthropic’s systems flagged the text, a human review team read it, and deputies arrived at Carli Michelle Heller’s door using information Anthropic supplied. She was detained without incident and charged with a second-degree felony under Florida Statute 836.10. She has not been convicted, and everything in that report remains an allegation.

I expected to write about a privacy scandal. Having read the coverage, I think the reviewer probably made a defensible call. My complaint is with what surrounds that call. A chat window is now one end of a pipeline that runs to a police station, and the only people who know its parameters work at the labs.

Here is the pipeline as far as the public can reconstruct it. The arrest report says Anthropic’s platform monitors for key phrases and threatening content. Severe material goes to a human review team. If that team judges the threat credible, Anthropic can hand over user details under a policy reserved for rare emergencies, where it thinks sharing them could stop someone being killed or badly hurt. Coverage notes this matches Anthropic’s published safeguards, but the company has not said how this specific conversation was processed. So we know the shape of the machine and none of its settings: which phrases, what score sends a chat to a human, how many referrals per month, how many led to nothing.

The detail that surprised me sits in the statute. Section 836.10 only applies if the threat was sent in a way that somebody else could see it. Heller says she used Claude as a diary. Nobody reads a paper diary, but a chatbot with a human review queue has readers by design. I would expect her defence to lean on this, and I think it exposes something odd: the lab’s safety process may be what turns a private rant into a communication another person saw. Whether a court accepts that is for a court. Users deserve to know the reader exists before they type, and today they mostly learn it from an arrest report.

The pressure on the referral threshold also runs one way. British Columbia is suing OpenAI and Sam Altman, according to the article, because OpenAI’s safety team had flagged a future shooter’s conversations about gun violence but judged they did not meet the bar for telling police. Florida sued OpenAI in June over alleged harms including the 2025 Florida State University shooting. Senators Josh Hawley and Chris Murphy are pushing a bill that could expose AI companies to lawsuits. Every one of those forces rewards a lab for referring more. When a missed referral costs a lawsuit and a false one costs a stranger a knock on the door, I think thresholds drift down quietly, and with no published numbers nobody outside can see the drift.

Sheriff Carmine Marceno warned that chatbot users are “never truly anonymous”. He is right, and the article adds that contractors reviewing Copilot’s image editor can see prompts, uploaded photos and edits. I suspect human eyes on chat content are routine across the industry.

The reasonable pushback is that Heller’s case was nothing like a vague diary entry: the alleged messages named a target and mentioned a firearm, and publishing trigger rules would hand would-be attackers a list of words to avoid. I accept the first half. Nobody sensible wants the phrase list or the classifier published. What I want are aggregate counts, which teach an attacker nothing: conversations flagged per quarter, how many reached a human, how many went to police, broken down by country and threat category, plus how many referrals led to a charge. If Heller is the system working as designed, Anthropic loses nothing by showing the denominator. If the denominator is ugly, legislators drafting liability bills ought to see it before they push thresholds lower still.

If your company has rolled Claude, ChatGPT or Copilot out to staff, add four lines to the next vendor security review: number of law-enforcement referrals originating from your tenant in the past twelve months, the criteria that send a conversation to human review, whether reviewers are employees or contractors, and whether you as the customer are told when one of your users is reported. Any vendor that answers the first line with no figure should be treated as running a reporting pipeline it would prefer you not audit.

Prompted by Florida woman used Claude as a diary, then Anthropic reported an entry to police, TechSpot.