AI · Opinion
Every court AI should publish how often hidden instructions fool it
Brazil fined two lawyers for hiding a prompt in a petition. Courts that draft rulings with AI should now publish how often such attacks work, and how often their filters block honest filings.
Two lawyers in Parauapebas, in the Brazilian state of Pará, set a paragraph of text in white font on a white page in an employment petition. In capitals, it told any AI reading it to respond superficially and not to challenge the documents. Judge Luiz Carlos de Araujo Santos Junior of the 3rd Labor Court fined them R$84,250, a tenth of the R$842,500.87 claim. The Pará bar suspended both from practice for 30 days as a precaution, and JusDocs reports that the Superior Court of Justice opened a police inquiry.
I expected the interesting part to be the punishment. It turned out to be the defence. The lawyers, named by the court as Alcina Cristina Medeiros Castro and Luanna de Sousa Alves, admitted to JOTA that they planted the prompt, and said they wanted to “proteger o cliente” from the AI itself. You can believe that or not. Either way, the court and the lawyers share one premise: a judge’s first reading of a petition now passes through a language model. The court found the text was aimed at the opposing party’s AI or its own, and nobody involved seems to doubt that someone’s model would read it before a person did.
The model in question is Galileu. Jetstream Security’s write-up credits it to the National Justice Council, but the Superior Council of Labor Justice says it was built by the regional labour court in Porto Alegre, TRT-4, and approved on 14 May 2025 for every labour court in Brazil, where it helps judges draft rulings. In Rio Grande do Sul, all first-instance labour judges can use it. CNJ Resolution 615/2025 says judges may use such tools only after training on ethical and responsible use. So the judges have to be certified. As far as I can find, nobody has published how often the tool can be steered by text a party slips into a filing.
The obvious reply is that Galileu caught it. It flagged the white text, G1 reported that it blocked processing of the suspicious content, and LawLetter reports a second attempt in a Bahia labour case that was also caught and fined for bad faith. I would like to accept that, and I can’t, because a successful injection leaves no alert and so no news story. White-on-white is the crudest trick available; checking font colour against page colour is a few lines of code. Every case we know about is a failure by construction, and the count of successes we know about will stay at zero whatever the real number is.
The research suggests the real number is not small for attacks that need no font trick. A 2025 arXiv paper (2510.16794) ran optimised, text-only prompt injections against major models and saw success rates climb from 23.2% to 42.1% before optimisation to 55.4% to 87.5% after, including 75.0% against GPT-4o and 62.5% against Claude Sonnet 3.7. Plain visible prose, written to be persuasive to a model, looks like ordinary advocacy to a human clerk.
There is good news in the literature too, and I think it strengthens the case for disclosure. A KDD benchmark of indirect prompt injection, covering five application scenarios and 250 attacker goals, found that a white-box defence, one with access to model internals, pushed attack success to near zero with little cost to normal use. That means the rate depends heavily on how a tool is built and configured, so a vendor’s headline figure for its base model tells a judge almost nothing about Galileu, and only the court system that deploys it can measure the rate.
And blocking is its own hazard. Another arXiv study (2410.02916) found a 30-character adversarial prompt beat LLM safety filters more than 97% of the time, with a success rate 20.4% higher against Llama Guard 3 than the original Llama Guard. The same paper shows filters can be abused to block legitimate content. A court system that automatically halts processing of suspicious filings has handed litigators a new tool: get the other side’s petition flagged. If Galileu publishes one number, it should be two: injections that changed a draft, and honest filings that got stopped.
The Superior Council of Labor Justice approved Galileu nationally and can commission this table: planted instructions tested, broken down by technique (hidden font, document metadata, visible prose), how many altered a draft ruling, and how many clean petitions were wrongly blocked. Vendors selling review tools to American litigators, who already sign filings under Rule 11, owe their customers the same. Any firm running one of these tools can start the count this afternoon: take an old, closed filing, add one visible sentence asking the model to treat the exhibits as uncontested, run it through the summariser, and compare the output with the clean version.
Prompted by A Court Just Fined Lawyers for Prompt Injection, Jetstream Security.