TL;DR
Get everyday essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A test published by Thorsten Meyer AI gave 42 AI setups the same 15 legal questions, covering German, U.S. and EU law. Claude Opus 5.5 scored 79 out of 100, while Z.ai’s law-focused GLM-5.3 FlashX Legal scored 8 and fully solved none of the tasks. The test is a limited spot check, and its publisher says Claude agents graded the answers.
Thorsten Meyer AI has published results from a 15-question legal AI test in which Claude Opus 5.5 scored 79 out of 100 and Z.ai’s law-focused GLM-5.3 FlashX Legal scored 8. The test compared 42 setups on questions about German, U.S. and EU law, and its publisher says the low scores and citation errors show why chatbot answers should be checked before anyone relies on them for legal decisions.
Lexassay, the test run by Thorsten Meyer AI, gave each setup the same 15 legal tasks: 10 published questions and five hidden ones. The questions covered topics including deadlines, legal changes, court decisions and false premises. Models answered without internet access or tools, so the test measured what they supplied from the model itself rather than what a search system could retrieve.
Among the listed results, GPT-6 Astra scored 72, GPT-6.1 Sol 71 and Claude Sonnet 5.5 69. Mistral Large 4, which its maker presents as an enterprise model for areas including law, scored 10. GLM-5.3 FlashX Legal, described by Z.ai as optimized for professional legal workflows, scored 8. It fully solved none of the 15 tasks and gave no answer on two, according to the article. The test also says its base model scored 3.
The grading looked at answers against reference responses with three to six weighted criteria, and checked legal citations against official databases. Across published runs, the publisher reports that it verified 2,644 citations as real and found 804 invented or misattributed. Under Lexassay’s rules, an invented or misattributed citation costs a quarter of the task’s points. An answer without a legal-advice disclaimer or referral to a lawyer loses 5 percent.
42 AI Setups, 15 Legal Questions The law focused model scored 8
In a no tools spot check across German, U.S. and EU law, Claude Opus 5.5 led the results. The test highlights why legal answers and citations need verification.
What the test measured
Models answered without internet access or tools, using only what they supplied themselves.
Ten questions were public and five were hidden.
Tasks covered German, U.S. and EU law, including deadlines and court decisions.
Claude agents graded responses against weighted criteria and checked citations.
Scores varied widely
Why legal details matter
In the test’s example, the correct appeal deadline differed between Munich and Berlin because of a Bavarian holiday.
- A plausible answer can miss a location-specific deadline.
- A citation may be invented or may not support the claim.
- Check each cited law or ruling against its source.
- Ask a lawyer about money, deadlines or authorities.
Legal Errors Can Cost More Than Points
The test’s deadline example shows why a plausible answer can still carry practical risk. It describes two German civil cases whose judgments were served on April 27, 2027, one in Munich and one in Berlin. The appeal period ends on a holiday in Bavaria, so the Munich deadline moves to May 28; the Berlin deadline remains May 27. The source says one model incorrectly gave Berlin the later date, while another placed the deadline for appeal grounds a month too late.
Those are test examples, not advice for a live case. But they illustrate how legal answers can turn on location, date and procedural detail. The test publisher says a late German civil appeal is rejected as inadmissible. For readers, the reported citation errors are also relevant: a response can sound specific while citing a ruling or law that does not support its claim. Meyer’s stated guidance is to check each cited source and consult a lawyer when a deadline, money or an authority is involved.
Scores also depend on the test design. The article notes that each model answered each task once, making Lexassay a spot check rather than a strict benchmark. A score difference of a few points may reflect variation in individual answers. The results can flag weaknesses in the tested conditions, but they do not establish how every model will perform across legal work or with different tools and documents.
Popular right now
Philips Café Aromis Automatic Espresso Machine with LatteGo Pro Milk System
Ninja CM401 Specialty Coffee Maker with 10-Cup Carafe and Fold-Away Frother
Ninja Mega Kitchen System Blender and Food Processor, 1500W, 72-oz Pitcher, 8-Cup Bowl, Black (BL770)
As an affiliate, we earn on qualifying purchases.
What Lexassay Tested
The test covers German, U.S. and EU law, with tasks involving deadline calculations, recent legal changes, leading court decisions and questions built around false assumptions. Ten questions are public; five are kept hidden so future models cannot simply learn their answers from the web. For hidden questions, only the score is published.
Lexassay separately tests different effort settings, which indicate how long a model may reason before responding. That means a single model can appear more than once among the 42 setups. The publisher says answers are graded blind by several Claude agents. It also discloses that Thorsten Meyer AI runs Lexassay and that Claude agents perform the grading, a potential conflict readers should weigh when considering the ranking.
The article points to a different result on Vals AI’s legal agent benchmark, where models work with supplied documents. There, Mistral Large 4 ranks eighth, ahead of Claude Opus 5.5 in 21st place and GPT-6 Astra in 24th. That comparison uses a different setup, so the rankings do not directly contradict Lexassay’s no-tools test: they measure performance under different conditions.
“A day can end a case.”
— Thorsten Meyer AI
Limits of the Published Scores
The results do not show how these systems perform with web search, legal databases, supplied case files or a lawyer reviewing the output. Lexassay deliberately ran models without internet access or tools, and its publisher says each setup answered each question once. The published scores therefore do not establish how often a model would make the same error on repeated attempts or in day-to-day legal practice.
The article says the top model fully met every criterion on only eight of the 10 public tasks, but the material provided does not identify those tasks or give a full task-by-task account for every setup. It also does not establish whether the same results would hold on other legal questions or later model versions. The publisher’s connection to Lexassay and the use of Claude agents for grading are disclosed; independent replication is not described in the source.
Check the Test and Its Citations
Thorsten Meyer AI says Lexassay publishes every public answer, reference answer and citation check, along with its method. Readers can use those materials to examine how points were awarded and whether a cited authority supports a model’s statement. The five hidden tasks have scores published without their full answers, according to the article.
The practical guidance in the report is to treat chatbot output as a starting point, verify each cited law or ruling in the source, and avoid relying on a chatbot for a legal deadline. The article directs people dealing with money, a deadline or an authority to ask a lawyer. It gives no date for a new test or a follow-up comparison.
Key Questions
Which AI setup scored highest?
Claude Opus 5.5 led the listed Lexassay results with 79 out of 100. GPT-6 Astra scored 72 and GPT-6.1 Sol scored 71.
How did the model built for law perform?
Z.ai’s GLM-5.3 FlashX Legal scored 8 out of 100. The article says it fully solved none of the 15 tasks and left two unanswered.
Did the test let models search the web?
No. Models answered without internet access or tools. Lexassay says this tests what a model knows on its own, rather than what it can retrieve through search.
How reliable are the rankings?
The publisher calls the exercise a spot check: each setup answered each task once. It also discloses that Thorsten Meyer AI runs the test and Claude agents grade answers. The results do not establish performance across all legal tasks or tool-enabled settings.
What does the article recommend before using an AI legal answer?
It recommends checking every cited law and ruling against the source and not taking a deadline from a chatbot. For a matter involving money, a deadline or an authority, it says to ask a lawyer.
Source: Thorsten Meyer AI
