Why does an AI invent a ruling instead of saying it cannot find one?

Because it is not built to look things up, but to complete them. It works out what text is likely to come next and writes it. If what is likely at that point is a citation, it produces a citation. It has no mechanism that lets it stop and acknowledge that it does not have the information.

Do specialised legal tools hallucinate too?

There is a language model underneath most of them, so the risk is always there. What changes is where they draw their material from: a tool that answers only from a specific set of documents has far less room to build a reference that is not in them. None guarantees a hundred per cent, and it is wise to distrust anyone who promises it.

How do I know whether a citation it gave me is real?

By checking it at the source, without exception. It is the part of the work that cannot be delegated and the one that takes the least time: it is a spot check, not a full read.

Can it be avoided entirely?

Not entirely, but it can be reduced a great deal. The 2 factors that matter most are what material the tool works on and whether it lets you check where each statement comes from. An answer that arrives with the document and the exact point it came from is verified in seconds.

What happens if I file a brief with an invented citation?

It can have professional consequences, on top of the obvious effect on the matter and on the relationship with the client. The signature belongs to whoever signs, and the failure to check is the responsibility of whoever skips it, not of the tool that produced the text.

Practice18 September 20266 min read

Why an AI invents a court ruling (and how to avoid it)

A lawyer would say «I don't know, I need to check». A language model does not have that option.

Desiree OriaChief Executive Officer

A language model is not designed to give you the right answer. It is designed to build the most likely next word, and on top of that to always answer, and to answer convincingly. When one of its sentences usually ends with a citation to a ruling and it cannot find one, it does not stop: it builds one. This is not a system failure. It is the system working as conceived. Knowing that changes quite a lot about how you work with it.

It is not a database: it is a language model

Let us start with the name, which says it all.

A language model is exactly that: a system trained to handle language. That is why it understands you even if you type in a hurry, with typos or mixing languages. And that is why what it returns has impeccable syntax and sounds good. That is its competence, and at that it is extraordinary.

The second characteristic is the one almost nobody explains: it works by probability. It has processed an enormous amount of text and learned which patterns repeat. When it answers, it is not consulting a file. It is working out which word usually comes after the previous ones, and then the next, and the next.

From which the important part follows: its goal is not accuracy.

That is not a criticism, it is a description. A legal database is built to return the exact record; if it does not have it, it returns nothing. A language model is built to produce plausible language. They are 2 different tools, and we have spent 2 years asking them for the same thing.

And it is also designed to always answer

Here is the part that really throws people.

It always replies. It does not have the option of saying «I don’t know». A lawyer uses it constantly: I don’t know, I need to look it up, let me check. It is one of the things that mark out a good professional. The model does not have it.

And it replies convincingly. It is trained to satisfy whoever asks, so what it returns sounds confident even when there is no reason to be. There is no hesitation, no nuance, no «this would need confirming».

The combination of the 2 is what produces the problem. If the pattern it has learned says that this kind of paragraph ends by citing a ruling, it cites one. Whether it exists makes no difference at all. And it comes out in full format: chamber, date, appeal number, everything in its place.

The same happens with legislation. It may tell you that article 267 of the Civil Procedure Act says something that is actually said by article 215, or 268. It is not that it lies: it is that, for the calculation it is making, the right number and the one next to it are equally likely.

And it does not care whether the rule is still in force. It has seen it cited many times in the texts it read, and that is enough for it.

What deceives is not the error: it is the format

An obvious error deceives no one. If an answer says something outrageous, it is caught on the first read and nothing happens.

What deceives is an invented citation in impeccable format, placed exactly where a real one would go, in the paragraph where you yourself would have put one. It does not jar. It fits.

That is why it is worth learning to recognise a text written this way. The signs are fairly constant:

It starts at the beginning and ends at the end, leaving nothing out. A thoroughness no professional has when writing against the clock. Human texts skip things, take others for granted and get to the point on what matters.

The length bears no relation to what was asked for. 3 pages for something that a paragraph would have settled.

The references are where they should be but do not survive checking. Either they do not exist, or they exist and say something else.

The tone is uniformly confident. Without the nuances of someone who knows the case, who knows which point is weak and which is not.

Anyone receiving briefs from third parties is already noticing it. A formal notice arrives at the firm and you can tell on the first read that it was not drafted by a lawyer. Not because of what it says: because of how it is built.

«It is quicker if I do it myself»

This is the real objection, and it deserves a serious answer.

The reasoning goes like this: if I am going to have to check every ruling, every article and every statement, I am saving nothing. I will do it myself and sleep easy.

It is a reasonable calculation. If verifying means redoing the work from scratch, it genuinely does not pay off. Someone who thinks this is not being distrustful: they are doing the maths.

But verifying is not redoing, and the difference lies in knowing where to look.

What must always be checked, without exception: every citation of a rule, of a decision or of a specific figure. Everything with a number in it. That is not negotiable and it does not take as long as it seems, because it is a spot check, not a read.

What does not demand the same level of review: the structure of an argument, the order in which the facts are presented, the wording of a paragraph you are going to sign after reading it. The risk there is low, because the error shows on reading.

And what changes the calculation entirely: where the tool draws what it tells you from.

If it works on all the text it has read on the internet, it can build a citation out of a pattern. If it works only on documents you have given it, it has nowhere to pull a ruling from that is not in them. The risk does not disappear, but it moves: it stops being in the citations and starts being in the interpretation of what is in your documents. And a lawyer knows how to look at that, because it is what they do every day.

So the useful question is not does this hallucinate?, but where is it getting what it is telling me from?

What cannot be delegated

However much the tools improve, and they have improved considerably, one thing does not move: responsibility for the brief lies with whoever signs it.

Delegating a hundred per cent is not a shortcut. It is negligence, and it is judged as such.

And one practical recommendation worth more than any checklist: read it all the way to the end, even if it looks perfect. Especially when it looks perfect, which is precisely the moment one lets their guard down and signs without looking at the last page.

If you would like to see it with one of your own briefs in front of you, get in touch.

Frequently asked questions

Why does an AI invent a ruling instead of saying it cannot find one?: Because it is not built to look things up, but to complete them. It works out what text is likely to come next and writes it. If what is likely at that point is a citation, it produces a citation. It has no mechanism that lets it stop and acknowledge that it does not have the information.
Do specialised legal tools hallucinate too?: There is a language model underneath most of them, so the risk is always there. What changes is where they draw their material from: a tool that answers only from a specific set of documents has far less room to build a reference that is not in them. None guarantees a hundred per cent, and it is wise to distrust anyone who promises it.
How do I know whether a citation it gave me is real?: By checking it at the source, without exception. It is the part of the work that cannot be delegated and the one that takes the least time: it is a spot check, not a full read.
Can it be avoided entirely?: Not entirely, but it can be reduced a great deal. The 2 factors that matter most are what material the tool works on and whether it lets you check where each statement comes from. An answer that arrives with the document and the exact point it came from is verified in seconds.
What happens if I file a brief with an invented citation?: It can have professional consequences, on top of the obvious effect on the matter and on the relationship with the client. The signature belongs to whoever signs, and the failure to check is the responsibility of whoever skips it, not of the tool that produced the text.