# Does "statistically significant" have a specific legal meaning?

I occasionally write expert opinions as a statistician or econometrician for legal cases in the United States.

On a few different cases, I have been hired to determine whether a given sample should be considered statistically significant, usually at the request of a judge. In other words, does it have enough observations/records?

In standard statistical terminology, this is nonsense. A sample can't be statistically significant or not. Statistical significance is something that applies to the results of hypothesis tests, not samples, and the same sample can be used to conduct hypothesis tests that turn out statistically significant and also tests that turn out not to be.

So far, I've just assumed that what they really want is for me to calculate the margin of error, and I do so. I've had no complaints.

But seeing the same thing multiple times makes me wonder if I'm missing something. Is there a specific legal meaning of "statistical significance", separate from the typical statistical meaning, that I should be looking at?

• Does this answer your question? Can a person be found guilty based purely on statistical evidence? Commented Aug 30, 2022 at 7:38
• No, that question is also about statistics, but is otherwise unrelated. Thanks though! Commented Aug 30, 2022 at 7:48
• Judges and lawyers operate in context, according to "agreed facts" and expert testimony, and are always looking to precedent to guide future interpretations. So "statistical significance" can literally have any meaning assigned to it for any particular case.
– user608
Commented Aug 31, 2022 at 1:14
• It is questionable whether it has a useful statistical meaning, so I would hope not! ;o) Commented Aug 31, 2022 at 13:44
• If you answer the literal statistical question without carefully explaining, in a way that is clear to non-experts, its assumptions, implications, and appropriateness to answer the underlying question, you may end up like the guy who wrote the Texas v. Pennsylvania declaration. Don't be that guy. Commented Aug 31, 2022 at 14:01

## Most judges went to law school

Therefore, my null hypothesis is that they have next to no knowledge of statistics and couldn’t tell a median from a mean on their best, let alone, their average day - mean, median, or mode.

Courts turn to experts to fill in the gaps in their knowledge. If they are asking the wrong questions, then it is the expert’s job to, respectfully, tell them so and guide them to the right questions. However, you shouldn’t guess what the right question is.

“Excuse me, your honour, the term “statistical significance” can’t be applied to a data set, only to the result of an a priori null-hypothesis tested against that data set.” Feel free to explain p hacking if necessary.

Ask the judge to explain, in layman’s term, what they want from you. Feel free to admit that just like you don’t understand their legal jargon, you don’t expect them to understand your statistical jargon; if everyone speaks plain English, you can go and do your statistics thing and they can go and do their law-talking thing.

• Can I file an amicus brief featuring xkcd's explanation of significance? Commented Aug 30, 2022 at 17:25
• Wiki link on "p hacking" in case the next reader isn't a statistician ...... en.wikipedia.org/wiki/Data_dredging Commented Aug 31, 2022 at 0:49
• This is also how quite a lot of completely BS rulings came to be: The judges being fooled by statistics Commented Aug 31, 2022 at 8:31
• I would have thought that judges are required to perform inference under uncertainty as part of their job. While they don't need to be statisticians, they ought to have sufficient basic understanding of statistics (as that is the main tool for inference under uncertainty), at least sufficient to understand basic concepts such as statistical significance. If they don't know what it means, they shouldn't be asking about it (which is what the OP suggests is happening). They should indeed ask a question without the jargon term, so it is clear what they are asking - spot on! Commented Sep 1, 2022 at 9:37
• thegardenisland.com/2020/08/21/hawaii-news/… Commented Sep 1, 2022 at 13:40

You may be interested in this article, which asks if proof of statistical significance is relevant in law (US) – it clearly is, but still, nobody understands it. It is packed full of relevant legal case citations.

Apparently, this peculiarity didn't exist until two Supreme Court cases in 1977, Castaneda v. Partida, 430 U.S. 482 and Hazelwood v. US, 433 U.S. 299. In the former case, the reasoning of the case relied on the observation that "The discrepancy between the expected and observed values is more than 12 standard deviations", and in the latter case, that "The difference between the observed and expected values was more than six standard deviations in 1972-1973 and more than five standard deviations in 1973-1974", citing the earlier Castenada case. It is worth noting that the court talks of statistics, and significant discrepancies, but does not use the expression "statistically significant".

Then, in Moultrie v. Martin, 690 F.2d 1078, the court held (as "the main legal conclusion", not a throw-away comment) a rule that "in all cases involving racial discrimination, the courts of this circuit must apply a standard deviation analysis such as that approved by the Supreme Court in Hazelwood before drawing conclusions from statistical comparisons." (They also note the relationship between confidence level and statistical significance.)

FRE 403 is an evidentiary rule that can be used to preclude admitting relevant evidence if it confuses the issues or misleads the jury. (Of course, a witness can at most point out this connection to the relevant attorney who will then move to exclude, or not.)

The customary threshold for, and most common meaning of the phrase "statistical significance" in the sciences and social science is a difference of at least two standard deviations (a.k.a. two sigma a.k.a. a "p-value" of 0.05 or less) from a null hypothesis.

A related phase is that data that is not statistically significant is "consistent with" the null hypothesis.

The law doesn't have special definition of the term, but expert testimony could usually be adduced to that effect, for example, for the purpose of contract interpretation if a contract requires a statistically significant result without defining the term.

An obvious place for experts to duel would be over whether one is looking for "local" statistical significance considering only the data set at hand, or "global" statistical significance, which also takes into account the "look elsewhere effect" (i.e. the notion that if you run enough experiments, some of them will be locally statistically significant as a result of random chance even if the null hypothesis is true).

• Comments are not for extended discussion; this conversation has been moved to chat.
– Dale M
Commented Sep 1, 2022 at 9:54

"Statistical significance" doesn't have any special legal meaning. It seems most likely that your interpretation is correct, as an inquiry into how reliable the data set is would speak to its reliability, which is part of the Daubert test for admitting expert opinions.

If the legal definition of "statistically significant" is different from the statistical one, then it is very likely to be a deeply misleading one, and one that suggests the evidence is much stronger than it actually is. In this case, described in the OP, the question about statistical significance is clearly from someone that has no idea what it actually means, and the best approach is to see if the question can be posed in layman's terms avoiding statistical jargon, so it is clear what they really want to know. Dean's answer is a very good one. However, telling people that they are wrong sometimes goes down badly, but I would hope that judges and lawyers would have the good sense to allow themselves to be corrected by a domain expert when they are wrong.

I thought it might be useful to explain why this phrase is likely to be misunderstood - it is because statisticians (at least those performing null hypothesis statistical tests) and laypersons are likely to have fundamentally different ideas of what is meant by a probability.

The original definition of a probability in statistics was similar to the everyday meaning - it is a numerical indication of the relative plausibility of a proposition, e.g. there is a probability of 30% that it will rain today. However, there is a "subjective" element to this, as the plausibility may depend on your prior beliefs (or equivalently your state of knowledge). A new branch of statistics came about, called "frequentism" in the 20th century that aimed to eliminate this "subjective" element and have a purely objective means of performing inference. They did this by defining a probability only in terms of long-run frequencies. This is perfectly reasonable, but it does mean you cannot assign a probability to a particular event, only to populations of events. For instance, a frequentist fundamentally cannot tell you the probability of a particular DNA match being a false-positive - it either is a false-positive or it isn't - it doesn't have a non-trivial long run frequency, so you can't assign a probability to it. So what do we do? Instead of "probability" we speak of "confidence" (as in "confidence interval") or "significance" (as in "statistically significant"). I think originally, this was intended to highlight that it isn't a probability and avoid misinterpretations, but sadly this is not how it has turned out.

The key problem is that we normally want to ask a question about a particular case (e.g. this DNA match), but a frequentist statistician cannot answer that question, so they substitute an answer about some (often fictitious) population (e.g. on average over a large number of DNA tests of which this one is representative in some way that is most often left explicitly unspecified). The questioner hears this and interprets it as an old-fashioned (degree of plausibility) based answer about the specific case - because that is what they were expecting (not unreasonably).

So if there is a legal definition of "statistical significance", it is very likely based on this "degree of belief" definition of probability, and be a bad misinterpretation of what was actually meant by the person that conducts the test. This isn't particularly the fault of the legal profession, statisticians often shy away from explaining what things mean because the audience is often very hostile to it.

Bayesian statistics is likely to be a better match for the needs of the Law as it is based on the same basic definition of a probability that the layperson uses, and hence is less likely to be misinterpreted.

The way statistical hypothesis test should be used is as a minimal hurdle that prevents you from making a fool of yourself by getting carried away with your research hypothesis (the thing you want to be true). A statistically significant outcome just means you have stumbled over the hurdle somehow and you can still talk about your research hypothesis. If it wasn't significant, you should not talk about it as if it were true (yet). It really doesn't mean very much, and is most useful to your when non-significant.

The definition in law - if it exists - would have to be compatible with what statisticians understand by that phrase. Otherwise statisticians would not be called in as expert witnesses in such cases.

But you must find a layman's phrasing for this concept so that everyone in court is on the same understanding as you.

It is also likely that the attorney engaging you will also need an explanation of what this phrase means (and indeed any other statistics jargon you propose to use during your expert witness statement) so he can counter any efforts by the opposing attorney to undermine your assertions.

It's hard at the beginning to explain to people from a humanities/arts/business primary degree background how statistics work. But it has to be done lest you end up tied in knots at court. Use everyday examples simple words to present the concept and take them through it slowly.

• "But you must find a layman's phrasing" the actual meaning of "statistical significance" is very subtle and you will struggle to find an explanation in layman's terms, a lot of people who teach statistical significance don't fully understand it, see Haller and Krauss (2002) researchgate.net/profile/Heiko-Haller/publication/… Commented Aug 31, 2022 at 14:01

In the legal field, statistical significance is often used as a measure of the strength of the evidence in a particular case. This can be helpful in determining whether or not to pursue a case, or in evaluating the likely outcome. However, it is important to keep in mind that the legal definition of statistical significance may be different from the statistical definition. For example, in the legal field, a study may be considered statistically significant if it shows a difference that is large enough to be considered meaningful by a judge or jury. In contrast, in statistics, a study may be considered statistically significant if it shows a difference that is small enough to be considered meaningful by the standards of mathematics. As a result, it is important to consult with an attorney or other legal expert before relying on statistical significance as a measure of evidence in a legal case.

• The American Statistical Association on what is a p-value states: "By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis." tandfonline.com/doi/full/10.1080/00031305.2016.1154108 . If a study is large enough to be considered meaningful by a judge they should just say the study "significant". Saying "statistically significant" is making it look rigorous when it isn't (and misleading appropriation of jargon from a relevant field). Commented Aug 31, 2022 at 18:27