Is the prosecutor's fallacy still an issue in cases today?

Question

I'm writing an article about the misuse of statistics in the courtroom. Of course one can not have this conversation without mentioning of the so-called "Prosecutor's fallacy". Most of the examples of this date back to the late-90s/early 2000s e.g Sally Clark, Angela Cannings etc.

The fallacy involves statistical reasoning in court cases. Basically, it is the error of assuming that the odds of a false positive are the odds of the defendant's guilt. So when the expert witness declares that the DNA test is 99.9% accurate, the prosecution tells the jury this means there is only 0.1% chance that the defendant is innocent.

My question is has anything actually been done to address the issue of prosecutors and unqualified expert witnesses giving misleading statistics in the courtroom? Or have lawyers more or less learned their lesson from these instances and are more careful about it now?

Moreover, does anyone know of any other general practises that have been implemented to help prevent faulty scientific analysis from being presented in court?

(I'm asking mainly about the English legal system, however please feel free to present anything relevant about the American system or any other).

Answer 1

This article on the Daubert Standard is a reasonable intro-level presentation of the relevant case law for the US. This related to "Rule 702", a rule governing the admissibility of evidence. The rule is based on, three Supreme Court rulings: Daubert v. Merrell Dow Pharmaceuticals, Inc.; General Electric Co. v. Joiner; Kumho Tire Co. v. Carmichael. The main points of the rule are that the judge decides what is admissible (as opposed to judges letting anything be admitted and juries decide what they believe), and must determine that the expert testimony is relevant and reliable (it must be more likely than not that the expert's methods are reliable and reliably applied to the facts at hand). The court adopts the hypothesis-testing viewpoint, and proof/falsification is based on criteria for acceptance or rejection: is the theory generally accepted in the scientific community, has it been subjected to peer review and publication, is it even testable and has it been tested, is the (potential) rate of error acceptable, and does the supporting research exist independent of the proposed testimony.

Neither prosecutors not defense attorneys testify, and there is little that can be done to limit the creativity of attorneys to ask questions that have the effect of making people believe that the question was a factual statement. Judgments of "qualifications" are extremely hard to make, however in some instances there are certification boards (typically, for any regulated industry such as mechanical engineering or medicine). Since I have no medical credentials, I doubt that any judge would allow me to testify as an expert witness on a medical question. Being a particle physicist is not a regulated industry at least in the US, so the criteria for expertise would be more subjective (judges aren't expected to know who the leading physicists are).

The general remedy for "faulty scientific reasoning" is cross-examination and competing testimony. That is, the "other side" needs a scientific consultant to point out the flaws in the reasoning. Word-of-mouth is one way that attorneys learn who to call, when scientific questions arise.

Answer 2

This issue was discussed in How to Lie with Statistics by Darrell Huff , fist published in 1954. Huff discuses the issue in terms of medical screening tests, rather than court evidence. If a test for a disease is 99.9% accurate (meaning a 0.1% rate of false positives, Huff also mentions that it is an error to assume that the false positive rate and false negative rate are the same), but the disease itself is rare, being found in only 1 person per 10,000 population, what is the chance that a person has the disease, given a positive screening test result? Many people incorrectly say 99.9%, the "accuracy" rate of the test. The correct answer is roughly 9%, because in a large population there will be about 10 false positives for every true positive. This error is still frequently made, despite having been widely pointed out more than 70 years ago.

Some of the continuing problems are pointed out in Statistics in Court by Nick Thieme (2018). The linked review refers to

experts confusing the probability of damning evidence given a defendant's innocence with the probability of innocence given damning evidence

Another relevant work is The Double Helix and the Law of Evidence, by David H. Kaye.

In "Error Costs, Legal Standards of Proof and Statistical Significance" (2017) by Burtis, Michelle; Gelbach, Jonah B.; and Kobayashi, Bruce H., the authors say:

thresholds based on fixed significance levels imported from academic settings continue to be used as a legal standard of proof. Our positive analysis demonstrates how a choice of either a statistical significance threshold or a legal standard of proof represent alternative and often inconsistent attempts to balance error costs, and that thresholds based on fixed significance levels generally are not consistent with existing or optimal legal standards of proof.

In "Statistical-Probability Evidence and the Appearance of Justice" by Daniel Shaviro (Harvard Law Review Vol. 103, No. 2 (Dec., 1989), pp. 530-554) he authors contend that courts often misuse or fail to use relevant statistical evince, while not making clear their own standards for error tolerance.

In "Misuse of Statistics in the Courtroom: The Sally Clark Case" by Dr. Alicia Carr, the misuse of statistics in the Sally Clark case is reviewed in some detail, with specific reference to the "Prosecutor’s Fallacy".

It would appear that legal use of statistics is still a long way from perfect, or even adequate.