Examples of Unethical Experimental Studies

Question

I am teaching ethics in a statistics class. I am looking for examples when deceptive statistical evidence was caught or attempted to be used.

For example, is there an example of a party who had several surveys or studies performed and tried to only publicize the findings that was beneficial to them?

I am looking for court case examples. Thank you.

---

Edit

I am aware that using misleading statistics can be punished by being fired or publicly disgraced in the scientific community, which is not a legal matter. However, some statistics is actually used for important decisions or claims. I am looking for when bad statistics contributed to a decision that caused damages that led to a criminal or civil case.

I am not part of the law community, so I cannot say whether I am looking for fraud, bad faith, or incompetence in any technical sense. But here is an example.

Say a company pressured researchers to conclude a product was safe or exceeded its true capability. Then that product caused damage to some party who sued. If evidence came that a biased sample was purposefully used, I think that would show bad faith on that company and have bearing upon the court's decision. Or if a too small sample was taken and the data was presented without confidence intervals, that could show incompetence and the company failed to perform due diligence.

Outright falsification of evidence would clearly show unethical practices, but I am looking for more interesting examples in the "grey area."

I did not find any case examples, so I assume there are not many. That is why I asked it here.

Answer 1

Statistics deceive courts all the time

Most judges and jurors know no statistics. Most doctors and social scientists use statistics without really understanding them. Therefore, innocently or otherwise, perfectly valid statistics are introduced into courts in ways and manners that result in mistakes even before we consider the possibility of deliberate deception.

Perhaps the most egregious example was that of the wrongful conviction of Sally Clark for the murder of her two children who died of Sudden Infant Death Syndrome. The conviction led to her spending 3 years in gaol and probably contributed to her premature death of alcohol poisoning in 2007.

The case was flawed for two reasons: 1) evidence that clearly pointed to death by natural causes of her second child was withheld from the defence (and, to be fair, from most of the prosecution team) and 2) misuse of statistics.

The expert evidence of paediatrician Dr Roy Meadow was flawed in these ways:

1. "He claimed that, for an affluent non-smoking family like the Clarks, the probability of a single cot death was 1 in 8,543, so the probability of two in the same family was around "1 in 73 million" (8543 × 8543). Given that there are around 700,000 live births in Britain each year, Meadow argued that a double cot death would be expected to occur once every hundred years."

   As any undergraduate statistician (or advanced high-school mathematician) could articulate this is total bollocks. Combining probabilities by multiplying them together is only applicable if the events are independent - there is no evidence that the deaths of 2 young children with the same parents living in the same house at roughly the same time are uncorrelated and plenty of reasons for suspecting that they are highly correlated and probably in non-linear ways. Indeed, subsequent studies have shown that the risk of a second SIDS death in the same family increases by a factor of 5 to 10 - this is a very high correlation on what is still a very rare event.

   It is ironic and tragic that the argument that led to the charges (that the deaths are correlated by a single murderer) relies on the exact opposite (and equally vacuous) argument that led to the conviction (that SIDS deaths within a family are uncorrelated).

2. Further to the statement, the rate of SIDS in the UK is 1 in 1,300; not 1 in 8,500. Dr Medlow factored in the non-smoking and socio-economic elements of the household that lowered the chances but failed to factor in those that increased the chances: like the fact that both children were boys who have a higher rate of SIDS than girls.

3. Further, it's likely the court committed the Prosecutor's fallacy. Specifically, the court failed to understand the difference between conditional and unconditional probability.

   Double SIDS deaths are rare. But so are double infanticides. Even if Dr Medlow's figure of 1 in 73 million for double SIDS is accurate, it says nothing about the chances of double homicide (let alone double homicide by whom). We are not interested in the likelihood of 2 deaths by SIDS or homicide in the general population - we are interested in the chance of SIDS OR homicide OR something else (noting the second child likely died of a bacterial infection - so neither SIDS nor murder played a role) given that 2 deaths have occurred. For that, we need to turn to the good work of Reverand Bayes:

Further examples can be found here.

Answer 2

"Deceptive statistics" is not a matter over which a person can be sued, so coming up with a court case will be hard. Fabricating data is not deceptive statistics – data fabrication would easily fall under the general rubric of fraud. Likewise, falsifying the application of a test is basically fraud (claiming to have performed an ANOVA on a set of numbers when in fact the results were just made up). We can group these two kinds of cases together as "falsified statistics". If I attempt to induce people to enter into a contract with me based on falsified statistics, I will have committed fraud. I would not be surprised to find cases where falsified statistics was an element of a fraud case.

It is also possible (and common) for legitimate data to be honestly run through the computation, where you get different results depending on which computation you run it through. Then the question is which of these computations is one legally allowed to report, and which others are you legally obligated to also report? It would depend, first, on who you are. Specifically, are you "just some guy", or do you have a professional obligation to a client? If I hire a professional statistician to sort out some data and the statistician treats 5 categories encoded with numbers 1-5 as a continuous variable, that is kind of incompetent and might be the grounds for a breach of contract suit. If I myself do that and then make a claim based on bad statistical thinking, I'm not liable to anyone for making that claim (even if I know that this is incorrect).

In order to sue a person for what you might call an unethical choice in the use of statistical methods, that person has to have some duty to you. Moreover, the choice has to be "clearly wrong", i.e. violates professional standards of that profession. Leaving aside fraud using falsified statistics, even technical incompetence is not actionable except in the context of a professional duty to a client. But, as a contractual matter, it is possible that a statistician could be held liable for damages to a client for incompetent application of the wrong statistical test. Your scenario suggests not just incompetence but deliberate cherry-picking for personal gain, so I don't know how to integrate that with the basic principles of law that I sketched.

---

In light of the recent edit, there are many cases of product liability where the company knew of a defect but did not fix the problem. Some of these no doubt involve misrepresentation of the product's safety, when the product has been subject to some safety test. In that group, we should exclude "scientifically wrong test" because e.g. tendency to explode is not calculated by testing pH.

"Bad statistics" could include numerous things, such as bad machines or bad procedures (the machine doesn't measure what it is claimed to measure; numerous flaws of the numerical analysis of the data). Again focusing on the contribution of statistical analysis, a person may have actual knowledge of sample bias but disregards that knowledge and took no steps to mitigate the bias. As I imagine you know, the world is rife with biased samples, many of them of the form "Oh, I didn't think of that". The manager making the decision about the sample may have a weak understanding of statistics, so may not have actual knowledge that the sample is invalid. If the analysis is outsourced to a statistician, the statistician may be unaware of the sampling problem. The court will ask whether the company took reasonable steps to test the product's safety, and whether an ordinary prudent person would know that cell phone interviewing biases the sample of the nation's population. A professional statistician may know that but Joe Manager probably does not.

I very much doubt that a failure to report confidence intervals would contribute to a civil action. Ordinary people never encounter a confidence interval, and consumers do not directly rely on confidence intervals to make decisions. However, there is an indirect avenue for confidence intervals in consumer decisions, via "unproven claims". If you claim that a product does X, in some cases you may have to prove that claim in advance (drugs for example). I suspect the best avenue for finding "knowingly bad statistics" in court will be snake oil claims. But snake oil claims are most likely disposed of administratively, without there being a lawsuit.