I work at a financial institution in the United States and have been researching the regulatory impacts of Artificial Intelligence recently - specifically the simple purpose-specific systems we have today like Alexa, Siri, or the AI that navigates your Tesla when you switch on autopilot. Based on some googling [1] [2], the current standards seem to be that there aren't a lot of specific regulations that apply to the AI agent (with the exception of self-driving cars), but the normal regulations that apply when using a human agent would apply if you substitute an AI-based decision process. The top answer on a related topic here on StackExchange seems to support the idea that the regulations applied to the human apply to the AI agent as well (at least in Hong Kong).

We're currently shying away to research into the legal implications of AIs that can learn any task, much like a human. These are sometimes referred to as Artificial General Intelligence (AGIs) and the most aggressive estimate I've seen in my research places them emerging sometime in the next 100 - 200 years. A problem for future generations!

The Question:

This raises a couple of questions in my mind about a couple of legal concepts that have clear definitions when applied to a human or certain kinds of AI, but get fuzzy with other ones. Specifically:

  • How do you determine an AI agent's intent?
  • How can you tell if an AI agent is biased in its decisions?
  • How would the concept of discrimination apply to an AI agent? How would you prosecute it?

A Specific Example:

For instance, take the anti-discrimination provisions of the Equal Credit Opportunity Act (ECOA). In the case of a decision-tree based AI agent discrimination is obvious - if you add in a rule that makes different decisions based on race, gender, age, or other protected factors you have a discriminatory model.

In the case of a statistically driven Machine Learning model, this is less obvious. For instance, a neural network based model could accept address data as part of its decision. After examining hundreds of thousands of credit applications, the machine learning model could potentially derive information about income distribution from address or similar fields and end up discriminating against applicants that match certain traits. In this case you accept no information on age, sex, or ethnicity but still discriminate based on these factors since the model found some problematic trends in the data that it was trained on.


With this in mind, my questions are:

  • What are your thoughts on regulations that refer to intent or bias as they relate to AI agents - particularly ones that don't operate on explicitly defined rules?
  • Is there any precedent in this space that we could look to to guide decisions on AI risk?
  • Does anyone have thoughts on where regulation in this space might head?

Update (7/18)

I've removed the portions of this question with a technical lean and posted a question about the technical aspects of detecting and fixing a discriminatory model to ai.stackexchange.

  • The ideas of code review are very unrealistic. Pretty much the only way is to 1) verify that the algorithm has access to the information that can be used for discrimination and 2) feed the algorithm data with different discriminatory factors, but all other things equal, and see if it gives different results.
    – user626528
    Commented Oct 6, 2023 at 21:33

3 Answers 3


Firstly, true artificial intelligence does not yet exist. The term "artificial intelligence" is a bit of a buzzword, used to refer to things like neural networks and decision trees, which are really just elaborate statistical calculations. They do not have a "mind of their own" by any stretch of the imagination, though it is possible to make them appear as if they do, in the time-tested tradition of constructing convincing but simple automatons that goes back centuries. Once true AI appears and begins to act as an independent agent in the public sphere, likely new legislation will be produced to address it, so our speculation based on the present situation is unlikely to be proven accurate in the end.

The law will tend to view AI as a tool or calculation of the human operators, therefore liability for whatever damage or offense the AI causes rests with the operators. This is little different from prosecuting the person who pulls the trigger, rather than the gun. But two caveats apply:

  • Because computer systems are often very complex, it is easy for operators to argue that any harm is unintentional. Especially machine learning approaches that have become popular recently, which often produce unpredictable or unintuitive, if not downright non-deterministic results (indeed, they are often employed to find solutions that intuition could not), are very complex. While understanding the principles of their operation is quite easy for a technical person, predicting the exact result for any given input is much harder, due to sheer volume of numerical calculation involved. So it is hard to prove any outcome is intentional, unless someone went on record stating the intent.
  • Because such systems are often implemented by an organization rather than individuals, it is often the organization that ends up being liable, not the programmer.

As for your specific example, it is actually not that important whether the model discriminates explicitly or implicitly. Even a model that makes decision completely at random can still be considered discriminatory in certain settings. This is because US law has a concept called disparate impact, where impact on a protected group alone can be sufficient to argue a policy is discriminative.

Is there any precedent in this space that we could look to to guide decisions on AI risk?

I think autopilots (both planes and cars) are a good example. Another one is automated fraud detection and credit scoring used by financial and insurance industries. In particular you would want to look at disputes where the client disagrees with the results from the statistical model.

By my argument that what is commonly referred to as AI is not really AI but dumb software, it is worth looking at disputes including much simpler devices: For example the controversy surrounding the reliability of breathalyzer devices and their admissibility in court.

Does anyone have thoughts on where regulation in this space might head?

In the immediate future, I think it would be nice to see more accountability from programmers and designers. For example, programs which can cause significant damage (like car autopilots or forensic tools such as breathalyzers) should be required to pass more stringent QC than usual, and they should be produced by specially trained, qualified and certified programmers. I think it is also too easy for corporations to escape responsibility from harm caused by their software tools by just shrugging and claiming it was an accident. Ultimately no one can guarantee a complex program will always perform 100% as intended, but that is not an excuse to write very sub standard programs and have inadequate testing. Thus courts should be more skeptical when an "AI" like a self-driving car causes an accident and the company claims it's not their fault. As complexity and ubiquity of such "AI" systems in everyday life grows, accidents due to software bugs or design issues should become more and more common, so I expect that eventually legislation will be produced to address them (or alternatively, lobbying will overwhelm the process sufficiently so as to bring about a sort of unregulated cyber-anarchy for big tech corporations).

  • I agree that the types of AI you reference in your first paragraph don't exist. I usually refer to programs that can learn any task with no prior training as Artificial General Intelligence (AGIs) and the most aggressive estimate I've seen puts them more than 100 years out. I'm more concerned with the primitive AIs we have today - simple stuff like neural networks and random forests that can figure one thing like "recognize emotions from an image", I'll update the question. Your note on disparate impact is a great point for these models. Commented Jul 19, 2018 at 19:31

How can you tell if an AI agent is biased in its decisions? How would the concept of discrimination apply to an AI agent? How would you prosecute it?

A subject matter expert would scrutinize or audit the implementation of the algorithm and testify accordingly (see hszmv's allusion to the independent reviewer). From there, counsel representing the meritless party(-ies) will adopt the vexatious role as usual.

Weapons of Math Destruction by Cathy O'Neil "denounces" the type of discrimination scenario sketched in your [original] inquiry. Basically, the book's argument is that decision algorithms are implemented in a way that only reinforces patterns of disparity and discrimination.

Does anyone have thoughts on where regulation in this space might head?

Let me address a different, non-speculative question: where regulation should head. The legislation and judiciary should refrain from intruding in aspects like this. That restrain will allow the market economy to deal with the state of affairs naturally (that is, free from distortions).

O'Neil's general criticism is directed at the application of computer based decision-making. In that sense, her criticism is misplaced because she appears to miss that what she characterizes as "discrimination" might be prevented by fine-tuning the (values of) input parameters of an otherwise robust implementation of the algorithm.

O'Neil also addresses the issue you described in the inquiry as you initially formulated it: the risk of ending up discriminating on impermissible grounds despite not being provided with the prohibited kind of information. But we need to come to terms with the impossibility of dissociating some variables: variables that are and will remain correlated unless a government decides to control a greater portion of private or personal affairs.

For instance, O'Neil mentions the situation where a retail (?) business reportedly sets higher prices to certain high-crime areas. Clearly the algorithm cannot be omniscient of whether those areas might coincidentally also be low-income areas. What is the policymaker to do?

  • Will the government order relocation of families to prevent the clustering of households with homogeneous economic status?
  • Will the policymaker micromanage the retail business so that a human being is tasked with the pricing for a list of neighborhoods?
  • Will the government prohibit a difference in pricing despite the [presumably] greater risk the business and/or deliverer face in the high-crime area?
  • Will the policymaker enact laws toward an artificial and dubious dissociation of correlated variables?

In a (free-)market economy, significant differences in the pricing of goods and services tend to disappear because suppliers who are less risk-averse will take advantage of what essentially is an arbitrage opportunity. In the context of free competition, prices will eventually adjust without need for governmental intrusion.

At most, judicial scrutiny of automated decision-making would be pertinent only in few matters. For example, the importance of preventing the incarceration of innocent people justifies all the necessary scrutiny of the implementation and usage of algorithms.

There are other topics where legislation needs to be enforced and be brought up to speed, rather than insist on doing a halfway job (note I'm being generous here) in emerging fields. Likewise, our society would be better off if the judiciary purges itself and extirpates pernicious, outdated doctrines such as judicial immunity or the attorney-client privilege.

  • 1
    How is attorney-client privilege outdated?
    – JAB
    Commented Jul 19, 2018 at 2:34
  • @JAB Nowadays people have free online access to the sources of law, aka authorities; and illiteracy is largely eradicated: A person of average intelligence can attain proficiency in legal concepts and theories.Thus, the value that attorneys "add" to society has decreased, yet the drawback of privilege remains: Confidentiality forecloses the oftentimes only possible way to discover the truth, while culprits' lawyer works to prevent wrongs from being remedied. Wrongdoers' confiding with their lawyer about their offense or fraud boosts the victims' equitable entitlement to the use of that info. Commented Jul 19, 2018 at 10:52
  • 1
    "A person of average intelligence can attain proficiency in legal concepts and theories." This is not true, and even if it was, lots of litigants need lawyers because they are of below average intelligence. The approximately level of education/ intelligence/literacy to figure out the law in my experience with pro se parties is roughly a bachelor's degree with above average grades, or a grad degree. In accord, you need about a 145 LSAT (i.e. IQ 117 (86th percentile, average college grad) to have any real shot at passing the bar. LSAT 165, IQ 133, 98th percentile is needed to consistently pass.
    – ohwilleke
    Commented Jul 19, 2018 at 21:27
  • 1
    @IñakiViggers "a victim of unlawful acts is equitably entitled to use in court proceedings the confession wrongdoers willfully make to their spouse, priest, or lawyer." I think that I understand your meaning in the quoted material, but for the clarity of future readers, this is what you believe that the law should be, not what the law actually is right now.
    – ohwilleke
    Commented Jul 20, 2018 at 22:33
  • 1
    @ohwilleke If you have a genuine issue about the legal semantics of "equitable", I encourage you to click on "Ask a Question" and start from there. So far, your comments here prove the very first paragraph of my answer: You --an attorney-- evidently have nothing to say/inquire about the contents of my answer, so instead you opt to interpose distractor after distractor, first by discussing LSAT scores, then a redundant "remark" of what is the law vs my opinion, and now this. The SE Comments feature is not intended for obfuscating others' answers by bringing up increasingly unrelated "issues". Commented Jul 21, 2018 at 12:07

There is a really good example of how this could happen, look up the footage of Watson getting the wrong answer on Jeopardy. Watson worked by weighting the inputs of the question and the response as a confidence value that it was the right response. In the failure, the question was about which American city has two Airports that were named after a WWII pilot... and Watson answered "What is Toronto?"

This was hilarious because Toronto is out of the scope... it's a Canadian City not an American one right? Well, the problem was that American can be used in a sense of "Of the United States of America" or "Of the continents of North and/or South America"... Watson read it as the latter meaning, which put Toronto in scope and gave a more confident score. Jeopardy used the former meaning. (in a data dump, Chicago, the correct answer, was Watson's second choice and it was pretty close). For a more human example, Norm on Cheers was correct, as those people had never been in his kitchen, but wrong because that's not the Jeopardy Answer or interpretation of the answer.

In order for your specific answer to occur, the address has to be weighted to produce a certain score of the neighborhood... in order to to prove that it's being used in discriminate manner, one has to look at the code to see if the areas were weighted unfairly based on legal discrimination... You would have to get an independent reviewer to look at the code, and assess if the code is weighted properly, and if that weight changes based on factors. Then you would need to assess if the error is deliberate or if bug testing wasn't thorough enough.

The guilty party would likely be the bank, and possibly the coder if you can prove the coder did it without the bank's knowledge OR if the bank instructed them to break the law and they complied.

  • 2
    What's this about Norm on Cheers? My entire thought process got derailed when I read that part, as you've got no link for it and I, like I'm guessing many other people, do not share your understanding of Norm and Cheers. This answer also seems extremely light on things actually concerning the law. Commented Jul 18, 2018 at 19:26
  • I'm very interested in the example from Cheers, would you mind dropping a link to the scene or a season and episode number so I can look it up? Human examples always help me with explaining odd AI concepts to the nontechnical folks! Thanks for the Watson example as well, I've shared with my team and they got a good laugh out of it. Commented Jul 19, 2018 at 19:35
  • I think I've tracked down the Cheers reference - It looks like it was Cliff in s8e14 "What is... Cliff Clavin?" - here's the scene Commented Jul 19, 2018 at 22:45
  • @BrucePulley: That's the clip... I don't even need to see it (and I can't do videos from my computer, hence the lack of reference). The example was basically a human version of a "right" answer that is still wrong because the person answering the question weight the answer incorrectly.
    – hszmv
    Commented Jul 20, 2018 at 15:54
  • 1
    As for legal matters, machine learning is not AI in the sense that the computer is capable of forming irrational biases... any biases is dependent on a mathmatical formula. Thus, if it was actually applying discriminatory lending practices, it is either a bug or a feature of the code... and it would then have to be discovered how it got into the system, and if the latter, who authorized it.
    – hszmv
    Commented Jul 20, 2018 at 15:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .