How do concepts like intent, discrimination, and bias apply to an Artificial Intelligence agent?

Question

Background

I work at a financial institution in the United States and have been researching the regulatory impacts of Artificial Intelligence recently - specifically the simple purpose-specific systems we have today like Alexa, Siri, or the AI that navigates your Tesla when you switch on autopilot. Based on some googling [1] [2], the current standards seem to be that there aren't a lot of specific regulations that apply to the AI agent (with the exception of self-driving cars), but the normal regulations that apply when using a human agent would apply if you substitute an AI-based decision process. The top answer on a related topic here on StackExchange seems to support the idea that the regulations applied to the human apply to the AI agent as well (at least in Hong Kong).

We're currently shying away to research into the legal implications of AIs that can learn any task, much like a human. These are sometimes referred to as Artificial General Intelligence (AGIs) and the most aggressive estimate I've seen in my research places them emerging sometime in the next 100 - 200 years. A problem for future generations!

The Question:

This raises a couple of questions in my mind about a couple of legal concepts that have clear definitions when applied to a human or certain kinds of AI, but get fuzzy with other ones. Specifically:

• How do you determine an AI agent's intent?
• How can you tell if an AI agent is biased in its decisions?
• How would the concept of discrimination apply to an AI agent? How would you prosecute it?

A Specific Example:

For instance, take the anti-discrimination provisions of the Equal Credit Opportunity Act (ECOA). In the case of a decision-tree based AI agent discrimination is obvious - if you add in a rule that makes different decisions based on race, gender, age, or other protected factors you have a discriminatory model.

In the case of a statistically driven Machine Learning model, this is less obvious. For instance, a neural network based model could accept address data as part of its decision. After examining hundreds of thousands of credit applications, the machine learning model could potentially derive information about income distribution from address or similar fields and end up discriminating against applicants that match certain traits. In this case you accept no information on age, sex, or ethnicity but still discriminate based on these factors since the model found some problematic trends in the data that it was trained on.

Recap:

With this in mind, my questions are:

• What are your thoughts on regulations that refer to intent or bias as they relate to AI agents - particularly ones that don't operate on explicitly defined rules?
• Is there any precedent in this space that we could look to to guide decisions on AI risk?
• Does anyone have thoughts on where regulation in this space might head?

Update (7/18)

I've removed the portions of this question with a technical lean and posted a question about the technical aspects of detecting and fixing a discriminatory model to ai.stackexchange.

Answer 1

Firstly, true artificial intelligence does not yet exist. The term "artificial intelligence" is a bit of a buzzword, used to refer to things like neural networks and decision trees, which are really just elaborate statistical calculations. They do not have a "mind of their own" by any stretch of the imagination, though it is possible to make them appear as if they do, in the time-tested tradition of constructing convincing but simple automatons that goes back centuries. Once true AI appears and begins to act as an independent agent in the public sphere, likely new legislation will be produced to address it, so our speculation based on the present situation is unlikely to be proven accurate in the end.

---

The law will tend to view AI as a tool or calculation of the human operators, therefore liability for whatever damage or offense the AI causes rests with the operators. This is little different from prosecuting the person who pulls the trigger, rather than the gun. But two caveats apply:

• Because computer systems are often very complex, it is easy for operators to argue that any harm is unintentional. Especially machine learning approaches that have become popular recently, which often produce unpredictable or unintuitive, if not downright non-deterministic results (indeed, they are often employed to find solutions that intuition could not), are very complex. While understanding the principles of their operation is quite easy for a technical person, predicting the exact result for any given input is much harder, due to sheer volume of numerical calculation involved. So it is hard to prove any outcome is intentional, unless someone went on record stating the intent.
• Because such systems are often implemented by an organization rather than individuals, it is often the organization that ends up being liable, not the programmer.

As for your specific example, it is actually not that important whether the model discriminates explicitly or implicitly. Even a model that makes decision completely at random can still be considered discriminatory in certain settings. This is because US law has a concept called disparate impact, where impact on a protected group alone can be sufficient to argue a policy is discriminative.

Is there any precedent in this space that we could look to to guide decisions on AI risk?

I think autopilots (both planes and cars) are a good example. Another one is automated fraud detection and credit scoring used by financial and insurance industries. In particular you would want to look at disputes where the client disagrees with the results from the statistical model.

By my argument that what is commonly referred to as AI is not really AI but dumb software, it is worth looking at disputes including much simpler devices: For example the controversy surrounding the reliability of breathalyzer devices and their admissibility in court.

Does anyone have thoughts on where regulation in this space might head?

In the immediate future, I think it would be nice to see more accountability from programmers and designers. For example, programs which can cause significant damage (like car autopilots or forensic tools such as breathalyzers) should be required to pass more stringent QC than usual, and they should be produced by specially trained, qualified and certified programmers. I think it is also too easy for corporations to escape responsibility from harm caused by their software tools by just shrugging and claiming it was an accident. Ultimately no one can guarantee a complex program will always perform 100% as intended, but that is not an excuse to write very sub standard programs and have inadequate testing. Thus courts should be more skeptical when an "AI" like a self-driving car causes an accident and the company claims it's not their fault. As complexity and ubiquity of such "AI" systems in everyday life grows, accidents due to software bugs or design issues should become more and more common, so I expect that eventually legislation will be produced to address them (or alternatively, lobbying will overwhelm the process sufficiently so as to bring about a sort of unregulated cyber-anarchy for big tech corporations).

Answer 2

How can you tell if an AI agent is biased in its decisions? How would the concept of discrimination apply to an AI agent? How would you prosecute it?

A subject matter expert would scrutinize or audit the implementation of the algorithm and testify accordingly (see hszmv's allusion to the independent reviewer). From there, counsel representing the meritless party(-ies) will adopt the vexatious role as usual.

Weapons of Math Destruction by Cathy O'Neil "denounces" the type of discrimination scenario sketched in your [original] inquiry. Basically, the book's argument is that decision algorithms are implemented in a way that only reinforces patterns of disparity and discrimination.

Does anyone have thoughts on where regulation in this space might head?

Let me address a different, non-speculative question: where regulation should head. The legislation and judiciary should refrain from intruding in aspects like this. That restrain will allow the market economy to deal with the state of affairs naturally (that is, free from distortions).

O'Neil's general criticism is directed at the application of computer based decision-making. In that sense, her criticism is misplaced because she appears to miss that what she characterizes as "discrimination" might be prevented by fine-tuning the (values of) input parameters of an otherwise robust implementation of the algorithm.

O'Neil also addresses the issue you described in the inquiry as you initially formulated it: the risk of ending up discriminating on impermissible grounds despite not being provided with the prohibited kind of information. But we need to come to terms with the impossibility of dissociating some variables: variables that are and will remain correlated unless a government decides to control a greater portion of private or personal affairs.

For instance, O'Neil mentions the situation where a retail (?) business reportedly sets higher prices to certain high-crime areas. Clearly the algorithm cannot be omniscient of whether those areas might coincidentally also be low-income areas. What is the policymaker to do?

• Will the government order relocation of families to prevent the clustering of households with homogeneous economic status?
• Will the policymaker micromanage the retail business so that a human being is tasked with the pricing for a list of neighborhoods?
• Will the government prohibit a difference in pricing despite the [presumably] greater risk the business and/or deliverer face in the high-crime area?
• Will the policymaker enact laws toward an artificial and dubious dissociation of correlated variables?

In a (free-)market economy, significant differences in the pricing of goods and services tend to disappear because suppliers who are less risk-averse will take advantage of what essentially is an arbitrage opportunity. In the context of free competition, prices will eventually adjust without need for governmental intrusion.

At most, judicial scrutiny of automated decision-making would be pertinent only in few matters. For example, the importance of preventing the incarceration of innocent people justifies all the necessary scrutiny of the implementation and usage of algorithms.

There are other topics where legislation needs to be enforced and be brought up to speed, rather than insist on doing a halfway job (note I'm being generous here) in emerging fields. Likewise, our society would be better off if the judiciary purges itself and extirpates pernicious, outdated doctrines such as judicial immunity or the attorney-client privilege.

Answer 3

There is a really good example of how this could happen, look up the footage of Watson getting the wrong answer on Jeopardy. Watson worked by weighting the inputs of the question and the response as a confidence value that it was the right response. In the failure, the question was about which American city has two Airports that were named after a WWII pilot... and Watson answered "What is Toronto?"

This was hilarious because Toronto is out of the scope... it's a Canadian City not an American one right? Well, the problem was that American can be used in a sense of "Of the United States of America" or "Of the continents of North and/or South America"... Watson read it as the latter meaning, which put Toronto in scope and gave a more confident score. Jeopardy used the former meaning. (in a data dump, Chicago, the correct answer, was Watson's second choice and it was pretty close). For a more human example, Norm on Cheers was correct, as those people had never been in his kitchen, but wrong because that's not the Jeopardy Answer or interpretation of the answer.

In order for your specific answer to occur, the address has to be weighted to produce a certain score of the neighborhood... in order to to prove that it's being used in discriminate manner, one has to look at the code to see if the areas were weighted unfairly based on legal discrimination... You would have to get an independent reviewer to look at the code, and assess if the code is weighted properly, and if that weight changes based on factors. Then you would need to assess if the error is deliberate or if bug testing wasn't thorough enough.

The guilty party would likely be the bank, and possibly the coder if you can prove the coder did it without the bank's knowledge OR if the bank instructed them to break the law and they complied.