How to Explain Model Bias to Regulators

Illustration by Gary Clement
apple model bias regulators illustration by gary clement
Illustration by Gary Clement

What models are we talking about?

Have you ever wondered how Netflix recommends movies suited to your taste? Ever thought about how self-driven cars have come into picture? Or how credit card agencies are so quick in detecting transaction frauds? The answer to each question is Machine Learning and Artificial Intelligence (AI). 

With the availability of endless amounts of data and the competency of today’s computers to process it, businesses have taken a huge shift towards data-driven decision-making. While these solutions are deemed to be more accurate and robust than conventional methodologies, the need to understand that models are biased is still prevalent. By bias, we try to figure out whether our models and predictions discriminate based on some acquired demographic characteristics which include: race, gender, educational background, disability, color, social status, etc. These considerations are important when it comes to regulatory bodies that have stringent laws that do not permit any kind of discrimination; intentional or otherwise. 

What is model bias and why is it undesirable – some examples

One of the best examples to understand bias is COMPAS (short for Correctional Offender Management Profiling for Alternative Sanctions). It is a case management tool used by US courts to predict the likelihood of a criminal committing a future crime. It came under scrutiny and controversy when the results of the underlying algorithm said that black origin defendants were often deemed to be more prone to criminal behavior in the future, although all the other parameters were the same. This is a classic example of biased models and shows how an acquired parameter (that of color) led to injustice and discrimination. 

Other examples to understand bias is the credit lending sector in financial services. A study conducted at UC Berkeley showed that lending in the United States was biased against Latinx/African-American borrowers. The statistics tell us that they charged 7.9 and 3.6 basis points more than other borrowers and were required to pay 765 million USD more of interest in aggregate per year. These examples provide us sufficient insights that although automated models are designed to improve the discrimination and fallacies of human decisions, they have not been able to perform a great job at that. While one can argue that the overall decision making is far better than that involving human intervention, the problem of bias is not catered as we would have liked.

 Women ended up with a much smaller bite of Apple’s credit card.

The New York Department of Financial Services is opening a probe into the new Apple Card’s algorithms build in collaboration with Goldman Sachs which determine credit limits, after a series of tweets from a technology entrepreneur for alleged gender discrimination. He criticized the Apple Card for giving him 20 times the credit limit that his wife got leading to possible sex discrimination.

Goldman has responded by saying that it would be difficult to discriminate based on gender given that they don’t know the applicant’s gender during the application process. This statement led to sparking conversation about black-box algorithms and the inherent biases in those systems of Goldman and Apple.

So how do we know there isn’t an issue with the machine-learning algorithm when no one can explain how this decision came?

Where does explainability come in all this?

One of the root causes of bias originates from the fact that businesses try to develop models for accuracy and profits. The basic premise of modeling does not account for fairness at all. Moreover, historical data plays a vital role in deciphering the bias of any model. Models are only as good or as unbiased as the training data they were trained upon. If the training data belongs to a particular category of individuals, such that these individuals are representative of a particular acquired characteristic (such as gender, race, color, social status, etc.), the models and applications developed using this data have a high probability of getting biased. Another reason can be the features on which the model is built upon. Does the feature set contain any demographic parameter? Or does it have an attribute that is not explicitly demographic but is leading to bias?

The more complex the model, the less explainable it will be. Thus, a more likely chance of not being able to understand its bias. So, to explain the model and to find its bias, we should strike a good relationship between the complexity of the model and the accuracy of its predictions. We should be in a position to explain the weights/significance of the features as given by the model. 

Are there any tools available to combat model bias? 

Despite all our efforts to omit any bias from the origination of the model development, it might be impossible to build a bias-free model, which is why researchers have introduced tools to determine and quantify the amount of fairness in a given model.

‘Lime’ is a widely used module that determines which inputs have more weightage in the predictions and then tries to evaluate the nature of change in the prediction after removing these inputs, in the process, understanding whether some parameters have an extraordinary bias. ‘AI Fairness 360’ developed by IBM works with the help of bias mitigation algorithms such as optimized pre-processing, disparate impact remover, and equalized odds post processing; and even has an interactive module to let users see and analyze various metrics which may cause bias. Aequitas is yet another product that is widely used in loan underwriting departments. It assists in stopping certain groups of people from getting disadvantaged and ensures that loans are sanctioned in a non-discriminatory pattern to everyone. Other toolboxes include the What-if plugin by Google and the FairML toolbox.

Whenever there are fallacies in model performance, it leads to monetary loss of money in the form of fines for a company. The regulatory team often conducts audits to understand the reasons behind these issues, whether validation steps were in place before production, and what recovery steps should be taken to alleviate the loss. In order to mitigate the risks associated with these audits, we need a model management and governance platform to be a step ahead of the curve. This is where Datatron comes in. Biased models can go into production, so by monitoring the responses of the model and feedback given by the user, Datatron’s platform can alert users and provide monitoring and evidence to quickly show auditors what steps were taken in production when your models are biased. Read more about Datatron here or visit our website.

Summarizing, the presence of any kind of bias in models is not ideal. Not only from regulatory points of view but also because it drives discrimination in the society. While restricted use of demographic attributes goes a long way in eliminating this, the utilization of the toolkits mentioned above can ensure minimal presence of bias. 

Check out our previous article about statistical bias and why it is so important to data science.


Let's Discuss