A recent study conducted by Pennsylvania's Lehigh University has revealed significant racial bias in how chatbots make loan suggestions for mortgage applications. By examining 6,000 sample applications based on data from the 2022 Home Mortgage Disclosure Act, the study found that chatbots recommended more denials for Black applicants than for identical white applicants. Additionally, these bots not only labelled Black and Hispanic borrowers as "riskier" but also suggested higher interest rates for Black applicants.
In comparison, white applicants were found to have a higher likelihood of approval, being 8.5% more likely to receive loan approval than Black applicants with the same financial profile. This discrepancy was even more pronounced among applicants with "low" credit scores of 640, where white applicants were approved 95% of the time, whereas Black applicants were approved less than 80% of the time.
The study was part of an initiative to examine how financial institutions are utilising artificial intelligence (AI) algorithms, machine learning, and large language models to streamline processes such as lending and underwriting. According to Donald Bowen, an assistant fintech professor at Lehigh, these "black box" systems—whose algorithmic workings remain opaque to users—present an opportunity for financial savings but also carry risks related to flawed training data and historical biases.
AI discrimination in the financial sector occurs primarily through decision-making tools and large language models. These systems, which are applied across various industries including healthcare and justice, rely on classification models. For instance, a machine learning algorithm considers inputs such as age, income, and credit history to determine loan approval. However, biased data sets and programming errors can lead to discriminatory outcomes, mirroring systemic racism.
Bowen's interest in exploring racial bias in these models stemmed from a smaller-scale student assignment that revealed similar results. Given that the lending industry is regulated to exclude race from consideration, the findings are particularly concerning. For their study, Bowen's team tested multiple commercial large language models such as OpenAI's GPT-3.5 Turbo and GPT-4, finding consistent biases against non-white applicants.
Despite not explicitly collecting racial data, algorithms may still replicate societal biases through factors like credit scores and zip codes, which themselves can be affected by discriminatory practices like redlining. Even without explicit racial identifiers, these models can infer race from contextual clues, resulting in higher denial rates and interest costs for minority applicants.
The broader implications of AI biases are evident beyond finance, notably in hiring practices where algorithms screen applications, potentially filtering candidates based on discriminatory criteria. Legal cases in the US, such as the $365,000 settlement involving iTutorGroup Inc., underscore the prevalence and impact of these biases. AI is also utilised in the judicial system for risk assessment, highlighting the need for caution due to its potential to influence life-altering decisions.
Some US states, including Utah and California, have enacted legislation to curb algorithmic discrimination. Utah's HB 366 emphasises the need for human oversight in justice-related decisions enhanced by AI tools. Furthermore, the FAIRR Act, introduced in Congress, aims to regulate the financial sector's use of AI, recognising its systemic risks and calling for coordinated oversight.
Professor Bowen suggests that while these technologies continue to grow, fairness can be improved through prompt engineering and routine bias audits of AI systems. He and others advocate for stronger human involvement and government regulation to ensure equitable application of AI tools. Pending legislation like H.R. 6936, which encourages federal agencies to adopt the National Institute of Standards and Technology's AI Risk Management Framework, reflects a move towards improving the trustworthiness and accountability of AI applications.
These findings highlight the ongoing need for vigilance in the use of AI to prevent perpetuating, or exacerbating, societal biases that technology, if unchecked, might propagate.
Source: Noah Wire Services