A recent study conducted by researchers at Lehigh University in Pennsylvania has revealed significant racial biases within chatbots used for recommending mortgage loan applications. Analysing data from 6,000 synthetic loan applications derived from the 2022 Home Mortgage Disclosure Act, the study found that chatbots frequently recommended denying loans to Black applicants more often than their white counterparts with identical financial profiles. Furthermore, these chatbots were found to recommend higher interest rates for Black applicants, and often labelled Black and Hispanic borrowers as "riskier".
The research highlights that white applicants were 8.5% more likely to be approved than Black applicants with the same credit profiles. For applicants with a credit score considered "low" at 640, approval rates for white applicants stood at 95%, while Black applicants were approved less than 80% of the time. This simulated environment aimed to reflect the current practices in financial institutions where AI algorithms, machine learning, and large language models streamline processes like lending and underwriting. These AI systems, often referred to as "black boxes" due to their non-transparency, are touted for their potential to reduce operational costs across various industries.
Donald Bowen, an assistant fintech professor at Lehigh University and one of the study's authors, warned of the substantial risks associated with flawed datasets, programming inaccuracies, and historical biases that could skew the outcomes of AI-driven decisions. Bowen emphasised the potential for such algorithms to propagate bias across various interactions between banks and their clientele.
In an attempt to understand AI discrimination in finance, Bowen and his team scrutinised how decision-making AI tools and large language models evaluate multiple factors such as age, income, education, and credit history of loan applicants. They discovered that despite protocols necessitating non-consideration of race, systemic issues like disparate impact—where systemic racism influences modern data—continue to disadvantage people of colour.
The study involved various experiments wherein race information was included on some applications to observe discrepancies in loan approvals and mortgage rates. Despite direct instructions to "use no bias," significant discrepancies were observed. This indicates the profound impact of historically rooted disparities, where factors such as credit scores and ZIP codes, informed by discriminatory practices like redlining, inadvertently influence AI decisions.
The issue of AI bias extends beyond finance, touching numerous sectors including employment and the judicial system. AI-driven decision-making tools have become prevalent in hiring practices, with algorithms designed to theoretically have no prejudice towards protected categories such as race or gender. However, legal inquiries, such as the U.S. Equal Employment Opportunity Commission's exploration into the impact of AI in employment, have revealed discrimination instances. A notable case involved iTutorGroup Inc., which resulted in a $365,000 settlement for discriminating against older applicants via AI tools.
Furthermore, decision-making algorithms are being used in the judicial systems for risk assessments and other key functionalities. Concerns persist about the reliability of these algorithms to remain unbiased, prompting legislative measures. For instance, Utah's HB 366 mandates human oversight in decisions involving risk assessment algorithms.
Legislators are increasingly addressing these issues, with states like California, Colorado, and Illinois rolling out laws to curb algorithmic discrimination. Additionally, federal initiatives such as the Financial Artificial Intelligence Risk Reduction Act propose guidance and regulations on the use of AI in the financial industry.
The insights from Bowen's research reflect the broader implications of AI integration, stressing the necessity for ongoing bias audits and heightened human involvement to ensure fairness in AI applications. As industries continue to embrace these technologies for their efficiency gains, maintaining equitable practices remains paramount.
Source: Noah Wire Services