A recent study conducted by researchers at Pennsylvania’s Lehigh University has uncovered significant racial biases in how chatbots recommend loan decisions, particularly in the context of mortgage applications. The study, utilising 6,000 sample loan applications drawn from the 2022 Home Mortgage Disclosure Act, revealed that Black and Hispanic applicants were consistently disadvantaged compared to their white counterparts, despite having identical financial profiles.
The research team discovered that chatbots recommended the denial of loans to more Black applicants than white ones and advised higher interest rates for Black applicants. Furthermore, Black and Hispanic borrowers were often labelled as "riskier." The study plays a critical role in highlighting how AI-based decision-making systems used in financial services can perpetuate existing societal biases. For instance, white applicants with “low” credit scores of 640 were approved 95% of the time, whereas Black applicants with the same score were approved less than 80% of the time.
The implications of these findings are vast, given the increasing reliance on AI algorithms and machine learning systems within various industries, including finance. These decision-making tools, often referred to as “black box” systems due to their opaque nature, are designed to streamline processes such as lending but can inadvertently reinforce historical biases.
The lead researcher, Donald Bowen, an assistant fintech professor at Lehigh University, emphasized the importance of recognising the potential for these algorithms to propagate biases. Bowen was motivated to conduct this study after a smaller experiment with his students revealed similar discriminatory patterns in AI recommendations. The comprehensive study involved running thousands of loan applications through several large language models, such as OpenAI’s GPT 3.5 Turbo, GPT 4, Anthropic’s Claude 3 Sonnet and Opus, and Meta’s Llama 3-8B and 3-70B.
One of the key revelations of the study was the persistence of bias even when race data was not overtly included in the decision-making process. This occurs due to indirect influences like credit scores, geographic indicators, and other socio-economic factors that have been historically affected by systemic racism, such as redlining. These factors inadvertently serve as proxies for race and ethnicity in AI algorithms, continuing the cycle of discrimination.
Beyond the financial sector, AI decision-making tools are seeing widespread application in areas like employment, healthcare, and the judicial system. Legal challenges have arisen, revealing underlying biases in hiring algorithms and sentencing recommendations. A notable case involved iTutorGroup Inc., which settled a lawsuit over its AI hiring tool's alleged discrimination against older job applicants.
Recognising these challenges, several states have begun taking legislative action to curb algorithmic discrimination. Utah, California, Colorado, and Illinois have passed laws aimed at preventing AI systems from contributing to discrimination based on inherent characteristics such as race, gender, or disability. These legal frameworks complement federal initiatives like the proposed "FAIRR Act," which seeks to regulate AI use within the financial industry to mitigate risks.
The research from Lehigh University underscores the need for financial institutions and other sectors to implement regular audits and establish mechanisms to ensure fairness in AI-driven decision-making. Experts call for increased human oversight and regulatory guidance to ensure these technologies are deployed responsibly. There is a growing consensus among academics, such as Michael Wellman from the University of Michigan, on the importance of developing standards like the National Institute of Standards and Technology's AI Risk Management Framework to manage the biases inherent in AI systems.
As industries become more reliant on such technologies, the call for transparency, accountability, and fairness in AI continues to gain urgency. This study serves as a pivotal resource for organisations aiming to balance the efficiency benefits of AI while addressing the critical need for ethical and unbiased applications.
Source: Noah Wire Services