Concerns Emerge Over AI Transcription Tool Used in Health Facilities
San Francisco—The implementation of OpenAI's artificial intelligence-based transcription tool, Whisper, in various sectors, especially healthcare, has sparked significant concern among experts. Whisper, which has been branded by OpenAI as close to "human-level robustness and accuracy," is reportedly prone to producing fabricated text segments, a phenomenon known as hallucinations within the tech community. Such errors include concocted discussions that may involve racial or violent rhetoric or even imaginary medical procedures.
These hallucinations are especially troubling in the healthcare industry, where accurate transcript details are critical. Hospitals and medical centres are increasingly employing Whisper-based systems to transcribe patient consultations with doctors. This is despite OpenAI's cautionary advisement against deploying the tool in "high-risk domains" due to potential error implications.
The extent of the hallucinations, while difficult to quantify, has been highlighted by various tech professionals and academics. For instance, a University of Michigan researcher noted hallucinations in approximately 80% of the audio transcriptions he reviewed during a study on public meetings. Similarly, a machine learning engineer identified fabrication in nearly half of the more than 100 hours of transcription data analyzed, while another expert found inaccuracies in almost all the 26,000 transcripts he developed using Whisper.
The issue persists even with clear and concise audio samples. A recent investigation conducted by computer scientists yielded 187 errors in over 13,000 examined audio snippets, implying that widespread transcription inaccuracies could occur across millions of audio files.
Experts have noted the potentially grave consequences of these errors. Alondra Nelson, a professor at the Institute for Advanced Study and former leader of the White House Office of Science and Technology Policy, stressed the danger posed by such inaccuracies, particularly in healthcare where misdiagnoses based on erroneous data could have severe repercussions.
Given the gravity of these inaccuracies, there have been calls from researchers, advocates, and even ex-employees of OpenAI for regulatory oversight on AI technologies by the federal government. William Saunders, a San Francisco-based research engineer and former OpenAI employee, emphasized the need for the company to focus on resolving this flaw. Saunders resigned from OpenAI following concerns over its strategic direction.
Despite these significant concerns and OpenAI's own advice against the tool's use in decision-making scenarios, where errors could critically impact outcomes, Whisper-based applications have been adopted by over 30,000 medical professionals and 40 healthcare systems globally. This includes notable institutions such as Mankato Clinic in Minnesota and Children's Hospital Los Angeles. These facilities are utilising a Whisper-derived transcription tool developed by Nabla, a firm with branches in France and the US.
Nabla's Chief Technology Officer, Martin Raison, acknowledged the occurrence of hallucinations in Whisper's AI model, noting that the tool has been specifically adjusted for medical contexts to better serve patient-doctor interactions. However, the issue of hallucinations persists. Furthermore, the inability to compare AI-generated transcripts with original audio recordings raises additional concerns. Nabla, citing "data safety reasons," deletes the original audio, thus preventing clinicians from verifying the accuracy of the transcripts. This tool has already been employed to document an estimated seven million medical consultations.
The practice of erasing original recordings, compounded with the potential for unchecked transcription errors, is particularly worrisome to experts such as Saunders. He underscores the danger inherent in this process if healthcare professionals cannot access the initial recording for verification purposes, thereby risking reliance on possibly flawed transcriptions in making medical decisions.
Source: Noah Wire Services