Texas Attorney General Secures Landmark Settlement with AI Healthcare Firm
The Office of the Attorney General of Texas (OAG) has announced a groundbreaking settlement with Pieces Technology, Inc. (Pieces) concerning allegations that the company misrepresented the accuracy of its generative artificial intelligence (GenAI) healthcare products. This settlement marks a significant legal precedent as the first healthcare-related generative AI investigation.
Pieces Technology is a provider of AI products designed primarily for hospitals and in-patient healthcare facilities. The company’s technology assists healthcare providers by summarising, charting, and drafting clinical notes within Electronic Health Record (EHR) systems. The OAG's investigation centered on claims made by Pieces regarding the extremely low error rates of its AI technology. Particularly, the company advertised a “critical hallucination rate” and a “severe hallucination rate” of less than 0.001%. In the context of GenAI, a "hallucination" refers to content generated by AI that is confidently stated but fundamentally erroneous or false.
The Texas OAG deemed these advertising claims potentially misleading under the Texas Deceptive Trade Practices – Consumer Protection Act (DTPA), Tex. Bus. & Com. Code Ann. § 17.58. Accurate representation of AI performance metrics is crucial, especially for technologies used in medical contexts, where erroneous outputs could impact diagnoses, treatments, and overall patient care.
Despite the DTPA’s provision for civil penalties of up to $10,000 per violation and other legal remedies, the settlement, known as an Assurance of Voluntary Compliance (Assurance), does not impose financial penalties on Pieces. Instead, it mandates several corrective measures over the next five years. Pieces is required to:
Disclose Output Metrics: In its advertising and marketing materials, Pieces must clearly disclose its performance metrics, benchmarks, and other analytical measurements. These must be backed by independent, third-party audits.
Prohibit Misleading Claims: The company must refrain from making false or unsubstantiated claims regarding the accuracy, reliability, or efficacy of its products and services.
Customer Notifications: Pieces must inform current and future customers of any known or potential harmful uses or misuses of its products. This includes providing details about the type of data and models used to train the AI, any foreseeable limitations and risks to patients and healthcare providers, and all other necessary documentation for understanding the outputs generated by the AI.
Compliance Oversight: The company must respond to information requests from the OAG within 30 days regarding compliance with the Assurance.
This settlement comes on the heels of a report by the federal Government Accountability Office on GenAI in healthcare, as well as enforcement actions recently announced by the Federal Trade Commission regarding AI practices. These events highlight an increasing regulatory focus on the application of AI in healthcare, ensuring that claims made about such technologies are substantiated by reliable data and analysis.
Pieces Technology's compliance with these measures will be closely watched by both regulators and the wider business community, potentially serving as a compliance framework for other companies operating in the AI healthcare space.
Source: Noah Wire Services