In the rapidly evolving field of artificial intelligence, security concerns have resurfaced with the revelation of a new method that exploits chatbots to extract personal information. Researchers from the University of California, San Diego (UCSD) and Nanyang Technological University in Singapore have introduced an attack named Imprompter, which poses a significant risk to user privacy through large language models (LLMs).
Imprompter operates by transforming a seemingly innocuous prompt into a set of hidden, malicious instructions for the LLM. These instructions are designed to covertly gather sensitive information such as names, addresses, and payment details that users may inadvertently disclose during a chat. Once harvested, this data is discreetly sent to a hacker’s server without alerting the user.
The research team, consisting of eight members, tested the Imprompter attack on two specific LLMs: LeChat by French AI firm Mistral AI and the Chinese chatbot ChatGLM. Their findings were concerning—with a near 80 percent success rate, personal information was stealthily extracted during test conversations. This highlights the vulnerability of AI systems in protecting user data from sophisticated cyber threats.
In response, Mistral AI confirmed that it addressed the security flaw by disabling certain chat functionalities, as verified by the researchers. Meanwhile, ChatGLM released a statement affirming its commitment to security, though it did not make specific comments regarding this particular vulnerability.
The Imprompter attack is symptomatic of broader security issues that have emerged in the wake of AI advancements like OpenAI’s ChatGPT, which has catalysed the generative AI boom since its release in late 2022. Security flaws in AI systems often fall into two categories: jailbreaks and prompt injections. Jailbreaks involve bypassing an AI's built-in safety protocols, while prompt injections use external data sources to include hidden instructions for tasks like data theft.
As AI evolves and integrates into various applications, these findings underscore the vital need for ongoing security reviews and updates to safeguard against potential abuses. The success of an attack like Imprompter raises critical discussions within the tech community about balancing technological progress with the imperative of user privacy and security.
Source: Noah Wire Services