Researchers Alert: Open-Source AI Models at Risk of Criminal Exploitation
Recent research has unveiled alarming security risks associated with open-source large language models (LLMs). Hackers and criminals can easily take control of computers running these models, bypassing the security measures of major artificial intelligence platforms. This vulnerability poses significant threats, as researchers highlighted on Thursday.
According to the findings, hackers could exploit these LLMs to conduct spam operations, create phishing content, or launch disinformation campaigns, all while evading established platform security protocols.
The research, conducted by cybersecurity firms SentinelOne and Censys over a span of 293 days, sheds light on the extensive potential for illicit activities stemming from thousands of open-source LLM deployments. These activities range from hacking and hate speech to personal data theft and even child sexual abuse material.
While there are numerous open-source LLM variants available, a considerable number of these models are based on Meta’s Llama, Google DeepMind’s Gemma, and others. Although some models come with built-in guardrails, researchers found hundreds of instances where these safety measures had been deliberately removed.
Juan Andres Guerrero-Saade, executive director for intelligence and security research at SentinelOne, emphasized that discussions within the AI industry regarding security controls often overlook this surplus capacity being utilized for various purposes—some legitimate, others clearly criminal. He likened the situation to an “iceberg” that remains largely unaccounted for within both the industry and the open-source community.
The research specifically analyzed publicly accessible deployments of open-source LLMs through Ollama, a tool that enables individuals and organizations to run their own versions of various large language models. The researchers discovered that they could access system prompts—instructions that dictate model behavior—in about 25% of the LLMs they examined. Alarmingly, they found that 7.5% of these models could potentially facilitate harmful activities.
Geographically, approximately 30% of the observed hosts are based in China, while around 20% are located in the United States.
Rachel Adams, CEO and founder of the Global Center on AI Governance, noted that once open models are released, the responsibility for their subsequent use is shared across the ecosystem, including the originating labs. She stated, “Labs are not responsible for every downstream misuse (which are hard to anticipate), but they retain an important duty of care to anticipate foreseeable harms, document risks, and provide mitigation tooling and guidance, particularly given uneven global enforcement capacity.”
A spokesperson for Meta declined to comment on the responsibilities of developers regarding the downstream abuse of open-source models. However, they did mention the company’s Llama Protection tools and the Meta Llama Responsible Use Guide.
Ram Shankar Siva Kumar, Microsoft AI Red Team Lead, expressed that while open-source models play a crucial role in various fields, they can also be misused if released without appropriate safeguards. Microsoft conducts pre-release evaluations to assess risks associated with internet-exposed, self-hosted, and tool-calling scenarios, and they actively monitor for emerging threats and misuse patterns. “Ultimately, responsible open innovation requires shared commitment across creators, deployers, researchers, and security teams,” he added.
As of now, Ollama, Alphabet’s Google, and Anthropic have not responded to requests for comment.
Interested in AI?
Get automatic alerts for this topic.
Recent research has unveiled alarming security risks associated with open-source large language models (LLMs). Hackers and criminals can easily take control of computers running these models, bypassing the security measures of major artificial intelligence platforms. This vulnerability poses significant threats, as researchers highlighted on Thursday.
According to the findings, hackers could exploit these LLMs to conduct spam operations, create phishing content, or launch disinformation campaigns, all while evading established platform security protocols.
The research, conducted by cybersecurity firms SentinelOne and Censys over a span of 293 days, sheds light on the extensive potential for illicit activities stemming from thousands of open-source LLM deployments. These activities range from hacking and hate speech to personal data theft and even child sexual abuse material.
While there are numerous open-source LLM variants available, a considerable number of these models are based on Meta’s Llama, Google DeepMind’s Gemma, and others. Although some models come with built-in guardrails, researchers found hundreds of instances where these safety measures had been deliberately removed.
Juan Andres Guerrero-Saade, executive director for intelligence and security research at SentinelOne, emphasized that discussions within the AI industry regarding security controls often overlook this surplus capacity being utilized for various purposes—some legitimate, others clearly criminal. He likened the situation to an “iceberg” that remains largely unaccounted for within both the industry and the open-source community.
The research specifically analyzed publicly accessible deployments of open-source LLMs through Ollama, a tool that enables individuals and organizations to run their own versions of various large language models. The researchers discovered that they could access system prompts—instructions that dictate model behavior—in about 25% of the LLMs they examined. Alarmingly, they found that 7.5% of these models could potentially facilitate harmful activities.
Geographically, approximately 30% of the observed hosts are based in China, while around 20% are located in the United States.
Rachel Adams, CEO and founder of the Global Center on AI Governance, noted that once open models are released, the responsibility for their subsequent use is shared across the ecosystem, including the originating labs. She stated, “Labs are not responsible for every downstream misuse (which are hard to anticipate), but they retain an important duty of care to anticipate foreseeable harms, document risks, and provide mitigation tooling and guidance, particularly given uneven global enforcement capacity.”
A spokesperson for Meta declined to comment on the responsibilities of developers regarding the downstream abuse of open-source models. However, they did mention the company’s Llama Protection tools and the Meta Llama Responsible Use Guide.
Ram Shankar Siva Kumar, Microsoft AI Red Team Lead, expressed that while open-source models play a crucial role in various fields, they can also be misused if released without appropriate safeguards. Microsoft conducts pre-release evaluations to assess risks associated with internet-exposed, self-hosted, and tool-calling scenarios, and they actively monitor for emerging threats and misuse patterns. “Ultimately, responsible open innovation requires shared commitment across creators, deployers, researchers, and security teams,” he added.
As of now, Ollama, Alphabet’s Google, and Anthropic have not responded to requests for comment.
Interested in AI?
Get automatic alerts for this topic.
