02. May 2025
Ai Code Hiccups Imperil Global Software Supply Chains
The Rise of AI Code Hallucinations: A Growing Threat to Software Supply Chains
Recent years have seen a significant increase in the use of Artificial Intelligence (AI) in software development, with Large Language Models (LLMs) being particularly popular among developers due to their ability to generate code quickly and efficiently. However, a new study has highlighted the risks associated with these models, specifically the phenomenon known as “package hallucinations.” These hallucinations occur when LLMs produce outputs that reference nonexistent third-party libraries, creating an opportunity for supply-chain attacks that can compromise software security.
A study using 16 of the most widely used large language models to generate 576,000 code samples found that a staggering 440,000 package dependencies contained references to nonexistent libraries. This represents approximately 19.7% of the total package references in the generated code samples. Furthermore, among these hallucinated packages, 205,474 had unique package names.
Open source models were found to be more prone to hallucinations than closed-source models, with 21% of their package dependencies linking to nonexistent libraries. This is particularly concerning, as open source models are widely used and relied upon by developers worldwide. The use of these models in software development can lead to a false sense of security, as developers may rely on the model’s generated code without properly verifying its accuracy.
Dependency confusion attacks are a type of attack that exploits this vulnerability. These attacks work by causing a software package to access the wrong component dependency, often by publishing a malicious package with the same name as a legitimate one but with a later version stamp. Software that depends on the package may choose the malicious version rather than the legitimate one because it appears more recent.
The risk of package hallucination flashbacks is real and has been demonstrated in recent proof-of-concept exploits. In 2021, a team of researchers published a paper showcasing an exploit that executed counterfeit code on networks belonging to major companies such as Apple, Microsoft, and Tesla. This attack was made possible by the existence of package hallucinations, which allowed malicious actors to publish fake packages with the same name as legitimate ones.
To understand how these attacks work, it’s essential to delve into the world of software supply chains. A dependency is an essential code component that a separate piece of code requires to work properly. Dependencies save developers the hassle of rewriting code and are a critical part of modern software development. However, when dependencies become muddled, as they can with package hallucinations, it becomes increasingly difficult for developers to ensure the integrity of their software.
When a malicious actor publishes a package under a fictional name that closely resembles an existing library, they rely on a large language model to generate code that references this fabricated library. If a developer trusts the output of the LLM and installs the package without proper verification, the malicious payload hidden within the package is executed on their system.
According to Joseph Spracklen, a University of Texas at San Antonio PhD student and lead researcher on the study, “Once the attacker publishes a package under the hallucinated name, containing some malicious code, they rely on the model suggesting that name to unsuspecting users.” If a user trusts the LLM’s output and installs the package, the malicious payload is executed, potentially compromising the software.
The rise of AI code hallucinations serves as a stark reminder that even the most seemingly innocuous technologies can pose significant risks if not properly vetted. Developers must remain vigilant when using these models, carefully verifying the accuracy of any generated code before deploying it to production.
Improving the robustness of LLMs themselves could be a potential solution to mitigate this risk. By incorporating additional checks and verifications into the model’s output, developers can reduce the likelihood of package hallucinations occurring in the first place. Additionally, developers must stay informed about emerging trends and take proactive steps to improve the security of their software.
The study’s findings serve as a warning bell for the software development community. As AI continues to shape the way we develop software, it is crucial that we prioritize security and verify the accuracy of any generated code before deploying it to production. By doing so, we can prevent devastating consequences and protect our software from malicious actors.
In the world of software development, trust is a luxury few can afford. As we navigate the complexities of modern software supply chains, it’s essential that we remain vigilant and proactive in addressing potential threats. The rise of AI code hallucinations highlights the need for developers to stay informed and take steps to improve the security of their software.
Ultimately, the responsibility for mitigating these risks lies with developers themselves. By staying informed about emerging trends and taking steps to improve the robustness of our software, we can ensure that our applications remain secure and trustworthy. The future of software development depends on it.