How LLMs Are Powering Next-Gen Malware: The New Cyber Frontier
Welcome to the AI cyber arms race, dear reader.
What is the state-of-the-art in LLM-generated malware?
Disclaimer: The following information is for educational and lawful purposes only. Misusing LLMs for unauthorized activities is illegal and unethical. By proceeding, you agree to adhere to all relevant laws and regulations.
Introduction: The biggest LLM risk
The rapid advancement of AI, in particular Large Language Models (LLMs), presents revolutions in cybersecurity, which is defense, and cyber, which is offense. In defensive cybersecurity jargon, which you might be most familiar with, “cyber tools” are called “malware.” Ever wonder what the state-of-the-art is in cyber tools?
Some cybersecurity concerns with LLMs include data privacy, discrimination and bias, attacks on AI systems themselves, misuse of synthetic AI-generated or edited media, using AI to support social engineering cyber-attacks, a lack of transparency and power asymmetry between users and AI companies, and ethical considerations, such as having a human “on” the loop controlling autonomous weapons.
This article is about another LLM risk – one covered much less – using LLMs to develop advanced cyber tools (i.e., malware) that are extremely challenging to detect and defend against. Quoting ChatGPT: “ChatGPT can potentially generate harmful content, including phishing emails, social engineering attacks, or even malicious code. Malicious actors may try to exploit the technology to create malware or other attack vectors.”
How LLMs Generate Malware:
Cyber actors exploit LLMs to generate malware in three ways: they manipulate general-purpose LLMs, buy custom LLMs tailored for cyber-attacks, and train LLMs.
Cyber actors use LLM jailbreaks sold on the dark web as a part of a criminal economy. A more benign “hacktivist” version of an LLM jailbreak is explored here.
Alternatively, cyber actors buy pre-trained LLMs specifically designed for cyber-attacks, such as PhishingGTP, which uses an LLM to generate tailored phishing cyber tools rapidly. These LLMs specialize in one type of cyber tool and do not have the content moderation controls you see in ChatGPT and Gemini. This specialization can make them effective for their intended purposes even if they lack OpenAI or Google’s vast resources.
Lastly, advanced cyber actors, besides buying and chaining together pre-trained LLMs, build their own LLMs to generate cyber tools. Because of the computational resources and technical expertise involved, this is a challenging project, but well within the capabilities of many advanced threat actors. According to Microsoft threat intelligence, Iran has been doing exactly this with LLM-supported social engineering, LLM-generated code, and LLM-enhanced anomaly detection evasion.
Growth areas for LLM cyber tools
LLM-assisted cyber tooling use cases include reconnaissance and vulnerability research, threat generation, personalized attacks, adaptation, and enhanced evasion techniques and security feature bypass.
The initial stages of cyber-attacks, such as reconnaissance, can be automated using LLMs. They can refine proven network management tools, like Nmap, and innovate new methods for vulnerability discovery. Additionally, LLMs can tailor cyber-attack tools to specific targets, making the attacks more precise and effective.
Welcome to Day Zero of the LLM cyber revolution
One interesting use of cyber LLMs is to discover vulnerabilities in target systems and generate exploits for them. The LLM can analyze the entirety of the source code and configuration of the software you use and leverage these vulnerabilities to create cyber tool exploits and attack sequences. This has consequences for zero-day and other time-sensitive exploits.
For commercial software, where the source may be unavailable for LLM training, be aware that cyber LLMs can be trained on disassembled code (assembly code, for example) as easily as they can read open-source code, thanks to tools like Ghidra. Ghidra is a well-known software reverse engineering (SRE) framework developed by the US National Security Agency (NSA) and released to the public back in 2019.
A zero-day attack exploits a previously unknown hardware, firmware, or software vulnerability. This means that a patch to address it does not exist.
Between a patch's release and consumer installation, a one-day attack can take place. The name "one day" suggests a short time frame, but in reality, because patching is delayed, these vulnerabilities remain in every organization for months. The most notable example of a long-running vulnerability is Heartbleed, which Neel Mehta and Michaël Ras of Google's Project Zero security team discovered in 2014. Despite the availability of a vendor provided patch, cyber actors continued to exploit the vulnerability for years due to delays in patching (it was not a straightforward remediation).
LLMs accelerate the discovery of time-sensitive vulnerabilities and increase the effectiveness of cyber tools to exploit them. Cyber actors are eager to launch their attacks before the vulnerabilities are patched in the system they are targeting. Using LLMs reduces the time it takes to make a viable cyber tool (i.e., exploit) enabling cyber actors to successfully attack even systems that are patched quickly.
The forefront in LLMs cyber:
The most advanced use of LLMs in cyber tooling lies not in their ability to create zero day exploits, but in their ability to enhance evasion techniques. AI can generate malware variants with unique characteristics that bypass traditional detection methods. By generating many malware variants, each with unique but functionally equivalent profiles, AI challenges most widely adopted anti-virus detection systems that rely on known patterns. Advanced evasion techniques include content obfuscation through encryption and encoding, polymorphism, metamorphism, and anti-analysis tactics.
Metamorphism is malware capable of rewriting its entire code from scratch in new ways to avoid detection, making it impossible for antivirus software to recognize the threat using traditional methods that search for specific malware signatures. AI's practical application in evasion is not only theoretical, but a reality, with automated and adaptive strategies already in use.
It might be that behavior based cybersecurity tools can help, if the LLMs cannot easily circumvent them as well as the go tos of patching, Continuous Monitoring and Response, Zero trust, Layered Security Advanced Threat Intelligence and cybersecurity awareness training.
This is just a partial list of how and why cyber actors are using LLMs to create and improve cyber tools.
Welcome to the AI cyber arms race
AI-generated malware poses a formidable challenge to traditional cybersecurity due to its rapid evolution, ability to mimic legitimate software, and potential for high-severity attacks. AI's automation capabilities make it difficult to keep up with the constant threat, while the unknown nature of AI algorithms can hinder the development of effective countermeasures. This combination of factors means cybersecurity professionals need innovative tools that leverage AI for cybersecurity to develop effective countermeasures: Welcome to the AI cyber arms race. It’s the only way they have a chance.
Knowledge is power when it comes to protecting digital data and processing. By staying informed about the latest tactics used by cyber actors, we can develop more robust defenses. Understanding cyber actor tactics helps in developing stronger defenses. Resources like Microsoft's threat intelligence reports, which publicly acknowledge the use of LLMs by adversarial nation-states including Russia, China, and Iran, can help us stay ahead of the curve. Additionally, tools such as MITRE ATLAS, dedicated to understanding the adversarial landscape of AI systems, can help improve safeguards.
Welcome to the AI cyber arms race, dear reader.
What is the state-of-the-art in LLM-generated malware?
Disclaimer: The following information is for educational and lawful purposes only. Misusing LLMs for unauthorized activities is illegal and unethical. By proceeding, you agree to adhere to all relevant laws and regulations.
Introduction: The biggest LLM risk
The rapid advancement of AI, in particular Large Language Models (LLMs), presents revolutions in cybersecurity, which is defense, and cyber, which is offense. In defensive cybersecurity jargon, which you might be most familiar with, “cyber tools” are called “malware.” Ever wonder what the state-of-the-art is in cyber tools?
Some cybersecurity concerns with LLMs include data privacy, discrimination and bias, attacks on AI systems themselves, misuse of synthetic AI-generated or edited media, using AI to support social engineering cyber-attacks, a lack of transparency and power asymmetry between users and AI companies, and ethical considerations, such as having a human “on” the loop controlling autonomous weapons.
This article is about another LLM risk – one covered much less – using LLMs to develop advanced cyber tools (i.e., malware) that are extremely challenging to detect and defend against. Quoting ChatGPT: “ChatGPT can potentially generate harmful content, including phishing emails, social engineering attacks, or even malicious code. Malicious actors may try to exploit the technology to create malware or other attack vectors.”
How LLMs Generate Malware:
Cyber actors exploit LLMs to generate malware in three ways: they manipulate general-purpose LLMs, buy custom LLMs tailored for cyber-attacks, and train LLMs.
Cyber actors use LLM jailbreaks sold on the dark web as a part of a criminal economy. A more benign “hacktivist” version of an LLM jailbreak is explored here.
Alternatively, cyber actors buy pre-trained LLMs specifically designed for cyber-attacks, such as PhishingGTP, which uses an LLM to generate tailored phishing cyber tools rapidly. These LLMs specialize in one type of cyber tool and do not have the content moderation controls you see in ChatGPT and Gemini. This specialization can make them effective for their intended purposes even if they lack OpenAI or Google’s vast resources.
Lastly, advanced cyber actors, besides buying and chaining together pre-trained LLMs, build their own LLMs to generate cyber tools. Because of the computational resources and technical expertise involved, this is a challenging project, but well within the capabilities of many advanced threat actors. According to Microsoft threat intelligence, Iran has been doing exactly this with LLM-supported social engineering, LLM-generated code, and LLM-enhanced anomaly detection evasion.
Growth areas for LLM cyber tools
LLM-assisted cyber tooling use cases include reconnaissance and vulnerability research, threat generation, personalized attacks, adaptation, and enhanced evasion techniques and security feature bypass.
The initial stages of cyber-attacks, such as reconnaissance, can be automated using LLMs. They can refine proven network management tools, like Nmap, and innovate new methods for vulnerability discovery. Additionally, LLMs can tailor cyber-attack tools to specific targets, making the attacks more precise and effective.
Welcome to Day Zero of the LLM cyber revolution
One interesting use of cyber LLMs is to discover vulnerabilities in target systems and generate exploits for them. The LLM can analyze the entirety of the source code and configuration of the software you use and leverage these vulnerabilities to create cyber tool exploits and attack sequences. This has consequences for zero-day and other time-sensitive exploits.
For commercial software, where the source may be unavailable for LLM training, be aware that cyber LLMs can be trained on disassembled code (assembly code, for example) as easily as they can read open-source code, thanks to tools like Ghidra. Ghidra is a well-known software reverse engineering (SRE) framework developed by the US National Security Agency (NSA) and released to the public back in 2019.
A zero-day attack exploits a previously unknown hardware, firmware, or software vulnerability. This means that a patch to address it does not exist.
Between a patch's release and consumer installation, a one-day attack can take place. The name "one day" suggests a short time frame, but in reality, because patching is delayed, these vulnerabilities remain in every organization for months. The most notable example of a long-running vulnerability is Heartbleed, which Neel Mehta and Michaël Ras of Google's Project Zero security team discovered in 2014. Despite the availability of a vendor provided patch, cyber actors continued to exploit the vulnerability for years due to delays in patching (it was not a straightforward remediation).
LLMs accelerate the discovery of time-sensitive vulnerabilities and increase the effectiveness of cyber tools to exploit them. Cyber actors are eager to launch their attacks before the vulnerabilities are patched in the system they are targeting. Using LLMs reduces the time it takes to make a viable cyber tool (i.e., exploit) enabling cyber actors to successfully attack even systems that are patched quickly.
The forefront in LLMs cyber:
The most advanced use of LLMs in cyber tooling lies not in their ability to create zero day exploits, but in their ability to enhance evasion techniques. AI can generate malware variants with unique characteristics that bypass traditional detection methods. By generating many malware variants, each with unique but functionally equivalent profiles, AI challenges most widely adopted anti-virus detection systems that rely on known patterns. Advanced evasion techniques include content obfuscation through encryption and encoding, polymorphism, metamorphism, and anti-analysis tactics.
Metamorphism is malware capable of rewriting its entire code from scratch in new ways to avoid detection, making it impossible for antivirus software to recognize the threat using traditional methods that search for specific malware signatures. AI's practical application in evasion is not only theoretical, but a reality, with automated and adaptive strategies already in use.
It might be that behavior based cybersecurity tools can help, if the LLMs cannot easily circumvent them as well as the go tos of patching, Continuous Monitoring and Response, Zero trust, Layered Security Advanced Threat Intelligence and cybersecurity awareness training.
This is just a partial list of how and why cyber actors are using LLMs to create and improve cyber tools.
Welcome to the AI cyber arms race
AI-generated malware poses a formidable challenge to traditional cybersecurity due to its rapid evolution, ability to mimic legitimate software, and potential for high-severity attacks. AI's automation capabilities make it difficult to keep up with the constant threat, while the unknown nature of AI algorithms can hinder the development of effective countermeasures. This combination of factors means cybersecurity professionals need innovative tools that leverage AI for cybersecurity to develop effective countermeasures: Welcome to the AI cyber arms race. It’s the only way they have a chance.
Knowledge is power when it comes to protecting digital data and processing. By staying informed about the latest tactics used by cyber actors, we can develop more robust defenses. Understanding cyber actor tactics helps in developing stronger defenses. Resources like Microsoft's threat intelligence reports, which publicly acknowledge the use of LLMs by adversarial nation-states including Russia, China, and Iran, can help us stay ahead of the curve. Additionally, tools such as MITRE ATLAS, dedicated to understanding the adversarial landscape of AI systems, can help improve safeguards.
