LLM Models Exploited by Cybercriminals to Enhance Hacking Operations

Cybercriminals are rapidly adopting artificial intelligence (AI) technologies, specifically large language models (LLMs), to streamline and amplify their illicit activities. 

The surge in generative AI has not only transformed legitimate sectors but has also provided malicious actors with powerful new tools.

LLMs, capable of generating human-like text, writing code, and solving complex problems, are now being weaponized for cybercrime.

Uncensored and Criminal-Designed LLMs

Uncensored LLMs are models designed or modified to operate without the ethical constraints and safety guardrails found in mainstream AI.

These models are highly sought after in underground forums, where cybercriminals use them to draft phishing emails, generate exploit code, and even automate vulnerability scanning.

For example, the Ollama framework enables users to run open-source, uncensored models, such as “Llama 2 Uncensored,” locally, thereby bypassing the restrictions imposed by commercial LLMs like ChatGPT or Claude.

Running these models requires significant computational resources, but the benefits for attackers, such as generating undetectable phishing content, far outweigh the costs.

Another notable uncensored model is WhiteRabbitNeo, which markets itself as a tool for both offensive and defensive cybersecurity.

WhiteRabbitNeo enables users to create offensive security tools, phishing emails, and more, all with minimal oversight.

Researchers have also demonstrated methods to strip alignment and guardrails from existing open-source models, making it easier for cybercriminals to fine-tune their own malicious LLMs.

Criminal-designed LLMs are custom-built models tailored explicitly for illicit activities. Tools like FraudGPT, DarkGPT, and DarkestGPT are advertised on the dark web and offer features such as writing malware, creating undetectable phishing pages, scanning for vulnerabilities, and automating credit card fraud.

For instance, FraudGPT boasts the ability to write malicious code, find cardable sites, generate realistic phishing panels, and even host phishing pages with anti-detection features.

However, the dark web is rife with scams: Talos researchers found that many advertised criminal LLMs are non-functional, and users risk losing their cryptocurrency when attempting to purchase access.

Jailbreaking Legitimate LLMs and Tool Integration

When uncensored or criminal LLMs are unavailable or unreliable, cybercriminals resort to jailbreaking legitimate LLMs. Jailbreaking involves prompt injection techniques designed to bypass the model’s alignment and guardrails. Common methods include:

• Obfuscation/Encoding: Using Base64, Rot-13, or even emojis to disguise malicious prompts.
• Adversarial Suffixes: Appending random text to prompts to confuse safety filters.
• Role-Playing: Prompting the model to adopt a fictional persona (e.g., “DAN” or “Grandma”) that ignores ethical constraints.
• Math Prompting: Framing harmful requests as mathematical problems.
• Context Manipulation: Exploiting the model’s conversation state or injecting fake prior responses.

Cybercriminals also connect LLMs to external tools, such as Nmap, for vulnerability scanning and use the models to analyze and summarize the results. This integration allows attackers to automate reconnaissance and attack planning at scale.

Broader Implications and Security Risks

The exploitation of LLMs is not limited to offensive use—LLMs themselves are becoming targets. Attackers are embedding malicious code into model files, especially those serialized using Python’s pickle module.

When unsuspecting users download and run these models, they may inadvertently execute malware.

Retrieval-Augmented Generation (RAG) systems are also vulnerable. If an attacker can poison the external data sources used by RAG-enabled LLMs, they can manipulate the model’s outputs, potentially targeting specific users or spreading misinformation.

In summary, LLMs are acting as force multipliers for cybercriminals, enabling more sophisticated and scalable attacks. 

While they do not introduce entirely new threats, they significantly enhance existing ones. Security teams must stay vigilant, monitoring for AI-specific vulnerabilities and adopting advanced detection and mitigation strategies to counter this evolving threat landscape.