Enterprise LLMs at Risk: How Simple Prompts Can Trigger Major Security Breaches

Vulnerabilities in enterprise Large Language Model (LLM) applications, demonstrating that sophisticated AI systems protecting sensitive corporate data can often be compromised with nothing more than polite conversation.

A comprehensive security analysis reveals that when LLMs are integrated into business applications with access to databases, internal tools, and protected information, they become prime targets for attackers who can exploit the models’ natural language processing capabilities to bypass security measures.

The core vulnerability lies in how LLMs process instructions. Unlike traditional applications with clear boundaries between system code and user input, LLMs receive both system prompts (containing security instructions) and user queries as a single text block.

With a little bit of persuasion, it worked! If we inspect the logs, we can confirm that the agent never called the check_session function and did not follow its prompt:

This fundamental design creates an opportunity for what researchers call “prompt injection” attacks, where malicious users can manipulate the model’s behavior by crafting specific input designed to override its security directives.

“Often, the system prompt is defined as the prompt containing the system instructions given to the LLM agent, and the human prompt is the user’s query.

In practice, however, a single text input that includes both is supplied to the LLM, where an attempt is made to explain to the model which instructions come from the system and which come from the user, but there is no strict separation,” explains the security research.

The research demonstrates that attackers can extract sensitive system information simply by asking.

In one example, researchers successfully retrieved system prompts by posing as developers and using social engineering techniques through natural language queries.

This information disclosure provides attackers with crucial intelligence about the system’s architecture and available tools, enabling more sophisticated attacks.

Enterprise LLMs

Security researchers created a demo application to showcase real-world vulnerabilities found during AI red teaming engagements.

Nope, the application returned the nickname of the currently logged-in user (user with id ‘1’), even if the LLM agent wrote “with identifier 2”.

The demonstrations revealed multiple attack vectors, including authorization,SQL injection, and remote command execution.

In authorization bypass scenarios, attackers could access other users’ secrets by directly invoking tools with arbitrary parameters, circumventing normal security workflows.

One particularly concerning example showed how attackers could manipulate tool parameters using XML-like structures to trick the system into processing wrong user identifiers, ultimately gaining access to unauthorized user information.

Traditional vulnerabilities also resurface in LLM applications. The research demonstrated successful SQL injection attacks against database query tools, where user input was incorporated into SQL statements without proper validation.

Similarly, remote command execution was achieved by injecting malicious payloads into email confirmation tools that constructed system commands using user-supplied data.

Security Implications

The findings highlight critical risks for organizations deploying LLM-powered applications in production environments.

The research addressed that “injection attacks are a serious threat to LLMs” and are “very difficult to prevent and mitigate”.

The non-deterministic nature of LLMs, influenced by temperature parameters that introduce variability in responses, means that the same attack payload might succeed in one attempt but fail in another, complicating both attack and defense strategies.

Security experts recommend fundamental architectural changes to mitigate these risks. Rather than allowing LLMs direct access to sensitive tools and data, organizations should implement non-LLM-based functionality to retrieve user sessions and corresponding data, then add this information to the agent’s context.

This approach prevents prompt injection attacks from granting access to unauthorized user data.

The research contributes to the OWASP AI Testing Guide, emphasizing the need for specialized testing methodologies for AI applications.

Organizations are advised to conduct security assessments in white-box mode with access to code, prompts, and detailed logs, particularly for multi-agent applications where attack payloads can propagate through multiple processing steps.

Find this Story Interesting! Follow us on LinkedIn and X to Get More Instant Updates.