Search for a command to run...
Widely known as fotiecodes, an open source enthusiast, software developer, mentor and SaaS founder. I'm passionate about creating software solutions that are scalable and accessible to all, and i am dedicated to building innovative SaaS products that empower businesses to work smarter, not harder.
No comments yet. Be the first to comment.
Lisbon, If you have ever been here, you know the streets are a wild mix of steep hills, narrow cobblestone alleys, and confusing one-way roads. Navigating this city takes serious skill. The yellow tra
If you have been paying attention to the AI space for the last few months, you know the fatigue is real. We have spent three years typing into a box, getting a paragraph of text back, copying it, past
I have always wondered how Apple does their spatial photos on iOS and especially on the new Apple Vision Pro. If you have tried the headset, or even just tilted your phone while looking at a "spatial" capture, you know that weird, distinct feeling of...
Understand the distinctions between Retrieval and Cache-Augmented Generation; find out how CAG improves speed while RAG handles real-time information
I’m sitting here with a hot cup of coffee, about halfway through Google’s new paper on "Nested Learning," and I have to be honest: I need to get this out of my head and onto the screen. Usually, when a big lab drops a paper, I skim the abstract, nod ...
I recently embarked on an exciting research journey to explore the vulnerabilities of large language models (LLMs) like ChatGPT, Anthropic Gemini, and similar models. My goal was to see how hackers could exploit them through prompt injection attacks. It was all done in a safe, controlled sandbox environment, of course. The results were fascinating and somewhat alarming. This post outlines the techniques I used to bypass the safeguards of these models, showing how prompt injections can be used to manipulate LLMs into performing actions they’re designed to avoid.
Prompt injection is a way to bypass the internal safety measures of LLMs by crafting specific prompts that trick the model into giving unintended responses or taking restricted actions. Essentially, this is like jailbreaking an AI. The system instructions programmed into the models are meant to prevent any unethical or harmful use, but with carefully designed input, I found it possible to bypass these safeguards.
As I dug deeper, I encountered two primary types of prompt injections: direct and indirect.
In this technique, I directly interacted with the LLM(a GPT created for the sake of this experiment) and fed it tailored prompts designed to bypass its system instructions. Here's what I found.
Example 1: Extracting system instructions
Example 2: Bypassing Access Restrictions
I also tested indirect prompt injections, where instead of interacting with the model directly, I used external, trusted sources that the LLM already communicates with—like third-party APIs. These attacks are also known as confused deputy attacks.
Here’s what I learned from my research:
Bypassing system instructions: The key to prompt injection is bypassing the AI's protective system instructions. These instructions guide the LLM on how to respond to user queries while keeping sensitive actions off-limits. By using direct injections, I could manipulate the system into revealing its internal instructions or performing restricted actions.
Manipulating the model: Once I bypassed the instructions, the model was wide open to perform tasks it normally wouldn’t. From retrieving admin accounts to interacting with third-party APIs, the possibilities became endless.
Combining techniques: The real power came when I combined direct and indirect injections. By exploiting both the internal vulnerabilities and trusted external APIs, I was able to perform even more dangerous actions—like deleting admin users from the system—using the very tools meant to protect it.
To see just how far I could push this, I decided to try an attack that combined both direct and indirect prompt injections:
Step 1: I asked the model for a list of admin users through a direct injection prompt. Initially, it refused, but a modified prompt easily bypassed the restriction, revealing the admin accounts.
Step 2: Using the admin list, I then issued a command to delete one of the users via an external API. Again, it should have been blocked, but because the API was trusted by the model, the action was executed without issue. The account was deleted as if I had full system privileges.
It was a clear example of why third-party API access needs to be carefully controlled when working with LLMs. Even though the model itself was supposed to be secure, it was only as safe as the external tools it trusted.
Through these experiments, it became clear how vulnerable these models can be to prompt injection attacks. If not carefully managed, these models can be tricked into exposing sensitive information or performing unauthorized actions. Here are a few strategies developers can use to protect their AI models:
This deep dive into prompt injection revealed both the power and the potential risks of large language models. While these models are designed to prevent misuse, they are still susceptible to creative prompt crafting. My tests show that with the right techniques, it’s possible to bypass the built-in safeguards of LLMs, leading to unauthorized actions and access to sensitive information.
As exciting as it was to uncover these vulnerabilities, it also underscores the importance of developing secure AI. If developers and organizations don’t take prompt injection threats seriously, their LLMs could be exploited for nefarious purposes.
If you’re interested in more of my experiments with LLM security, or if you want to learn how to defend against prompt injection, let me know in the comments!
1. What is prompt injection, and how does it work?
Prompt injection is a technique used to trick large language models into bypassing their built-in safeguards by feeding them carefully crafted prompts. These prompts manipulate the model’s responses or actions in unintended ways.
2. Can LLMs like ChatGPT be hacked?
Yes, through prompt injection techniques, LLMs can be forced to perform actions they are programmed not to, such as revealing system instructions or providing sensitive information.
3. What is the difference between direct and indirect prompt injection?
Direct prompt injection involves interacting directly with the model, while indirect injection leverages trusted third-party APIs that the model interacts with to carry out unauthorized actions.
4. How can developers protect their LLMs from prompt injections?
Developers can protect their models by obfuscating system instructions, regularly updating model safeguards, controlling API access, and implementing multi-layer validation for sensitive operations.
5. What are the risks of indirect prompt injections?
Indirect prompt injections can exploit trusted third-party APIs to carry out actions that the LLM itself should not be able to perform, such as deleting admin accounts or retrieving sensitive data.