Prompt injection allows third-party to inject a system prompt with simple message, by inserting special HTML-like sequence (details below). Some of the issues are well-known and pretty old (almost 2 years for Transformers library)
The problem is that for tools like Ollama there is no solution except of to fix it by Ollama developers
More context
Why this is important: prompt injections are pretty dangerous and as long as user input is an instruction to the model it could be decided as code injections vulnerability. And in combination with long-memory and multi-agentic runtimes this vulnerability could stay in system for a long time. And it has not been decided as a serious security vulnerability by the global community yet
Temporal Solution
So if you're running local models just make sure to throw an error when there is a special sequence in user input. For Gemma family it is <|turn> and for tiktoken-based models it's <|im_start|>. And would be nice to see more solutions
Disclaimer
I'm not a security expert
The companies were notified 30 days ago about the issue. Only Google responded with a feedback on the issue (swiftly)
Prompt injection allows third-party to inject a system prompt with simple message, by inserting special HTML-like sequence (details below). Some of the issues are well-known and pretty old (almost 2 years for Transformers library)
Issues:
• Ollama: https://github.com/ollama/ollama/issues/15931
• HuggingFace: https://github.com/huggingface/transformers/issues/29279 (labeled as feature request) https://github.com/huggingface/transformers/issues/47822 (with a bug label)
• Gemma4: https://github.com/google-deepmind/gemma/issues/768
The problem is that for tools like Ollama there is no solution except of to fix it by Ollama developers
More context
Why this is important: prompt injections are pretty dangerous and as long as user input is an instruction to the model it could be decided as code injections vulnerability. And in combination with long-memory and multi-agentic runtimes this vulnerability could stay in system for a long time. And it has not been decided as a serious security vulnerability by the global community yet
Temporal Solution
So if you're running local models just make sure to throw an error when there is a special sequence in user input. For Gemma family it is <|turn> and for tiktoken-based models it's <|im_start|>. And would be nice to see more solutions
Disclaimer
I'm not a security expert
The companies were notified 30 days ago about the issue. Only Google responded with a feedback on the issue (swiftly)