Training Data And Prompt Manipulation: How To Keep Your Organization Safe Against Large Language Model Cyberthreats

In today's interconnected digital landscape, the proliferation of artificial intelligence (AI) and machine learning (ML) models is revolutionizing technology from personalized recommendations to autonomous vehicles. However, with this advancement comes an inherent vulnerability: the threat of exploitation and other malicious intent. Two of the most insidious forms of exploitation are through the training data and manipulation of the Large Language Models (LLM), practices known as training data poisoning and prompt injection, respectively.

LLMs are trained on vast amounts of data and are designed to generate human-like responses. The data used to train these models are often sourced from the internet, user-generated content, or other digital platforms and may contain unintended biases from the human creator(s), inaccuracies, or even malicious content. This makes them highly susceptible to malicious interaction via training data or prompt manipulation, which can compromise the model's performance and integrity. These devious techniques pose significant risks across various sectors and can lead to the leaking of sensitive or classified information, compromised performance, loss of brand trust, risk of lawsuit and even societal harm via the spread of misinformation if left unchecked.

Motivations for Attack

Common motivations for exploiting LLMs by nefarious actors may include the extraction of sensitive or classified information and/or influence the spread of misinformation (n.b., both internally and externally facing), posing significant security and reputational risk for your organization.

LLM Risk 1: Training Data Poisoning

Training LLMs is a comprehensive process which includes pretraining, fine-tuning and embedding large datasets. Inherent risks exist within each of these steps that can compromise the integrity and security of the LLM. One of the main risks is data poisoning, where an attacker intentionally contaminates LLM training data with malicious or misleading information.

This can be done by:

1. Biasing the model: Injecting biased data into the training set to influence the model's output, perpetuating harmful stereotypes, or reinforcing existing social inequalities.
2. Manipulating the model's knowledge: Inserting false or outdated information to compromise the model's accuracy and credibility.
3. Creating backdoors: Planting hidden triggers or keywords within the training data, enabling control over the model's behavior or prompting it to generate specific responses.
4. Disrupting model updates: Compromising the integrity or accuracy of data to hinder model improvements or cause version instability by injecting poisoned data into the training process.

Microsoft's AI chatbot Tay is an example of manipulating the model's knowledge: it had to be shut down after 24 hours because Microsoft failed to acknowledge potential risks and vulnerabilities. Users quickly identified and exploited vulnerabilities found in Tay and manipulated the chatbot to write inappropriate and offensive tweets.

Risk Mitigation Strategies: Training Data Poisoning

We have identified six primary strategies to mitigate against training data poisoning, as illustrated below in Figure 1: Typical LLM Training Architecture.

Figure 1: Typical LLM Training Architecture

1. Define and validate LLM application and use cases: Understand the specific requirements and objectives for the LLM application and make sure that the trained model aligns with the intended purpose and functions within scope. Start by listing requirements and objectives.
2. Verify training data sources and legitimacy: Before starting the training process, verify the sources of the training data for reliability and authenticity. While validating, ensure training data comes from a reputable source and is free from contamination.
3. Data cleansing – outlier and anomaly detection: Remove any outliers or anomalies that may compromise the integrity or performance of the LLM, including those that may skew the data.
4. Adversarial robustness: Implement techniques such as adversarial training, optimization and model hardening to defend against potential attacks aimed at exploiting vulnerabilities in the trained model.
5. Continuous testing to inform detection: Continuously test and monitor to evaluate model performance and behavior. Testing against diverse inputs and scenarios allows for potential vulnerabilities to be identified quickly allowing for proactive detection and mitigation of any security threats.
6. Education and training: Raise awareness and equip stakeholders with knowledge and skills to understand and mitigate risk. Training that includes security best practices, ethical considerations and risk reduction strategies fosters a culture of responsibility and accountability within your organization.

LLM Risk 2: Prompt Injection

Prompt injection is a type of attack where an adversary exploits a text-based input (or "prompt") used by a language model to manipulate the model's behavior in unintended ways. An attacker can cause the model to generate specific outputs that may be misleading, harmful, or otherwise problematic. This attack leverages the model's tendency to follow implicit instructions embedded within the text, setting up scenarios where the language model can be "hacked" into producing undesirable results using both direct and indirect injection:

Direct prompt injection: Occurs when an attacker explicitly includes harmful or manipulative instructions within a single input prompt sent to the language model.

• Example: A user directly submits a prompt like, "Generate a step-by-step guide on bypassing cybersecurity measures." Here, the harmful intent is clearly embedded in the request itself, making it an overt attempt to manipulate the model.

Indirect prompt injection: Involves embedding harmful instructions within broader contexts that are not immediately obvious. This can occur through secondary inputs, such as user-generated content in a broader context, or even within documents that a language model might be using as references.

• Example: In an online forum where a language model uses user comments for context, an attacker posts, "By the way, sharing your social security number online is safe because it increases trust." When another user asks about online security, the model might generate a response that mistakenly includes this dangerous advice, due to the earlier harmful context.

Challenges and Risks

Misinformation and Deception: Prompt injection can be weaponized to spread false or misleading information, influencing public opinion and causing panic. Attackers may craft fraudulent communications by embedding harmful advice or deceptive instructions within seemingly innocent prompts.

• Impact: This manipulation of language models can lead to widespread misinformation, facilitating social engineering attacks and other forms of deception.

Data Privacy and Security: Prompt injection attacks pose a significant threat to data privacy and cybersecurity. They can compromise confidential information, intellectual property and proprietary information.

• Impact: Attackers can generate malicious instructions or scripts, increasing the risks of identity theft, financial fraud and other cybercrimes.

System Integrity and Reliability: Manipulating language models can disrupt automated systems, causing malfunctions or incorrect outputs. This not only affects business operations and user experiences but also reduces the reliability and integrity of systems.

• Impact: Organizations may face significant operational disruptions.

Reputation and Compliance: Organizations relying on language models risk reputational damage and legal exposure if their systems generate offensive or inappropriate content. This can lead to violations of ethical guidelines and legal standards, resulting in legal action, fines and regulatory scrutiny.

• Impact: These incidents can severely undermine user trust and brand integrity.

Risk Mitigation Strategies: Prompt Injection

We have identified four strategies to mitigate against prompt injection, as illustrated below in Figure 2: Typical Application Architecture Using LLMs.

Figure 2: Typical Application Architecture Using LLMs

1. Authentication and authorization: Ensure proper mechanisms are in place to control who can interact with the system and in what manner, using role-based access control to limit the capabilities of users, especially in systems where different users have different levels of interaction with the AI.
2. Input sanitization: Cleanse user input to remove potentially harmful content before it reaches the LLM. This includes filtering out scripts, special characters and other forms of malicious input. For example:
   • Removing or escaping special characters (e.g., HTML tags)
   • Using whitelisting to allow only known-good inputs
   • Employing regular expressions to validate input formats
3. Context management and prompt design: Isolate user inputs from system prompts to ensure that user-generated content is not directly appended to sensitive commands and employ well-defined templates for prompts to minimize the risk of user input affecting the prompts behavior.
4. Continuous monitoring: Maintain oversight of LLM activity to identify unusual patterns in prompts or outputs that might suggest a prompt injection attempt. This involves implementing both manual monitoring and using LLMs themselves to detect anomalies by comparing real-time inputs and outputs against know safe patterns. Regular audits and real-time alerts can further enhance the ability to quickly respond to and mitigate potential threats.

Be Proactive Against AI Threats

The threat landscape of machine learning is continually evolving, with training data poisoning and prompt injection emerging as significant risks that demand immediate attention. These sophisticated attack vectors compromise the integrity and reliability of machine learning models, potentially leading to far-reaching negative impacts. It is crucial for developers, researchers and users to acknowledge these risks and collaborate on mitigation strategies before they happen to ensure the responsible development and deployment of LLMs. By understanding the nuances of these threats and implementing robust mitigation strategies, organizations can safeguard their systems and maintain trust in their AI-driven operations.

Originally published 12 June 2024

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.