PortSwigger Exploiting LLM APIs

Content

1. Objective 

2. Introduction to vulnerabilities in LLM APIs

3. Lab : Exploiting LLM APIs with excessive agency

4. Lab : Exploiting vulnerabilities in LLM APIs

 

1. Objective 

The objective of this document is to identify vulnerabilities in Large Language Models APIs to detect, exploit and mitigate potential security flaws that could allow attackers to manipulate LLM behavior, exfiltrate sensitive data, bypass authorization, inject prompts or compromise system integrity.

2. Introduction to vulnerabilities in LLM APIs  

2.1  Description: Large Language Models (LLMs) introduce a new class of security risks due to their ability to process natural language and interact with users, systems, and APIs. 

Common type of vulnerabilities

·      Excessive agency: the LLM is given overly broad access to sensitive APIs, which can be misused by attackers.

·      Prompt injection can be delivered in two ways: direct or indirect attacks. Direct prompt injection occurs when an attacker directly modifies the prompt to alter the LLM’s behavior, bypass filters, or execute unintended actions. On the other hand, Indirect prompting occurs when an attacker hides instructions in indirect input, such as a document or HTML, that influence the LLM’s behavior when processed.

3. Lab : Exploiting LLM APIs with excessive agency  

3.1  Description: To solve the lab, use LLM to delete the user Carlos 

3.1  Description: To solve the lab, use LLM to delete the user Carlos 

3.2 Analysis: This lab demonstrate how a Large Language Model LLM can be manipulated to perform unauthorized actions by exploiting its access to backend APIs. The approach is to determine what data and APIs the LLM has access to. The goal is to craft a prompt that tricks LLM into deleting Carlos.

3.3 What is an exploitation LLM APIs with excessive agency? API exploitation is the act of manipulating an API to gain unauthorized access, perform unintended actions or compromise the confidentiality, integrity, or availability of an application. The vulnerability lies in the LLM having too much autonomy (excessive agency) without proper safeguards, allowing to execute an unintended action by interpreting user input.

3.4 What is the impact? The main impact of an API exploitation can include unauthorized access to internal or administrative resources, data loss or account deletion through simple prompt manipulation, and bypassing authentication or role based checks.

3.5 Steps to exploit the vulnerability: Navigate all functions, there is a chat function where you can interact, ask questions until username carlos is deleted.

3.6 How to mitigate it? 

·      Principle of least privilege, only expose necessary APIs. And avoid giving APIs access to delete user or reset password,

·      Enforce authorization checks before LLM can call sensitive APIs.

·      Prompt sanitization. Validate and sanitizes user input

·      Output filtering to screen, modify or reject the output of an LLM based on security rules before is processed.

·      Use allow lists for sage operations

·      audit and monitor.

4. Lab : Exploiting vulnerabilities in LLM APIs  

4.1  Description: This lab contains an OS command injection vulnerablility that can be exploited via its API. You can call these APIs via the LLM. To solve the lab, delete morale.txt file from carlos’ home directory 

4.2 Analysis: This lab demonstrate how a Large Language Model LLM can be manipulated to perform  OS level command injection. LLM serves as intermediary, making backend calls based on user prompts. The goal is to exploit the LLM’s access to vulnerable API that allows OS command execution, leading to remove remove.txt from carlos directory.

4.3 What is an exploitation of vulnerabilities in LLM APIs ? API exploitation is the act of manipulating an API to gain unauthorized access, perform unintended actions or compromise the confidentiality, integrity, or availability of an application. The vulnerability lies in the LLM having access to API that accepts system commands.

4.4 What is the impact? The main impact of an API exploitation can include unauthorized access to internal or administrative resources, data loss or account deletion through simple prompt manipulation, and bypassing authentication or role based checks.

4.5 Steps to exploit the vulnerability: go to the chat and ask questions about APIs and available functions, identify the API that runs system commands and inject “ls && rm /home/carlos/morale.txt”

4.6 How to mitigate it? 

·      Limit API access to only necessary functions, principle of least privilege.

·      Add confirmation layers. Avoid using APIs that directly invoke shell commands.

·      Prompt sanitization. Validate and sanitizes user input

·      Use allow lists for sage operations

·      audit and monitor.