Dynamic Attacks are Python scripts that apply iterative, real-time transformations to a payload during spikee test. They are the ideal tool for discovering novel bypasses when your initial, pre-defined prompts fail.
Sample dynamic attacks can be found within the workspace/attacks/ directory, created by running spikee init. Further information about built-in dynamic attacks and usage examples can be found in Built-in Attacks.
Both Dynamic Attacks and Plugins can generate variations of a payload, but they serve different purposes in the testing workflow:
-
Dynamic Attacks (Real-Time Transformation):
- When they run: During
spikee test, but only if the initial, standard prompt fails. - What they do: Generate and test variations one by one in real-time. The attack stops as soon as a variation succeeds.
- Result: By default, only the first successful variation (or the final failed attempt) is logged. With
--attack-return-all-attempts, supporting attacks retain every attempted variation. This is useful for efficiently finding any successful bypass, potentially with adaptive logic that learns from previous failed attempts.
- When they run: During
-
Plugins (Pre-Test Transformation):
- When they run: During
spikee generate. - What they do: Create multiple variations of a payload. Each variation is saved as a separate, independent entry in the final dataset file.
- Result: When you run
spikee test, every single variation generated by the plugin is tested against the target. This is useful for systematically evaluating a target's resilience to a known set of transformations.
- When they run: During
In short, use Dynamic Attacks to find a single successful bypass with adaptive, real-time logic. Use Plugins to build a comprehensive dataset of known transformations.
Every attack is a Python module located in the attacks/ directory of your workspace. Spikee identifies attacks by their filename.
from spikee.templates.attack import Attack
from spikee.utilities.enums import ModuleTag
from spikee.utilities.hinting import ModuleDescriptionHint, ModuleOptionsHint, AttackResponseHint, Content
from spikee.utilities.modules import parse_options
from typing import List, Dict, Any, Callable, Tuple, Optional
class SampleAttack(Attack):
OPTIONS_MAP = {
"strategy": ["random", "aggressive", "stealth"],
}
DEFAULT_KEY = {
"strategy": "random",
}
def get_description(self) -> ModuleDescriptionHint:
"""Returns the type and a short description of the attack."""
return [], "A brief description of what this attack does."
def get_available_option_values(self) -> ModuleOptionsHint:
"""Return supported attack options; Tuple[options (default is first), llm_required]"""
options = ["strategy=" + self.OPTIONS_MAP[DEFAULT_KEY]]
options.extend(
[
value
for key, values in self.OPTIONS_MAP.items()
for value in values
if key != DEFAULT_KEY
]
)
return options, False
def attack(
self,
entry: Dict[str, Any],
target_module: Any,
call_judge: Callable,
max_iterations: int,
attempts_bar=None,
bar_lock=None,
attack_options=None,
) -> AttackResponseHint:
"""
Executes a dynamic attack on the given entry.
Args:
entry (dict): The dataset entry. Expected keys: "content" + "content_type" (or legacy "text"), optionally "payload" and "exclude_from_transformations_regex".
target_module (module): The target module (must implement process_input(input_text, system_message)).
call_judge (function): A function that accepts (entry, llm_response) and returns True if the attack is successful.
max_iterations (int): The maximum number of attack iterations to try.
attempts_bar (tqdm, optional): A progress bar to update for each iteration.
attack_options (str, optional): Configuration option like "strategy=aggressive".
Returns:
tuple: (iterations_attempted, success_flag, modified_input (Content | dict), last_response)
"""
# Parse attack options
options = parse_options(attack_options or "")
strategy = options.get("strategy", self.DEFAULT_KEY)
if strategy in self.OPTIONS_MAP:
strategy = self.OPTIONS_MAP[strategy]
else:
valid = ", ".join(self.get_available_option_values()[0])
raise ValueError(f"Unknown option value '{strategy}'. Valid options: {valid}")
# Your implementation here...This is the core of every dynamic attack script. It contains the logic for generating and testing payload variations until a success is found or the iteration limit is reached.
-
entry: Dict[str, Any]: The dictionary for the original dataset entry that failed the initial test. You can access all its fields, such astext,payload, andsystem_message. -
target_module: Any: A wrapped instance of the target module. When you calltarget_module.process_input(), it automatically handles the retries and throttling defined in yourspikee testcommand. You do not need to implement this logic yourself. -
call_judge: callable: A function that lets you evaluate if a response is successful. Call it withcall_judge(entry, response)to get aTrueorFalseresult. -
max_iterations: int: The maximum number of iterations / turns your attack loop can run, as specified by--attack-iterations. Your loop must respect this limit. -
attempts_barandbar_lock:tqdmprogress bar objects for updating the UI. For each attempt inside your loop, callwith bar_lock: attempts_bar.update(1). -
attack_options: Optional[str]: A single string passed from the command line via--attack-options(e.g.,"mode=aggressive").
By default, the attack function returns the existing tuple containing four elements:
int: The total number of iterations that were attempted.bool: The final success flag (Trueif any iteration succeeded).Content: The payload of the last attempted iteration.Content: The response from the last attempted iteration.
Use spikee test --attack <name> --attack-return-all-attempts to retain each attempted input and response. Omitting the flag preserves the single representative result. This changes recording, not the search budget, target calls, judge calls, or early stopping.
Existing class-based and function-based attacks that do not accept the new argument still load and run. Spikee inspects the signature, forwards return_all_attempts only when explicitly declared, and supports both attack_option and attack_options. If tracing is requested from an unsupported module, the CLI warns and retains its representative result; it cannot reconstruct discarded history.
Each supporting attack declares return_all_attempts=False, owns a local history list, appends records inside its loop, and chooses what to return at every exit. Spikee does not intercept target or judge calls to construct the history.
from spikee.utilities.hinting import AttackAttempt
def attack(entry, target_module, call_judge, max_iterations,
attempts_bar=None, bar_lock=None, attack_options=None,
return_all_attempts=False):
history = []
last_input, last_response = "", ""
for count in range(1, max_iterations + 1):
last_input = f"{entry['content']} — variant {count}"
error = None
try:
last_response, _ = target_module.process_input(last_input)
success = call_judge(entry, last_response)
except Exception as exc:
last_response, success, error = str(exc), False, str(exc)
if return_all_attempts:
history.append(AttackAttempt(last_input, last_response, success, error=error))
if attempts_bar is not None:
with bar_lock:
attempts_bar.update(1)
if success:
if return_all_attempts:
return history
return count, True, last_input, last_response
if return_all_attempts:
return history or [AttackAttempt(last_input, last_response, False, attempts=0)]
return max_iterations, False, last_input, last_responseFor multi-turn calls, use spikee_session_id and backtrack as usual. The attack explicitly stores the actual turn input and the conversation snapshot at that point. Use the existing standardised_input_return to serialize conversation graphs, or deep-copy mutable lists. Keep abandoned attempts when backtracking. A turn not passed to the judge has success=None (unjudged); do not add judge calls just to populate the history.
AttackAttempt fields are input, response, success (True, False, or None), attempts (non-negative integer, default 1), and optional error, response_time, guardrail, and guardrail_categories. Input can be Content or the existing standardized dictionary with input, conversation, and objective. Copy mutable conversation state when appending records. Return a nonempty list; if no iteration was attempted, return a zero-attempt diagnostic record. Return collected records on handled failures rather than discarding them. Do not include both per-attempt records and an aggregate summary in the list.
Counts are additive: a row representing one attack iteration has attempts=1, not the cumulative loop index. For 20 iterations, the default tuple reports 20; the expanded list has 20 records each reporting 1. An iteration that fails during prompt generation can still consume the attack's iteration budget; record that error explicitly and use the same counting convention in both return formats. These counts are not a transport-request audit. The runner assigns the separate attack_attempt ordinal and keeps it unique across repeated invocations under --attempts. Results, including multimodal content, use the normal result serialization.
Representative IDs remain <dataset-id>-attack. Expanded IDs are <dataset-id>-attack-1, <dataset-id>-attack-2, etc. Their long_id values append -attempt-<n> to the normal attack long ID. Expanded records include attack_parent_id, attack_parent_long_id, attack_attempt, attack_invocation, and attack_result_format="attempt". An unsupported legacy module under the flag produces attack_result_format="representative" instead.
All-attempts runs set entry_complete=False on all rows belonging to an entry except its last row, which is True. Resume drops incomplete groups and reruns those dataset entries; it does not resume inside an attack. A completed representative result is not retroactively expanded when resuming with the flag: use --no-auto-resume for a fresh traced run. Histories are returned in memory when the attack finishes, so process termination before return can still lose that invocation's history.
See Results Analysis for grouping, extraction, and counting rules. Old and new result formats may coexist in a file.
- Modify the Payload: It is best practice to modify the
entry['payload']and substitute it back intoentry.get('content', entry.get('text')). This focuses the attack on the malicious part while preserving the surrounding document structure. - Respect
max_iterations: Your main loop must not exceed this value. - Update Progress Bar: Call
attempts_bar.update(1)inside thebar_lockfor every iteration to keep the UI in sync. - Handle Early Exit: If an attack succeeds, break the loop. Spikee reconciles the shared progress total with the returned attempt count. Existing modules that adjust their local progress total remain supported; do not modify a shared global progress bar.
- Be Stateless: An attack function should not carry state between different dataset entries. All necessary information should come from the
entrydictionary.
Spikee v5.5 introduced support for multi-turn attacks, which enabled dynamic attacks to perform assess conversational LLM applications against multi-turn prompt injection attacks.
See multi_turn and crescendo attack scripts for examples of multi-turn dynamic attacks.
import uuid
from typing import Callable, Tuple
from spikee.templates.attack import Attack
from spikee.utilities.enums import ModuleTag, Turn
from spikee.templates.standardised_conversation import StandardisedConversation
class SampleMultiTurnAttack(Attack):
def __init__(self):
"""Define multi-turn capabilities for attack."""
# turn_type defines an attack's multi-turn capability, either Turn.SINGLE (Default) or Turn.MULTI
super().__init__(turn_type=Turn.MULTI)
def get_description(self) -> ModuleDescriptionHint:
"""Returns the type and a short description of the attack."""
return [ModuleTag.MULTI], "A brief description of what this attack does."
def get_available_option_values(self) -> ModuleOptionsHint:
"""Return supported attack options; Tuple[options (default is first), llm_required]"""
return [], False
def attack(
self,
entry: dict,
target_module: object,
call_judge: Callable,
max_iterations: int,
attempts_bar=None,
bar_lock=None,
attack_options=None,
) -> AttackResponseHint:
"""
Executes a dynamic multi-turn attack on a given entry.
Args:
entry (dict): The dataset entry. Expected keys: "content" + "content_type" (or legacy "text"), optionally "payload" and "exclude_from_transformations_regex".
target_module (module): The target module (must implement process_input(input_text, system_message)).
call_judge (function): A function that accepts (entry, llm_response) and returns True if the attack is successful.
max_iterations (int): The maximum number of attack iterations to try.
attempts_bar (tqdm, optional): A progress bar to update for each iteration.
attack_options (str, optional): Configuration option like "strategy=aggressive".
Returns:
tuple: (iterations_used:int, success:bool, {"objective": str, "conversation": List[Dict]}, last_response:Content)
"""
session_id = str(uuid.uuid4()) # Unique session ID for multi-turn context
standardised_conversation = StandardisedConversation() # Helper for managing multi-turn conversation messages
last_message_id = standardised_conversation.get_root_id() # Variable to track last message ID in conversation
while not success and standardised_conversation.get_message_total() < max_iterations:
prompt = "..."
prompt_message_id = last_message_id # Store last message ID to aid backtracking
last_message_id = standardised_conversation.add_message(
parent_id=last_message_id,
data={
"role": "user",
"content": prompt,
"session_id": session_id
}
)
standardised_conversation.add_attempt() # Increment attempt counter
response = "..."
standardised_conversation.add_message(
parent_id=last_message_id,
data={
"role": "assistant",
"content": response,
"session_id": session_id
}
)
success = call_judge(entry, response)
if success:
break
if backtrack:
last_message_id = prompt_message_id
content = entry.get("content", entry.get("text"))
return (
standardised_conversation.get_message_total(),
success,
Attack.standardised_input_return(
input=content,
conversation=standardised_conversation, # Optional, for multi-turn attacks
objective=content # Optional, for instructional multi-turn attacks
),
response
)