← Back to Library

Structured Visual Narratives Undermine Safety Alignment in Multimodal Large Language Models

Authors: Rui Yang Tan, Yujia Hu, Roy Ka-Wei Lee

Published: 2026-03-23

arXiv ID: 2603.21697v1

Added to Library: 2026-03-24 03:02 UTC

Red Teaming

📄 Abstract

Multimodal Large Language Models (MLLMs) extend text-only LLMs with visual reasoning, but also introduce new safety failure modes under visually grounded instructions. We study comic-template jailbreaks that embed harmful goals inside simple three-panel visual narratives and prompt the model to role-play and "complete the comic." Building on JailbreakBench and JailbreakV, we introduce ComicJailbreak, a comic-based jailbreak benchmark with 1,167 attack instances spanning 10 harm categories and 5 task setups. Across 15 state-of-the-art MLLMs (six commercial and nine open-source), comic-based attacks achieve success rates comparable to strong rule-based jailbreaks and substantially outperform plain-text and random-image baselines, with ensemble success rates exceeding 90% on several commercial models. Then, with the existing defense methodologies, we show that these methods are effective against the harmful comics, they will induce a high refusal rate when prompted with benign prompts. Finally, using automatic judging and targeted human evaluation, we show that current safety evaluators can be unreliable on sensitive but non-harmful content. Our findings highlight the need for safety alignment robust to narrative-driven multimodal jailbreaks.

🔍 Key Points

  • Introduction of ComicJailbreak, a comic-template benchmark, consisting of 1,167 instances that manipulate Multimodal Large Language Models (MLLMs) through structured visual narratives, exposing vulnerabilities in model safety.
  • Findings show that comic-template attacks yield attack success rates comparable to traditional rule-based jailbreaks, achieving ensemble success rates exceeding 90% in several cases, highlighting the effectiveness of visual narratives in evading safety measures.
  • Defense mechanisms, while effective in reducing harmful outputs, result in high refusal rates for benign prompts, illustrating the trade-off between model safety and usability that needs to be addressed in future frameworks.
  • Evaluation of automated safety judges revealed significant discrepancies in reliability, particularly on benign prompts, indicating the necessity for human assessment in safety evaluations.
  • The study emphasizes the need for improved safety alignment methodologies that can robustly handle narrative-driven vulnerabilities without overly compromising the model's helpfulness.

💡 Why This Paper Matters

This paper is significant as it sheds light on the novel vulnerabilities introduced by narrative-driven attacks on MLLMs, showcasing the ease with which harmful intentions can be embedded in seemingly innocuous visual contexts. It raises critical awareness about the limitations of existing safety mechanisms and the over-refusal phenomenon that can impair the usability of AI systems.

🎯 Why It's Interesting for AI Security Researchers

The paper's focus on multimodal attacks through structured visual narratives is pertinent for AI security researchers as it uncovers new avenues of exploitation within large language models. The findings urge the community to reconsider current safety protocols, particularly in handling edge cases involving narrative framing that may lead to harmful outputs, thereby advancing the field's understanding of resilience against sophisticated adversarial techniques.

📚 Read the Full Paper

📄 View on arXiv 📎 Download PDF