← Back to Library

Generalization Limits of Reinforcement Learning Alignment

Authors: Haruhi Shida, Koo Imai, Keigo Kansa

Published: 2026-04-03

arXiv ID: 2604.02652v1

Added to Library: 2026-04-06 02:05 UTC

Red Teaming

📄 Abstract

The safety of large language models (LLMs) relies on alignment techniques such as reinforcement learning from human feedback (RLHF). However, recent theoretical analyses suggest that reinforcement learning-based training does not acquire new capabilities but merely redistributes the utilization probabilities of existing ones. In this study, we propose ``compound jailbreaks'' targeting OpenAI gpt-oss-20b, which exploit the generalization failures of alignment. This approach combines multiple attack techniques -- each individually defended against -- to saturate the instruction hierarchy maintenance process. Our evaluation shows that the attack success rate (ASR) increased from 14.3\% with individual methods to 71.4\% with the combined approach. These results provide empirical evidence for the hypothesis that safety training does not generalize as broadly as model capabilities, highlighting the need for multifaceted safety evaluations using compound attack scenarios.

🔍 Key Points

  • Empirical validation of the theoretical limits of RLHF, suggesting it merely redistributes existing capabilities rather than fostering new ones.
  • Introduction of 'compound jailbreaks' which effectively exploit structural vulnerabilities in safety mechanisms by combining multiple attack techniques.
  • Demonstration of significant increase in attack success rate (ASR) from 14.3% to 71.4% through the use of combined attack strategies, highlighting the inadequacy of current individual defenses.
  • Identification of instruction hierarchy breaches and vulnerabilities in tool misuse due to 'contextual inertia' in agent scenarios.
  • Proposals for enhanced safety measures revealing the need for structural changes rather than solely reliance on model-level adjustments.

💡 Why This Paper Matters

This paper is highly relevant in the context of AI safety as it provides empirical evidence regarding the vulnerabilities in alignment techniques used in large language models. By highlighting the ineffectiveness of existing safety measures against compound attacks, the authors underscore the urgent need for more comprehensive and multifaceted safety evaluations in AI systems. The findings contribute to the discourse on improving LLM safety, making it a critical read for researchers in the field.

🎯 Why It's Interesting for AI Security Researchers

AI security researchers will find this paper particularly important as it addresses the limitations of reinforcement learning from human feedback (RLHF) in alignment processes, which is a cornerstone of current AI safety methodologies. By exposing how compound attack strategies can bypass traditional defenses, it calls attention to potential oversight in AI safety mechanisms, prompting future research into more robust techniques. Additionally, the empirical results provide a clear metric for assessing the vulnerabilities of language models, establishing a framework for testing and improving security measures.

📚 Read the Full Paper

📄 View on arXiv 📎 Download PDF