← Back to Library

On Surjectivity of Neural Networks: Can you elicit any behavior from your model?

Authors: Haozhe Jiang, Nika Haghtalab

Published: 2025-08-26

arXiv ID: 2508.19445v1

Added to Library: 2025-08-28 04:02 UTC

Red Teaming

📄 Abstract

Given a trained neural network, can any specified output be generated by some input? Equivalently, does the network correspond to a function that is surjective? In generative models, surjectivity implies that any output, including harmful or undesirable content, can in principle be generated by the networks, raising concerns about model safety and jailbreak vulnerabilities. In this paper, we prove that many fundamental building blocks of modern neural architectures, such as networks with pre-layer normalization and linear-attention modules, are almost always surjective. As corollaries, widely used generative frameworks, including GPT-style transformers and diffusion models with deterministic ODE solvers, admit inverse mappings for arbitrary outputs. By studying surjectivity of these modern and commonly used neural architectures, we contribute a formalism that sheds light on their unavoidable vulnerability to a broad class of adversarial attacks.

🔍 Key Points

  • The paper proves many modern neural architectures, such as those employing pre-layer normalization and linear attention modules, are almost always surjective, allowing them to generate virtually any output from some input.
  • It establishes that widely used models like GPT-style transformers and certain diffusion models admit inverse mappings for arbitrary outputs, raising potential safety risks.
  • By applying differential topology, the authors provide a robust mathematical framework to analyze the input-output behavior of neural networks, contributing necessary tools for future research in this area.
  • The findings highlight vulnerabilities in model safety, showing that trained models can theoretically produce harmful or undesirable content, which is crucial for understanding and mitigating adversarial attacks.
  • The authors distinguish between theoretical surjectivity and practical concerns, noting that while models are almost always surjective, efficiently finding suitable inputs for harmful outputs remains a computational challenge.

💡 Why This Paper Matters

This paper is relevant and important because it addresses a fundamental aspect of neural networks – their surjectivity – and its implications for model safety. By demonstrating that modern architectures can theoretically reproduce any output, including harmful ones, it sheds light on the vulnerabilities of generative models. The findings encourage further scrutiny of AI safety measures and highlight the need for robust mitigation strategies against potential adversarial attacks.

🎯 Why It's Interesting for AI Security Researchers

AI security researchers will find this paper of significant interest as it tackles the inherent risks associated with the surjectivity of neural networks. The insights gained regarding the ability of models to generate harmful outputs can inform better safety protocols, monitoring systems, and design practices for AI systems. Understanding these vulnerabilities is essential for developing countermeasures against adversarial attacks and ensuring responsible deployment of generative models in sensitive applications.

📚 Read the Full Paper

📄 View on arXiv 📎 Download PDF