← Back to Library

Executable Governance for AI: Translating Policies into Rules Using LLMs

Authors: Gautam Varma Datla, Anudeep Vurity, Tejaswani Dash, Tazeem Ahmad, Mohd Adnan, Saima Rafi

Published: 2025-12-04

arXiv ID: 2512.04408v1

Added to Library: 2025-12-05 03:05 UTC

Risk & Governance

📄 Abstract

AI policy guidance is predominantly written as prose, which practitioners must first convert into executable rules before frameworks can evaluate or enforce them. This manual step is slow, error-prone, difficult to scale, and often delays the use of safeguards in real-world deployments. To address this gap, we present Policy-to-Tests (P2T), a framework that converts natural-language policy documents into normalized, machine-readable rules. The framework comprises a pipeline and a compact domain-specific language (DSL) that encodes hazards, scope, conditions, exceptions, and required evidence, yielding a canonical representation of extracted rules. To test the framework beyond a single policy, we apply it across general frameworks, sector guidance, and enterprise standards, extracting obligation-bearing clauses and converting them into executable rules. These AI-generated rules closely match strong human baselines on span-level and rule-level metrics, with robust inter-annotator agreement on the gold set. To evaluate downstream behavioral and safety impact, we add HIPAA-derived safeguards to a generative agent and compare it with an otherwise identical agent without guardrails. An LLM-based judge, aligned with gold-standard criteria, measures violation rates and robustness to obfuscated and compositional prompts. Detailed results are provided in the appendix. We release the codebase, DSL, prompts, and rule sets as open-source resources to enable reproducible evaluation.

🔍 Key Points

  • Introduction of Policy-to-Tests (P2T), a framework that automates the translation of natural-language AI policies into executable, machine-readable rules, significantly reducing manual effort and errors.
  • The framework employs a domain-specific language (DSL) and a pipeline that includes deterministic checks and LLM extraction, yielding high-quality rules that closely match human baselines.
  • Empirical evaluation shows that AI-generated rules improve compliance under real-world conditions, as demonstrated by a case study implementing HIPAA-derived guardrails in an AI agent, resulting in reduced violation rates.
  • Comprehensive evaluation metrics and methods are utilized, including span-level and rule-level metrics, human benchmarking, and detailed annotation agreements, ensuring the robustness of generated rules.

💡 Why This Paper Matters

This paper presents a significant advancement in the operationalization of AI governance by providing a reliable method to convert regulatory prose into actionable rules. By automating this process, the authors address critical challenges of scalability and error-proneness, which are essential for timely and effective AI governance in rapidly evolving technologies.

🎯 Why It's Interesting for AI Security Researchers

AI security researchers will find this paper of interest due to its focus on translating AI governance policies into enforceable and verifiable rules. The innovative methodology of using LLMs in rule extraction and the demonstrated improvements in AI agent behavior under guardrails significantly contribute to the field of safe AI deployment. Understanding how to operationalize regulations efficiently directly impacts the development of security measures that protect against the misuse of AI systems.

📚 Read the Full Paper