← Back to Library

RECAP: Reproducing Copyrighted Data from LLMs Training with an Agentic Pipeline

Authors: André V. Duarte, Xuying li, Bin Zeng, Arlindo L. Oliveira, Lei Li, Zhuo Li

Published: 2025-10-29

arXiv ID: 2510.25941v1

Added to Library: 2025-10-31 04:01 UTC

Red Teaming

📄 Abstract

If we cannot inspect the training data of a large language model (LLM), how can we ever know what it has seen? We believe the most compelling evidence arises when the model itself freely reproduces the target content. As such, we propose RECAP, an agentic pipeline designed to elicit and verify memorized training data from LLM outputs. At the heart of RECAP is a feedback-driven loop, where an initial extraction attempt is evaluated by a secondary language model, which compares the output against a reference passage and identifies discrepancies. These are then translated into minimal correction hints, which are fed back into the target model to guide subsequent generations. In addition, to address alignment-induced refusals, RECAP includes a jailbreaking module that detects and overcomes such barriers. We evaluate RECAP on EchoTrace, a new benchmark spanning over 30 full books, and the results show that RECAP leads to substantial gains over single-iteration approaches. For instance, with GPT-4.1, the average ROUGE-L score for the copyrighted text extraction improved from 0.38 to 0.47 - a nearly 24% increase.

🔍 Key Points

  • Introduction of RECAP, an innovative agentic pipeline that utilizes a feedback-driven loop to extract memorized training data from large language models (LLMs).
  • Development of EchoTrace, a benchmarking dataset that includes various written works to evaluate LLM's memorization abilities, consisting of 30 books and research papers.
  • Substantial performance improvement in copyrighted content extraction, with average ROUGE-L scores rising from 0.38 to 0.47 using RECAP compared to traditional single-pass approaches.
  • Implementation of jailbreaking methods that effectively reduce alignment-induced response refusals in models, enabling better access to copyrighted content.
  • Thorough evaluation of RECAP indicates minimal contamination from non-training data, strengthening its reliability for assessing LLM memorization of copyrighted material.

💡 Why This Paper Matters

This paper presents significant advancements in the detection of memorization in large language models, particularly regarding copyrighted content. The introduction of RECAP represents a crucial step toward ensuring compliance and ethical use of AI technologies. By allowing researchers and companies to understand and verify what data models have been exposed to, it lays down a foundational framework for responsible AI development and regulatory adherence.

🎯 Why It's Interesting for AI Security Researchers

This paper is particularly relevant for AI security researchers as it delves into the issues surrounding data leakage and copyright violations by LLMs. Understanding how these models memorize and replicate copyrighted content is crucial for developing strategies to mitigate risks associated with model outputs. Additionally, the methodologies proposed in RECAP could inform security frameworks and protocols for ensuring compliance with copyright laws in AI deployments.

📚 Read the Full Paper

📄 View on arXiv 📎 Download PDF