← Back to Library

From Transcripts to AI Agents: Knowledge Extraction, RAG Integration, and Robust Evaluation of Conversational AI Assistants

Authors: Krittin Pachtrachai, Petmongkon Pornpichitsuwan, Wachiravit Modecrua, Touchapon Kraisingkorn

Published: 2026-01-26

arXiv ID: 2602.15859v1

Added to Library: 2026-02-19 03:01 UTC

Red Teaming

📄 Abstract

Building reliable conversational AI assistants for customer-facing industries remains challenging due to noisy conversational data, fragmented knowledge, and the requirement for accurate human hand-off - particularly in domains that depend heavily on real-time information. This paper presents an end-to-end framework for constructing and evaluating a conversational AI assistant directly from historical call transcripts. Incoming transcripts are first graded using a simplified adaptation of the PIPA framework, focusing on observation alignment and appropriate response behavior, and are filtered to retain only high-quality interactions exhibiting coherent flow and effective human agent responses. Structured knowledge is then extracted from curated transcripts using large language models (LLMs) and deployed as the sole grounding source in a Retrieval-Augmented Generation (RAG) pipeline. Assistant behavior is governed through systematic prompt tuning, progressing from monolithic prompts to lean, modular, and governed designs that ensure consistency, safety, and controllable execution. Evaluation is conducted using a transcript-grounded user simulator, enabling quantitative measurement of call coverage, factual accuracy, and human escalation behavior. Additional red teaming assesses robustness against prompt injection, out-of-scope, and out-of-context attacks. Experiments are conducted in the Real Estate and Specialist Recruitment domains, which are intentionally challenging and currently suboptimal for automation due to their reliance on real-time data. Despite these constraints, the assistant autonomously handles approximately 30 percents of calls, achieves near-perfect factual accuracy and rejection behavior, and demonstrates strong robustness under adversarial testing.

🔍 Key Points

  • Proposes an end-to-end framework for constructing conversational AI assistants using historical call transcripts, which addresses challenges such as noisy conversational data and accurate human escalation.
  • Introduces a systematic grading and filtering process utilizing the PIPA framework, enhancing the quality of transcripts used in training AI systems.
  • Employs a Retrieval-Augmented Generation (RAG) pipeline to ground AI responses in extracted structured knowledge from transcripts, improving the factual accuracy of interactions.
  • Demonstrates effective prompt tuning through iterative design, enabling modular orchestration of AI responses, which is essential for robust performance across different domains.
  • Evaluates the framework's performance against adversarial testing, showcasing strong resilience to various manipulation attempts, which is crucial for reliable deployment in sensitive environments.

💡 Why This Paper Matters

This paper is relevant as it provides a comprehensive solution to the pressing challenges faced in the deployment of conversational AI in real-world applications. By focusing on transcript-driven development and rigorous evaluation methodologies, it paves the way for reliable AI systems that can assist in customer service settings while ensuring safety and operational efficiency.

🎯 Why It's Interesting for AI Security Researchers

This paper is of significant interest to AI security researchers due to its detailed examination of adversarial testing methods like red teaming, which assess the robustness of conversational AI against various manipulation strategies. The exploration of potential weaknesses in AI prompt structures and the strategies to mitigate them contributes valuable insights into the security of AI deployments, making it essential reading for professionals in the field.

📚 Read the Full Paper