← Back to Library

Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioning

Authors: Seung hee Choi, MinJu Jeon, Hyunwoo Oh, Jihwan Lee, Dong-Jin Kim

Published: 2026-03-12

arXiv ID: 2603.11460v1

Added to Library: 2026-03-13 03:02 UTC

📄 Abstract

Existing retrieval-augmented approaches for Dense Video Captioning (DVC) often fail to achieve accurate temporal segmentation aligned with true event boundaries, as they rely on heuristic strategies that overlook ground truth event boundaries. The proposed framework, \textbf{STaRC}, overcomes this limitation by supervising frame-level saliency through a highlight detection module. Note that the highlight detection module is trained on binary labels derived directly from DVC ground truth annotations without the need for additional annotation. We also propose to utilize the saliency scores as a unified temporal signal that drives retrieval via saliency-guided segmentation and informs caption generation through explicit Saliency Prompts injected into the decoder. By enforcing saliency-constrained segmentation, our method produces temporally coherent segments that align closely with actual event transitions, leading to more accurate retrieval and contextually grounded caption generation. We conduct comprehensive evaluations on the YouCook2 and ViTT benchmarks, where STaRC achieves state-of-the-art performance across most of the metrics. Our code is available at https://github.com/ermitaju1/STaRC

🔍 Key Points

  • The paper introduces a comprehensive security threat analysis framework, specifically targeting the operational lifecycle of autonomous LLM agents like OpenClaw, structured across five stages: initialization, input, inference, decision, and execution.
  • It identifies critical threats such as indirect prompt injection, memory poisoning, and intent drift, demonstrating their prevalence and impact through detailed case studies.
  • The authors highlight the limitations of existing defense mechanisms, emphasizing their inadequacy against cross-temporal and multi-stage threats typical in autonomous LLM operations.
  • A layered defense architecture is proposed, integrating measures across all lifecycle stages, which is critical for effective mitigation of identified threats.
  • The paper emphasizes the need for holistic security architectures that enforce strict controls at every stage of the agent's lifecycle.

💡 Why This Paper Matters

This paper is vital as it addresses an important gap in the understanding of security vulnerabilities in autonomous LLM agents like OpenClaw. By elaborating on the complex threat landscape and proposing a structured framework for analysis and defense, it provides critical insights for developing more resilient AI systems. The findings underscore the urgent need for robust security measures to handle the sophisticated attacks that such agents may face in real-world applications.

🎯 Why It's Interesting for AI Security Researchers

AI security researchers will find this paper highly relevant because it not only elucidates the specific vulnerabilities present in autonomous LLM agents but also offers a novel framework for thinking about security in this context. The insights into multi-stage attacks and the proposed defense mechanisms are crucial for developing future AI security protocols. Moreover, the emphasis on lifecycle-aware defenses aligns closely with current trends towards comprehensive and proactive security solutions in AI systems.

📚 Read the Full Paper