← Back to Library

SHERLOCK: Towards Dynamic Knowledge Adaptation in LLM-enhanced E-commerce Risk Management

Authors: Nan Lu, Yurong Hu, Jiaquan Fang, Yan Liu, Rui Dong, Yiming Wang, Rui Lin, Shaoyi Xu

Published: 2025-10-10

arXiv ID: 2510.08948v1

Added to Library: 2025-10-13 12:02 UTC

Risk & Governance

📄 Abstract

The growth of the e-commerce industry has intensified the adversarial dynamics between shadow economy actors and risk management teams. Companies often conduct risk investigations into suspicious cases to identify emerging fraud patterns, thereby enhancing both preemptive risk prevention and post-hoc governance. However, the sheer volume of case analyses imposes a substantial workload on risk management analysts, as each case requires the integration of long-term expert experience and meticulous scrutiny across multiple risk dimensions. Additionally, individual disparities among analysts hinder the establishment of uniform and high-standard workflows. To address these challenges, we propose the SHERLOCK framework, which leverages the reasoning capabilities of large language models (LLMs) to assist analysts in risk investigations. Our approach consists of three primary components: (1) extracting risk management knowledge from multi-modal data and constructing a domain knowledge base (KB), (2) building an intelligent platform guided by the data flywheel paradigm that integrates daily operations, expert annotations, and model evaluations, with iteratively fine-tuning for preference alignment, and (3) introducing a Reflect & Refine (R&R) module that collaborates with the domain KB to establish a rapid response mechanism for evolving risk patterns. Experiments conducted on the real-world transaction dataset from JD.com demonstrate that our method significantly improves the precision of both factual alignment and risk localization within the LLM analysis results. Deployment of the SHERLOCK-based LLM system on JD.com has substantially enhanced the efficiency of case investigation workflows for risk managers.

🔍 Key Points

  • Introduction of the SHERLOCK framework that integrates LLMs with risk management in e-commerce, addressing adversarial dynamics.
  • Development of a domain knowledge base (KB) for effective knowledge extraction from multi-modal data to enhance LLM capabilities.
  • Implementation of a Data Flywheel mechanism for continuous learning and optimization of risk assessment models.
  • Introduction of the Reflect & Refine (R&R) module to improve the accuracy of LLM outputs by integrating expert knowledge and adapting to evolving risk patterns.
  • Empirical validation through real-world experiments demonstrating increased precision in risk detection and significant reductions in investigation times.

💡 Why This Paper Matters

The SHERLOCK framework signifies a substantial advancement in integrating AI with e-commerce risk management, offering a robust solution that enhances both efficiency and accuracy in detecting fraudulent activities. Its successful deployment in a real-world environment underscores its practical applicability and potential for broader adoption across various industries.

🎯 Why It's Interesting for AI Security Researchers

This paper is particularly relevant to AI security researchers as it explores the intersection of large language models and risk management, providing insights into developing adaptive, efficient systems capable of handling security threats in dynamic environments. The methodologies introduced, such as the data flywheel and the R&R module, present innovative approaches to improving AI interpretability and decision-making in security contexts.

📚 Read the Full Paper