Automated Data Enrichment using Confidence-Aware Fine-Grained Debate among Open-Source LLMs for Mental Health and Online Safety
Safety
📄 Abstract
Real-world indicators are important for improving natural language processing (NLP) tasks such as life events for mental health analysis and risky behaviour for online safety, yet labelling such information in NLP training datasets is often costly and/or difficult given the dynamic nature of such events. This paper compares several LLM-based data enrichment methods and introduces a novel Confidence-Aware Fine-Grained Debate (CFD) framework in which multiple LLM agents simulate human annotators and exchange fine-grained evidence to reach consensus. We describe two new expert-annotated datasets, a mental health Reddit wellbeing dataset and an online safety Facebook sharenting risk dataset. Our CFD framework achieves the most robust data enrichment performance compared to a range of baselines and we show that this type of data enrichment consistently improves downstream tasks. Enriched features incorporated via debate transcripts yield the largest gains, outperforming the non-enriched baseline by 10.1% for the online safety task.
🔍 Key Points
- Introduction of the Confidence-Aware Fine-Grained Debate (CFD) framework that utilizes multiple LLM agents to enhance data enrichment by simulating human annotators.
- Development of two expert-annotated datasets focused on mental health and online safety, specifically the emotional wellbeing of Reddit users and sharenting risks associated with Facebook posts.
- The CFD framework significantly outperforms baseline models, achieving a 10.1% performance improvement in a specific online safety task, demonstrating its efficacy for real-world applications.
- Utilization of fine-grained confidence metrics to enhance annotation quality during debates among language models, leading to better consensus outcomes on multi-label predictions.
- Release of the annotated datasets as open source, facilitating further research in the NLP domain, particularly concerning mental health and online safety issues.
💡 Why This Paper Matters
This paper is relevant and important as it addresses a critical issue in NLP concerning the difficulty and cost of annotating real-world datasets, particularly those related to sensitive topics like mental health and online safety. By leveraging LLMs and introducing novel methodologies for data enrichment, the findings can greatly enhance the development of more robust and context-aware NLP solutions.
🎯 Why It's Interesting for AI Security Researchers
For AI security researchers, this paper is of interest because it explores how enriched data can improve model performance on tasks related to identifying risks in online behavior and mental health conditions, which are crucial for creating safer online environments. The insights into multi-agent debates and confidence calibration provide useful frameworks for tackling challenges in data integrity and algorithmic decision-making under uncertainty.