Advancements and Challenges in Video-Based Deception Detection: A Systematic Literature Review of Datasets, Modalities, and Methods

Video-based deception detection has emerged as a promising field that leverages advances in computer vision, machine learning, and multimodal analysis to capture a wealth of nonverbal cues for identifying deceptive behavior. However, the field faces significant challenges related to dataset development, methodological approaches, and ethical considerations. This systematic literature review (SLR) aims to provide a comprehensive analysis of video-based deception detection research, with five distinct contributions: 1) an unprecedented analysis of 21 datasets, revealing critical gaps and opportunities in data resources; 2) a novel evaluation framework for assessing dataset quality and ecological validity; 3) a systematic comparison of multimodal integration approaches, identifying optimal strategies for combining visual, audio, and textual cues; 4) a critical examination of temporal modeling techniques for capturing the dynamic nature of deceptive behavior; and 5) a roadmap for addressing ethical challenges in deployment. Following the PRISMA guidelines, we reviewed studies published between 2019 and 2024 in major databases, including IEEE Xplore, ACM Digital Library, ScienceDirect, and Springer Link. The review process involved a rigorous two-stage screening, which resulted in the inclusion of 42 primary research papers. Our analysis revealed several key findings: 1) only 52.4% of identified datasets are publicly accessible, highlighting a critical gap in research reproducibility; 2) multimodal approaches consistently outperform unimodal methods, with accuracy improvements of 10-15%; 3) deep learning architectures, particularly LSTM variants and attention mechanisms, demonstrate superior performance in capturing temporal aspects of deception; 4) the Real-Life Trial Dataset emerged as the most frequently used dataset (65% of studies), indicating a preference for high-stakes ecologically valid data; and 5) significant ethical challenges remain unaddressed, particularly regarding privacy, bias, and cross-cultural validity. This review makes several novel contributions to advance the field: 1) provides a comprehensive framework for dataset evaluation and development; 2) identifies optimal strategies for multimodal integration and temporal modeling; 3) presents a structured approach to addressing ethical considerations; and 4) offers a detailed roadmap for future research priorities. These contributions will guide researchers in developing more robust, ethical, and generalizable deception detection systems, while addressing critical gaps in current methodologies and datasets.

Background & Context

Video-based deception detection represents a rapidly evolving frontier in applied artificial intelligence, combining computer vision, machine learning, and multimodal analysis to identify deceptive behavior through nonverbal cues. This field has emerged as promising by leveraging advances in computer vision, machine learning, and multimodal analysis to capture a wealth of nonverbal cues for identifying deceptive behavior. Unlike traditional polygraph examinations that rely on physiological sensors, video-based approaches analyze facial expressions, body language, vocal patterns, and linguistic content from recordings.

Despite significant technological progress, the field faces fundamental challenges that have hindered widespread deployment. Only 52.4% of identified datasets are publicly accessible, highlighting a critical gap in research reproducibility. This lack of data sharing impedes scientific validation and cross-laboratory replication. Additionally, ethical concerns regarding privacy, algorithmic bias, and cross-cultural validity remain largely unaddressed in current research.

This systematic review fills a critical void by providing the first comprehensive evaluation of video-based deception detection datasets, methodologies, and ethical frameworks. Previous reviews have examined deception detection broadly but lacked rigorous assessment of dataset quality, ecological validity, and optimal multimodal integration strategies for video-based approaches specifically.

Research Design & Methodology

Following the PRISMA guidelines, the researchers reviewed studies published between 2019 and 2024 in major databases, including IEEE Xplore, ACM Digital Library, ScienceDirect, and Springer Link. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) protocol ensures rigorous, transparent, and reproducible systematic reviews.

The review process employed a two-stage screening methodology that initially identified hundreds of potentially relevant articles. After removing duplicates and applying strict inclusion criteria focused on video-based deception detection with machine learning or deep learning approaches, 42 primary research papers were included in the final analysis. The researchers systematically extracted data on datasets, algorithms, performance metrics, and methodological approaches.

The analysis framework included five key dimensions:

• Dataset analysis: Comprehensive evaluation of 21 datasets for accessibility, size, ecological validity, and modality coverage
• Multimodal integration: Systematic comparison of unimodal versus multimodal approaches across visual, audio, and textual modalities
• Temporal modeling: Assessment of techniques for capturing dynamic deception patterns over time
• Algorithm performance: Evaluation of machine learning and deep learning architectures
• Ethical considerations: Critical examination of privacy, bias, and cross-cultural validity challenges

Results & Key Findings

The review revealed that multimodal approaches consistently outperform unimodal methods, with accuracy improvements of 10-15%. This finding demonstrates the value of integrating visual facial cues, vocal characteristics, and linguistic content rather than relying on single-modality analysis. Studies combining video, audio and transcripts achieve better results in multimodal classification than in unimodal classification from videos, showcasing the strength of combining multiple types of data.

Key statistical findings include:

• 52.4% dataset accessibility: Less than half of the 21 identified datasets are publicly available, creating reproducibility barriers
• 10-15% accuracy gain: Multimodal fusion consistently outperforms single-modality approaches
• 65% dataset dominance: The Real-Life Trial Dataset emerged as the most frequently used dataset (65% of studies), indicating a preference for high-stakes ecologically valid data
• LSTM superiority: Deep learning architectures, particularly LSTM variants and attention mechanisms, demonstrate superior performance in capturing temporal aspects of deception

The Real-Life Trial Dataset's prominence reflects researchers' preference for ecologically valid, high-stakes scenarios that better approximate real-world deception detection applications. This dataset contains courtroom testimony videos where individuals face genuine consequences for deception, unlike laboratory settings with minimal stakes.

Temporal modeling emerged as critical for deception detection success. LSTM networks and attention mechanisms excel at capturing the dynamic evolution of deceptive behavior across time, recognizing that deception manifests through temporal patterns in micro-expressions, speech hesitations, and gesture sequences rather than static features alone.

Discussion & Significance

This review establishes that effective video-based deception detection requires multimodal integration and temporal modeling. The 10-15% accuracy improvement from multimodal fusion represents a substantial gain that could differentiate between operational utility and inadequate performance in real-world applications. This finding aligns with psychological research showing that deception manifests across multiple behavioral channels simultaneously.

The dataset accessibility crisis poses serious implications for scientific progress. With less than half of datasets publicly available, independent validation of reported results becomes impossible, potentially leading to an irreproducibility crisis similar to other AI domains. The field would benefit from coordinated efforts to create standardized, freely accessible benchmark datasets with diverse populations and scenarios.

The dominance of the Real-Life Trial Dataset signals appropriate prioritization of ecological validity. However, this reliance on a single dataset raises concerns about generalizability. Algorithms optimized for courtroom testimony may not transfer to other high-stakes contexts like security screenings, employment interviews, or investigative interrogations. The field needs diverse datasets spanning multiple deception scenarios, cultural contexts, and demographic groups.

The superior performance of LSTM variants and attention mechanisms validates theoretical models emphasizing deception's temporal dynamics. These architectures can learn complex sequential dependencies, capturing how truthful and deceptive narratives unfold differently over time through subtle variations in behavioral timing and sequencing.

Limitations & Considerations

The review identifies several critical limitations in current video-based deception detection research. Significant ethical challenges remain unaddressed, particularly regarding privacy, bias, and cross-cultural validity. Algorithmic bias poses particular concern, as training datasets predominantly feature Western populations, potentially encoding culture-specific deception cues that fail to generalize across cultural contexts.

Additional methodological limitations include:

• Limited ecological diversity: Heavy reliance on a single dataset (Real-Life Trial) may not represent the full spectrum of deception scenarios
• Cross-domain generalization: Unclear whether models trained on courtroom testimony transfer to other contexts
• Baseline comparisons: Inconsistent reporting of human expert performance makes it difficult to assess whether AI approaches offer practical advantages
• Temporal resolution: Optimal time windows for analysis remain understudied, with unclear guidance on analyzing micro-expressions versus extended narrative patterns

The 2019-2024 timeframe, while capturing recent advances, may have excluded foundational earlier work. The focus on published research may also introduce publication bias, as negative results often go unreported.

Practical Applications

This systematic review provides actionable guidance for practitioners developing video-based deception detection systems. The consistent 10-15% multimodal advantage strongly supports investment in integrated systems combining visual, acoustic, and linguistic analysis rather than single-channel approaches. Practitioners should prioritize LSTM-based architectures with attention mechanisms for temporal modeling.

For operational deployments, the ethical challenges identified—privacy violations, algorithmic bias, and cross-cultural invalidity—demand immediate attention. Developers must implement rigorous fairness testing across demographic groups and cultural contexts before deployment. Transparency about system limitations and human oversight requirements remains essential, particularly given that even the best systems likely fall short of perfect accuracy. The review's findings caution against treating video-based deception detection as definitive evidence, suggesting instead its use as one investigative tool among many, always subject to human judgment and corroborating evidence.