Affect-Aware Machine Learning Models for Deception Detection

This research presents a novel analysis of the potential for including dimensional representations of facial affect—specifically valence and arousal—in machine learning models for detecting deception. Using real-world courtroom video data, the study demonstrates that affect-aware models achieve substantially higher accuracy than previous approaches, with multimodal fusion methods reaching 91% AUC.

The study employed supervised and unsupervised machine learning approaches including Support Vector Machines and Deep Belief Networks trained on facial valence, arousal, and multimodal features extracted from real-world courtroom testimony videos (approximately 121 videos from the Real-life Trial Dataset).

Background & Context

Automated deception detection systems have significant potential to enhance societal well-being across health, social work, and legal domains, yet traditional approaches to this challenge have fallen short. Existing computational approaches for detecting deception have not leveraged dimensional representations of affect, specifically valence and arousal, expressed during communication—a notable gap given extensive psychological theories linking emotional states to deceptive behavior.

This research emerged as part of Mathur's undergraduate thesis work in Computer Science at the University of Southern California, representing a novel integration of affective computing with deception detection. The work utilized a video dataset of people communicating truthfully or deceptively in real-world, high-stakes courtroom situations, addressing one of the field's most pressing challenges: detecting deception in authentic, consequential contexts rather than laboratory simulations.

The innovation lies in applying dimensional models of emotion—measuring facial valence (pleasantness) and arousal (activation)—rather than discrete emotion categories or traditional behavioral cues alone. This approach bridges psychology, computer vision, and machine learning to create more robust deception detection systems.

Research Design & Methodology

The study employed the Real-life Trial Deception Detection Dataset, consisting of approximately 60 truthful videos and 61 deceptive videos (roughly 28 seconds per video) of people speaking in real-world courtrooms. This dataset represents the current benchmark for multimodal high-stakes deception detection in videos, with ground truth labels verified by police investigations.

The research leveraged a state-of-the-art deep neural network trained on the Aff-Wild database to extract continuous representations of facial valence and facial arousal from speakers. This automated emotion recognition system enabled scalable extraction of affect features from video without manual annotation.

The methodological approach included multiple experimental paradigms:

• Unimodal Support Vector Machines (SVM) and SVM-based multimodal fusion methods to identify effective features, modalities, and modeling approaches
• Affect-aware unsupervised Deep Belief Networks (DBN) trained on facial valence, facial arousal, audio, and visual features
• A novel DBN training procedure using facial affect as an aligner of audio-visual representations
• Integration of visual, vocal, and verbal modalities alongside facial affect measurements

Results & Key Findings

Unimodal models trained on facial affect achieved an AUC of 80%, demonstrating that affect features alone provide substantial discriminative power for deception detection. Even more impressive, facial affect contributed toward the highest-performing multimodal approach (adaptive boosting) that achieved an AUC of 91% when tested on speakers who were not part of training sets.

Key performance outcomes included:

• 91% AUC for multimodal AdaBoost fusion incorporating facial affect—higher than existing automated machine learning approaches that used interpretable visual, vocal, and verbal features but did not use facial affect
• 80% AUC for unsupervised approaches using facial valence and visual features—outperforming human ability and performing comparably to fully-supervised models
• Comparison to previous best automated approach achieving AUC of 87.7% using head, face, eye movements, MFCC coefficients, and GloVe word embeddings

Across all videos, deceptive and truthful speakers exhibited significant differences in facial valence and facial arousal, contributing computational support to existing psychological theories on affect and deception. This empirical validation strengthens the theoretical foundation linking emotional expression to deceptive behavior.

Discussion & Significance

This research makes multiple important contributions to deception detection science. First, it demonstrates that dimensional representations of affect—valence and arousal—provide powerful discriminative features for identifying deception in high-stakes, real-world contexts. The substantial performance gains over prior methods validate the importance of incorporating emotional dimensions into automated systems.

Second, the development of unsupervised approaches achieving 80% AUC while outperforming human ability addresses a critical practical challenge. Labeled datasets to train supervised deception detection models can rarely be collected for real-world, high-stakes contexts, making unsupervised methods essential for real-world deployment.

The demonstrated importance of facial affect in these models informs and motivates the future development of automated, affect-aware machine learning approaches for modeling and detecting deception and other social behaviors in-the-wild. This work opens new avenues for integrating affective computing with behavioral analysis across domains beyond deception detection.

Limitations & Considerations

Deep learning approaches for deception detection have been considered inadvisable in this dataset due to its small size, which constrained some modeling choices. The Real-life Trial Dataset, while representing authentic high-stakes deception, contains approximately 121 videos from 56 unique individuals—a relatively small sample that limits the complexity of models that can be reliably trained and tested.

The dataset's courtroom context, while ecologically valid, may limit generalizability to other deception contexts. Courtroom testimony occurs under unique psychological and social pressures that may not fully represent deception in interrogations, security screenings, or interpersonal interactions. Additionally, automated facial affect recognition systems trained on general emotion databases may have variable performance across different demographic groups and cultural contexts, though this was not explicitly evaluated in the study.

Practical Applications

This research provides a foundation for developing practical deception detection systems that could assist in forensic interviews, security screenings, and investigative contexts. The multimodal fusion approach achieving 91% AUC suggests that operational systems combining facial affect with other behavioral modalities could provide valuable decision support for trained professionals.

The unsupervised methods are particularly promising for deployment in contexts where labeled training data is unavailable or unethical to collect. These approaches could enable organizations to develop customized deception detection systems for their specific contexts without requiring extensive labeled datasets. However, any deployment should recognize that automated systems are decision-support tools rather than definitive lie detectors, and human judgment remains essential in high-stakes contexts.