AFFAKT: A Hierarchical Optimal Transport Based Method for Affective Facial Knowledge Transfer in Video Deception Detection

The scarcity of high-quality large-scale labeled datasets poses a huge challenge for employing deep learning models in video deception detection. To address this issue, inspired by the psychological theory on the relation between deception and expressions, we propose a novel method called AFFAKT in this paper, which enhances the classification performance by transferring useful and correlated knowledge from a large facial expression dataset. Two key challenges in knowledge transfer arise: 1) how much knowledge of facial expression data should be transferred and 2) how to effectively leverage transferred knowledge for the deception classification model during inference. Specifically, the optimal relation mapping between facial expression classes and deception samples is firstly quantified using proposed H-OTKT module and then transfers knowledge from the facial expression dataset to deception samples. Moreover, a correlation prototype within another proposed module SRKB is well designed to retain the invariant correlations between facial expression classes and deception classes through momentum updating. During inference, the transferred knowledge is fine-tuned with the correlation prototype using a sample-specific re-weighting strategy. Experimental results on two deception detection datasets demonstrate the superior performance of our proposed method. The interpretability study reveals high associations between deception and negative affections, which coincides with the theory in psychology.

Background & Context

Video-based deception detection represents a critical frontier in forensic psychology, security screening, and credibility assessment. However, the scarcity of high-quality large-scale labeled datasets poses a huge challenge for employing deep learning models in video deception detection. The Real-Life Trial (RTL) dataset consists of only 121 videos with an average length of 28 seconds, while DOLOS includes 1,675 video clips featuring 213 subjects—both far smaller than datasets available for facial expression recognition.

This 2025 study introduces AFFAKT (Affective Facial Knowledge Transfer), a novel approach grounded in psychological theory. Motivated by insights from psychological theory and previous research that certain facial movements and expressions are associated with deception, the researchers developed a method to transfer knowledge from large-scale facial expression datasets to improve deception detection with limited training data. This addresses a fundamental challenge: how to leverage the relationship between emotional expressions and deceptive behavior to enhance classifier performance.

The work fills a critical gap by bridging affective computing and deception detection. Traditional deep learning methods fail when confronted with small deception datasets, while transfer learning approaches have not adequately quantified how much knowledge should be transferred or how to effectively utilize it during inference.

Research Design & Methodology

The objective of AFFAKT is to transfer and leverage the knowledge properly from a large-scale video facial expression recognition (VFER) dataset (source domain) to enhance video deception detection (target domain) model. The researchers used three large VFER datasets as source domains: DFEW containing 16,372 samples, FERV39K with 38,935 samples, and MAFW with 10,045 samples, with 7, 7, and 11 expression categories respectively.

The architecture comprises four key modules working in sequence:

• H-OTKT (Hierarchical Optimal Transport Knowledge Transfer): Quantifies the potential correlation between the class of facial expressions and deception samples via hierarchical optimal transport, determining how much knowledge from different categories is transferred to each sample
• SRKB (Sample-specific Re-weighting Knowledge Bank): Learns the invariant class-level relation between VFER and video deception detection datasets via momentum updated correlation prototype during training, and fine-tunes correlation mapping during testing using a sample-specific re-weighting strategy
• Pre-trained encoders: VideoMAE and W2V2 were used as backbones to encode visual and audio data separately, with MAE-DFER providing the most compact and separable embedding space for VFER features
• Multi-modal fusion: Visual, audio, and combined modalities were evaluated

Comparisons were made with several deception detection methods on RTL and DOLOS in visual, audio and fused modalities to validate AFFAKT. The experimental framework was implemented using Python 3.11 with PyTorch 2.0.0 on RTX 3090 24GB and A800 80GB GPUs.

Results & Key Findings

AFFAKT demonstrated superior performance across both benchmark datasets. Experimental results on two video deception detection datasets demonstrate the superior performance of the proposed method, with particularly strong gains on the smaller RTL dataset. Compared to larger DOLOS dataset, AFFAKT outperforms deep learning based methods on smaller RLT more significantly, indicating that AFFAKT is able to show better detection performance on datasets with fewer samples.

Key performance metrics included:

• Deep learning based models achieved better results compared to machine learning based models, indicating that deep learning model exhibits better feature extraction ability
• Extra facial expression knowledge proved helpful to deception detection, with AFFAKT achieving higher classification accuracy than transfer learning based methods, demonstrating that better knowledge of facial expression data could be transferred and leveraged
• AFFAKT achieved the best performance on both deception datasets in both audio and fused modalities, with highest accuracies and F1 scores achieved when source domains MAFW and FERV39K were used

Interpretability analysis revealed psychologically valid patterns. Sad was significantly related to deceptive behavior for both RTL and DOLOS, while happy was remarkably correlated to truthful behavior, supported by psychological theory that liars will be less positive and pleasant than truth tellers. Additionally, fear was more correlated with deceptive than truthful on both datasets, consistent with findings that deceivers in high-stakes situations are likely to associate with fear.

Discussion & Significance

This research makes several important contributions to deception detection methodology. The hierarchical optimal transport framework provides a principled mathematical approach to quantifying cross-domain knowledge transfer—a problem that has plagued transfer learning applications in forensic contexts. Unlike naive transfer learning that treats all source knowledge equally, AFFAKT adaptively weights facial expression categories based on their relevance to deception.

The method automatically establishes proper relations between classes of facial expressions and deception, which helps in leveraging transferred knowledge from facial affective datasets, with interpretability studies revealing high associations between deception and negative affections that coincide with psychological theory. This alignment with established psychological findings (e.g., DePaulo et al., 2003) validates the computational approach and suggests the model has learned meaningful rather than spurious patterns.

The work demonstrates that AI-based deception detection can benefit from indirect training data when direct deception samples are scarce. This has practical implications for jurisdictions or agencies that lack large proprietary deception databases but can access public facial expression datasets. The modular architecture also allows for updating source datasets as larger VFER corpora become available, potentially improving performance without requiring additional deception data collection.

Limitations & Considerations

Several methodological limitations warrant consideration. First, when certain source datasets like FERV39K or MAFW were used, accuracies increased for RTL data with visual modality but decreased slightly for others, attributed to different representative abilities in different pre-trained MAE-DFER encoders. This suggests performance is sensitive to source dataset selection and encoder quality.

The study evaluated only video-based behavioral cues, excluding the multimodal physiological measures (cardiovascular, electrodermal) traditionally used in polygraph examinations. While the RTL baseline achieved classification accuracies in the range of 60-75% using multimodal features, these accuracy levels fall well below the standards required for forensic applications. The datasets used—court trial videos and gameshow clips—may not generalize to controlled interrogation settings where traditional polygraphy is applied.

Furthermore, the reliance on pre-trained encoders introduces potential biases from the original training data. The interpretability analysis, while psychologically grounded, represents correlational rather than causal relationships between expressions and deception.

Practical Applications

For polygraph professionals, this research demonstrates both opportunities and boundaries for AI-augmented credibility assessment. The method could potentially serve as a pre-screening tool in security contexts where video data is available but physiological monitoring is impractical. The automatic correlation between negative affect (sadness, fear) and deception aligns with examiner training on behavioral observation, potentially providing objective quantification of visual cues.

However, the technology is not a replacement for comprehensive psychophysiological detection of deception. The accuracy levels, dataset constraints, and absence of physiological measures mean AFFAKT should be viewed as complementary technology rather than a standalone solution. Agencies considering implementation should note the method's particular advantage with small sample sizes—important when building agency-specific models with limited training data. The open-source code availability facilitates reproducibility and adaptation, though deployment would require careful validation on population-specific data and rigorous testing for bias across demographic groups.