Detecting Deepfakes Through Emotion?: Facial Expression and Emotional Contagion as Dual Indicators of Deepfake Credibility

ABSTRACT
This study examines the differences in emotional expressions between face‐swap deepfake videos featuring real human subjects and their authentic counterparts and investigates how these discrepancies influence viewers' emotional reactions and perceptions of video credibility. Using a two‐step approach, we first applied computer‐based facial expression analysis to compare emotional displays between deepfake and authentic videos. Next, guided by emotional contagion and emotion‐as‐information frameworks, we conducted an audience response analysis to assess how displayed emotions in the two types of videos transfer to viewers' emotional experiences and their subsequent assessments of the videos. Results indicate that deepfakes generally exhibit lower overall and negative emotions compared to authentic counterparts. Notably, audiences' reduced emotional responses to deepfakes were associated with higher perceived credibility. These findings underscore the importance of emotion‐based signals for detecting fabricated videos and highlight the relationship between viewers' emotional responses and their perceived trust in AI‐generated content.

Two-step mixed factorial design using AFFDEX 2.0 facial expression analysis on 4 YouTube-sourced deepfake/authentic video pairs, followed by between-subjects audience response study (N=60) measuring self-reported emotions and credibility judgments with mediation analysis via PROCESS Model 4.

Background & Context

The emergence of AI-generated deepfake videos presents a critical gap in our understanding: while one line of inquiry examines discrepancies in emotional expressions between deepfakes and authentic videos, another documents how viewers' emotional responses to such discrepancies relate to perceptions of video credibility—yet little research has attempted to bridge these two perspectives. This study addresses a fundamental question in deception detection research: can emotion serve as a diagnostic signal for identifying fabricated content?

Emotions serve as a primary channel of nonverbal communication and significantly shape how audiences perceive and process messages, underscoring the intricate connection between emotions and cognition and how emotions impact key cognitive processes like perception, attention, and learning. While computer science has focused on algorithmic emotion detection in deepfakes, this research brings a human-centered psychological perspective to understanding how emotional signals in synthetic media influence viewer perceptions.

The study builds on theoretical foundations of emotional contagion—the phenomenon where emotions displayed in videos transfer to viewers—and emotion-as-information theory, which posits that individuals use their emotional states as a source of information when forming judgments, essentially asking themselves "How do I feel about this?" and using these feelings to evaluate message content.

Research Design & Methodology

The study implemented a two-step approach: the first step involved a computer-based analysis of facial expressions in both deepfake and authentic videos to assess emotional expressions, and the second step relied on audience response analysis, where participants' emotional and cognitive reactions were evaluated through surveys following the viewing of each video. This dual methodology uniquely bridges computational and behavioral approaches to deepfake detection.

The study utilized a mixed factorial design, incorporating 2 versions of video (Deepfake vs. Authentic, between-subjects) × 4 topics (Videos 1–4, within-subjects) structure, and sourced four videos from user-uploaded content on YouTube to represent the types of deepfake content that are frequently disseminated online. The videos featured well-known figures including Cody Ko, Chris Rock, Morgan Freeman, and Taylor Swift.

For computational analysis, the AFFDEX 2.0 facial expression analysis algorithm was used, paired with version 10 of the iMotions biometrics data collection platform, which offers real-time detection of emotions based on facial action units derived from FACS, categorized into seven discrete emotions—anger, sadness, disgust, joy, surprise, fear, or contempt. The audience response study involved a final sample of 60 participants, which provided adequate statistical power for the mixed-design ANOVA.

Key methodological elements:

• Computer-based facial expression analysis using validated AFFDEX 2.0 algorithm
• Between-subjects experimental design (deepfake vs. authentic conditions)
• Within-subjects repeated measures across 4 different video stimuli
• Self-report measures of emotional responses and credibility perceptions
• Mediation analysis using PROCESS Model 4 to test indirect effects

Results & Key Findings

Deepfakes exhibited significantly lower emotional expressivity compared to authentic videos. Results indicate that deepfakes generally exhibit lower overall and negative emotions compared to authentic counterparts. The computational analysis revealed consistent patterns of emotional attenuation across the deepfake stimuli, though inferential statistics were not applied to the video-level data.

Counterintuitive credibility paradox: Notably, audiences' reduced emotional responses to deepfakes were associated with higher perceived credibility. This finding directly contradicts assumptions that emotional deficits would signal inauthenticity to viewers.

Mediation analysis findings:

• The mediation pathway via negative emotional responses to video credibility was significant (effect = 0.29, BootSE = 0.16, 95% CI = 0.02–0.62)
• The mediation pathway from video type to perceived credibility through overall emotion was significant (effect = 0.28, BootSE = 0.16, 95% CI = 0.004–0.63), indicating that lower overall emotion levels correspond to higher perceived credibility
• These patterns suggest that emotional attenuation in deepfake videos corresponds with greater perceived credibility

The emotional contagion effect was confirmed: differences in displayed emotions transferred to viewers' own emotional experiences, which then influenced their credibility judgments—but in the unexpected direction of muted emotions increasing perceived authenticity.

Discussion & Significance

This research makes a unique contribution by revealing a troubling psychological mechanism: media literacy programs can help audiences recognize how muted emotional cues in synthetic media—particularly attenuated negative emotions—may heighten perceived credibility by reducing cognitive vigilance. The finding challenges conventional wisdom that technical imperfections in deepfakes will make them easy to detect.

Negative media coverage has been shown to erode trust by amplifying negative emotional responses, and these effects can be attributed to the capacity of negative emotions to motivate individuals to critically scrutinize content—when experiencing negative emotions, people tend to allocate greater cognitive resources in evaluating information. The current findings suggest deepfakes may exploit this mechanism in reverse: by reducing negative emotional content, they paradoxically reduce the scrutiny that would expose their inauthenticity.

While findings provide valuable insights into current deepfake technology, these patterns may evolve as deepfakes become increasingly capable of reproducing authentic emotional nuances, and findings should be interpreted as reflecting broader psychological principles—namely, that emotional cues in videos and audiences' emotional responses serve as mechanisms shaping credibility perceptions in synthetic media. This positions emotion as a critical but complex factor in synthetic media detection.

The study bridges disciplinary divides by integrating computational emotion recognition with psychological theories of information processing, offering both technical and behavioral insights into deepfake detection challenges.

Limitations & Considerations

Based on simulation-based recommendations, the sample size offers sufficient power (> 0.80) to detect hypothesized medium-to-large indirect effects using bias-corrected bootstrapping, although power to detect smaller mediation effects may be limited—a limitation the authors acknowledge. The moderate sample size may limit generalizability of subtle effects.

Because computational data represent inherent characteristics of the videos themselves rather than repeated observations from a sample with a definable N, statistical assumptions necessary for a t-test were not met; each video provided only a dataset of computationally derived emotional characteristics, making it inappropriate to apply inferential statistics designed for comparing distributions of observations, so the computational analysis descriptively identified and compared emotional patterns. This limits the strength of conclusions about video-level differences.

Additional considerations:

• Study used publicly available YouTube deepfakes, which may vary in quality and technical sophistication
• All stimuli featured celebrity subjects, limiting generalizability to non-celebrity deepfakes
• Cross-sectional design prevents assessment of how detection abilities change with repeated exposure
• Self-report measures of emotion may not capture unconscious affective responses

Practical Applications

For polygraph professionals and forensic practitioners, this research highlights that emotional signals in audiovisual evidence require careful interpretation. The finding that reduced emotional expressivity can increase perceived credibility suggests that relying on intuitive judgments of "something feels off" may be insufficient or even misleading when evaluating potentially manipulated media.

The study underscores the need for structured media literacy interventions that specifically educate the public about the counterintuitive relationship between emotional muting and synthetic media. Training programs should emphasize that artificially calm or emotionally flat presentations may actually indicate fabrication rather than authenticity. This has implications for consumer education around deception detection and evaluation of video evidence in investigative contexts.