LegalEye: Multimodal Court Deception Detection Across Multiple Languages

This study introduces LegalEye, a multimodal machine-learning model developed to detect deception in courtroom settings across three languages: English, Spanish, and Tagalog. The research investigates whether integrating audio, visual, and textual data can enhance deception detection accuracy and reduce bias in diverse legal contexts. LegalEye uses neural networks and late fusion techniques to analyze multimodal courtroom testimony data. The dataset was carefully constructed with balanced representation across racial groups (White, Black, Hispanic, Asian) and genders, with attention to minimizing implicit bias. Performance was evaluated using accuracy and AUC across individual and combined modalities. The model achieved high deception detection rates—97% for English, 85% for Spanish, and 86% for Tagalog. Late fusion of modalities outperformed single-modality models, with visual features being most influential for English and Tagalog, while Spanish showed stronger audio and textual performance. The Tagalog audio model underperformed due to frequent code-switching. Dataset balancing helped mitigate demographic bias, though Asian representation remained limited. LegalEye shows strong potential for language-adaptive and culturally sensitive deception detection, offering a robust tool for pre-trial interviews and legal analysis. While not suited for real-time courtroom decisions, its objective insights can support legal counsel and promote fairer judicial outcomes. Future work should expand linguistic and demographic coverage.

Background & Context

Perjury in court poses a serious threat to the fairness and credibility of the judicial system, with false testimony potentially distorting legal outcomes and increasing the risk of wrongful convictions. The challenge of detecting deception in legal settings has historically relied on human judgment, which research shows rarely exceeds chance-level accuracy. Traditional polygraph techniques, while widely used in investigative contexts, have faced significant limitations in multilingual and multicultural courtroom applications.

This study represents a significant advance by addressing a critical gap in deception detection technology: most existing models have been tested primarily on English-language data within the U.S. legal context, limiting their applicability to multilingual, multicultural courtroom settings. The LegalEye framework responds to growing concerns about algorithmic bias in legal AI systems by explicitly designing for demographic balance and cultural sensitivity across three languages.

LegalEye serves as the first multimodal deception detection model evaluated across English, Spanish, and Tagalog with race- and gender-balanced datasets for all three languages, positioning it as a groundbreaking tool for global judicial systems. The research builds on prior University of Michigan work demonstrating that neural networks with late fusion techniques outperform traditional classifiers in deception detection.

Research Design & Methodology

LegalEye uses neural networks and late fusion techniques to analyze multimodal courtroom testimony data, integrating three distinct input streams: audio (speech patterns and prosodic features), visual (facial expressions, gaze direction, hand gestures), and textual (transcript-based linguistic analysis). The architecture employs separate neural network pathways for each modality before combining their outputs through late fusion.

The research team compiled datasets in English, Spanish, and Tagalog—three languages that use Romanized scripts, allowing consistency in textual feature training, with native speakers on the team transcribing videos, and each video focused on the subject's face, figure, and voice. All video data was sourced from publicly available YouTube courtroom trial footage and interviews where ground truth (truthfulness vs. deception) could be established from trial outcomes.

Key methodological features include:

• Dataset construction with balanced representation across racial groups (White, Black, Hispanic, Asian) and genders, with attention to minimizing implicit bias
• Use of OpenFace for visual feature extraction including facial action units, gaze patterns, and micro-expressions
• CNN architecture for textual modality processing transcript-level linguistic features
• LSTM networks for temporal sequence modeling of audio and visual features
• Performance evaluation using accuracy and AUC across individual and combined modalities

Results & Key Findings

The LegalEye system achieved remarkably high accuracy rates: 97% for English, 85% for Spanish, and 86% for Tagalog. These results significantly exceed human baseline performance in deception detection tasks and approach or exceed many laboratory-based polygraph accuracy rates.

Late fusion of modalities outperformed single-modality models, with visual features being most influential for English and Tagalog, while Spanish showed stronger audio and textual performance. This language-specific finding reveals important cultural and linguistic differences in deception cues—English and Tagalog speakers may exhibit more detectable visual deception signals, while Spanish testimony showed stronger vocal and linguistic markers.

Key performance findings include:

• Multimodal fusion consistently outperformed any single modality alone across all three languages
• The Tagalog audio model underperformed due to frequent code-switching between Tagalog and English, highlighting a technical challenge in multilingual environments
• Dataset balancing helped mitigate demographic bias, though Asian representation remained limited
• Visual features (facial expressions, gaze, gestures) showed strong cross-linguistic consistency as deception indicators

Discussion & Significance

This research makes a landmark contribution to AI-based deception detection by demonstrating that machine learning systems can achieve high accuracy across multiple languages and cultural contexts. The findings emphasize the effectiveness of multimodal fusion in capturing the nuanced and context-sensitive characteristics of deceptive behavior, validating theories that deception manifests across multiple behavioral channels simultaneously.

The language-specific modality differences discovered in this study have important theoretical implications. The stronger visual cue reliance in English and Tagalog versus audio-textual dominance in Spanish suggests that cultural display rules, linguistic structure, or courtroom behavior norms may shape how deception manifests across languages. This finding argues against one-size-fits-all deception detection approaches and supports culturally adaptive systems.

LegalEye shows strong potential for language-adaptive and culturally sensitive deception detection, offering a robust tool for pre-trial interviews and legal analysis, though it is not suited for real-time courtroom decisions. The system's objective analytical capabilities could complement rather than replace human judgment, particularly in supporting legal counsel during case preparation and witness credibility assessment.

Limitations & Considerations

Several important limitations constrain the generalizability of these findings. Asian representation remained limited in the dataset, potentially affecting model performance with Asian subjects and limiting conclusions about cross-racial bias mitigation. The reliance on YouTube courtroom footage introduces selection bias—cases available on YouTube may not represent typical court proceedings, and video quality/camera angles vary substantially.

The Tagalog audio model's underperformance due to frequent code-switching exposes a significant challenge for multilingual applications. Many Tagalog speakers naturally blend English words into speech, creating acoustic and linguistic complexity that current models struggle to parse. This limitation extends to other code-switching language communities worldwide.

Additional considerations include:

• Ground truth establishment relies on trial outcomes, which may not perfectly reflect actual truthfulness
• Ecological validity concerns—courtroom behavior under public scrutiny may differ from private deception
• Translating these advancements into real-world courtroom applications will require careful attention to ethical, legal, and procedural concerns
• The system has not been validated in prospective, real-time applications

Practical Applications

LegalEye offers a robust tool for pre-trial interviews and legal analysis, with objective insights that can support legal counsel and promote fairer judicial outcomes. Prosecutors and defense attorneys could use the system during witness preparation, credibility assessment, and case strategy development. The multilingual capability makes it particularly valuable for international legal cooperation and immigration proceedings where language diversity is common.

The system's explicit focus on bias mitigation addresses critical fairness concerns in legal AI deployment. By balancing datasets across racial and gender categories and evaluating performance disparities, LegalEye provides a model for responsible AI development in high-stakes legal contexts. However, the researchers appropriately caution against real-time courtroom use—the technology should support rather than supplant human decision-making, functioning as an analytical tool for counsel rather than evidence presented to juries.