Automated Verbal Credibility Assessment of Intentions: The Model Statement Technique and Predictive Modeling

This study tested automated verbal credibility assessment for detecting deceptive intentions using the model statement technique and machine learning approaches. While initial accuracy exceeded 80%, independent validation revealed significant performance drops and limitations in real-world application.

Two online experiments had participants write truthful or deceptive statements about planned weekend activities after viewing model statements, with automated linguistic analysis and machine learning classification.

The research addressed automated deception detection for future intentions rather than past events, combining model statement techniques with machine learning approaches for potential security screening applications. Using natural language processing and automated linguistic analysis, the study extracted named entities and psycholinguistic features from participant statements about planned activities. Initial machine learning classification exceeded 80% accuracy, but crucial independent validation in Experiment 2 showed performance dropped to 63%, revealing the limitations of within-sample cross-validation. The counterintuitive finding that liars included more person and location references suggests an "over-preparation" effect where deceptive individuals attempt to appear more convincing than naturally honest truth-tellers.

Results highlight the critical importance of independent validation in automated deception detection systems and suggest that over-reliance on within-sample performance may lead to overestimation of real-world effectiveness in security screening applications.

Background & Context

The detection of deceptive intentions represents one of the most critical challenges in security screening, particularly in contexts like airport passenger assessment where threats must be identified before harmful actions occur. Traditional verbal credibility assessment research has focused primarily on past events, but this study extends these techniques to the detection of deceptive intentions about future activities, combining three emerging approaches: detecting deceptive intentions, using model statements, and applying predictive modeling.

The study addresses a fundamental practical barrier: large-scale deception detection in settings like prospective airport passenger screening requires automated data collection and analysis, as manual human coding is limited in its scalability and makes real-time assessment of vast numbers of statements virtually impossible. This research bridges the gap between traditional human-coded verbal analysis and fully automated computational approaches, testing whether machine learning can reliably distinguish truthful from deceptive statements about planned activities.

The model statement technique, wherein interviewees are shown an example of a detailed truthful statement before providing their own account, has shown promise in eliciting richer verbal content. Research indicates that model statements may improve verbal deception detection when examining certain verbal aspects, and importantly, no study has found that model statements impede detection.

Research Design & Methodology

The research employed two complementary experiments using an online data collection platform. In Experiment 1, participants read a model statement and then wrote either a truthful or deceptive statement about their planned weekend activities. The study also included a comparison condition examining statements about past weekend activities to contrast deception detection for intentions versus past events.

The methodological approach included several key elements:

• Automated linguistic analysis using spaCy natural language processing to extract named entities across 16 categories (persons, locations, dates, times, organizations, events, etc.), with the proportion of unique named entities relative to word count serving as the primary outcome variable
• Machine learning classification using linguistic features, achieving initial accuracies exceeding 80% in Experiment 1
• Manual blind coding of a random subset of 147 statements by two independent coders to validate the automated analysis
• Experiment 2 as a crucial independent-sample replication to test whether findings generalized beyond the initial dataset

This two-experiment design with cross-validation represents a rigorous approach to evaluating both within-sample performance and external validity of the automated detection system.

Results & Key Findings

Using machine learning with linguistic features, Experiment 1 achieved classification accuracy exceeding 80% in distinguishing truthful from deceptive statements. However, the independent validation revealed important limitations.

The key empirical findings included:

• Experiment 2's classification accuracy remained above chance level but dropped to 63%, demonstrating that within-sample cross-validation may overestimate real-world performance
• A counterintuitive pattern emerged: liars included significantly more person and location references than truth-tellers, suggesting an "over-preparation" effect
• This finding suggests that liars may over-prepare their statements, attempting to appear more convincing than truth-tellers who assume their honesty will be naturally apparent
• Predictive modeling with psycholinguistic features yielded promising above-chance results, though independent validation showed that within-sample cross-validation can overestimate classification accuracies

The replication in Experiment 2 corroborated the core finding about liars' enrichment strategies while highlighting the critical importance of external validation in machine learning applications to deception detection.

Discussion & Significance

This research makes several important contributions to deception detection science. The finding that liars mentioned more persons and locations than truth-tellers reveals a strategic over-preparation effect where deceivers create answers more detailed than most truth-tellers are willing or able to provide about their intentions. This challenges traditional assumptions from Reality Monitoring theory that liars produce less detailed accounts due to fabrication demands and cognitive load.

The study provides novel insights into liars' strategies, highlights the promise of machine learning for deception detection, and critically emphasizes the need for proper validation of predictive deception detection analyses. The substantial drop from over 80% to 63% accuracy between initial and replication samples serves as an important methodological lesson for the field about avoiding over-optimistic estimates from within-sample validation.

The automation of verbal credibility assessment represents a potential breakthrough for practical applications. Unlike human coders who can process only limited numbers of statements, automated approaches make large-scale deception detection applicable when data can be collected and analyzed automatically, opening possibilities for high-throughput screening applications while maintaining above-chance detection accuracy.

Limitations & Considerations

The study acknowledges several important limitations. Most significantly, independent validation demonstrated that within-sample cross-validation can substantially overestimate classification accuracies. The 17-percentage-point accuracy drop between experiments underscores the critical importance of external validation before deploying automated systems in applied contexts.

Additional considerations include the laboratory nature of the deception scenario, where participants fabricated statements about weekend plans rather than concealing genuine malicious intentions with serious consequences. The generalizability to high-stakes security contexts remains an empirical question. The study also focused specifically on written statements in English, and performance may differ across languages, modalities (written versus spoken), or cultural contexts.

The automated approach, while scalable, may lack the contextual understanding that human coders bring to nuanced verbal analysis. Future research must address whether the efficiency gains from automation justify any potential loss of interpretive sophistication.

Practical Applications

This research demonstrates that automated verbal credibility assessment of intentions is technically feasible and can achieve above-chance accuracy using commercially available natural language processing tools and machine learning algorithms. For security screening applications, the ability to rapidly analyze large volumes of statements about travel plans or future activities could provide a scalable first-stage triage system, flagging potentially deceptive accounts for further human review.

However, practitioners should approach implementation cautiously. The 63% accuracy in independent validation, while statistically above chance (50%), indicates substantial error rates that would be unacceptable as a sole decision criterion. The system shows promise as a decision-support tool within a comprehensive assessment protocol rather than as a standalone detector. Understanding that liars may over-prepare by including excessive specific details provides investigators with a novel behavioral cue to consider during credibility assessment.