Can a Large Language Model Judge a Child’s Statement?: A Comparative Analysis of ChatGPT and Human Experts in Credibility Assessment

The findings reveal a profound gap between the nuanced, contextual reasoning of human experts and the pattern-recognition capabilities of the LLM tested. The study concludes that this type of AI, in its current, prompt-engineered form, cannot reliably replicate expert judgment in the complex task of credibility assessment. While not a viable autonomous evaluator, it may hold potential as a "cognitive assistant" to support expert workflows. The assessment of child testimony credibility remains a task that deeply requires professional judgment and appears far beyond the current capabilities of such generative AI models.

Background & Context

The assessment of child witness credibility in sexual abuse cases represents one of the most sensitive and consequential challenges in forensic practice. Criteria-Based Content Analysis (CBCA) has emerged as a widely used method for assessing statement credibility, providing structured criteria to evaluate the authenticity of children's testimony when physical evidence is limited or absent.

As artificial intelligence systems increasingly demonstrate capabilities across complex cognitive tasks, questions have emerged about whether Large Language Models (LLMs) could augment or assist in forensic evaluations. This study investigates the inter-rater reliability between human experts (a forensic psychologist and a social worker) and a large language model in the assessment of child sexual abuse statements, representing a pioneering examination of AI's potential role in this ethically fraught domain.

The research fills a critical gap by moving beyond hypothetical assessments to test whether current-generation AI can meaningfully replicate the nuanced judgment required for credibility assessment. Given the high stakes of child protection cases, understanding the actual capabilities and severe limitations of AI systems in this context is essential for preventing premature or inappropriate deployment.

Research Design & Methodology

Sixty-five anonymized transcripts of forensic interviews with child sexual abuse victims (N = 65) were independently evaluated by three raters: a forensic psychologist, a social worker, and a large language model (ChatGPT, GPT-4o Plus). This comparative design allowed direct assessment of whether AI could achieve reliability comparable to trained human evaluators.

Each statement was coded using the 19-item CBCA framework, which assesses credibility through criteria including logical structure, quantity of details, contextual embedding, descriptions of interactions, reproduction of conversations, and accounts of subjective mental states. The CBCA system represents decades of forensic practice and has established inter-rater reliability benchmarks in the literature.

Key methodological features included:

• Three independent raters: two human experts and GPT-4o Plus
• 19 CBCA criteria assessed for each transcript
• Statistical analysis using Intraclass Correlation Coefficient (ICC), Cohen's Kappa (κ), and other agreement statistics
• Comparison of human-human reliability versus human-AI reliability

The study employed rigorous statistical methods to quantify agreement, allowing precise measurement of how LLM evaluations deviated from expert consensus.

Results & Key Findings

A high degree of inter-rater reliability was found between the two human experts, with the majority of criteria showing "good" to "excellent" agreement (15 of 19 criteria with ICC > .75). This strong human-human agreement established a benchmark demonstrating that the CBCA criteria can be reliably applied by trained professionals.

In stark contrast, a dramatic and significant decrease in reliability was observed when the AI model's evaluations were compared with those of the human experts. The LLM consistently failed to achieve even moderate agreement with human raters across most CBCA dimensions.

Key findings demonstrate:

• 15 of 19 criteria achieved ICC > .75 for human-human agreement
• Significant reliability drop for human-AI comparisons across nearly all criteria
• A profound gap between the nuanced, contextual reasoning of human experts and the pattern-recognition capabilities of the LLM tested
• AI demonstrated inability to capture subtle credibility indicators that human experts consistently identified

The study concludes that this type of AI, in its current, prompt-engineered form, cannot reliably replicate expert judgment in the complex task of credibility assessment. The findings reveal fundamental limitations in how LLMs process and evaluate the contextual, developmental, and psychological dimensions critical to CBCA.

Discussion & Significance

These findings carry profound implications for the responsible development and deployment of AI in forensic contexts. The stark difference between human expert agreement and AI performance suggests that current LLMs lack the capacity for the type of holistic, contextually-grounded judgment required for credibility assessment.

The results align with broader concerns about AI systems' limitations in tasks requiring deep contextual understanding, developmental knowledge, and integration of subtle psychological cues. While LLMs excel at pattern recognition in textual data, the nuanced, contextual reasoning of human experts operates on dimensions that extend beyond surface-level linguistic patterns.

While not a viable autonomous evaluator, the LLM may hold potential as a "cognitive assistant" to support expert workflows. This distinction is critical: the study's findings do not preclude all AI involvement but firmly establish that autonomous AI credibility assessment is inappropriate and unreliable given current technology.

The assessment of child testimony credibility remains a task that deeply requires professional judgment and appears far beyond the current capabilities of such generative AI models. This conclusion underscores the irreplaceable role of human expertise in high-stakes forensic evaluations where errors can have devastating consequences for child welfare and justice.

Limitations & Considerations

The study focused on a single LLM (GPT-4o Plus) at a specific point in technological development. While this model represents state-of-the-art capabilities as of 2026, rapid AI advancement means future models may demonstrate different performance characteristics. However, the fundamental challenges identified—contextual reasoning, developmental sensitivity, psychological nuance—represent deep limitations unlikely to be resolved through incremental improvements.

The study examined CBCA application in child sexual abuse cases specifically. Generalization to other forensic contexts or credibility assessment frameworks requires additional research. The 65-transcript sample, while substantial for this type of intensive forensic analysis, represents a specific population and interview context.

Prompt engineering approaches could potentially influence LLM performance, though the study used carefully crafted prompts designed to optimize AI capabilities. The core reliability gap suggests that prompt refinement alone cannot overcome fundamental limitations in how current LLMs process credibility-relevant information.

Practical Applications

For forensic practitioners, these findings provide clear guidance: AI systems should not be used as autonomous credibility evaluators in child abuse cases. The profound reliability gap documented in this study demonstrates that current LLMs cannot substitute for trained human judgment in CBCA application.

Potential supportive roles for AI might include administrative tasks (transcript organization, preliminary coding checks for human review), educational applications (training scenarios), or research tools (pattern analysis across large datasets). However, any such applications must maintain human oversight and recognize AI limitations. Practitioners should approach claims about AI-assisted credibility assessment with appropriate skepticism and demand rigorous validation before considering any form of implementation.

For policymakers and child protection systems, this research reinforces the need for human expertise and appropriate training in credibility assessment. Investment in skilled forensic interviewers and evaluators remains essential, as technological shortcuts are not viable given current AI capabilities.