Meta-Analysis of the Comparison Question Test

The first meta-analysis of the Comparison Question Test, examining moderator effects of subject type, incentives, and decision policy on CQT accuracy across laboratory studies.

Meta-analytic review of 14 CQT laboratory studies. Examined moderator effects using standard meta-analytic techniques. Variables included participant type (student vs. community), motivation/incentive level, and decision rule stringency.

Kircher, Horowitz, and Raskin conducted the pioneering meta-analysis of CQT research, establishing the methodological template for all subsequent quantitative reviews in the field. Their analysis of 14 laboratory studies identified three significant moderator variables: subject type (community volunteers vs. students), incentive level (monetary or other consequences), and decision policy (conservative vs. liberal cutscores). All three variables predicted accuracy but showed high covariations, making it difficult to isolate individual contributions. This finding highlighted the importance of ecological validity in polygraph research — studies with more realistic conditions (community participants, higher stakes, standard decision rules) produced more representative accuracy estimates. This groundbreaking work laid the methodological foundation for the next three decades of CQT meta-analyses.

As the first CQT meta-analysis, this study established that polygraph accuracy is systematically influenced by study design features. The high covariation among moderators suggests that studies maximizing ecological validity across all dimensions provide the most representative accuracy estimates. This insight remains relevant for interpreting modern meta-analytic results and designing future research.

Background & Context

The Kircher, Horowitz, and Raskin (1988) meta-analysis represents a watershed moment in polygraph research—it was the first quantitative synthesis of Comparison Question Test (CQT) accuracy ever conducted. Prior to this groundbreaking work, polygraph research consisted of isolated studies with wildly inconsistent findings, and no systematic effort had been made to identify why accuracy rates varied so dramatically across experiments.

This study emerged at a critical juncture for the polygraph profession. The 1983 Office of Technology Assessment report had raised serious questions about polygraph validity, and the field desperately needed rigorous meta-analytic evidence. By examining 14 laboratory mock crime studies, the authors sought to answer a fundamental question: What methodological features systematically influence CQT accuracy? Their findings would establish the template for interpreting polygraph research for the next three decades.

Research Design & Methodology

Kircher and colleagues conducted a systematic meta-analysis of 14 mock crime laboratory studies that employed standard guilty and innocent treatment conditions with the CQT. The research team applied meta-analytic techniques developed by Hunter, Schmidt, and Jackson (1982) to quantify variance across studies and identify moderator variables that predicted detection accuracy.

The analysis examined three primary moderator variables across the included studies:

• Subject Type: Whether participants were students or community volunteers drawn from non-student populations
• Incentives: The nature and strength of motivation provided (minimal vs. stronger monetary or other consequences for convincing the examiner)
• Decision Policy: The scoring and decision rules used (standard field cutoff criteria vs. other/liberal policies)

The researchers developed a Detection Efficiency Coefficient (rdec)—a novel accuracy metric that correlated binary ground truth (guilty/innocent) with tri-categorical test outcomes (deceptive/inconclusive/truthful). This innovation allowed them to account for inconclusive results in a single coefficient, reducing their impact compared to outright errors.

Results & Key Findings

The meta-analysis revealed that accuracy rates across the 14 studies ranged from chance levels to 100% correct—a stunning degree of variability that demanded explanation. The analysis successfully identified systematic sources of this variance through moderator effects.

Approximately 24% of the variance in detection rates could be attributed to sampling error, while the three moderator variables showed substantial correlations with detection accuracy:

• Subject Type: r = .61 (student vs. non-student populations)
• Incentives: r = .73 (minimal vs. stronger motivation)
• Decision Policy: r = .67 (standard field vs. other methods)

The highest diagnostic accuracies were obtained from nonstudent subject samples, when both guilty and innocent subjects were offered monetary incentives to convince the examiner of their innocence, and when conventional field methods were used for interpreting the physiological recordings and diagnosing truth and deception. Together, differences in Subjects, Incentives, and Decision Policies may account for as much as 65% of the observed variance in detection rates.

Critically, all three variables were found to be predictive of accuracy, but all three showed high covariations within the studies, making it impossible to isolate the independent contribution of each factor. Studies with high ecological validity tended to maximize all three factors simultaneously—using community participants, offering substantial incentives, and applying field-standard decision rules.

Discussion & Significance

This pioneering work fundamentally transformed how the polygraph field understood laboratory research. The high covariation among moderators revealed that ecological validity is not unidimensional—studies approximating real-world conditions did so across multiple design features simultaneously. This insight explained why some laboratory studies showed near-perfect accuracy while others performed at chance: they differed systematically in how closely they approximated field conditions.

The findings validated a crucial assumption underlying polygraph research: laboratory experiments can provide meaningful estimates of field accuracy, but only when they incorporate realistic participant populations, meaningful consequences for test outcomes, and field-appropriate decision standards. Studies using college students completing psychology experiments for course credit, with no meaningful stakes, would inevitably underestimate the CQT's operational accuracy.

This was the only review to use meta-analytic techniques to examine moderator variables until decades later, establishing Kircher et al. (1988) as the methodological gold standard. Every subsequent polygraph meta-analysis—including the influential 2003 National Research Council report and the 2021 Honts comprehensive meta-analysis—built directly upon the conceptual framework and moderator variables identified in this seminal work. The Detection Efficiency Coefficient introduced by the authors has been adopted in subsequent research as a sophisticated method for handling inconclusive outcomes.

Limitations & Considerations

The analysis was limited to only 14 laboratory studies, reflecting the relatively small body of controlled CQT research available in 1988. This small sample size, while understandable given the historical context, limited statistical power for detecting nuanced moderator effects and made it impossible to conduct multivariate analyses that could disentangle the independent effects of the three highly correlated moderators.

The exclusive focus on laboratory mock crime studies meant the analysis could not directly address field accuracy or generalizability to actual criminal investigations. While this design choice enhanced internal validity and allowed for careful experimental control, it left open questions about whether the identified moderator effects would hold in operational polygraph settings where base rates, examiner-examinee dynamics, and stakes differ fundamentally from laboratory conditions. The high covariation among moderators, while theoretically informative, also meant that the moderator effects were confounded and difficult to interpret regarding their unique contributions to accuracy.

Practical Applications

This meta-analysis provided crucial guidance for both researchers and practitioners. For researchers, it established clear design principles: validity studies should employ community participants, provide meaningful incentives, and use field-standard scoring criteria. Studies that cut corners on ecological validity would produce misleadingly low accuracy estimates that misrepresent the CQT's operational performance.

For polygraph examiners and agencies, the findings reinforced that test accuracy is not fixed but depends on examination conditions. The work suggested that field examinations—which naturally incorporate motivated examinees and standardized scoring—should achieve accuracy rates at the upper end of the observed distribution. For consumers evaluating polygraph evidence, the study highlighted that not all research is created equal: studies maximizing ecological validity across multiple dimensions provide the most trustworthy accuracy benchmarks for real-world applications.