Building a second-opinion tool for classical polygraph

This 2023 study developed a machine learning-based second-opinion tool to detect human examiner errors in classical polygraph interpretations. The researchers trained their model on 2,094 field polygraph screening recordings to identify when examiners incorrectly interpret physiological data, rather than attempting to improve polygraph accuracy directly.

Researchers built a machine learning model trained on historical field polygraph screening recordings with known Deception Indicated attributes to detect patterns of examiner interpretation errors.

This 2023 study by Sberbank researchers addressed quality assurance concerns in polygraph practice by developing an automated second-opinion tool to detect human examiner errors. Using machine learning on 2,094 field screening records from bank personnel, the team trained classifiers to flag cases where automated analysis disagreed with human examiner conclusions. The approach was unconventional in that it deliberately avoided incorporating polygraph examiner rules to prevent the model from making the same errors as humans. Field validation involved manual double quality assurance by independent examiners on flagged cases, successfully identifying genuine examiner mistakes in historical records.

This research provides a practical quality assurance solution for polygraph operations, offering a cost-effective way to identify potential examiner errors without requiring full second examiner reviews. The approach represents a novel application of machine learning to improve polygraph practice reliability while acknowledging the fundamental limitations of polygraph methodology itself.

Background & Context

The polygraph has long been criticized by the scientific community for its susceptibility to human error, yet it remains widely used in high-stakes contexts including banking, law enforcement, and federal agencies. Classical polygraph screenings are routinely used by critical businesses, but a major concern is that screening errors stem not only from the method itself but also from human examiner error. The challenge of quality assurance in polygraph practice has historically been limited by the cost and impracticality of having second examiners review every test—a quality assurance review takes at least half the time of performing the original screening, with an average screening taking at least two hours.

This 2023 study by researchers at Sberbank of Russia represents the first detailed disclosure of building and testing a second-opinion tool for classical polygraph. Rather than attempting to improve polygraph accuracy directly, the research addresses a more specific problem: detecting when human examiners make errors in interpreting physiological data. The researchers explicitly state that their results neither justify nor solidify the practice of classical polygraph screenings, but rather provide a temporary and partial patch to eliminate a specific type of error.

The study builds on earlier work by Slavcovic (2002), which analyzed raw polygraph data but warned about potential examiner errors without directly attempting to identify them. Both studies work with raw classical polygraph data collected in the field, with accuracies roughly equivalent, but the current study specifically aimed at finding examiner errors and succeeded at catching mistakes in historical records.

Research Design & Methodology

The researchers built a basic second-opinion model by training on historical data from 2,094 field polygraph screening recordings (PSRs) with Deception Indicated (DI) attributes set by the examiners who conducted the screenings. These screenings were performed on bank personnel with critical roles, covering topics including drug abuse, gambling addiction, insider trading, corruption, bribery, and fraud. The data was in NCCA ASCII standard text format with a 31 Hz sampling rate and included raw signal data, alternative data (sex and age), magnetic storm data, and weather data.

The machine learning approach was unconventional: the researchers hypothesized that the model would not learn to make the same errors as human examiners because the share of examiner errors is minor, and they deliberately avoided implementing polygraph examiner rules as features to avoid the model being trapped in the same way human examiners are. Using stratified fivefold cross-validation, the researchers trained binary classifiers to flag cases where the model's conclusion contradicted the examiner's determination.

The experimental framework included several key elements:

• Basic model: Features based on 10 raw polygraph signals plus examinee age and sex
• Novel features tested: Alternative data including job position, weather conditions, and geomagnetic storm data
• Topic-specific models: Separate training for individual topics (e.g., drug abuse) versus universal models
• Field validation: Manual double quality assurance by two independent examiners on cases flagged by the model

Primary quality metrics were ROC AUC and TPR for an FPR at 0.05, though these were acknowledged as indirect measures because they assess how well the model mimics examiner conclusions—counterintuitively, a perfect model predicting 100% of examiner conclusions would flag no candidates for errors.

Results & Key Findings

The machine learning tool successfully identified genuine examiner errors in field polygraph screenings. Based on model scores, researchers selected 40 conclusions in 36 screenings as candidates for examiner errors and handed these to two examiners for thorough blind quality assurance. By QA-ing 39 examiner conclusions in 35 screenings, they identified 30 problematic conclusions where either plain examiner errors or hard-call cases requiring a concilium were found, with only 9 conclusions being model errors.

Model performance metrics and key findings included:

• The benchmark AUC of 0.85 from Slavcovic's work (obtained on criminal investigation data) was considered an upper bound, as it was known to have much more predictive power than job screenings
• Training a topic-specific model for drug abuse gained a +2% ROC AUC (6% relative uplift) compared to the universal model
• A significant portion of flagged cases (17 of 39 conclusions) required a concilium where examiners had to discuss conflicting conclusions
• In several problematic screenings, QA examiners noted that examinees appeared to practice countermeasures

The validation phase represented a major methodological challenge: manual field validation by double QA-ing covered only 39 conclusions but required approximately 80 examiner hours, not including concilium time to resolve discrepancies. Despite this limited sample, given the assumption that examiner error rate might be between 0.0 and 1.0% of all screenings—an extraordinary and critical situation—finding at least one examiner error in 39 conclusions inside 35 screenings was considered successful.

Discussion & Significance

This study represents a paradigm shift in polygraph quality control by demonstrating that machine learning can serve as a scalable screening mechanism for human examiner errors. The ML tool allows for semi-automatic double-checking of all new and historical reports without hiring additional examiners, with the added benefit that if examiners know their work will be QA-ed, they will make decisions more carefully. The approach is particularly valuable given that industrial internal security departments QA screenings infrequently or not at all due to cost.

The researchers demonstrated novel feature engineering that improved model performance. Beyond standard physiological signals, they explored unconventional predictors including weather conditions and geomagnetic storms on the date of screening. More significantly, the decision to avoid implementing traditional polygraph scoring rules as features represents a methodological departure from prior work that may have helped the model identify patterns human examiners systematically miss.

Perhaps most important is the study's intellectual honesty about the polygraph's limitations. The authors explicitly frame their work not as validation of polygraph practice but as damage control—considering their results a temporary and partial patch that helps eliminate a specific type of error until better methods are devised, representing a step toward rethinking classical polygraph practices. The researchers planned a one-year pilot where 100% of new screenings (approximately one hundred per day) would be scored, with manual QAs mandated when conflicts arose, beginning at the end of 2022.

Limitations & Considerations

The study faces fundamental methodological challenges acknowledged by the authors. The training set is contaminated with few and unknown examiner errors, and creating a gold standard trainset would involve double QA-ing hundreds of screenings. Until field validation, the researchers risked finding no examiner errors because models might learn to replicate examiner mistakes. The validation sample of 39 conclusions is extremely small, and even this tiny test required approximately 80 examiner hours.

The study has generated significant scientific controversy. Subsequent critiques by Kotsoglou and Biedermann (2024) challenged the fundamental premise, arguing that without true ground truth (actual deception status), the system merely identifies disagreements between examiners rather than genuine errors. The critics note that the data are not obtained under controlled conditions with known ground truth but are convenience data labeled with examiner opinions—something more arbitrary than rational. The context matters: this research was conducted by industry-based researchers at a Russian bank for internal security purposes, not as academic validation of polygraph science.

Additional limitations include:

• Data from a single organization in one country, limiting generalizability
• Screening context (pre-employment and routine employee checks) differs significantly from criminal investigation applications
• Data are from civil personnel screenings versus Slavcovic's army criminal investigations, with distributions of classes differing significantly and data nature being very different
• No standardized test format—screenings covered 14 different topics with varying base rates of deception

Practical Applications

For organizations that use polygraph screening despite scientific controversy, this research offers a practical harm-reduction approach. The machine learning tool provides a triage mechanism to identify the small percentage of cases most likely to contain examiner errors, making comprehensive quality assurance feasible without proportional increases in examiner staff. The automated flagging system could be particularly valuable for reviewing historical archives of polygraph data where second opinions were never obtained.

Polygraph examiners and administrators should recognize both the potential and the limits of this approach. The tool does not validate polygraph methodology itself or improve the fundamental accuracy of deception detection. Rather, it addresses examiner consistency—identifying cases where an examiner's conclusion deviates from patterns in the training data. This may catch genuine errors, but it may also flag cases where an examiner correctly identified deception that was atypical or where individual differences made interpretation difficult. The high rate of cases requiring concilium resolution (17 of 39) suggests many flagged cases represent genuinely ambiguous calls rather than clear errors.