Analysis of variance versus bootstrap procedures for analyzing dependent observation in small group research

Small group investigators have been plagued by the problem of observational dependency. This problem exists when data collectedfrom members of the same group are more similar to each other than they are to data collected from another small group receiving identical treatment. Observational dependency can result in inflated Type I error rates. This study demonstrates the effect of different levels of observational dependency on Type I error rates for ANOVA and introduces an alternative statistical procedure to address the problem. Bootstrapping is shown to be superior to ANOVA in minimizing the effect of Type I error rates due to observational dependency.

Monte Carlo simulation study systematically manipulating levels of observational dependency (intraclass correlation) in simulated small group datasets and comparing Type I error rates between conventional ANOVA and bootstrap resampling procedures across different group structures.

Background & Context

This 1994 methodological paper emerged from a critical statistical challenge that plagued small group research, including polygraph studies where multiple participants might be tested together or share experimental conditions. Observational dependency—the phenomenon where data from members of the same group are more similar to each other than to data from different groups—creates a serious threat to statistical validity that researchers had long struggled to address.

The problem is particularly insidious because it inflates Type I error rates, leading researchers to conclude that effects exist when they may not. In polygraph research, this issue arises when multiple examinees are exposed to the same examiner, testing conditions, or mock crime scenarios. Traditional parametric tests like ANOVA assume independence of observations, an assumption routinely violated in group-based experimental designs.

The bootstrap—a computer-intensive resampling method pioneered by Efron—offered a promising alternative that could accommodate dependency structures without requiring strict parametric assumptions. This paper represented an important bridge between statistical methodology and applied psychophysiological research.

Research Design & Methodology

The study employed Monte Carlo simulation methods to systematically examine how different levels of observational dependency affect Type I error rates under both ANOVA and bootstrap approaches. The authors manipulated the degree of intraclass correlation (dependency) within simulated small groups to model realistic research scenarios.

Key design elements included:

• Simulation of data sets with varying levels of observational dependency (intraclass correlations)
• Comparison of conventional ANOVA F-tests versus bootstrap resampling procedures
• Systematic assessment of actual Type I error rates under nominal alpha levels
• Evaluation across different group sizes and sample configurations typical of psychological research

The bootstrap procedure involved resampling entire groups (rather than individual observations) to preserve the dependency structure, then calculating test statistics from thousands of resampled datasets to empirically derive sampling distributions and p-values.

Results & Key Findings

Bootstrap procedures substantially outperformed ANOVA in controlling Type I error rates when observational dependency was present. As the level of within-group correlation increased, ANOVA became increasingly liberal, rejecting true null hypotheses far more often than the nominal alpha level would suggest.

Specific findings included:

• ANOVA showed dramatically inflated Type I error rates as observational dependency increased
• Bootstrap methods maintained appropriate error rates even under moderate to high dependency
• The magnitude of error inflation in ANOVA was directly related to the intraclass correlation coefficient
• Bootstrap procedures proved robust across different group structures and sample sizes

The authors demonstrated that ignoring observational dependency could lead researchers to claim statistically significant findings when none actually existed—a form of methodological artifact that had potentially contaminated the small group research literature for decades.

Discussion & Significance

This paper made an important methodological contribution to both small group research generally and polygraph research specifically. By demonstrating bootstrap's superiority in handling dependent observations, the authors provided researchers with a practical tool for more valid statistical inference. The work was particularly timely as computer power made bootstrap methods increasingly feasible.

For polygraph research, the implications were substantial. Many laboratory studies involve multiple participants exposed to the same examiner, mock crime scenario, or experimental manipulation—creating the exact dependency structures this paper addressed. Subsequent polygraph studies by Honts, Kircher, and colleagues incorporated bootstrap methods for analyzing data where traditional parametric assumptions were violated.

The paper also contributed to a broader movement toward resampling and computer-intensive statistical methods in psychological science, anticipating debates about replication and statistical validity that would intensify in following decades.

Limitations & Considerations

The study focused primarily on Type I error control and did not extensively examine statistical power under different conditions. Bootstrap methods, while more appropriate for dependent data, can be computationally intensive—a more significant concern in 1994 than today.

The simulation parameters, while representative of typical small group research, may not have captured all realistic dependency structures encountered in practice. Additionally, the paper did not provide extensive guidance on when practitioners should prefer bootstrap over traditional methods, leaving some ambiguity about practical implementation.

The generalizability of findings to very small sample sizes (common in some polygraph field studies) was not fully explored. Bootstrap methods themselves have finite-sample limitations that require consideration.

Practical Applications

This research provided polygraph practitioners and researchers with concrete guidance on statistical analysis when examining data with group-level dependencies. The message was clear: researchers must account for observational dependency or risk drawing invalid conclusions.

For contemporary polygraph research, the paper supports best practices including: treating experimental groups as the unit of analysis when appropriate, using multilevel modeling techniques, and employing resampling methods when parametric assumptions are violated. The work underscores that methodological rigor in statistical analysis is as important as careful experimental design and instrumentation in producing valid polygraph research.