Quantifying Sampling Variability of IRT-Based Classification Accuracy and Consistency

In high-stakes testing, classification accuracy (CA) and classification consistency (CC) provide critical validity evidence for criterion-referenced decisions. However, these indices are conventionally reported as point estimates, obscuring their statistical uncertainty. This study focuses on quantifying the uncertainty associated with item response theory-based classification indices. Because these indices are functions of estimated item parameters, they are subject to sampling variability from parameter estimation. We derive analytical expressions for the asymptotic standard errors of conditional and marginal CA and CC indices under the two-parameter logistic model. In addition, a parameter-drawing multiple imputation approach is implemented as an alternative method for propagating parameter uncertainty. A Monte Carlo simulation study varying sample size and test length evaluates the empirical performance of both estimators. Results indicate that the proposed standard errors closely approximate empirical standard deviations and yield confidence intervals near nominal coverage rates.