Classical Test Theory (CTT)

Most people who build or use tests have never thought explicitly about measurement error. Classical Test Theory makes it unavoidable — it starts from the premise that every observed score is imperfect, and builds a framework for understanding how imperfect it is and why.

CTT is old. The foundations were laid in the early 20th century and formalized by Charles Spearman and later by Gulliksen and Lord & Novick. That age is worth noting: CTT was designed to be tractable with pencil, paper, and basic arithmetic. Modern IRT requires computational tools and large samples. CTT works with smaller samples and simpler math — which is part of why it remains the dominant framework for most practical test development.

The True Score Model

The core equation is simple:

X = T + E

Where X is the observed score, T is the true score (the score a person would get if there were no measurement error), and E is error. Error here is conceptualized as random — unsystematic variance due to momentary fluctuations in attention, guessing, environmental noise, and every other factor that isn’t the construct you’re measuring.

Because E is random, it’s assumed to average out to zero across repeated measurements. This means the expected value of X equals T. In practice, you never measure T directly — you can only estimate it through the properties of your test.

Reliability

Reliability is the ratio of true score variance to observed score variance. A perfectly reliable test would have no error variance — everyone’s score would reflect only their true level on the construct. In reality, reliability coefficients fall between 0 and 1, with most well-constructed psychological measures landing between 0.70 and 0.95.

There are several ways to estimate reliability:

Test-retest reliability measures the correlation between scores at two time points. It captures stability over time, but conflates genuine trait change with measurement error.

Internal consistency measures how well items within a test correlate with each other. The most widely reported statistic is Cronbach’s alpha (α). Alpha is often misunderstood as measuring “unidimensionality” — it doesn’t. It measures the average inter-item correlation adjusted for test length. A test can have high alpha while measuring multiple correlated constructs. McDonald’s omega is a more defensible alternative for most modern uses.

Inter-rater reliability applies when scoring involves human judgment (e.g., coding open-ended responses). Measured via Cohen’s kappa, intraclass correlation, or similar statistics.

Item Statistics

CTT gives you two primary item-level statistics:

Item difficulty (p-value): The proportion of respondents who answer an item correctly (for ability tests) or endorse it in the keyed direction (for personality tests). A p-value of 0.80 means 80% got it right — probably too easy if you want differentiation. A p-value of 0.20 means only 20% got it right — too hard for most populations. Items in the 0.30–0.70 range tend to maximize variance and discrimination.

Item discrimination (point-biserial correlation): The correlation between item response (right/wrong) and total test score. A highly discriminating item sorts people in the same way the overall test does. Items with point-biserials below 0.20 are usually worth revisiting — they’re adding noise, not signal.

What CTT Does Well

CTT is accessible. The statistics are interpretable without specialized software, communicable to non-statisticians, and computable from relatively small samples. For rapid test development, early-stage validation, or situations where you don’t have the sample sizes needed for IRT calibration, CTT is entirely appropriate.

It also maps directly to what most stakeholders care about: “Is this test consistent? Do the items work? Is it reliable enough to make decisions with?”

The Limitations

CTT’s core limitation is sample dependence. The statistics describe how a test performed in a specific sample. Administer the same test to a different population and the numbers change. An item that’s “medium difficulty” for one group might be easy or hard for another. This makes it harder to compare scores across test forms, equate scores across administrations, or generalize findings to populations outside the calibration sample.

CTT also assumes equal measurement error across the trait range — the Standard Error of Measurement is a single number applied to all scores. In reality, most tests measure more precisely in the middle of the score distribution than at the extremes. IRT makes this explicit; CTT averages over it.

CTT vs. IRT in Practice

The choice isn’t always either/or. Many well-designed assessments use CTT for initial item screening and IRT for final calibration and scoring. CTT gets you far enough to identify bad items quickly; IRT gives you the precision to build adaptive tests and make fine-grained comparisons.

A practical rule: if you have fewer than 300–500 respondents per item, IRT calibration will be unstable and CTT is the safer choice. If you’re building a large-scale or adaptive assessment, investing in IRT is worth it. The assessments I’ve built at Gyfted and the HSE Career Quiz draw on both frameworks at different stages of development.