The Problem with Single Metrics

The Earnings Premium is about to become the law of the land. A single metric will determine the fate of thousands of college programs. The idea is straightforward and seductively simple: measure how much more graduates of a particular program earn compared to a baseline—usually high school graduates. Higher earnings suggest higher value.

But there’s a catch. The metric only looks at graduates. It tells us how well colleges do conditional on success. It says nothing about how many students actually make it to graduation.

Imagine we judged car manufacturers this way. Suppose Manufacturer A produces cars that, when they run, get 35 miles per gallon—but seven out of ten never start. Manufacturer B produces cars that average 28 miles per gallon, but 9 out of 10 work reliably. If we measure performance only by fuel efficiency, Manufacturer A looks better. But any reasonable person would prefer Manufacturer B.

.

This is, in essence, a Bayesian problem. The Earnings Premium measures

P(High Earnings | Graduated)

but what students, families, investors and the public care about is closer to

P(High Earnings | Enrolled).

Bayes’ rule reminds us that these are not the same. The overall likelihood of ending up with high earnings depends both on the probability of graduating and the earnings conditional on graduation.

A college with a high premium but a low graduation rate might look good on paper but deliver poor odds in practice. That’s why relying on a single metric—no matter how elegant—is highly problematic. A complete picture requires multiple lenses: completion, cost, and outcomes.

For more data, analysis, and discussion, visit the EDU Accountability Lab.