The selective labels problem

Context. To ask whether a model would make better bail decisions than judges, one needs outcomes for the defendants the model would release, but outcomes exist only for those the judge released. The labels are selected by the very decisions being compared against, by people who see things the model does not (demeanor, who came with the defendant). Every standard fix, from imputation to doubly robust estimation (Dudík et al., next in this section), assumes no unmeasured confounding and is biased here. Lakkaraju et al. impute nothing and use the variation between decision-makers who see comparable cases. The most lenient one's released set contains an outcome for every case any stricter policy on that caseload would release, provided the stricter policy also refuses what the lenient one refused. So a model can be evaluated by contracting that released set, and the error this leaves depends only on how much the model disagrees with those refusals, which can be computed from the data. Kleinberg et al. (2018, QJE) make the same design a central tool in their study of bail decisions in New York City.

Setting. Data $\{(x_i,j_i,t_i,y_i)\}$ : recorded features $x_i$ , decision-maker $j_i$ , decision $t_i\in\{0,1\}$ (1 = release, which produces a label), and outcome $y_i$ observed only when $t_i=1$ . Decision-makers also see unobservables $z_i$ that may affect $y$ . A black-box model $B$ , trained on separate data, scores risk from $x$ alone. The acceptance rate is the fraction released, and the failure rate is bad outcomes among those released divided by all cases judged, so the two must be read together as a curve. Contraction needs many decision-makers, cases assigned to them as if at random, and acceptance rates that vary.

Contraction and its error bound. Let $q$ be the decision-maker with the highest acceptance rate $\psi$ , with caseload $D_q$ and released set $R_q$ . To evaluate $B$ at acceptance rate $r\le\psi$ , keep everything $q$ detained detained, also detain the cases in $R_q$ that $B$ scores riskiest until the rate is $r$ , and report $\hat u(r)=\frac{\#\{\text{failures among the cases still released}\}}{|D_q|}.$ Let $a$ be the fraction of $q$ 's detained cases that $B$ , ranking all of $D_q$ on its own, would also detain at rate $r$ . Then the error relative to $B$ 's true failure rate on $D_q$ is at most $\big|\hat u(r)-u_B(r)\big|\le\frac{(1-a)\,|D_q\setminus R_q|}{|D_q|}.$ The machine curve $r\mapsto\hat u(r)$ is compared with a human curve built by binning decision-makers by acceptance rate.

Step 1

Evaluating on labeled cases alone, or imputing the rest, overstates the model's performance. Released cases are a population the judge has already screened using $z$ . If a family's presence in court predicts a good outcome for young defendants and judges use it, the labeled data show few failures among the young, and a model trained on them learns that youth is safe and fails on the young defendants the judges would have detained. Imputation from $x$ assumes $y\perp t\mid x$ ; when $z$ drives both, detained cases are riskier than released cases with the same $x$ , so imputed outcomes understate failures. Contraction counts only observed outcomes, so unobservables can enter only through the swapped cases the bound counts.

Step 2

Random assignment lets one caseload stand in for another, and leniency supplies the labels. The model runs on $q$ 's cases but is compared with judge $k$ at $k$ 's rate, which is fair only if $D_q$ and $D_k$ are draws from one distribution of $(x,z,y)$ . Since $q$ has labeled a superset of what a stricter policy that keeps $q$ 's refusals would release, the model can be evaluated at any rate up to $\psi$ and none above, which would release cases $q$ detained. One condition is implicit: a released case's outcome must not depend on who released it, or the outcomes in $R_q$ are $q$ 's outcomes and not those any policy would see.

Step 3

The error is bounded by the number of swapped cases, which the data determine. $B$ 's own detentions at rate $r$ include $a|D_q\setminus R_q|$ of $q$ 's detainees, so it releases the other $m=(1-a)|D_q\setminus R_q|$ and detains correspondingly more from $R_q$ than contraction does. The two released sets therefore differ by exactly $m$ cases on each side, with unknown outcomes on one side; failures in each group lie between 0 and $m$ , which gives the bound, and at $a=1$ contraction is exact. Since $a$ uses only $q$ 's decisions and $B$ 's scores on $D_q$ , the interval's width is known before looking at a label. It is a worst case over the unknown outcomes, conditional on the sample, and says nothing about sampling error in $D_q$ .

Step 4

On synthetic data with a strong unobservable, contraction tracks the truth and imputation reverses the conclusion. With 100 judges of 500 cases each, failure depends on $X+Z+0.2W$ with $Z$ seen only by judges, who detain by nearly that index at acceptance rates from 0.1 to 0.9, and the model is a logistic regression trained on the labeled cases of a training half. Contraction follows the true failure-rate curve closely, while labeled-only evaluation and five imputation methods, propensity matching and doubly robust estimation among them, understate the model's failure rate enough to put it below the judges' curve, when in truth the judges, who see $Z$ , do better. The imputation error grows with the weight on $Z$ , and at weight 1 contraction's mean absolute error is already 6.4 times below the best imputation method's. Its error rises as the lenient rate or the agreement falls, as the bound predicts, and as the lenient caseload shrinks, which is sampling error. The simulation favors contraction, since judges rank by nearly the outcome index, so swapped cases are both high-risk and their failures largely cancel: in my rerun with a model ranking by $X$ , agreement at $r=\psi=0.9$ is only about 0.4, yet the actual error is about a tenth of the bound.

Step 5

On three real data sets the model beats the humans, with agreement high enough to trust the estimate. On bail decisions, asthma treatment and insurance underwriting, with gradient-boosted trees trained on the labeled training cases, the contraction curve lies below the human curve at every acceptance rate, and the labeled-only curve lower still, which the paper reads as over-optimism. Agreement rates are 0.891, 0.932 and 0.718, computed at the lenient decision-maker's own rate, where agreement is lowest. An F-test fails to reject random assignment on all three, but it checks balance on recorded features only, and balance on $z$ is assumed. With no ground truth, these results rest on the assumptions and the bound, whose width the paper does not report.

Consequences and limits. Contraction replaces "no unmeasured confounding" with "as-if random assignment plus varying leniency", the structure of the judge-leniency designs in economics. It reaches only rates up to $\psi$ , with the precision of one caseload, and evaluates a fixed model without saying how to train on selectively labeled data. Payment fraud has the same structure: declined transactions never get an outcome, so scoring a new model on approved traffic alone is the labeled-only evaluation shown here to overstate its performance. If traffic is split at random across policies or review queues that decline at different rates, the most permissive one plays the lenient judge, and the swap bound says how much of a comparison rests on outcomes nobody saw; a small randomly approved slice, which the paper rules out on ethical grounds for bail, is often an acceptable cost there and gives labels directly. The failure labels inside approved traffic are themselves late chargebacks with unreported fraud counted as clean, so every failure rate is an undercount, and contraction compares policies fairly only if that undercount does not differ between the released sets being compared.

Terms

Selective labels: outcomes observed only for cases the decision-maker accepted
Unobservables $z$: information the decision-maker uses that the data do not record; breaks $y\perp t\mid x$
Acceptance rate: fraction of a decision-maker's cases accepted (released, approved)
Failure rate: bad outcomes among accepted cases divided by all cases judged
$q$ , $\psi$: the most lenient decision-maker and its acceptance rate; the model can be evaluated only at rates up to $\psi$
Contraction: keep $q$ 's refusals, refuse further cases from $R_q$ by model score, count observed failures
Agreement $a$: share of $q$ 's refusals the model would also refuse at rate $r$ ; computable from features alone
Swap bound: $|\hat u-u_B|\le(1-a)|D_q\setminus R_q|/|D_q|$ ; exact when $a=1$
Random assignment: caseloads are draws from one distribution, so one caseload stands in for another
Labeled-only evaluation: scoring the model on accepted cases; optimistic because they were screened using $z$

Quiz

When is the contraction estimate exactly the model's failure rate on $q$ 's caseload?
Why can contraction not evaluate the model at acceptance rates above $\psi$ ?
The paper tests random assignment by regressing a predicted outcome on decision-maker identity. What can this test not detect?
Contraction counts observed failures in the cases a policy releases. If outcomes themselves are under-reported (some failures recorded as clean), when does the comparison of two policies stay fair?

Sources

Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables. KDD, 2017.

Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. Human Decisions and Machine Predictions. Quarterly Journal of Economics, 2018.