Building Fair Evaluation Sets Is a Combinatorial Problem
Overall accuracy is a weighted average — and the weights are whatever your evaluation set happens to contain. If your eval set is 90% group A and 10% group B, a model scoring 95% on A but only 60% on B still reports a comfortable 91.5% overall: Impact of …