Validation Set vs Test Set: The Student Who Aced Every Practice SAT But Bombed the Real One

You tuned your model using a "test set" and got 95% accuracy. You deployed it. Production accuracy was 82%. What happened? You used your test set as a validation set — peeking at the final exam while studying. The test set must be touched ONCE. Here's why and how this distinction saves your model from false confidence.