Task-Specific Evaluation: Code, SQL, and JSON Correctness

If you're fine-tuning a model to write code, generate SQL, or extract structured data, you have access to something Blue and Bert Score can only dream of. Ground truth you can execute. A function either passes its tests or it doesn't. A query either returns the right rows or it doesn't. A JSON field either matches the schema or it doesn't. No human judgment required. No embedding model in the loop. No reference text to argue about. Welcome back to the fine-tuning series. In previous videos, we built the evaluation stack and went deep on text similarity metrics, blue, rouge, meteor, BERT score. Today, we go one level more specific. taskspecific verifiable metrics, code correctness, SQL execution accuracy, and F1 scoring on JSON field extraction, plus the general framework for writing a custom evaluation script for whatever your domain actually is. These are, in my experience, the single most reliable signal you'll get from any automated eval. Let's build them. Let's establish why these three are different from everything in previous video. Three properties make task specific metrics special. First, they're binary and objective. A test passes or it doesn't. A query returns the right rows or it doesn't. There's no subjective threshold like is 0.87 BERT score good enough. Second, they're immune to phrasing. A correct SQL query that uses a different but equally valid algorithm still passes. Third, they're cheap. Running a unit test or executing a query takes milliseconds and zero GPU time. Compare that to calling an LLM judge on every example. The decision rule for this whole video. If you can write a checker, a function that takes the model's output and returns pass, fail, or a score with zero subjective judgment involved, write that checker before you reach for any text similarity metric or LLM judge. This is the single highest signal, lowest cost evaluation you can build. Let's start with code. The standard metric for code generation is pass at K. and getting it right requires more care than people expect. The setup for each coding problem, you generate K completions from the model, then run each one against a held out unit test suite. The naive version of pass at K just checks, did at least one of my K samples pass. But if you generated more than K samples and want pass at K for a smaller K, naively resampling introduces bias. The fix from the original codeex paper is the unbiased estimator. Generate n samples per problem where n is larger than k. Count c of them that pass and compute the expected pass rate over all possible ways to choose k from those n without actually rerunning. The formula and the numerically stable implementation are right here. This exact function is what human eval under the hood. Generate n equals at least 20 samples per problem if you're reporting pass at one more if you want pass at 10 or pass at 100. These numbers track how often the model can solve a problem at all versus how reliably it solves it on the first try. Running model generated code means running code you don't trust and that changes how you build the harness. Never call exec or eval directly on model output in your main evaluation process. A fine-tuned model can generate code that deletes files, opens network connections, or just hangs in an infinite loop. And it will eventually, especially early in training when outputs are noisy, run every generated solution in an isolated subprocess with a hard timeout, restricted file system access, and no network. A lightweight sandbox like a restricted subprocess with resource limits is the minimum bar. A full container per execution is safer if you're running this at scale or on completely untrusted checkpoints. On test design, write multiple test cases per problem, including edge cases, empty inputs, boundary values, large inputs, not just the happy path, and track partial credit alongside the binary pass fail. The percentage of test cases passed per problem tells you whether a failing solution is close or completely off track, which is useful signal you lose if you only log pass at K. For text to SQL, the naive evaluation mistake is comparing the predicted query text to a reference query text and it fails constantly because SQL has enormous syntactic freedom for the same semantic result. Select name, age from users and select age, name from users return the same information in different column order. A query with an inner domain written one way and the same join expressed as a wear clause can be functionally identical with completely different text. Blue or exact match on the query string will score these as wrong even when they're correct. The standard fix used by benchmarks like Spider is execution accuracy. Don't compare the query text at all. Execute both the gold query and the predicted query against the actual database and compare the result sets they return. If the results match, the prediction is correct regardless of how differently the SQL was written. The code here runs both queries on a SQLite connection and compares the returned rows. Execution accuracy sounds simple until you hit the wrinkles that make a naive implementation wrong. Row order. A select without an explicit order by can legally return rows in any order. So compare result sets after sorting both or compare as multis sets not as ordered lists. Column order. Select a comma b and select b comma a are equivalent. Compare by column name not position. Floating point. Aggregate functions like AVG can produce 3.0 in one engine and 2.9999.998 in another. Use a numeric tolerance, not exact equality. Nulls SQL's three valued logic means null doesn't equal null in a naive Python comparison the way you'd expect. Handle it explicitly and separate execution errors from wrong result errors in your logging. A query that fails to even run tells you the model produced invalid syntax. A query that runs but returns the wrong rows tells you the model misunderstood the question. Those need different fixes. For structured extraction, pulling fields out of a document into JSON. Exact match on the whole object is too punishing. One wrong field zeros out an otherwise perfect extraction. The standard fix is field level precision recall and F1. Treat every field value pair the model got right as a true positive. Every field it predicted that's wrong or extra as a false positive. And every field it missed as a false negative. Precision is correct predicted fields over all predicted fields. Recall is correct predicted fields over all fields in the gold answer. And F1 is the harmonic mean. This gives you partial credit and tells you whether the model's failures are mostly missing fields, a recall problem, or mostly wrong values, a precision problem. Before scoring field values at all, run a schema validation pass. Does the output even parse as JSON? Does it have the required keys? Are the types correct? report that as its own number, your schema validity rate, separate from F1, which should only be computed on the outputs that passed validation. Conflating the two hides whether your model's biggest problem is formatting or accuracy. Real JSON isn't flat, and a naive field comparison breaks on nested objects and arrays. The fix is to flatten both the gold and predicted JSON into dotted path keys before comparing. address city, address.zip, items.0.sku. So, a single wrong value three levels deep costs you exactly one field, not the entire nested object. The flattening function here does this recursively, handling both nested objects and arrays. Two more refinements matter in practice. First, optional fields. If a field is genuinely optional in your schema, the gold answer omitting it shouldn't count as something the model needs to predict. Only score fields the schema actually requires. Plus, whatever optional fields appear in this specific gold example. Second, value comparison needs to be type aare. Exact match makes sense for ids, enums, and dates, but free text fields need normalization. case, whites space, trailing punctuation before you call them a mismatch. And numeric fields need a tolerance, not exact equality. Define this comparison logic once per field in a small config and reuse it across your whole eval set instead of hard coding special cases. Per example, code, SQL, and JSON are three instances of the exact same four-step pattern. And you can apply this pattern to whatever your domain actually is. Step one, define a checker function. Output goes in, a score or pass fail comes out with zero subjective judgment and ideally zero LLM calls. Step two, decide your aggregation. Binary pass rate when partial correctness doesn't mean anything, like a SQL query that returns wrong rows. field or test level F1 when partial correctness is meaningful like JSON extraction or a multi-est calling problem. Step three, separate couldn't even attempted failures, parse errors, syntax errors, schema violations from attempted but wrong failures. They're different problems requiring different fixes and collapsing them into one number hides which one you actually have. Step four, version your checker like code. A bug fix in your checker function run against an old checkpoint will look exactly like a model regression if you're not tracking which checker version produced which score. If you can write down what correct means for your task in one sentence, you can probably write a checker for it. Most teams skip straight to an LLM judge for tasks like this, which is slower, costlier, and strictly less reliable than executionbased ground truth whenever ground truth is actually checkable. Five places custom checkers quietly go wrong. All of them I've personally shipped at some point. One, checker bugs masquerading as model regressions. If your checker score drops sharply between checkpoints, your first suspect should be a recent change to the checker itself, not the model. Unit test your checker against a small set of known correct and known incorrect examples, the same way you'd test any other code. Two, equality that's too strict. Exact string match on a free text field where capitalization or trailing punctuation legitimately varies will report failures that aren't real failures. This is the JSON field problem from two slides ago showing up again. Three, eval leakage. If your coding problems or SQL questions overlap with anything the model saw during training, your pass rate is measuring memorization, not capability. Keep your task specific eval set strictly held out. The same discipline as any other eval set in this series. Four, non-determinism code with randomness, SQL with no order by or floating point comparisons can make the identical model score differently across reruns. Fix random seeds where you can. Use tolerances where you can't. And if a task is inherently noisy, run it multiple times and report a mean and standard deviation instead of a single number. Five, sandboxing gaps. If generated code can read or write files or reach the network during your eval run, you have both a security problem and a repeatability problem. A passing solution that happened to read a cached file won't pass the same way next time. Here's how this fits into the pipeline you already built in part 9 and part 10. One dispatcher function, three taskspecific checkers, all logged the same way. The run task eval function takes a task type and routes to the right checker. Code problems go through the sandboxed pass at Krunner. SQL problems go through the execution accuracy comparison. JSON extraction goes through schema validation plus field F1. Every result, pass rate, execution accuracy, field F1, plus the error type breakdown from each logs to the same experiment tracker as your blue, rouge, and BERT score numbers from part 10. This is the metric you should gate deployment on whenever it's available. It's also exactly the kind of check that belongs in your CI regression suite from part 9. Task specific correctness checks are some of the highest value regression tests you can write because a regression here means the model got measurably worse at the thing you actually fine-tuned it to do. Seven takeaways. One, when the ground truth is checkable, prefer a deterministic checker to any similarity metric or LLM judge. It's faster, cheaper, and more reliable. Two, pass at K uses the unbiased estimator, not a naive count when you're sampling multiple completions per problem. Three, always sandbox generated code execution, hard timeouts, no inherited environment, no network access, never exec model output directly. Four, SQL correctness should compare executed result sets, not raw SQL query. Handle row order, column order, floating points, and nulls explicitly. Five, JSON extraction should be scored with field level precision, recall, and F1 after schema validation, not whole object exact match. Six. Separate couldn't even attempted failures, parse errors, syntax errors, schema violations from attempted but wrong failures. They're different problems requiring different fixes, and collapsing them into one number hides which one you actually have. Seven, unit test your checker itself, version it alongside your model checkpoints, and keep your task specific evolve problems strictly held out from training data. That's tasksp specific evaluation, code correctness, SQL execution accuracy, and JSON field F1. Three checkers built on the same idea that when correctness is checkable, you should check it, not approximated. your action item. Pick the one task in your pipeline where you currently rely on a text similarity metric or an LLM judge, but where correctness is actually checkable. A format requirement, an executable query, a value that should match exactly and write a deterministic checker for it this week. It will be the most reliable number in your entire dashboard. resources, the human eval repos for code eval patterns, including the pass at K estimator, the spider benchmark for text to SQL execution accuracy, and JSON schema for JSON validation. Like the video if it was useful. Subscribe for the rest of the series and drop a comment with a task in your domain you think needs a custom checker. I'll cover the best ones in a future video. Write the checker before you reach for the judge. See you in the next one.