Applied AI · 5 min read
Choose the model after you define the job
A model comparison becomes useful when it tests the task, the failure you cannot accept, and the conditions in which the system will run.
One of the code reviews in my comparison came back clean: zero comments, marked successful. It hadn’t reviewed anything. Its subtasks had timed out, and the harness running the test had folded ‘finished’ and ‘found nothing’ into a single status. It sat in my results as a pass until the next day, when I went behind the summary file and found every subtask had timed out. A month later the archive showed me a cousin of the same bug: a model I had rejected in early July for leaving zero comments on three diffs had actually been failing to produce tool calls my harness could read. The rejection stood. My reason for it had been wrong for six weeks.
That came from a small comparison I ran in July, and it’s the result I think about most. A score is only as honest as the machinery that produced it. It also changed the order I work in. I now define the job, and the failure I can’t accept, before I look at a single number.
A new model arrives with an impressive score, and the obvious next move is to try it. I’ve been evaluating local models for coding and review workflows, and a general ranking can’t tell me whether a particular model is a good reviewer under my time limits, with my tools, on the kinds of changes I actually need it to examine.
The job needs a definition before it gets a winner.
Name the failure that matters
For a code reviewer, finding plausible things to comment on is only part of the task. It also needs to catch the defect that makes a change unsafe to rely on. A polished review that misses that defect can leave the next person more confident than the evidence warrants.
In that July comparison, I tested two model variants, each with two template configurations, across three code changes. One candidate returned reviews faster and found useful issues, but missed a known issue on a change where the other caught it. The slower candidate also ran into the review tool’s deadlines. There was no clean winner for the job.
That was a limited experiment, not a ranking of those models for every task. Its value was identifying which tradeoff I actually faced: useful findings, missed defects, and completion under the conditions of the workflow.
The surprises go the other way, too. In March, a 2B model running on my own hardware turned out to be fast enough to stay out of the way and good enough to be useful. I didn’t expect that from something so small, and I wouldn’t have noticed if I’d only been reading leaderboards.
An empty review can mean two things
It can mean the model found no issues. It can also mean the model never finished examining the change. Until the test tells those outcomes apart, its summary gives you a misleading comparison.
Inspect what the system actually received, attempted, and returned. Record failed subtasks and partial completion. If one candidate keeps timing out, decide whether you’re measuring its reasoning, the serving setup, or the practical performance of the whole workflow. Each is a useful question. They lead to different conclusions.
Write the decision rule in advance
For a later comparison, I recorded the question, configurations, predictions, and disqualifying behavior before collecting the results. The question was narrow: could a challenger replace the reviewer already serving this machine?
That kept a broad score from becoming the whole decision. A model could perform well overall and still fail a requirement of the reviewer role. The existing model had to pass the same scrutiny. Familiarity doesn’t earn an exemption.
For your own workflow, choose representative tasks and examples of failures you need to detect. Write down what counts as correct, what counts as incomplete, and which result would stop you from using the model in that role. Then run the comparison without rewriting the criteria around the outcome you hoped for.
Compare the system you’ll actually use
The model name doesn’t fully describe the experiment. Inputs, instructions, software versions, resource limits, and simultaneous requests can all change what the workflow delivers. Record those conditions so you can tell what changed when the result changes.
There’s a cost to this care. Small teams can’t build a research program around every tool choice, and they shouldn’t have to. A small, repeatable set of relevant tasks can answer a bounded question. Be equally precise about what it leaves unanswered.
If you use an example to tune the setup repeatedly, add fresh examples before treating the result as evidence that the system handles unfamiliar work. Keep the difficult cases around for regression checks, but avoid teaching to the entire test.
I want an evaluation to end with a usable decision: keep the current setup, change a constraint, assign a narrower role, or keep testing. ‘No clean winner’ is useful when it explains the next move. A leaderboard position, on its own, rarely does.