logosmonn

What earns a check in AI-assisted coding

Published by Simon Ingeson

Most rules in my setup went through the same three stages. First, I wrote nothing down and corrected the agent every time I saw the mistake. Then I added the rule to the prompt. That mostly worked, until the agent forgot it in a long session and I was correcting it again. Finally, it became a check. I stopped reminding, and the agent stopped needing to remember.

The rule didn’t change. What changed was who had to hold it in their head.

That’s the first of two reasons I encode rules: to cut my mental load. The agent isn’t the only one who forgets. I lose track of a repo’s merge rules, which flag a tool takes, and why a hook exists. When the checks hold the details, I can focus on architecture, the big picture, and what good code looks like.

The second is to shape the output. Without a setup, you get whatever the model wants to do. Checks turn “whatever” into “this is how I want code structured.” They don’t make it deterministic. ESLint with the same config gives the same verdict every time, but two sessions under the same rules still write different code. The checks fix the boundary, not the path.

Halving the gap

Halve the distance, then halve what’s left, and again. In math, you never reach zero. Measure a mile that way, and soon there’s no distance left that a person could measure.

That’s the goal for my setup. I can’t remove the nondeterminism from AI-assisted coding, but I can keep halving it until what’s left doesn’t matter in practice. Every place where the agent still does whatever it wants is a gap, and each gap is the next half. The hard part isn’t closing one, since agents write hooks and tests fast. The hard part is deciding which gaps are worth closing. That’s what earns a check.

A gap shows up in one of four ways, each later than the last.

  • Friction. The agent feels it from inside the workflow, when a hook refuses its command and it has to retry, or when a rule of mine has a flaw it keeps tripping over.
  • Disappointment. I feel it from outside, when the path the agent took, or what it built, falls short of what I expected. At a previous job, the review skill kept flagging flaws that weren’t there. The fix was a second pass, where a fresh agent asked whether each finding was real, and real for the reason the review gave.
  • Another reviewer. I’ve read the diff, the agent has reviewed it, and I stand by it. Then a colleague asks, “did you think about this?” Neither of us had.
  • Production. Something breaks at runtime; worst case, an outage. The incident write-up ends with what changes so it doesn’t happen again, and sometimes that change belongs in the workflow.

The further down the list a gap is found, the more it has already cost.

Weighing a gap

So how much does a gap have to cost before it earns a check? I don’t have a rule for that. I have a scale.

On one side is what the next half costs. Writing the check is cheap (the agent writes it). What costs is everything around it: a new dependency, a change that breaks something else, a setup rework that ripples out, and every false refusal once it ships.

On the other side is how often the gap shows up and how much it hurts. Seen once, small, barely worth mentioning? It goes in a note and waits for later. I call that parking it. The fifth time it comes up, with at least one of those a reverted PR, a production bug, or an outage? The fix may still be expensive, but not fixing it costs more.

Parking isn’t forgetting. Each parked gap gets a trigger, the evidence that would move it up. The cheaper the fix, the lower the trigger. A gap doesn’t earn a check because I’m annoyed once. It earns one when its trigger fires.

Here’s one from my setup. A hook called cd-guard refuses cd in the agent’s shell commands, because a cd moves the working directory for every later call. One day it refused a command that wrote a file through a heredoc. Nothing in the command ran cd. The script inside the heredoc only mentioned it. The agent’s handoff note that day:

Parked because the Write tool is the documented way to create files, so it cost one retry.

Trigger: a second refusal of a heredoc or quoted argument that only mentions cd.

Over the next two days, in a batch of parallel sessions, the trigger fired: “about 9 refusals of heredoc bodies that only mention cd.” Two commits later, cd-guard ignored heredoc bodies. One retry wasn’t worth a fix. Nine were.

Evidence can also say no. I once wanted a hook that warned when the agent ran the same command a third time with no change (a common sign of an agent stuck in a loop). Before building it, I had an agent check 1,919 of my session transcripts. There were 117 cases in 81 sessions, and none came from a command that errored. The repeats were the agent rechecking its work, like rerunning tests after an edit somewhere else. The hook would have nagged the agent for doing the right thing, so I didn’t build it.

Evidence can loosen a check, too. In one week, cd-guard refused 34 calls, most of them a cd into the session’s scratch folder, where agents go to inspect a cloned repo. The rule exists so a stray working directory can’t send a later repo command to the wrong place. The scratch folder holds only throwaway clones, so a misdirected command lands somewhere that doesn’t matter, and cd-guard now allows it.

For me, a check loosens when its reason stops holding.

I should own a bias, though. In the history of my setup, no check has ever gone back to being a prompt. Checks have loosened, narrowed, and moved into a plugin. Two went away entirely.

One denied the agent reads of lock files. It also silently blocked the shell commands that read them for good reasons, with no clean way to tell the two apart. I deleted it. The other denied curl and wget. I dropped it once the harness shipped a classifier that judges those commands. That handed the rule to a model instead of back to a reminder.

When a check stops earning its place, I delete it. Maybe a rule that can’t be checked cleanly was never worth reminding about either. Or maybe I just haven’t met the rule that proves otherwise.

Where the halving stops

I think some gaps never close, because what’s missing is judgment. Review is where that judgment lives. Does the change match the spec? Did it touch files it shouldn’t? Does it add risk?

Not everything review catches needs judgment, though. Some of it is still waiting for the right tool to check it, and finding those is the next half.

A review makes a change less risky. It can’t make it safe, and nothing ever did. Every line of code is a liability. That was true long before AI.

Scope is my clearest example of a gap that stays open. At first, the agent wandered into tangents, and PRs ballooned. So I told the review to keep scope tight and turn anything outside the spec into a follow-up issue. It worked, and then it swung too far. Every small tangent turned into a listed follow-up. It went from no restraint to no taste.

I couldn’t write a check for “does this belong in this PR?” That’s the rule I hadn’t met. So I rewrote the prompt to ask, before filing a follow-up, whether the work belongs here, and to file one only when it’s truly needed. A rule written to fix one failure caused the opposite. The fix was a step back toward judgment, on purpose.

That’s where the halving stops for me. The checks hold the details, so I can hold the judgment.