The story of the guards we built earlier this week has a tidy shape: a misfile happened, we moved the safety property from prompts to code, the misfile cannot happen again the same way. That story is true. It is also incomplete.

Code-level guards have their own failure modes. A guard that refuses a write under one condition can let a write through under a related condition the author did not consider. A guard that checks state at time T can be wrong about state at time T+1. A guard that compares two values can be defeated by a subtle ambiguity in the comparison itself. These aren't bugs in the architecture — they're the shape of the architecture under closer inspection.

After the first batch of guards shipped, we did a structured audit: every assumption the code made, every place it could be wrong, every condition we hadn't tested. The audit produced a list of weaknesses. This post is about the four most important ones and how we addressed them in version 4.5.18 through 4.5.21.

1. Tag guards refused on certainty, not on mismatch

When the binding extractor reads a PDF, it picks an activity tag from the firm's catalog and reports a confidence: high, medium, or low. The original tag guard refused writes only when the certainty was high. Medium-certainty and low-certainty mismatches passed through with a stderr warning.

The reasoning at the time was reasonable: if the model isn't confident, the attorney's tag choice should win. The reasoning was also wrong. The model is least confident exactly on the documents where the tag matters most — ambiguous documents, hybrid documents, documents that straddle two categories. Letting low-confidence mismatches through with a warning meant the writes most likely to be wrong were the ones getting through.

We made it three-way. High-certainty mismatch refuses with exit code 3 (the existing behavior). Medium-certainty mismatch pauses with exit code 4 and prints a structured marker the calling skill can parse:

MERUS_FETCH_RESULT={"kind":"medium_tag_mismatch","upload_id":"...","expected":[...],"proposed":[...]}

The calling skill — mail processing, message processing, the task walker — is supposed to recognize exit 4, pause its loop, and prompt the attorney for explicit confirmation before retrying with an override token. Low-certainty mismatches still pass with a warning, because at that confidence level the model genuinely cannot help and the attorney is the better judge.

This is a small change. It also catches an entire class of misclassification — the borderline cases where the model is "pretty sure but not certain" — that the original guard let through.

2. Two terminals writing the same file would clobber each other

The binding file at ~/.aaicase/queue/bindings.json is read by every guard and written by every binding extraction. The original write pattern was read-modify-write: load the whole file, mutate the entry for one upload, write the file back.

This works as long as only one process is writing at a time. The instant two are — say, the attorney runs mail processing in one terminal while an audit-case sweep runs in another — they race:

Process A reads file (10 bindings)
Process B reads file (10 bindings)
Process A adds upload 100's binding, writes file (11 bindings)
Process B adds upload 200's binding, writes file (11 bindings) — clobbers A

Last writer wins. Upload 100's binding is silently lost. The next time a guard tries to verify upload 100, it sees no binding and refuses, even though aai-bind ran successfully. The attorney has no way to know the binding was lost; they just see a confusing "no binding" refusal on an upload they remember having bound.

We added an advisory lockfile around every bindings write. The lock is created with O_EXCL — the create operation is atomic at the syscall layer, so only one process succeeds. Others spin-wait briefly and retry. A 30-second stale-lock detection handles the case where the holding process crashed mid-write.

To prove the test caught the race, we ran the same stress test against the pre-fix code path and watched it lose 4 out of 10 parallel writes. With the lock, all 10 survived. We ran it again at 25 parallel writes: 12 lost without the lock, 0 lost with it. Stale-lock recovery clocked at 2 milliseconds. Lock contention measured 5 milliseconds from release to acquire.

Concurrent writes are not a theoretical problem. Mail processing and audit-case both write bindings, and an attorney can have both running in different terminals. The lock means they can.

3. The override switch was too powerful

Every guard had an escape hatch: set AAI_FORCE_BINDING_OVERRIDE=1 in the environment and the guard would let the write through with a stderr log entry. This existed because there are legitimate edge cases — a historical misfile being intentionally moved, a document where the extractor returned an ambiguous applicant. The override gave the attorney a way out.

The problem with a boolean environment variable as an override is that it's too powerful. Once set in a shell, every subsequent write in that shell bypasses every guard. The attorney enables it for one specific legitimate override, then forgets it's set, and the next typo silently bypasses guards on a different upload.

We replaced it with a per-write signed token. The new flow:

  1. Attorney encounters a guard refusal they need to bypass.
  2. They run aai-override with the guard kind and the specific parameters (upload_id, case_file_id, etc).
  3. The CLI prompts them to confirm, then mints an HMAC-SHA256 token bound to those exact parameters. The secret lives in ~/.aaicase/override-secret with mode 600.
  4. Attorney runs the original command with AAI_OVERRIDE_TOKEN=<the-minted-token>.
  5. Guard verifies the token matches the kind and parameters of the current write. If yes, bypass once. If no, refuse.

The same token used on a different upload, a different case, a different filename produces a signature mismatch and the guard refuses. Tampered tokens are rejected. Tokens are not time-bound — they're bound by parameters, so they're naturally scoped to one specific operation. Every mint, every redemption, and every rejection is audit-logged with the token's first 8 bytes for traceability.

During verification we found a subtle bug. Base64url encoding has unused bits in the last character — for a 32-byte HMAC, the token is 43 characters and the last character only uses 4 of its 6 bits. That means two different last characters can decode to the same 32 bytes. The signature comparison happens after decoding, so a tampered last character would have passed. We added a canonicality check that requires the input token to match the canonical re-encoding of its own bytes. Non-canonical encodings are rejected before the signature check.

The old boolean still works for backward compatibility, but emits a deprecation warning on every use. Skills and scripts are migrated as they're touched.

4. The Windows surface was untested

The attorney who started this whole sequence — by reporting the misfile that motivated the first guards — runs aaicase on Windows. Every guard we built since then was tested on macOS. None of it was tested on Windows. This is the kind of fact you don't want to be true and might prefer not to look at.

We did a static audit. Three real Windows-specific bugs:

Atomic file replacement. POSIX rename atomically replaces an existing file. Windows MoveFile without the REPLACE_EXISTING flag throws if the destination exists, and Node's renameSync doesn't pass REPLACE_EXISTING by default. This means every binding write after the first would have failed on Windows with EEXIST. The first binding extraction would succeed; the second would silently fail, and the attorney would see "no binding" refusals on every subsequent upload.

Fixed with a small helper that unlinks the target file first on Windows before renaming. The Windows path loses true atomicity (a reader could observe the missing target between unlink and rename), but combined with the advisory lock from fix #2, no other writer will be racing, and a reader catching the empty window just sees "no file" and returns the same empty state it would have seen if the read happened before the write started.

Subprocess invocation. The binding extractor and three related tools spawn claude as a subprocess. On macOS, claude is a symlink to a JavaScript file that Node can spawn directly. On Windows, claude is a .cmd shim that Node's spawn can't resolve without explicitly enabling shell mode. Without that, every Claude subprocess call would have failed with ENOENT — meaning no PDF extractions, no email classifications, no analysis subprocess could run.

Fixed with platform-conditional spawn options: shell: true on Windows, default on POSIX. The behavior is identical on macOS and Linux.

ANSI color codes. The task walker renders cards with ANSI escape codes for color and emphasis. Windows Terminal supports these; cmd.exe in some configurations does not. The walker now auto-detects: colors disabled when stdout isn't a TTY, when NO_COLOR or AAI_NO_COLOR is set, or when TERM=dumb. Output is clean text in those cases.

To verify these fixes — and to give a working diagnostic to the attorney to run on his actual machine — we wrote a 14-check compatibility script that ships in the tarball. He runs node scripts/test-windows-compat.mjs once after install. Green means the Windows surface is good. Red means the failed line points at the still-broken fix. We ran the script on macOS as a sanity check and got 13 of 14 green — the only red was "process.platform is win32", which correctly fails on darwin.

The pattern

Each of these four improvements addresses a weakness the original guards did not. None of them is glamorous. Each one is a small, specific, verifiable fix to a problem that becomes visible only after the first layer of safety is in place.

This is the second pass. The first pass solves the obvious failure mode. The second pass asks: under what conditions does the first pass not solve the failure mode? Tag certainty matters, not just the binding itself. Concurrent writes can lose data, even when each write is correct in isolation. A global override is too powerful for the legitimate use case. The platform we don't run on is still a platform we ship to.

You can keep doing second passes indefinitely — third passes, fourth passes, audits of the audits. At some point the marginal weakness is small enough that the work to close it isn't worth the cost. Where that line sits depends on what the system is doing. For an AI assistant that writes to a workers' compensation case management system, where misfiles harm clients and missed deadlines produce malpractice exposure, the line sits further out than it would for a generic productivity tool.

The list of weaknesses we identified isn't fully closed. There's still no rollback path for multi-step writes — if a sequence of file → activity → task partially fails, there's no atomic undo. The update-task parser silently drops unparseable fields rather than reporting them. The reassign picker can confuse staff who share first names. These are all known. They're scheduled.

The discipline we've adopted: every release ships with verification evidence, and every weakness audit becomes a backlog. We don't claim more safety than we've verified. We don't write blog posts about properties the code doesn't have. We don't ship a fix without proving the test catches the bug it's supposed to catch.

Four improvements, one day. Mostly mechanical work. None of it would have surfaced without the audit, and the audit only happened because the first layer of guards made it possible to ask "now what's still wrong?" instead of "is anything right yet?"

That's the architecture, in the smallest possible form. Build the guarantee. Audit the guarantee. Fix what the audit finds. Audit again.