One of the small tools in this codebase classifies text records using an LLM. The caller passes in a JSON array of records — each with an id and some text — and gets back a corresponding array of classification objects, each with the same id plus a boolean and a brief reason. The downstream code matches inputs to outputs by id and acts on the matched pairs.

The pattern is correct. The implementation had three silent failure shapes that the probe round surfaced this week. Each shape allowed the caller to receive a confidently-empty or partial result that wasn't actually wrong-shaped enough to throw.

Shape 1: typo'd task name + empty input

The tool accepts a --task argument naming one of four canonical tasks (find PD rating mentions, find TD mentions, find doctor records, find client-contact notes). Task names are checked inside the classify function, which is only called if there's something to classify. Sequence:

// Empty input → empty output, don't waste a claude call
if (records.length === 0) {
  console.log(JSON.stringify({ results: [] }));
  return;
}

// Chunk into 50-record batches
for (let i = 0; i < records.length; i += CHUNK) {
  const slice = records.slice(i, i + CHUNK);
  const results = classify(task, slice);  // task validated here
  allResults.push(...results);
}

The "don't waste a claude call" shortcut is reasonable on its own — empty input means empty output. But it sits in front of any task-name validation. A caller invoking the classifier with a typo in the task name (--task client-contat instead of --task client-contact) on a freshly-loaded empty array gets back the legitimate-looking response {"results":[]} and exit code 0. The caller treats it as "no client contacts found in this case."

The fix is to move the task validation up — check the task name immediately after argument parsing, before the empty-input shortcut. Bad task now exits 1 with a typed error listing all valid task names. The empty-input shortcut still works for legitimate empty cases.

Shape 2: input records without ids

The output schema requires every result to have an id. The caller matches results back to inputs by id. The implicit assumption: every input record has an id.

The implementation didn't check. If a record came through without an id field, it got forwarded to Claude with whatever other fields it had. Claude saw the schema requirement and either invented an id (frequently using the array index or making up a string) or omitted the record from its output. Either way, the caller's match-by-id failed silently — the input's not-an-id couldn't match anything in the output, so the input's row showed as "no classification found" even though Claude DID classify it.

The fix is an input-validation pass that checks every record has a non-empty string id, dies with the index of the first offender. This is the kind of defensive check that costs almost nothing and prevents a class of silent failures that are otherwise nearly impossible to debug.

Shape 3: dropped records in Claude's output

Claude is usually accurate but not always. For a 50-record batch, occasionally Claude will return 49 results — dropping one for reasons that range from "got confused by the structure" to "decided the record was malformed and skipped it." The schema doesn't forbid this; the schema just requires that each result OBJECT has its required fields. A response with 49 results matching the schema is schema-valid.

The classifier code didn't check. It pushed whatever Claude returned into the final array. The caller saw N inputs and N-1 outputs, but since the caller matches by id, the missing record's id just doesn't appear in the result set — same shape as Shape 2 from the caller's perspective. Silent gap.

The fix is a post-classify verification: build the expected-id-set from the input slice, build the returned-id-set from Claude's output, fail loud if any expected ids are missing. Use sets (not lengths) so a duplicate output doesn't mask a missing input.

The shape

I keep finding variants of this pattern in code that delegates work to the LLM and gets structured output back. The general shape is two collections, no correlation: the LLM's output is supposed to align row-by-row with the input, and there's an implicit contract that "row K of the output corresponds to row K of the input" or "id X in the output is the classification of id X in the input." The contract is in the developer's head. The code doesn't enforce it.

Every alignment failure shows up as a partial result. The output is technically valid — it parses, the schema passes, no exception fires — but it's misaligned with what the caller assumed. Downstream code then treats the partial result as authoritative and produces wrong answers.

The discipline is to make every alignment step explicit. If the contract is "every input id is present in the output," check it after every classify call. If the contract is "input records must have ids," validate the input before the classify. If the contract is "the task name must be one of N valid names," validate the task name before any code path that produces output.

None of these checks are clever. They're not interesting. The interesting part is that they weren't there, because the code was written assuming the LLM would honor the contract by being smart enough not to break it. LLMs are usually smart enough, which means the contract holds 95% of the time, which means the 5% failure cases sit in production indefinitely producing wrong answers that look right.

The general lesson

When you call an LLM for structured output, the schema is necessary but not sufficient. The schema enforces "each result has these fields" — it doesn't enforce "the result set covers the input set." That second check is the caller's responsibility, and it has to be explicit code, not an implicit assumption.

For a legal-grade tool where the classifier is used to decide what gets billed, what gets filed, what counts as client contact, the gap between "schema-valid output" and "correct alignment with input" is a real failure surface. The fixes shipped in this release close three specific instances. The general principle they encode is: if your code calls an LLM and processes the output, every assumption you make about the output's relationship to the input has to be a check the code runs and an error the code surfaces.

The version on the install URL is 4.5.104.