By the end of the previous day the system had shipped twenty-four verified releases. The original guards, the single-use override tokens, the rollback runner wired into every production write flow, the audit log rotation, the static analyzer that prevents content-interpretation regex from creeping back into the codebase, the task-delete guard moved to fire before the GET/POST split — all of it was in production and the regression suite was passing forty-six assertions cleanly.
The instinct after that kind of day is to declare the system done and stop. The instinct after twenty-four releases is to write a post saying the work converged. Neither instinct serves a workers' compensation case management tool that runs against a live production API on behalf of attorneys whose clients are injured workers waiting on benefits. The correct next move was to test more — not the safety mechanisms in isolation, which we had already exercised, but the skills the attorney actually triggers in a normal day. Status. Health check. Morning brief. Deadlines. Parties. Billing. The walker that steps through open tasks one at a time. The sequence runner against real records on a real case, with real rollback.
This post is about what that extended regression found. Two bugs. One in a skill that an attorney would have hit on the first day someone said "what are the costs on this case." One in the audit log itself — a class of write attempt that was disappearing entirely from the compliance trail.
The skill that called endpoints that don't exist
The first bug surfaced from a sweep that ran every read-only skill end-to-end against a real test case. Most of them returned sensible data. The billing skill returned a generic error: "Your account does not have sufficient privileges. Your Firm Administrator can adjust your permissions."
That message looks like a permissions problem. It is not. We had learned earlier in the project — and saved into the project's persistent memory — that the underlying API returns its generic "insufficient privileges" string when the endpoint you called does not exist at all. The error is a catch-all for unknown routes, not a signal that an admin needs to grant you something. Without that piece of context, the obvious response to the error message would be to open a ticket with the firm's admin, wait, and discover after the wait that the ticket cannot be resolved because the thing being requested does not exist.
The skill was calling two endpoint names that the API does not serve. A path that had an extra word in it ("Open" appended where it should not have been) and a different path that had been imagined from another product's documentation and never existed in this product at all. The correct endpoint name was already in the project's reference documentation. The skill had drifted off it. Nothing surfaces a drift like this until the skill actually runs against the API — which had not happened during normal development because most ledger-tracking is firm-wide work nobody had a reason to trigger.
The fix is the obvious shape: use the endpoint name the API actually serves, and filter by case on the client side because the server's filter parameter is silently ignored. Verified against two cases — one with ledger entries that came through correctly with type, amount, date and description, and one that legitimately had no entries. Both correct.
The lesson is older than this project: a generic error message that masks what is actually wrong is worse than a specific error message that points at the cause. The API's choice to collapse "endpoint missing" and "permission denied" into one string costs every caller a debugging round when they encounter it the first time, and risks sending firms in pursuit of resolutions that cannot exist. Defensive engineering against this looks like a memory entry that flags the error string as ambiguous, and a habit of probing the literal endpoint to confirm it exists before assuming the error is about permissions.
The audit gap
The second bug is the one that mattered more. It surfaced while testing the sequence runner against the live API, which had been verified in earlier releases but only against synthetic in-memory steps. We wanted the live version.
The pattern under test is a routine one in the production flow. The runner creates a new record. Some later step fails. The runner walks back through the completed steps and runs the compensating undo for each. For tasks, the compensating undo is a soft-delete: tasks/edit/<id> deleted=1. The runner had this pattern documented and the verification had passed in unit tests. The live test was meant to confirm the pattern under real network conditions against the real database.
The test ran. The runner returned ok: false, undone: [...] as expected. The task it created was gone afterward — we verified by re-fetching its ID and seeing the API report it as deleted. The compensating write had succeeded at the database level. So far, normal.
Then we read the audit log. The task creation was logged. The compensating soft-delete that followed was not. The audit trail showed the create but had no record of the undo.
This is the kind of finding that demands an investigation before it demands a fix, because the explanation matters more than the symptom. The audit log helper is well-tested. It does not silently drop writes. The runner explicitly calls the audit helper after each step. Something else was happening.
What was happening turned out to be a quirk in how the underlying API responds to the soft-delete operation. When you set deleted=1 on an edit of a not-yet-deleted record, the API both performs the deletion AND returns its "this record has been deleted" string as an error response. The error is not a refusal. It is the acknowledgment of the soft-delete, returned through the same channel the API uses for actual errors. The HTTP status is 200. The body contains an errors array.
The fetch helper saw the response, found errors in the body, printed the message to stderr, and called process.exit(1). The audit log call was the next statement after that exit branch — it never ran. From the audit trail's perspective, the write was never attempted. From the database's perspective, the write succeeded.
This is the kind of bug where the code is correct in isolation and wrong in composition. The error-check branch was right to print a diagnostic. The audit-log call was right to come after error handling — most writes you want to log are accepted writes. The two correct local decisions composed into a global property the system did not actually have: every write attempt is in the audit trail. The audit trail had a hole.
Why this is a legal-grade problem
For a system that operates on workers' compensation cases on behalf of injured workers, the audit log is not a debugging aid. It is the record of what was done, by whom, and when — the answer when someone asks, in deposition or in front of a judge, what actions the firm's tooling took on a given file. The audit log is not allowed to have invisible attempts.
An attempt that the API rejected is still an attempt. It tells you the attorney's intent at that moment: I tried to do X. The fact that the API responded "no" or "already done" does not change the intent. If the intent is invisible in the trail, a later reconstruction of what happened to a case will be subtly wrong in a way that's hard to detect — the visible trail will be the trail of accepted operations, and the rejected ones will exist only in stderr output that may or may not have been captured.
The fix is small. The audit call now runs before the error-check exit. Each entry has an explicit outcome field: accepted when the API accepted the write, rejected when the API returned a body-level error, with the API's error message preserved on the entry. The audit log is now closed under every write attempt that left the local process — accepted, rejected, refused by guard. There is no path from "the model called the fetch helper" to "no entry exists" except network failure before any HTTP request was sent.
The forward-path quirk that fell out
Fixing the audit hole made a second, related issue visible. Some sequences use forward soft-deletes — not as rollback compensation but as part of the planned operation. The "fix a misfiled record" flow does this: it soft-deletes the wrong-tagged activity and creates a fresh one with correct tags, in a single sequence with rollback if either step fails. The forward soft-delete uses the same deleted=1 pattern.
The undo path in the sequence runner already had logic to treat the API's "has been deleted" response as success — the desired post-condition (the record is gone) is met regardless of which code path the API used to acknowledge it. The forward path did not. So a sequence whose first step was a soft-delete and whose second step was the recreation would see the first step's response as a failure, declare the sequence failed, and roll back nothing (because there was nothing yet to roll back). The user would see "sequence failed" while the database showed the soft-delete had actually succeeded.
This had not been seen in production yet because the affected flows had been built but not run live on a record where the soft-delete was the first step. The unit tests passed because they ran against synthetic responses. The bug existed in the gap between the unit-test view and the live API's quirks.
The fix is symmetric with the existing undo-path logic. When a forward step has deleted=1 in its arguments and the response matches the "already gone" signature, the step is treated as success. Same detection rule, same justification: the desired post-condition is met. Verified by pre-creating a task on a real case, running a sequence that forward-soft-deletes it and creates a replacement, and confirming both steps succeeded, the original was gone, the replacement existed, and the audit log captured both attempts with their correct outcome tags.
What testing more bought
The version on the install URL the previous day worked. The forty-six-assertion regression suite passed. Every guard fired correctly in isolation. None of the bugs found this morning would have surfaced from the previous day's testing because the previous day's testing did not run every skill end-to-end against the live API, and did not exercise the live database's actual response shapes for the operations the sequences perform.
"Test more" sounds like a tautology of an instruction. The thing it actually did was shift the testing surface from "do the safety mechanisms work in isolation" to "do the skills the attorney triggers, against real cases, end-to-end, leave the world in the right state and the audit trail closed." The mechanisms were verified. The composition of the mechanisms with the live API was not. The composition was where the bugs were.
Two fixes shipped in 4.5.39. A real bug fixed in the billing skill in 4.5.38 that would have looked like a permissions problem to the next attorney who tried it. The combined regression: thirty-two skill files parse cleanly, twenty-five read-only skills run end-to-end against a real case, the walker boots and handles every option including graceful degradation when one API endpoint returns rate-limited errors, the sequence runner verified with real Merus writes for both the rollback path and the happy path, the override token's --exec flow verified end-to-end with the token never appearing on the terminal, the file lock serializing ten concurrent setBinding calls cleanly, the static analyzer passing zero violations.
The work continues at the pace of one verified release at a time. The version on the install URL is 4.5.39. The next test is whatever the next attorney run surfaces.