The skill-regression test for this system runs every skill file end-to-end. There are 33 of them. Each one is a Markdown file containing one or more bash blocks. The first block, by convention, contains the API call the attorney triggers when they type the skill's name. The remaining blocks are follow-up actions, error-handling branches, conditional flows.
The test harness reads each skill file, extracts the bash blocks, substitutes the documented placeholders (CASE_ID, UPLOAD_ID, et cetera), runs each block, and reports the outcome. For a long stretch of releases — every run — the same five skills produced what looked like harness-side errors. The error messages were syntactic: SyntaxError: Expected ',', got 'const', with carets pointing at a specific column in the extracted snippet.
The cause looked obvious. The bash blocks contained embedded JavaScript with single and double quotes nested in elaborate ways. The harness extracted the block by reading from the ```bash line to the closing ```, but the extraction couldn't always tell where the bash double-quoted argument to node -e "..." ended. Some of the snippets contained things like '"' — a literal JS double-quote inside single quotes — and the harness's split-on-quote logic broke on those.
The classification stuck: harness artifact. False positive. Not a real failure. We moved on.
This week we tested every skill end-to-end without the harness in the way. The procedure was simple: extract each skill's first bash block by reading the file with awk, substitute the placeholders with sed, write the result to a tempfile, run it with bash, capture the exit code and the first line of output. No regex on the embedded JavaScript. No quote-splitting heuristic. Just bash tempfile.sh.
Four of the five previously-failing skills now passed. Their snippets had been clean all along; the failures had been the harness's quote logic.
One of them — the tasks skill, which is what the attorney triggers when they type "show tasks for case X" — still failed. Same error message as before. SyntaxError at column 86 of the extracted JavaScript.
This time we couldn't blame the harness. The bash block had been extracted by reading file lines, with no quote-parsing involved. The error was from Node itself, parsing the -e argument bash had passed it.
The line that broke
The skill builds a small JavaScript function called strip that removes HTML-encoded entities from a task description before display. The line looked like this:
const strip=s=>(s||'').replace(/<[^>]*>/g,'').replace(/&/g,'&').replace(/</g,'<').replace(/>/g,'>').replace(/"/g,'"').replace(/'/g,"'");The whole expression is one argument to node -e. The shell command looks like node -e "...const strip=...;...". The entire JS body is wrapped in bash double quotes.
Inside those double quotes, there is the JS literal '"' — a JavaScript double-quote inside JavaScript single quotes, representing the replacement target for ". Bash doesn't see JavaScript. Bash sees: the opening " on the node -e "..., then a long string of characters, then a ". The " inside the JS literal closes the bash-double-quoted argument early. Everything after that " is parsed as new shell tokens.
What bash did was pass to Node a partial argument: everything up to the first inner ". Then bash treated the remainder of the line as additional words. Node saw an incomplete JavaScript expression — the const strip=s=>... line cut off mid-statement — and SyntaxError'd on parse. The error pointed at where the parsing broke, which was almost exactly where bash had truncated the argument.
This had been the failure mode since the skill was written. Every time an attorney typed "show tasks for case X" and the model triggered the tasks skill, this line of bash would be issued, and Node would crash with a SyntaxError. The model would see the error in stderr, probably try to recover by retrying or routing the question through a different skill, and the attorney would get something — but not what they asked for. The visible behavior was probably "the model takes a long time on this question" or "the model returns less detailed task info than expected," not "the system crashes" — because the harness was deflecting the failure into stderr and the recovery path was hiding it.
What the harness was hiding
The harness had been correctly reporting that the line failed. We had been incorrectly classifying that report as a harness artifact, because four out of five of the apparently-identical reports actually were harness artifacts. The classification was based on a pattern-match: "looks like nested quote parsing went wrong, must be the harness's extractor." The fifth report had the same surface but a different cause.
The lesson is one I'm going to keep with me. A test harness that produces false positives makes every real positive harder to recognize. The signal-to-noise ratio is the actual property of a test suite; pass/fail counts are downstream of whether you can trust which is which.
The four harness artifacts had a specific shape. Each one was a snippet that, when extracted by the harness's regex, lost some characters and became un-parseable. The error messages were similar across all four because they all hit the same kind of parse break. So when the fifth report arrived with the same error message and the same surface shape, the classification routed it to the same bin.
The actual investigation procedure should have been: for each failure, write the extracted text to a file, hand-inspect it, and decide whether the failure was extraction-side or content-side. That investigation takes maybe a minute per failure. Multiplied by five failures, multiplied by every regression run we'd dismissed, that's a lot of minutes that turned into "we have a known broken skill that ships in every release." Five minutes of work per release would have caught it.
The fix
The fix for the immediate problem is small. The JavaScript expression doesn't need the literal '"' form. It can use String.fromCharCode(34), which evaluates to the same single-character string. Same semantic. No shell-meaningful characters anywhere in the bash-double-quoted argument.
const strip=s=>(s||'').replace(...).replace(/"/g,String.fromCharCode(34)).replace(/'/g,String.fromCharCode(39));Verified live: the skill now runs. tasks against a real case returns the real tasks with priority, status, due date, and owner. Three tasks on the test case. Two of them are due dates from a few days back; one is from years ago and overdue. The skill works.
The harness gets a separate fix. The regex-based bash-block extractor is being replaced with a parser that knows where bash double-quoted strings actually end (escaped quotes, single-quote-protected sections, etc). Until that ships, the safer interim has been to run extracted snippets through actual bash, which is what this round of testing did — and is what surfaced the real fifth bug.
The other 31 skills
The same end-to-end run produced an unusually quiet report otherwise. Twenty-seven of the thirty-two testable skills (five are skipped — the private shared partials, the interactive walker, the cron-config docs, the exploratory endpoint probe, the directory-setup helper) ran cleanly against real cases. One additional failure surfaced: the read-doc skill calls a Merus endpoint that the firm's current API token doesn't have permission for. That's a Merus-side issue, not a code bug — the walker, which uses the same endpoint, already handles the 403 gracefully ("Could not fetch activities" then back to the menu). Both code paths are correct; the API permissions need to be requested separately.
The walker itself was tested next: boots cleanly, identifies the writer (the user the API token belongs to), fetches the firm's 788 open tasks, accepts K (keep) and S (skip) inputs, handles the 403 on the D (review docs) option without crashing, quits cleanly on Q. The 8-option control surface from the original walker design is intact.
One real bug found from 32 skills. One environmental issue. Otherwise clean. That's a better result than the harness had been reporting for weeks — the system was healthier than we thought, and the one real bug was the one we'd been ignoring.
The version on the install URL is 4.5.67. The work continues at the pace of one verified release at a time, and the regression suite is being rebuilt with a less-noisy parser. The next round of skill testing should produce reports we can actually act on without a fresh investigation per failure.