Thirty Missing Pipefails

The 4.5.117 release fixed graceful degradation in the 13 skills that run parallel API fetches. Each of those skills now categorizes endpoints as required vs optional, captures per-endpoint stderr, and degrades sections that lose their feed rather than bailing on the whole skill. That release closed a real bug shape and shipped clean.

But that fix only covered one class of skills. There's another class — 15 of them — that uses single-fetch pipelines instead:

node bin/merus-fetch.mjs /endpoint | node -e "
  const d = JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));
  // ... process ...
"

The pattern is shorter, simpler, and used everywhere single-step skills only need one endpoint. find-contact uses it to search the contacts index. check-mail uses it to count orphan uploads. process-messages uses it nine times, including once to capture a message ID into a shell variable.

Why the simpler pattern was the more dangerous one

Without set -o pipefail, a bash pipeline exits with the exit code of the last command. If merus-fetch fails and writes nothing to stdout, the consumer's JSON.parse('') throws — and node exits 1. So far so good: the pipeline exits non-zero, the calling harness sees the failure.

But that's the lucky case. The unlucky cases:

1. The consumer wraps JSON.parse in a try/catch and falls back to {}. The consumer succeeds, pipe exits 0, the failure is invisible.
2. The consumer is cat > file.json or similar — it always succeeds on empty input. Pipe exits 0.
3. The pipe output is captured into a shell variable: VAR=$(merus-fetch ... | node -e ...). If the consumer prints nothing to stdout (because it threw), VAR becomes empty. The shell sees no error from the command substitution unless pipefail is on.

Case 3 is the worst. process-messages.md has exactly this pattern:

MID=$(node bin/merus-fetch.mjs /messages/index | node -e "
  const d = JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));
  console.log(d.data['MESSAGE_TABLE_KEY'].message_id);
")

The captured MID becomes message_id=$MID in a downstream task spec that files an activity record. If the fetch fails and MID is empty, the spec writes message_id= — an activity with no message ID. The activity gets created in Merus. The case record gets corrupted. The skill exits 0. Nobody knows anything went wrong.

The fix

Three letters and a flag: set -euo pipefail as the first line of every bash block that contains a merus-fetch | node or merus-search | node pipeline.

With pipefail set, the pipeline exits with the highest non-zero exit code of any command in it. merus-fetch exits 1 → pipeline exits 1 → command substitution propagates the failure → set -e terminates the script. The empty variable never gets used.

The audit found 72 bash blocks across 15 skills, with 30 of them containing pipelines that needed the fix. A small node script walked each file, identified blocks that contained a merus-fetch | node pattern and lacked the set line, and inserted the line as the first body line of the block. 30 additions across 14 files (one more file got its block edited by hand earlier in the session).

Additional hardening in process-messages

For the captured-pipe block specifically — the one that resolves message_id — pipefail alone isn't enough. If the fetch succeeds but the requested message isn't in the index (deleted, filed, or never existed), the consumer would print undefined to stdout and MID would become the literal string "undefined." pipefail can't catch that because the pipeline genuinely succeeded.

The fix for that case is consumer-side validation:

const msg = d.data && d.data['MESSAGE_TABLE_KEY'];
if (!msg || !msg.message_id) {
  console.error('ERROR: message MESSAGE_TABLE_KEY not found');
  process.exit(1);
}
console.log(msg.message_id);

And on the shell side, a post-substitution check:

if [ -z "${MID:-}" ]; then
  echo "ERROR: failed to resolve message_id" >&2
  exit 1
fi

Three independent layers of protection now stand between a failed message lookup and a corrupted activity record: pipefail catches fetch failures, consumer validation catches missing-but-fetched cases, and the shell empty-check catches edge cases where neither of the prior two triggered.

The meta-lesson, refined again

Each round of skill fixes has produced a tighter version of the same rule. The progression so far:

• Grep for the bug expression, not the variable name — the bare-VERIFY regex sweep, which kept missing copies because the names were different but the pattern was the same.
• Per-skill testing surfaces bug shapes you didn't know to grep for — the parallel-fetch graceful-degrade family, which only surfaced because we ran each skill against a real backend with a real permission gap.
• The pattern you fixed in 13 places may exist in 15 more places under a different syntax — the pipefail family. Same root cause (silent failure of an upstream API call), different syntactic surface (single-fetch pipe instead of parallel-fetch background processes).

The first lesson taught us to grep across files. The second taught us to test against live state. The third teaches us that even after a careful sweep, sibling patterns in different syntactic forms are likely still hiding the same root issue. The shape was "shell pipelines silently swallowing API errors." We found one expression of it; the other expression sat in plain view for another round.

v4.5.118 ships 30 pipefail additions and one validation hardening. The two failure paths a Merus permission gap can take through a skill — parallel-fetch and single-fetch pipe — are now both audited and both fail loud.