An earlier release this month fixed a data-loss bug in one of the file locks in this codebase. The queue lock was losing 98% of writes under multi-process concurrent load. The fix was to drop a fallback in the stale-lock detection — the mtime-based check that was supposed to handle the case where a crashed lock holder hadn't written its PID yet. The fallback turned out to false-break the lock under tight contention.

After that fix, the queue lock was solid. Three other locks in the codebase used a similar pattern. I ran a probe round on the message-binding module, looking for unrelated bugs. The probe found the same race.

10 concurrent processes, each writing 30 message bindings, 300 expected. Pre-fix: 4 to 137 lost per run, averaging 90 missing or 30% of writes silently overwritten. The upload-binding sibling (same lock pattern, different module) lost 32 to 110 per run. The audit lock — also same pattern — did NOT lose anything. 300 of 300 every time.

Three locks with the same code, two losing writes, one stable. The difference is what runs inside the lock.

The race window

The lock acquisition has two sequential operations:

fd = openSync(LOCK_FILE, 'wx');           // creates file, empty body
writeFileSync(LOCK_FILE, String(pid));    // writes PID into the lock file

Between those two lines, the lock file exists with no content. Another process trying to acquire the lock fails the openSync (file already exists), falls into the catch, reads the lock file, finds empty content. The pre-fix code then checked mtime as a fallback signal — "if the file is older than 30 seconds, the holder is probably crashed and didn't write its PID; consider the lock stale and steal it."

On macOS with APFS under tight contention, the mtime check returned values that occasionally tripped the staleness threshold even though the lock had been created milliseconds ago. The lock was unlinked. Both processes ended up "holding" the lock simultaneously. Their writes raced. Last writer wins.

Why the audit lock didn't show the bug

The audit lock holds for very short durations. The work inside the lock is open-write-fsync-close — single-digit milliseconds at most. The race window between openSync and writeFileSync is even smaller because the writeFileSync happens immediately after openSync, with no work in between.

The binding locks hold longer. The work inside the lock is a JSON read-modify-write: load the bindings file, mutate the in-memory representation, serialize, atomic write. That's tens of milliseconds. The lock is held for the whole duration. Other processes spend more time waiting in the catch loop, more time stat'ing the lock file to check its mtime, and more chances to trip the false-stale detection.

The audit lock pattern works for audit appends. It doesn't work for binding writes. The work-inside-lock duration matters.

The fix

Same fix as the queue lock: drop the mtime fallback. Use PID-liveness only for stale detection.

let isStale = false;
try {
  const content = readFileSync(lockPath, 'utf8').trim();
  const parsed = parseInt(content, 10);
  if (Number.isInteger(parsed) && parsed > 0) {
    const alive = isPidAlive(parsed);
    if (alive === false) isStale = true;
  }
} catch { /* lock file just released — retry */ }

If the lock file has a PID and the PID is dead, the lock is stale. Otherwise (PID alive, PID unreadable, PID empty) keep waiting. A crashed lock holder whose PID was never written stays locked until the acquire timeout, at which point we throw with a typed message. That's the correct fail-loud behavior — far better than silently breaking a held lock.

The cost in practice is negligible. A crashed lock holder is rare. A crashed lock holder whose PID was never written is rarer still (the writeFileSync happens immediately after openSync — there's almost no window). And when it does happen, the lock-acquire timeout (5 seconds in the binding locks, 30 in the audit/queue locks) is short enough that the user just sees a brief delay before a clean error.

Post-fix: 0 lost across 5 runs of each binding lock. 3000 total writes, 3000 survived.

What I should have done after fixing the queue lock

This is the second time this month I've fixed the same lock bug. After fixing the queue lock in 4.5.99, I noted that the audit and binding locks had the same pattern but were "verified to remain at 300/300 under the same stress, so they're not being changed in this release." That verification was wrong for the binding locks. I tested the audit lock stress (which passed) and inferred the binding locks would also pass because they had the same code. They didn't, because the work-inside-lock was longer.

The lesson: when you fix a lock bug in one module and decide not to fix the same pattern elsewhere, the test for "elsewhere is fine" needs to use the actual work load of THAT module, not the proxy of "I tested the most-prominent module and it passed." The audit lock and the binding locks share the same lock code but exercise it very differently.

If I'd run the multi-process stress against the binding locks at the same time as 4.5.99, the bug would have shipped with the queue fix. Instead it shipped two weeks later, after a probe round on message-binding (where I happened to test it). The cost is that for two weeks, every binding write in production was losing 10-30% of its data under concurrent load.

The general lesson

When you fix a bug in code that's been copy-pasted into multiple modules, fix every copy at the same time. Don't trust that "the other copies have the same code, so they probably have the same behavior" — they might have different behavior because they're exercised differently. The probe round that confirmed the audit lock was fine didn't generalize to the binding locks; the only way to know was to run the actual binding-lock probe.

For shared lock code specifically, the right move is to factor it into a single helper that every caller uses. Then a fix to the helper applies to every caller automatically. The current code has the helper duplicated across four files. Worth a future refactor; for now, this release fixes the two copies that were actively losing data.

The version on the install URL is 4.5.109.