TrueMend vs. ShellCheck on Bash

Round 1 on nvm's Bash scripts had TrueMend ahead: 18.5% wrong-match on 195 judged findings against ShellCheck's 23.1%. A fresh, five-times-larger full-pool sample in round 2 put TrueMend at 33.2%, and ShellCheck was re-run alongside it at 27.3% on 194 judged findings — a real, current measurement, not the round-1 number this case study originally carried forward. Round 3 targeted the regression directly.

Round history

Round 3 — five line-scan disease patterns, 55 quarantines, 7 narrowings

Round 2's regression wasn't scattered noise — it clustered into five recurring failure modes, each root-caused against nvm's real source:

1. POSIX-contract blindness (~30 wrong). Detectors suggesting a bashism — [[ ]], (( )), brace expansion, bash arrays — for scripts that are POSIX-sh-portable by design. The "fix" would break the code. Quarantined test-single-bracket, arithmetic-in-test, seq-in-for-loop, and three more; narrowed three others behind a bash-shebang gate so they stay valid on POSIX scripts.
2. Guard-blind "missing X" detectors (~25 wrong). Asserting no cleanup/trap/error-check when the guard — a cleanup() function, an || return $?, a preceding [ -e ] && — sits on a line the line-scan can't see. Seven detectors quarantined.
3. Speculative-consequence detectors (~20 wrong). The matched construct is real but the asserted harm doesn't follow: uppercase_var alone produced 1,554 raw findings assuming every ALL_CAPS name clashes with an environment variable. Quarantined.
4. Factually-wrong-premise detectors (~8 wrong). string-length claimed ${#var} is bash-specific; it's POSIX. sc2181-exit-code flagged the exact if [ $? -ne 0 ] construct it should encourage. Quarantined.
5. Literal/style misidentification (~10 wrong). Backtick detection matching markdown code-quoting inside message strings; three pure-style detectors with no functional impact. Quarantined; three more (regex-sed shapes) narrowed rather than removed, verified to still fire on real $(ls *.txt) command substitution.

Result: a fresh, from-scratch TrueMend-only sample (42 findings, down from 2,240 raw in round 2 — the flooding detectors are gone), blind-judged by a 3-pass majority to cancel single-pass variance: 81.0% correct, 19.0% wrong-match, 9.5% real-rate.

142 of 198 Bash detectors stay active; the 56 quarantined were the systemic false-positive generators. The remaining 19.0% is dominated by irreducible edge-case noise on detectors that are genuinely valuable elsewhere — secret-detection firing on test-fixture tokens, eval-required dynamic indirection in POSIX sh — not fixable defects. No engine code changed this round; the win is entirely quarantines and narrowings.

Where this leaves the comparison

TrueMend: 19.0% wrong-match (round 3, TrueMend-only). ShellCheck: 27.3% — its round-2 number, not re-run this round. 19.0% beats that real, current figure outright, a 43% relative reduction from round 2's 33.2%. It is still above the <10% bar this benchmark holds every language to.

Still open

• ShellCheck needs a fresh round-3 re-measure to make this a same-round apples-to-apples comparison; 27.3% is carried forward from round 2, not re-run alongside TrueMend's round-3 sample.
• 19.0% is still above the <10% bar. The residual error is mostly irreducible secret-detection and dynamic-indirection noise, not categories waiting on a fix.
• 142 active detectors, most never re-verified this round — round 3 targeted the five worst failure clusters, not the full surface.