036/Shell tricks

Find out which stage of a pipeline actually failed

2026.10.05·004 min·macOS · Debian · Alpine · Shell

$ grep FATAL server.log | wc -l; echo "${PIPESTATUS[@]}"

anatomy

grep FATAL server.log

The stage whose exit status you care about. grep has three answers, not two: 0 means it found a match, 1 means it read the file and found nothing, and 2 means it could not read the file at all. A typo in the filename and a clean log both print zero lines downstream, so the exit code is the only place the difference survives.

| wc -l

The stage that hides the failure. A pipeline's exit status is the exit status of its last command, and wc succeeds whenever it can count its input, including zero lines of it. So the pipeline as a whole reports success no matter what grep said.

;

Run the next command after the pipeline finishes, regardless of how it went. It has to be the very next command, because bash replaces PIPESTATUS after every foreground command, including the echo you might have used to look at `$?` first.

${PIPESTATUS[@]}

A bash array holding the exit status of every command in the most recent foreground pipeline, left to right. Index 0 is the first stage. Where `$?` keeps only the last stage's answer, this keeps all of them, which is the only way to tell after the fact which link in the chain broke.

"..."

Quote the expansion out of habit. The values are always integers so nothing breaks here, but `"${array[@]}"` is the form that keeps elements intact in every other context, and the habit is cheaper than the exception.

Sample output

0
1 0

The first line is wc reporting zero matching lines, and the second line is the part $? would have hidden: grep exited 1, wc exited 0. A 1 from grep means it read the log and found nothing, so zero is a real answer. Misspell the filename and the first line is still 0, but the second becomes 2 0, and that 2 is the difference between “no fatal errors” and “never looked”.

When you would reach for it

A CI step that pipes through tee and turns green on a failed build, a cron job that pipes a dump through gzip and keeps producing tiny valid archives of nothing, or a monitoring check that counts matches and cannot tell an empty log from a missing one. Any time a pipeline’s success depends on something other than its last command, $? is answering a different question from the one you asked.

Gotchas

PIPESTATUS describes the most recent foreground command, and almost everything counts as one. echo $? on the line before replaces it with 0, and so does a [ ... ] test, so [[ ${PIPESTATUS[0]} -eq 0 ]] works once and then the array describes the test. Copy it into your own array on the very next line, rc=("${PIPESTATUS[@]}"), before doing anything else. echo $? "${PIPESTATUS[@]}" on one line also works, because both expansions happen before echo runs.
On macOS the default interactive shell is zsh, where ${PIPESTATUS[@]} expands to nothing. zsh’s equivalent is the lowercase $pipestatus, and its arrays start at index 1, so the first stage is $pipestatus[1]. The macOS /bin/bash is 3.2, which has PIPESTATUS, so scripts with a bash shebang behave the same as on Debian.
set -o pipefail is the blunter fix: the pipeline returns the status of the rightmost stage that failed. It also counts SIGPIPE as failure, which surprises people. yes | head -1 exits 0 normally, but yes was killed when head stopped reading, so PIPESTATUS is 141 0 (128 plus signal 13), and under pipefail the whole pipeline returns 141. Any producer that outruns a head can trip a pipefail script for no real reason.
Negating with ! flips $? but leaves PIPESTATUS alone. ! grep FATAL server.log | wc -l sets $? to 1 while the array still reads 1 0, so the two can disagree on purpose. Debian’s /bin/sh is dash, which has no PIPESTATUS and, in the version Debian 12 ships, rejects set -o pipefail as an illegal option. Alpine’s BusyBox ash supports pipefail but not the array, so install bash there if you need per-stage codes.

Variants

$ set -o pipefail; grep FATAL server.log | wc -l; echo $?

Let $? report the failure instead. With pipefail on, the pipeline returns grep’s 1 rather than wc’s 0. The option stays set for the rest of the shell session, so in an interactive terminal it changes every pipeline you run afterward

$ grep FATAL server.log | wc -l; rc=("${PIPESTATUS[@]}"); echo "grep=${rc[0]} wc=${rc[1]}"

Save the codes before anything else overwrites them. The array assignment captures every stage at once, and after that rc holds the values no matter how many tests and echoes come after it

$ ./run-tests.sh 2>&1 | tee test.log; exit "${PIPESTATUS[0]}"

Keep a log without letting tee swallow a failing test run. This belongs as the last line of a script or CI step, where exit hands the test runner’s status back to the caller. Typed at an interactive prompt, the exit closes your shell

lineage

The rule that a pipeline exits with the status of its last command goes back to the Bourne shell, and it was a reasonable choice when pipelines were mostly filters feeding a terminal: the thing you saw was the thing that ran last. It became a problem once pipelines moved into scripts and build systems, where `make | tee build.log` reported success for every failed build because tee had written the log perfectly. Bash added the PIPESTATUS array in version 2.0, released in 1996, as a way to look behind the last command without changing what `$?` meant. The `pipefail` option, which does change what `$?` means, came from the Korn shell, and bash adopted it in 3.0 in 2004. POSIX took longer: pipefail was standardized in the 2024 edition, which is why older dash builds still reject it. zsh went its own way and named its array `pipestatus`, in lowercase.

> the last command always takes the credit_