Find out which compressor actually wins on your data before you pick one

Sample output

gzip       704033
bzip2      446749
xz         606272

The input is a 7.7 MB nginx access log, and the result is not the one most people expect. xz is supposed to be the modern choice, yet bzip2 finishes 26% smaller than xz and 37% smaller than gzip. Access logs are thousands of near-identical lines with random numbers in fixed positions, which is exactly the shape the Burrows-Wheeler transform inside bzip2 is good at. Run the same loop on a tarball of binaries or a SQL dump and xz usually wins. That is the point of measuring.

When you would reach for it

You are setting up log rotation, archiving a year of database exports to cold storage, or deciding what to ship a large artifact in, and the choice will be paid for every day in storage or transfer. The received wisdom (“xz compresses best, gzip is fastest”) is a reasonable prior but not a measurement. Thirty seconds on a real sample of your own data replaces the argument with a number.

Gotchas

• Size is only half the trade. Wrap the pipeline in bash’s time keyword (see Variants) and the same log shows gzip at 0.24s, bzip2 at 0.46s, and xz at 1.16s to compress. Decompression flips the order: gzip 0.02s, xz 0.02s, bzip2 0.09s. If files are written once and read constantly, bzip2’s slow decompression can cost more than its smaller size saves.
• Small files make compressors look terrible. On a 993-byte log, all three produce output under 500 bytes, but bzip2 comes last because its header and block overhead dominate. Random or already-compressed data (JPEGs, .gz files, encrypted blobs) actually grows: 10,240 random bytes become 10,274 with gzip and 10,728 with bzip2. Test on a sample that looks like production, not whatever file is handy.
• xz -9 and xz -6 often produce identical output. The presets mostly change the dictionary size (8 MiB at -6, 64 MiB at -9), and a dictionary bigger than the input buys nothing, so on this 7.7 MB log both give 606272 bytes. Try xz -9e (extreme mode) instead, which spends more CPU on the same dictionary and got the log down to 574184.
• On macOS, gzip and bzip2 ship with the system but the xz command does not, so the loop prints xz: command not found and a count of 0 for that row. Install it with brew install xz. On Alpine, BusyBox provides an xz applet that only decompresses, so apk add xz bzip2 gzip for the full tools.

Variants

$ for c in gzip bzip2 xz; do TIMEFORMAT="$c %Rs"; time $c -9 -c access.log | wc -c; done

Size and wall-clock time together. time is a bash keyword, not the external program, so it times the whole pipeline, and TIMEFORMAT relabels its output with the compressor name

$ for c in gzip bzip2 xz; do printf '%-6s %10d\n' $c $(head -c 50000000 huge.log | $c -9 | wc -c); done

Benchmark a 50 MB sample of a multi-gigabyte file instead of the whole thing. With no file argument, each compressor reads stdin and writes stdout, so -c is implied

$ xz -9 -k -T0 access.log

Once you have picked a winner, write the real file. -k keeps the original and -T0 uses every core, at the cost of a few extra bytes because the input is split into independently compressed blocks. This writes access.log.xz to disk, and xz refuses with “File exists” rather than overwrite one already there. Adding -f forces the overwrite, so check before you reach for it