How the test files were selected
I was especially interested how well LZMA compression would fit in
-
binary package management of GNU/*/Linux distributions
-
distributing source code of free software
In both uses the files are compressed on one computer and decompressed many times by users around the world. In practice the most important factors are:
-
compressed size (faster to download; more packages fit into one CD or DVD)
-
time required in decompression (fast installation is nice)
-
memory requirements for decompression (should the user want to use the file on e.g. an old i486 with 8 MB RAM)
-
common format that everyone knows how to uncompress/install
Less important:
-
time wasted for compressing in the package build process; compiling software usually takes several minutes or even hours so spending one or two minutes to compress the package tightly increases the build time only little.
-
memory requirements for the compressing process; few people build packages on i486 or i586 class machines having 16 to 64 megs of RAM. However, no one wants to use a tool that needs hundreds of megabytes or even gigabytes of RAM to achieve good results.
Despite the many common factors, the contents of binary packages and source tarballs are quite different. Binary packages primarily contain executables and libraries while source tarballs contain mostly ASCII text of some programming language. Naturally both contain data files used by the program and (hopefully) some documentation.
Test conditions
Tests were run on a laptop:
-
AMD mobile Athlon XP2400+
-
512 MB RAM
-
Linux 2.6.12-rc4 (preempt, 4k stacks, regparm)
-
gzip 1.3.3, bzip2 1.0.3, LZMA SDK 4.17
bzip2 has two compression modes, one for normal use and another designed
for small memory footprint (which can be invoked with bzip2 --small).
Only the normal mode was tested because it’s faster.
Times are from the output of the command time (line user) and rounded.
Because of this, the compression and decompression time and speed tables
should be taken as suggestive and not as the absolute truth. In practice,
the bigger test files should be more reliable in terms of speed comparison.
When reading the tables, it is important to keep in mind which settings are the default in each program:
-
gzip -6(best speed/filesize ratio) -
bzip -9(best compression ratio) -
lzmash -7(excellent compression ratio and reasonable memory requirements) -
lzmash -e(the extreme mode) is only for reference in case someone wants to see how it affects the compression ratio.
The tables of the test results
Note: The first column with numbers 1…9 indicates
the compression setting passed to gzip, bzip2 and lzmash (e.g. gzip -9).
Tarball made from a full installation of OpenOffice.org 1.1.4 for Linux
Uncompressed size: 212664320 bytes (203 MB)
Compressed file size in bytes
gzip bzip2 lzmash lzmash -e
1 86322815 76147880 67456213 -
2 84858575 74320824 62085798 -
3 83561997 73467586 59547691 59278372
4 81312776 73044026 58245872 57964166
5 79798262 72762041 56694215 56411631
6 79179298 72540199 56182079 55859514
7 78995264 72512833 55535273 55269226
8 78816280 72314472 54678948 54405078
9 78768334 72223858 54068819 53769958
Compressed size / Uncompressed size * 100%
gzip bzip2 lzmash lzmash -e
1 40,6% 35,8% 31,7% -
2 39,9% 34,9% 29,2% -
3 39,3% 34,5% 28,0% 27,9%
4 38,2% 34,3% 27,4% 27,3%
5 37,5% 34,2% 26,7% 26,5%
6 37,2% 34,1% 26,4% 26,3%
7 37,1% 34,1% 26,1% 26,0%
8 37,1% 34,0% 25,7% 25,6%
9 37,0% 34,0% 25,4% 25,3%
Compression time
gzip bzip2 lzmash lzmash -e
1 11.5s 1m 26s 0m 58s -
2 12.0s 1m 40s 2m 7s -
3 13.7s 1m 54s 4m 58s 7m 37s
4 15.1s 2m 5s 5m 26s 8m 2s
5 18.4s 2m 11s 6m 47s 11m 18s
6 24.5s 2m 18s 7m 30s 12m 4s
7 29.4s 2m 25s 8m 24s 12m 59s
8 45.5s 2m 32s 10m 59s 20m 17s
9 66.9s 2m 37s 12m 20s 21m 56s
Decompression time
gzip bzip2 lzmash lzmash -e
1 3.3s 16.5s 11.3s -
2 3.3s 24.2s 10.5s -
3 3.3s 29.2s 10.5s 10.4s
4 3.3s 32.1s 10.4s 10.3s
5 3.2s 34.2s 10.2s 10.2s
6 3.2s 35.4s 10.2s 10.1s
7 3.2s 36.5s 10.1s 10.0s
8 3.2s 37.5s 10.0s 9.9s
9 3.1s 38.2s 10.0s 9.9s
Compression speed, MB/s of uncompressed data (1 MB = 1024 * 1024 bytes)
gzip bzip2 lzmash lzmash -e
1 18 2.4 3.5 -
2 17 2.0 1.6 -
3 15 1.8 0.68 0.44
4 13 1.6 0.62 0.42
5 11 1.5 0.50 0.30
6 8.3 1.5 0.45 0.28
7 6.9 1.4 0.40 0.26
8 4.5 1.3 0.31 0.17
9 3.0 1.3 0.27 0.15
Decompression speed, MB/s of uncompressed data (1 MB = 1024 * 1024 bytes)
gzip bzip2 lzmash lzmash -e
1 61 12 18 -
2 61 8.4 19 -
3 61 6.9 19 20
4 61 6.3 20 20
5 63 5.9 20 20
6 63 5.7 20 20
7 63 5.6 20 20
8 63 5.4 20 20
9 65 5.3 20 20
The Linux kernel 2.6.11.0 source tarball
Uncompressed size: 208250880 bytes (199 MB)
Compressed file size in bytes
gzip bzip2 lzmash lzmash -e
1 57860603 43873922 43933138 -
2 55274813 41108704 38871392 -
3 53416918 39791569 34863499 34823465
4 49695438 39040694 33545762 33513509
5 47775348 38395197 32481024 32445716
6 47004031 37975094 31686173 31661947
7 46797152 37676593 30881464 30841602
8 46578138 37365408 30295730 30261027
9 46578138 37075679 29809336 29780803
Compressed size / Uncompressed size * 100%
gzip bzip2
1 27,8% 21,1% 21,1% -
2 26,5% 19,7% 18,7% -
3 25,7% 19,1% 16,7% 16,7%
4 23,9% 18,7% 16,1% 16,1%
5 22,9% 18,4% 15,6% 15,6%
6 22,6% 18,2% 15,2% 15,2%
7 22,5% 18,1% 14,8% 14,8%
8 22,4% 17,9% 14,5% 14,5%
9 22,4% 17,8% 14,3% 14,3%
Compression time
gzip bzip2 lzmash lzmash -e
1 8.3s 1m 9s 0m 45s -
2 8.7s 1m 22s 1m 45s -
3 9.8s 1m 34s 5m 10s 8m 43s
4 11.1s 1m 45s 5m 43s 9m 41s
5 13.8s 1m 57s 7m 39s 14m 38s
6 17.8s 2m 2s 8m 23s 15m 32s
7 20.7s 2m 11s 9m 11s 16m 23s
8 29.7s 2m 21s 11m 34s 24m 47s
9 40.9s 2m 26s 12m 31s 25m 53s
Decompression time
gzip bzip2 lzmash lzmash -e
1 2.8s 12.8s 7.7s -
2 2.7s 19.4s 6.9s -
3 2.6s 23.8s 6.4s 6.6s
4 2.5s 26.4s 6.3s 6.3s
5 2.5s 28.3s 6.3s 6.3s
6 2.4s 29.6s 6.2s 6.3s
7 2.4s 30.6s 6.2s 6.2s
8 2.4s 31.3s 6.1s 6.1s
9 2.4s 32.1s 6.1s 6.1s
Compression speed, MB/s of uncompressed data (1 MB = 1024 * 1024 bytes)
gzip bzip2 lzmash lzmash -e
1 24 2.9 4.4 -
2 23 2.4 1.9 -
3 20 2.1 0.64 0.38
4 18 1.9 0.58 0.34
5 14 1.7 0.43 0.23
6 11 1.6 0.39 0.21
7 9.6 1.5 0.36 0.20
8 6.7 1.4 0.29 0.13
9 4.9 1.4 0.26 0.13
Decompression speed, MB/s of uncompressed data (1 MB = 1024 * 1024 bytes)
gzip bzip2 lzmash lzmash -e
1 71 16 26
2 74 10 29
3 76 8.3 31 30
4 79 7.5 32 32
5 79 7.0 32 32
6 83 6.7 32 32
7 83 6.5 32 32
8 83 6.3 33 33
9 83 6.2 33 33
In this test bzip2 is a tough adversary to lzmash in fast modes.
lzmash -e makes a few kB smaller files with the expense of a lot
longer compression time.
XMMS 1.2.10 binary package
XMMS 1.2.10 binary package (xmms-1.2.10-i486-2.tgz) from Slackware 10.1. The file was first gunzipped, resulting uncompressed size of 5498880 bytes (5.2 MB).
Compressed file size in bytes
gzip bzip2 lzmash lzmash -e
1 2160102 1803573 1431699 -
2 2112332 1611408 1140030 -
3 2072044 1539083 1034903 1038615
4 2031519 1487237 1004176 1007692
5 1992713 1464332 987189 988758
6 1979068 1433617 983305 983198
7 1973404 1431276 982125 983240
8 1972424 1414142 980836 983582
9 1970643 1385112 980836 983582
Compressed size / Uncompressed size * 100%
gzip bzip2 lzmash lzmash -e
1 39,3% 32,8% 26,0% -
2 38,4% 29,3% 20,7% -
3 37,7% 28,0% 18,8% 18,9%
4 36,9% 27,0% 18,3% 18,3%
5 36,2% 26,6% 18,0% 18,0%
6 36,0% 26,1% 17,9% 17,9%
7 35,9% 26,0% 17,9% 17,9%
8 35,9% 25,7% 17,8% 17,9%
9 35,8% 25,2% 17,8% 17,9%
Compression time
gzip bzip2 lzmash lzmash -e
1 0.3s 2.4s 1.4s -
2 0.3s 2.9s 2.7s -
3 0.4s 3.2s 6.2s 8.9s
4 0.4s 3.3s 6.6s 9.3s
5 0.5s 4.6s 8.2s 13.3s
6 0.7s 5.6s 8.5s 13.7s
7 0.8s 4.7s 8.6s 13.6s
8 1.1s 4.9s 10.5s 21.5s
9 1.8s 5.1s 10.5s 21.5s
Decompression time
gzip bzip2 lzmash lzmash -e
1 0.1s 0.4s 0.3s -
2 0.1s 0.6s 0.2s -
3 0.1s 0.7s 0.2s 0.2s
4 0.1s 0.8s 0.2s 0.2s
5 0.1s 0.9s 0.2s 0.2s
6 0.1s 0.9s 0.2s 0.2s
7 0.1s 0.9s 0.2s 0.2s
8 0.1s 1.0s 0.2s 0.2s
9 0.1s 1.0s 0.2s 0.2s
For some reason, bzip2 -6 took more time than even bzip -9. The
result didn’t change when the test was repeated. The extreme mode of
lzmash creates a few bytes bigger files; seems that using lzmash -e
makes compression both slower and less efficient with smaller files.
Speed tables are omitted because the smaller test file makes measuring
the elapsed time with 'time' command too inaccurate.
XMMS 1.2.10 source tarball
Uncompressed size: 15964160 bytes (15.2 MB)
Compressed file size in bytes
gzip bzip2 lzmash lzmash -e
1 4705710 3702465 3390291 -
2 4560441 3172615 2117511 -
3 4460478 2914692 1921894 1929077
4 4213705 2748562 1803104 1808532
5 4095300 2670185 1721301 1723689
6 4060060 2591439 1642013 1643645
7 4046707 2500735 1540827 1541735
8 4035433 2464688 1533283 1531514
9 4034855 2418265 1533283 1531514
Compressed size / Uncompressed size * 100%
gzip bzip2 lzmash lzmash -e
1 29,5% 23,2% 21,2% -
2 28,6% 19,9% 13,3% -
3 27,9% 18,3% 12,0% 12,1%
4 26,4% 17,2% 11,3% 11,3%
5 25,7% 16,7% 10,8% 10,8%
6 25,4% 16,2% 10,3% 10,3%
7 25,3% 15,7% 9,7% 9,7%
8 25,3% 15,4% 9,6% 9,6%
9 25,3% 15,1% 9,6% 9,6%
Compression time
gzip bzip2 lzmash lzmash -e
1 0.7s 6.1s 3.5s -
2 0.7s 7.3s 6.0s -
3 0.8s 8.5s 19.0s 30.8s
4 0.9s 9.9s 19.9s 31.2s
5 1.1s 11.2s 28.9s 1m 1s
6 1.4s 11.0s 30.1s 1m 2s
7 1.7s 12.5s 30.9s 1m 4s
8 2.5s 15.9s 41.7s 1m 56s
9 2.9s 17.5s 41.7s 1m 56s
Decompression time
gzip bzip2 lzmash lzmash -e
1 0.2s 1.0s 0.6s -
2 0.2s 1.5s 0.4s -
3 0.2s 1.9s 0.4s 0.4s
4 0.2s 2.1s 0.4s 0.4s
5 0.2s 2.3s 0.4s 0.4s
6 0.2s 2.5s 0.4s 0.4s
7 0.2s 2.6s 0.4s 0.4s
8 0.2s 2.7s 0.4s 0.4s
9 0.2s 2.8s 0.4s 0.4s
For some reason, in compression bzip2 -6 was a little faster than
bzip -5 but bzip -6 still created smaller file. Speed tables are
omitted because the smaller test file makes measuring the elapsed time
with time command too inaccurate.
Memory requirements
The memory requirements depend only on the used
compression mode (-1 … -9).
bzip2 has also a mode that uses less memory but is slower. This small memory
mode hasn’t been tested.
RAM usage on compression
gzip bzip2 lzmash lzmash -e
1 <1 MB 2 MB 2 MB -
2 <1 MB 2 MB 12 MB -
3 <1 MB 3 MB 12 MB 12 MB
4 <1 MB 4 MB 16 MB 16 MB
5 <1 MB 5 MB 26 MB 26 MB
6 <1 MB 5 MB 45 MB 45 MB
7 <1 MB 6 MB 83 MB 83 MB
8 <1 MB 7 MB 159 MB 159 MB
9 <1 MB 7 MB 311 MB 311 MB
RAM usage on decompression
gzip bzip2 lzmash lzmash -e
1 <1 MB 1 MB 1 MB -
2 <1 MB 2 MB 2 MB -
3 <1 MB 2 MB 1 MB 1 MB
4 <1 MB 2 MB 2 MB 2 MB
5 <1 MB 3 MB 3 MB 3 MB
6 <1 MB 3 MB 5 MB 5 MB
7 <1 MB 3 MB 9 MB 9 MB
8 <1 MB 4 MB 17 MB 17 MB
9 <1 MB 4 MB 33 MB 33 MB
Conclusions
Compression
When there’s need for a very fast compression, gzip is the clear winner. It has also very small memory footprint, making it ideal for systems with limited memory.
bzip2 creates about 15% smaller files than gzip. bzip2 compresses somewhat slower than gzip, but seems that it hasn’t prevented bzip2 from getting popular. Nowadays most source code is available as both gzip and bzip2 compressed tar archives.
lzmash -3 and lzmash -4 seem to be almost as fast (or slow);
same can be said for lzmash -5, lzmash -6 and lzmash -7.
However the memory requirements increase with every option meaning
that lzmash -3, lzmash -5 and lzmash -6 are usually useful
only if you (or the recipient) do not have enough memory for
lzmash -4 or lzmash -7.
lzmash -8 and lzmash -9 require lots of memory and are practical
only on newer computers; the files compressed with them are probably
a pain to decompress on systems with less than 32 MB or 64 MB of memory.
The extreme mode (lzmash -e) roughly doubles the compression time,
but especially with small files can lead to even worse compression
ratio than normal the mode. The extereme mode might be worth trying
if you want make as small files as possible, but in that case
forgetting lzmash wrapper script and playing with command line
options of lzma directly can lead to better results.
Decompression
In terms of speed, gzip is the winner again. lzma comes right behind it two to three times slower than gzip. bzip2 is a lot slower taking usually two to six times more time than lzma, that is, four to twelve times more than gzip. One interesting thing is that gzip and lzma decompress the faster the smaller the compressed size is, while bzip2 gets slower when the compression ratio gets better.
The memory usage of lzma stays competitive with bzip2 when files have
been compressed with lzmash -6 or with a smaller option. The files
compressed with the default lzmash -7 can still be decompressed, even
on machines with only 16 MB of RAM, but sometimes you don’t have even
that much memory available. If you compress with lzmash -8 or
lzmash -9, you should think if the users need to be able to
decompress your files also on “ancient” computers.
So what is the best?
Of course, it depends on the intended application. gzip is very fast and has small memory footprint. According to this benchmark, neither bzip2 nor lzma can compete with gzip in terms of speed or memory usage. bzip2 has notably better compression ratio than gzip, which has to be the reason for the popularity of bzip2; it is slower than gzip especially in decompression and uses more memory. However the memory requirements of bzip2 should be nowadays no problem even on older hardware.
Both gzip and bzip2 are bundled with practically all GNU/*/Linux distributions and *BSDs. Because everybody has the tools to handle gzip and bzip2 compressed files, they are by far the most commonly used formats to distribute e.g. source code of free software. However, the situation might change because better free (as in freedom) alternatives have become available.
LZMA clearly has potential to become the third commonly used general purporse compression format on *NIX systems. It mainly competes with bzip2 by offering significantly better compression ratio while still keeping decompressing speed relatively close to that of gzip. Its excellence has been already seen in Tukaani Linux package management system, and in software installers such as Nullsoft Scriptable Install System (NSIS), Inno setup and installers of MS-Windows versions of Mozilla products, including Firefox and Thunderbird.