|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
============================================================================
|
|
Packit Service |
5195f2 |
LZO -- a real-time data compression library
|
|
Packit Service |
5195f2 |
============================================================================
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Author : Markus Franz Xaver Johannes Oberhumer
|
|
Packit Service |
5195f2 |
<markus@oberhumer.com>
|
|
Packit Service |
5195f2 |
http://www.oberhumer.com/opensource/lzo/
|
|
Packit Service |
5195f2 |
Version : 2.08
|
|
Packit Service |
5195f2 |
Date : 29 Jun 2014
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Abstract
|
|
Packit Service |
5195f2 |
--------
|
|
Packit Service |
5195f2 |
LZO is a portable lossless data compression library written in ANSI C.
|
|
Packit Service |
5195f2 |
It offers pretty fast compression and very fast decompression.
|
|
Packit Service |
5195f2 |
Decompression requires no memory.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
In addition there are slower compression levels achieving a quite
|
|
Packit Service |
5195f2 |
competitive compression ratio while still decompressing at
|
|
Packit Service |
5195f2 |
this very high speed.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
The LZO algorithms and implementations are copyrighted OpenSource
|
|
Packit Service |
5195f2 |
distributed under the GNU General Public License.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Introduction
|
|
Packit Service |
5195f2 |
------------
|
|
Packit Service |
5195f2 |
LZO is a data compression library which is suitable for data
|
|
Packit Service |
5195f2 |
de-/compression in real-time. This means it favours speed
|
|
Packit Service |
5195f2 |
over compression ratio.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
The acronym LZO is standing for Lempel-Ziv-Oberhumer.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
LZO is written in ANSI C. Both the source code and the compressed
|
|
Packit Service |
5195f2 |
data format are designed to be portable across platforms.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
LZO implements a number of algorithms with the following features:
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
- Decompression is simple and *very* fast.
|
|
Packit Service |
5195f2 |
- Requires no memory for decompression.
|
|
Packit Service |
5195f2 |
- Compression is pretty fast.
|
|
Packit Service |
5195f2 |
- Requires 64 KiB of memory for compression.
|
|
Packit Service |
5195f2 |
- Allows you to dial up extra compression at a speed cost in the
|
|
Packit Service |
5195f2 |
compressor. The speed of the decompressor is not reduced.
|
|
Packit Service |
5195f2 |
- Includes compression levels for generating pre-compressed
|
|
Packit Service |
5195f2 |
data which achieve a quite competitive compression ratio.
|
|
Packit Service |
5195f2 |
- There is also a compression level which needs only 8 KiB for compression.
|
|
Packit Service |
5195f2 |
- Algorithm is thread safe.
|
|
Packit Service |
5195f2 |
- Algorithm is lossless.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
LZO supports overlapping compression and in-place decompression.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Design criteria
|
|
Packit Service |
5195f2 |
---------------
|
|
Packit Service |
5195f2 |
LZO was designed with speed in mind. Decompressor speed has been
|
|
Packit Service |
5195f2 |
favoured over compressor speed. Real-time decompression should be
|
|
Packit Service |
5195f2 |
possible for virtually any application. The implementation of the
|
|
Packit Service |
5195f2 |
LZO1X decompressor in optimized i386 assembler code runs about at
|
|
Packit Service |
5195f2 |
the third of the speed of a memcpy() - and even faster for many files.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
In fact I first wrote the decompressor of each algorithm thereby
|
|
Packit Service |
5195f2 |
defining the compressed data format, verified it with manually
|
|
Packit Service |
5195f2 |
created test data and at last added the compressor.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Performance
|
|
Packit Service |
5195f2 |
-----------
|
|
Packit Service |
5195f2 |
To keep you interested, here is an overview of the average results
|
|
Packit Service |
5195f2 |
when compressing the Calgary Corpus test suite with a blocksize
|
|
Packit Service |
5195f2 |
of 256 KiB, originally done on an ancient Intel Pentium 133.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
The naming convention of the various algorithms goes LZOxx-N, where N is
|
|
Packit Service |
5195f2 |
the compression level. Range 1-9 indicates the fast standard levels using
|
|
Packit Service |
5195f2 |
64 KiB memory for compression. Level 99 offers better compression at the
|
|
Packit Service |
5195f2 |
cost of more memory (256 KiB), and is still reasonably fast.
|
|
Packit Service |
5195f2 |
Level 999 achieves nearly optimal compression - but it is slow
|
|
Packit Service |
5195f2 |
and uses much memory, and is mainly intended for generating
|
|
Packit Service |
5195f2 |
pre-compressed data.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
The C version of LZO1X-1 is about 4-5 times faster than the fastest
|
|
Packit Service |
5195f2 |
zlib compression level, and it also outperforms other algorithms
|
|
Packit Service |
5195f2 |
like LZRW1-A and LZV in both compression ratio and compression speed
|
|
Packit Service |
5195f2 |
and decompression speed.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
+------------------------------------------------------------------------+
|
|
Packit Service |
5195f2 |
| Algorithm Length CxB ComLen %Remn Bits Com K/s Dec K/s |
|
|
Packit Service |
5195f2 |
| --------- ------ --- ------ ----- ---- ------- ------- |
|
|
Packit Service |
5195f2 |
| |
|
|
Packit Service |
5195f2 |
| memcpy() 224401 1 224401 100.0 8.00 60956.83 59124.58 |
|
|
Packit Service |
5195f2 |
| |
|
|
Packit Service |
5195f2 |
| LZO1-1 224401 1 117362 53.1 4.25 4665.24 13341.98 |
|
|
Packit Service |
5195f2 |
| LZO1-99 224401 1 101560 46.7 3.73 1373.29 13823.40 |
|
|
Packit Service |
5195f2 |
| |
|
|
Packit Service |
5195f2 |
| LZO1A-1 224401 1 115174 51.7 4.14 4937.83 14410.35 |
|
|
Packit Service |
5195f2 |
| LZO1A-99 224401 1 99958 45.5 3.64 1362.72 14734.17 |
|
|
Packit Service |
5195f2 |
| |
|
|
Packit Service |
5195f2 |
| LZO1B-1 224401 1 109590 49.6 3.97 4565.53 15438.34 |
|
|
Packit Service |
5195f2 |
| LZO1B-2 224401 1 106235 48.4 3.88 4297.33 15492.79 |
|
|
Packit Service |
5195f2 |
| LZO1B-3 224401 1 104395 47.8 3.83 4018.21 15373.52 |
|
|
Packit Service |
5195f2 |
| LZO1B-4 224401 1 104828 47.4 3.79 3024.48 15100.11 |
|
|
Packit Service |
5195f2 |
| LZO1B-5 224401 1 102724 46.7 3.73 2827.82 15427.62 |
|
|
Packit Service |
5195f2 |
| LZO1B-6 224401 1 101210 46.0 3.68 2615.96 15325.68 |
|
|
Packit Service |
5195f2 |
| LZO1B-7 224401 1 101388 46.0 3.68 2430.89 15361.47 |
|
|
Packit Service |
5195f2 |
| LZO1B-8 224401 1 99453 45.2 3.62 2183.87 15402.77 |
|
|
Packit Service |
5195f2 |
| LZO1B-9 224401 1 99118 45.0 3.60 1677.06 15069.60 |
|
|
Packit Service |
5195f2 |
| LZO1B-99 224401 1 95399 43.6 3.48 1286.87 15656.11 |
|
|
Packit Service |
5195f2 |
| LZO1B-999 224401 1 83934 39.1 3.13 232.40 16445.05 |
|
|
Packit Service |
5195f2 |
| |
|
|
Packit Service |
5195f2 |
| LZO1C-1 224401 1 111735 50.4 4.03 4883.08 15570.91 |
|
|
Packit Service |
5195f2 |
| LZO1C-2 224401 1 108652 49.3 3.94 4424.24 15733.14 |
|
|
Packit Service |
5195f2 |
| LZO1C-3 224401 1 106810 48.7 3.89 4127.65 15645.69 |
|
|
Packit Service |
5195f2 |
| LZO1C-4 224401 1 105717 47.7 3.82 3007.92 15346.44 |
|
|
Packit Service |
5195f2 |
| LZO1C-5 224401 1 103605 47.0 3.76 2829.15 15153.88 |
|
|
Packit Service |
5195f2 |
| LZO1C-6 224401 1 102585 46.5 3.72 2631.37 15257.58 |
|
|
Packit Service |
5195f2 |
| LZO1C-7 224401 1 101937 46.2 3.70 2378.57 15492.49 |
|
|
Packit Service |
5195f2 |
| LZO1C-8 224401 1 100779 45.6 3.65 2171.93 15386.07 |
|
|
Packit Service |
5195f2 |
| LZO1C-9 224401 1 100255 45.4 3.63 1691.44 15194.68 |
|
|
Packit Service |
5195f2 |
| LZO1C-99 224401 1 97252 44.1 3.53 1462.88 15341.37 |
|
|
Packit Service |
5195f2 |
| LZO1C-999 224401 1 87740 40.2 3.21 306.44 16411.94 |
|
|
Packit Service |
5195f2 |
| |
|
|
Packit Service |
5195f2 |
| LZO1F-1 224401 1 113412 50.8 4.07 4755.97 16074.12 |
|
|
Packit Service |
5195f2 |
| LZO1F-999 224401 1 89599 40.3 3.23 280.68 16553.90 |
|
|
Packit Service |
5195f2 |
| |
|
|
Packit Service |
5195f2 |
| LZO1X-1(11) 224401 1 118810 52.6 4.21 4544.42 15879.04 |
|
|
Packit Service |
5195f2 |
| LZO1X-1(12) 224401 1 113675 50.6 4.05 4411.15 15721.59 |
|
|
Packit Service |
5195f2 |
| LZO1X-1 224401 1 109323 49.4 3.95 4991.76 15584.89 |
|
|
Packit Service |
5195f2 |
| LZO1X-1(15) 224401 1 108500 49.1 3.93 5077.50 15744.56 |
|
|
Packit Service |
5195f2 |
| LZO1X-999 224401 1 82854 38.0 3.04 135.77 16548.48 |
|
|
Packit Service |
5195f2 |
| |
|
|
Packit Service |
5195f2 |
| LZO1Y-1 224401 1 110820 49.8 3.98 4952.52 15638.82 |
|
|
Packit Service |
5195f2 |
| LZO1Y-999 224401 1 83614 38.2 3.05 135.07 16385.40 |
|
|
Packit Service |
5195f2 |
| |
|
|
Packit Service |
5195f2 |
| LZO1Z-999 224401 1 83034 38.0 3.04 133.31 10553.74 |
|
|
Packit Service |
5195f2 |
| |
|
|
Packit Service |
5195f2 |
| LZO2A-999 224401 1 87880 40.0 3.20 301.21 8115.75 |
|
|
Packit Service |
5195f2 |
+------------------------------------------------------------------------+
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Notes:
|
|
Packit Service |
5195f2 |
- CxB is the number of blocks
|
|
Packit Service |
5195f2 |
- K/s is the speed measured in 1000 uncompressed bytes per second
|
|
Packit Service |
5195f2 |
- the assembler decompressors are even faster
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Short documentation
|
|
Packit Service |
5195f2 |
-------------------
|
|
Packit Service |
5195f2 |
LZO is a block compression algorithm - it compresses and decompresses
|
|
Packit Service |
5195f2 |
a block of data. Block size must be the same for compression
|
|
Packit Service |
5195f2 |
and decompression.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
LZO compresses a block of data into matches (a sliding dictionary)
|
|
Packit Service |
5195f2 |
and runs of non-matching literals. LZO takes care about long matches
|
|
Packit Service |
5195f2 |
and long literal runs so that it produces good results on highly
|
|
Packit Service |
5195f2 |
redundant data and deals acceptably with non-compressible data.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
When dealing with incompressible data, LZO expands the input
|
|
Packit Service |
5195f2 |
block by a maximum of 64 bytes per 1024 bytes input.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
I have verified LZO using such tools as valgrind and other memory checkers.
|
|
Packit Service |
5195f2 |
And in addition to compressing gigabytes of files when tuning some parameters
|
|
Packit Service |
5195f2 |
I have also consulted various 'lint' programs to spot potential portability
|
|
Packit Service |
5195f2 |
problems. LZO is free of any known bugs.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
The algorithms
|
|
Packit Service |
5195f2 |
--------------
|
|
Packit Service |
5195f2 |
There are too many algorithms implemented. But I want to support
|
|
Packit Service |
5195f2 |
unlimited backward compatibility, so I will not reduce the LZO
|
|
Packit Service |
5195f2 |
distribution in the future.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
As the many object files are mostly independent of each other, the
|
|
Packit Service |
5195f2 |
size overhead for an executable statically linked with the LZO library
|
|
Packit Service |
5195f2 |
is usually pretty low (just a few KiB) because the linker will only add
|
|
Packit Service |
5195f2 |
the modules that you are actually using.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
I first published LZO1 and LZO1A in the Internet newsgroups
|
|
Packit Service |
5195f2 |
comp.compression and comp.compression.research in March 1996.
|
|
Packit Service |
5195f2 |
They are mainly included for compatibility reasons. The LZO2A
|
|
Packit Service |
5195f2 |
decompressor is too slow, and there is no fast compressor anyway.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
My experiments have shown that LZO1B is good with a large blocksize
|
|
Packit Service |
5195f2 |
or with very redundant data, LZO1F is good with a small blocksize or
|
|
Packit Service |
5195f2 |
with binary data and that LZO1X is often the best choice of all.
|
|
Packit Service |
5195f2 |
LZO1Y and LZO1Z are almost identical to LZO1X - they can achieve a
|
|
Packit Service |
5195f2 |
better compression ratio on some files.
|
|
Packit Service |
5195f2 |
Beware, your mileage may vary.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Usage of the library
|
|
Packit Service |
5195f2 |
--------------------
|
|
Packit Service |
5195f2 |
Despite of its size, the basic usage of LZO is really very simple.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Let's assume you want to compress some data with LZO1X-1:
|
|
Packit Service |
5195f2 |
A) compression
|
|
Packit Service |
5195f2 |
* include <lzo/lzo1x.h>
|
|
Packit Service |
5195f2 |
call lzo_init()
|
|
Packit Service |
5195f2 |
compress your data with lzo1x_1_compress()
|
|
Packit Service |
5195f2 |
* link your application with the LZO library
|
|
Packit Service |
5195f2 |
B) decompression
|
|
Packit Service |
5195f2 |
* include <lzo/lzo1x.h>
|
|
Packit Service |
5195f2 |
call lzo_init()
|
|
Packit Service |
5195f2 |
decompress your data with lzo1x_decompress()
|
|
Packit Service |
5195f2 |
* link your application with the LZO library
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
The program examples/simple.c shows a fully working example.
|
|
Packit Service |
5195f2 |
See also LZO.FAQ for more information.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Building LZO
|
|
Packit Service |
5195f2 |
------------
|
|
Packit Service |
5195f2 |
As LZO uses Autoconf+Automake+Libtool the building process under
|
|
Packit Service |
5195f2 |
UNIX systems should be very unproblematic. Shared libraries are
|
|
Packit Service |
5195f2 |
supported on many architectures as well.
|
|
Packit Service |
5195f2 |
For detailed instructions see the file INSTALL.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Please note that due to the design of the ELF executable format
|
|
Packit Service |
5195f2 |
the performance of a shared library on i386 systems (e.g. Linux)
|
|
Packit Service |
5195f2 |
is a little bit slower, so you may want to link your applications
|
|
Packit Service |
5195f2 |
with the static version (liblzo2.a) anyway.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
For building under DOS, Win16, Win32, OS/2 and other systems
|
|
Packit Service |
5195f2 |
take a look at the file B/00readme.txt.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
In case of troubles (like decompression data errors) try recompiling
|
|
Packit Service |
5195f2 |
everything without optimizations - LZO may break the optimizer
|
|
Packit Service |
5195f2 |
of your compiler. See the file BUGS.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
LZO is written in ANSI C. In particular this means:
|
|
Packit Service |
5195f2 |
- your compiler must understand prototypes
|
|
Packit Service |
5195f2 |
- your compiler must understand prototypes in function pointers
|
|
Packit Service |
5195f2 |
- your compiler must correctly promote integrals ("value-preserving")
|
|
Packit Service |
5195f2 |
- your preprocessor must implement #elif, #error and stringizing
|
|
Packit Service |
5195f2 |
- you must have a conforming and correct <limits.h> header
|
|
Packit Service |
5195f2 |
- you must have <stddef.h>, <string.h> and other ANSI C headers
|
|
Packit Service |
5195f2 |
- you should have size_t and ptrdiff_t
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Portability
|
|
Packit Service |
5195f2 |
-----------
|
|
Packit Service |
5195f2 |
I have built and tested LZO successfully on a variety of platforms
|
|
Packit Service |
5195f2 |
including DOS (16 + 32 bit), Windows 3.x (16-bit), Win32, Win64,
|
|
Packit Service |
5195f2 |
Linux, *BSD, HP-UX and many more.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
LZO is also reported to work under AIX, ConvexOS, IRIX, MacOS, PalmOS (Pilot),
|
|
Packit Service |
5195f2 |
PSX (Sony Playstation), Solaris, SunOS, TOS (Atari ST) and VxWorks.
|
|
Packit Service |
5195f2 |
Furthermore it is said that its performance on a Cray is superior
|
|
Packit Service |
5195f2 |
to all other machines...
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
And I think it would be much fun to translate the decompressors
|
|
Packit Service |
5195f2 |
to Z-80 or 6502 assembly.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
The future
|
|
Packit Service |
5195f2 |
----------
|
|
Packit Service |
5195f2 |
Here is what I'm planning for the next months. No promises, though...
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
- interfaces to .NET and Mono
|
|
Packit Service |
5195f2 |
- interfaces to Perl, Java, Python, Delphi, Visual Basic, ...
|
|
Packit Service |
5195f2 |
- improve documentation and API reference
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Some comments about the source code
|
|
Packit Service |
5195f2 |
-----------------------------------
|
|
Packit Service |
5195f2 |
Be warned: the main source code in the 'src' directory is a
|
|
Packit Service |
5195f2 |
real pain to understand as I've experimented with hundreds of slightly
|
|
Packit Service |
5195f2 |
different versions. It contains many #if and some gotos, and
|
|
Packit Service |
5195f2 |
is *completely optimized for speed* and not for readability.
|
|
Packit Service |
5195f2 |
Code sharing of the different algorithms is implemented by stressing
|
|
Packit Service |
5195f2 |
the preprocessor - this can be really confusing. Lots of marcos and
|
|
Packit Service |
5195f2 |
assertions don't make things better.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Nevertheless LZO compiles very quietly on a variety of
|
|
Packit Service |
5195f2 |
compilers with the highest warning levels turned on, even
|
|
Packit Service |
5195f2 |
in C++ mode.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Copyright
|
|
Packit Service |
5195f2 |
---------
|
|
Packit Service |
5195f2 |
LZO is Copyright (C) 1996-2014 Markus Franz Xaver Oberhumer
|
|
Packit Service |
5195f2 |
All Rights Reserved.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
LZO is distributed under the terms of the GNU General Public License (GPL).
|
|
Packit Service |
5195f2 |
See the file COPYING.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
Special licenses for commercial and other applications which
|
|
Packit Service |
5195f2 |
are not willing to accept the GNU General Public License
|
|
Packit Service |
5195f2 |
are available by contacting the author.
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|
|
Packit Service |
5195f2 |
|