Blame doc/LZO.TXT

Packit Service 5195f2
Packit Service 5195f2
 ============================================================================
Packit Service 5195f2
 LZO -- a real-time data compression library
Packit Service 5195f2
 ============================================================================
Packit Service 5195f2
Packit Service 5195f2
 Author  : Markus Franz Xaver Johannes Oberhumer
Packit Service 5195f2
           <markus@oberhumer.com>
Packit Service 5195f2
           http://www.oberhumer.com/opensource/lzo/
Packit Service 5195f2
 Version : 2.08
Packit Service 5195f2
 Date    : 29 Jun 2014
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2
 Abstract
Packit Service 5195f2
 --------
Packit Service 5195f2
 LZO is a portable lossless data compression library written in ANSI C.
Packit Service 5195f2
 It offers pretty fast compression and very fast decompression.
Packit Service 5195f2
 Decompression requires no memory.
Packit Service 5195f2
Packit Service 5195f2
 In addition there are slower compression levels achieving a quite
Packit Service 5195f2
 competitive compression ratio while still decompressing at
Packit Service 5195f2
 this very high speed.
Packit Service 5195f2
Packit Service 5195f2
 The LZO algorithms and implementations are copyrighted OpenSource
Packit Service 5195f2
 distributed under the GNU General Public License.
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2
 Introduction
Packit Service 5195f2
 ------------
Packit Service 5195f2
 LZO is a data compression library which is suitable for data
Packit Service 5195f2
 de-/compression in real-time. This means it favours speed
Packit Service 5195f2
 over compression ratio.
Packit Service 5195f2
Packit Service 5195f2
 The acronym LZO is standing for Lempel-Ziv-Oberhumer.
Packit Service 5195f2
Packit Service 5195f2
 LZO is written in ANSI C. Both the source code and the compressed
Packit Service 5195f2
 data format are designed to be portable across platforms.
Packit Service 5195f2
Packit Service 5195f2
 LZO implements a number of algorithms with the following features:
Packit Service 5195f2
Packit Service 5195f2
 - Decompression is simple and *very* fast.
Packit Service 5195f2
 - Requires no memory for decompression.
Packit Service 5195f2
 - Compression is pretty fast.
Packit Service 5195f2
 - Requires 64 KiB of memory for compression.
Packit Service 5195f2
 - Allows you to dial up extra compression at a speed cost in the
Packit Service 5195f2
   compressor. The speed of the decompressor is not reduced.
Packit Service 5195f2
 - Includes compression levels for generating pre-compressed
Packit Service 5195f2
   data which achieve a quite competitive compression ratio.
Packit Service 5195f2
 - There is also a compression level which needs only 8 KiB for compression.
Packit Service 5195f2
 - Algorithm is thread safe.
Packit Service 5195f2
 - Algorithm is lossless.
Packit Service 5195f2
Packit Service 5195f2
 LZO supports overlapping compression and in-place decompression.
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2
 Design criteria
Packit Service 5195f2
 ---------------
Packit Service 5195f2
 LZO was designed with speed in mind. Decompressor speed has been
Packit Service 5195f2
 favoured over compressor speed. Real-time decompression should be
Packit Service 5195f2
 possible for virtually any application. The implementation of the
Packit Service 5195f2
 LZO1X decompressor in optimized i386 assembler code runs about at
Packit Service 5195f2
 the third of the speed of a memcpy() - and even faster for many files.
Packit Service 5195f2
Packit Service 5195f2
 In fact I first wrote the decompressor of each algorithm thereby
Packit Service 5195f2
 defining the compressed data format, verified it with manually
Packit Service 5195f2
 created test data and at last added the compressor.
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2
 Performance
Packit Service 5195f2
 -----------
Packit Service 5195f2
 To keep you interested, here is an overview of the average results
Packit Service 5195f2
 when compressing the Calgary Corpus test suite with a blocksize
Packit Service 5195f2
 of 256 KiB, originally done on an ancient Intel Pentium 133.
Packit Service 5195f2
Packit Service 5195f2
 The naming convention of the various algorithms goes LZOxx-N, where N is
Packit Service 5195f2
 the compression level. Range 1-9 indicates the fast standard levels using
Packit Service 5195f2
 64 KiB memory for compression. Level 99 offers better compression at the
Packit Service 5195f2
 cost of more memory (256 KiB), and is still reasonably fast.
Packit Service 5195f2
 Level 999 achieves nearly optimal compression - but it is slow
Packit Service 5195f2
 and uses much memory, and is mainly intended for generating
Packit Service 5195f2
 pre-compressed data.
Packit Service 5195f2
Packit Service 5195f2
 The C version of LZO1X-1 is about 4-5 times faster than the fastest
Packit Service 5195f2
 zlib compression level, and it also outperforms other algorithms
Packit Service 5195f2
 like LZRW1-A and LZV in both compression ratio and compression speed
Packit Service 5195f2
 and decompression speed.
Packit Service 5195f2
Packit Service 5195f2
 +------------------------------------------------------------------------+
Packit Service 5195f2
 | Algorithm        Length  CxB   ComLen  %Remn  Bits   Com K/s   Dec K/s |
Packit Service 5195f2
 | ---------        ------  ---   ------  -----  ----   -------   ------- |
Packit Service 5195f2
 |                                                                        |
Packit Service 5195f2
 | memcpy()         224401    1   224401  100.0  8.00  60956.83  59124.58 |
Packit Service 5195f2
 |                                                                        |
Packit Service 5195f2
 | LZO1-1           224401    1   117362   53.1  4.25   4665.24  13341.98 |
Packit Service 5195f2
 | LZO1-99          224401    1   101560   46.7  3.73   1373.29  13823.40 |
Packit Service 5195f2
 |                                                                        |
Packit Service 5195f2
 | LZO1A-1          224401    1   115174   51.7  4.14   4937.83  14410.35 |
Packit Service 5195f2
 | LZO1A-99         224401    1    99958   45.5  3.64   1362.72  14734.17 |
Packit Service 5195f2
 |                                                                        |
Packit Service 5195f2
 | LZO1B-1          224401    1   109590   49.6  3.97   4565.53  15438.34 |
Packit Service 5195f2
 | LZO1B-2          224401    1   106235   48.4  3.88   4297.33  15492.79 |
Packit Service 5195f2
 | LZO1B-3          224401    1   104395   47.8  3.83   4018.21  15373.52 |
Packit Service 5195f2
 | LZO1B-4          224401    1   104828   47.4  3.79   3024.48  15100.11 |
Packit Service 5195f2
 | LZO1B-5          224401    1   102724   46.7  3.73   2827.82  15427.62 |
Packit Service 5195f2
 | LZO1B-6          224401    1   101210   46.0  3.68   2615.96  15325.68 |
Packit Service 5195f2
 | LZO1B-7          224401    1   101388   46.0  3.68   2430.89  15361.47 |
Packit Service 5195f2
 | LZO1B-8          224401    1    99453   45.2  3.62   2183.87  15402.77 |
Packit Service 5195f2
 | LZO1B-9          224401    1    99118   45.0  3.60   1677.06  15069.60 |
Packit Service 5195f2
 | LZO1B-99         224401    1    95399   43.6  3.48   1286.87  15656.11 |
Packit Service 5195f2
 | LZO1B-999        224401    1    83934   39.1  3.13    232.40  16445.05 |
Packit Service 5195f2
 |                                                                        |
Packit Service 5195f2
 | LZO1C-1          224401    1   111735   50.4  4.03   4883.08  15570.91 |
Packit Service 5195f2
 | LZO1C-2          224401    1   108652   49.3  3.94   4424.24  15733.14 |
Packit Service 5195f2
 | LZO1C-3          224401    1   106810   48.7  3.89   4127.65  15645.69 |
Packit Service 5195f2
 | LZO1C-4          224401    1   105717   47.7  3.82   3007.92  15346.44 |
Packit Service 5195f2
 | LZO1C-5          224401    1   103605   47.0  3.76   2829.15  15153.88 |
Packit Service 5195f2
 | LZO1C-6          224401    1   102585   46.5  3.72   2631.37  15257.58 |
Packit Service 5195f2
 | LZO1C-7          224401    1   101937   46.2  3.70   2378.57  15492.49 |
Packit Service 5195f2
 | LZO1C-8          224401    1   100779   45.6  3.65   2171.93  15386.07 |
Packit Service 5195f2
 | LZO1C-9          224401    1   100255   45.4  3.63   1691.44  15194.68 |
Packit Service 5195f2
 | LZO1C-99         224401    1    97252   44.1  3.53   1462.88  15341.37 |
Packit Service 5195f2
 | LZO1C-999        224401    1    87740   40.2  3.21    306.44  16411.94 |
Packit Service 5195f2
 |                                                                        |
Packit Service 5195f2
 | LZO1F-1          224401    1   113412   50.8  4.07   4755.97  16074.12 |
Packit Service 5195f2
 | LZO1F-999        224401    1    89599   40.3  3.23    280.68  16553.90 |
Packit Service 5195f2
 |                                                                        |
Packit Service 5195f2
 | LZO1X-1(11)      224401    1   118810   52.6  4.21   4544.42  15879.04 |
Packit Service 5195f2
 | LZO1X-1(12)      224401    1   113675   50.6  4.05   4411.15  15721.59 |
Packit Service 5195f2
 | LZO1X-1          224401    1   109323   49.4  3.95   4991.76  15584.89 |
Packit Service 5195f2
 | LZO1X-1(15)      224401    1   108500   49.1  3.93   5077.50  15744.56 |
Packit Service 5195f2
 | LZO1X-999        224401    1    82854   38.0  3.04    135.77  16548.48 |
Packit Service 5195f2
 |                                                                        |
Packit Service 5195f2
 | LZO1Y-1          224401    1   110820   49.8  3.98   4952.52  15638.82 |
Packit Service 5195f2
 | LZO1Y-999        224401    1    83614   38.2  3.05    135.07  16385.40 |
Packit Service 5195f2
 |                                                                        |
Packit Service 5195f2
 | LZO1Z-999        224401    1    83034   38.0  3.04    133.31  10553.74 |
Packit Service 5195f2
 |                                                                        |
Packit Service 5195f2
 | LZO2A-999        224401    1    87880   40.0  3.20    301.21   8115.75 |
Packit Service 5195f2
 +------------------------------------------------------------------------+
Packit Service 5195f2
Packit Service 5195f2
 Notes:
Packit Service 5195f2
  - CxB is the number of blocks
Packit Service 5195f2
  - K/s is the speed measured in 1000 uncompressed bytes per second
Packit Service 5195f2
  - the assembler decompressors are even faster
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2
 Short documentation
Packit Service 5195f2
 -------------------
Packit Service 5195f2
 LZO is a block compression algorithm - it compresses and decompresses
Packit Service 5195f2
 a block of data. Block size must be the same for compression
Packit Service 5195f2
 and decompression.
Packit Service 5195f2
Packit Service 5195f2
 LZO compresses a block of data into matches (a sliding dictionary)
Packit Service 5195f2
 and runs of non-matching literals. LZO takes care about long matches
Packit Service 5195f2
 and long literal runs so that it produces good results on highly
Packit Service 5195f2
 redundant data and deals acceptably with non-compressible data.
Packit Service 5195f2
Packit Service 5195f2
 When dealing with incompressible data, LZO expands the input
Packit Service 5195f2
 block by a maximum of 64 bytes per 1024 bytes input.
Packit Service 5195f2
Packit Service 5195f2
 I have verified LZO using such tools as valgrind and other memory checkers.
Packit Service 5195f2
 And in addition to compressing gigabytes of files when tuning some parameters
Packit Service 5195f2
 I have also consulted various 'lint' programs to spot potential portability
Packit Service 5195f2
 problems. LZO is free of any known bugs.
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2
 The algorithms
Packit Service 5195f2
 --------------
Packit Service 5195f2
 There are too many algorithms implemented. But I want to support
Packit Service 5195f2
 unlimited backward compatibility, so I will not reduce the LZO
Packit Service 5195f2
 distribution in the future.
Packit Service 5195f2
Packit Service 5195f2
 As the many object files are mostly independent of each other, the
Packit Service 5195f2
 size overhead for an executable statically linked with the LZO library
Packit Service 5195f2
 is usually pretty low (just a few KiB) because the linker will only add
Packit Service 5195f2
 the modules that you are actually using.
Packit Service 5195f2
Packit Service 5195f2
 I first published LZO1 and LZO1A in the Internet newsgroups
Packit Service 5195f2
 comp.compression and comp.compression.research in March 1996.
Packit Service 5195f2
 They are mainly included for compatibility reasons. The LZO2A
Packit Service 5195f2
 decompressor is too slow, and there is no fast compressor anyway.
Packit Service 5195f2
Packit Service 5195f2
 My experiments have shown that LZO1B is good with a large blocksize
Packit Service 5195f2
 or with very redundant data, LZO1F is good with a small blocksize or
Packit Service 5195f2
 with binary data and that LZO1X is often the best choice of all.
Packit Service 5195f2
 LZO1Y and LZO1Z are almost identical to LZO1X - they can achieve a
Packit Service 5195f2
 better compression ratio on some files.
Packit Service 5195f2
 Beware, your mileage may vary.
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2
 Usage of the library
Packit Service 5195f2
 --------------------
Packit Service 5195f2
 Despite of its size, the basic usage of LZO is really very simple.
Packit Service 5195f2
Packit Service 5195f2
 Let's assume you want to compress some data with LZO1X-1:
Packit Service 5195f2
   A) compression
Packit Service 5195f2
      * include <lzo/lzo1x.h>
Packit Service 5195f2
        call lzo_init()
Packit Service 5195f2
        compress your data with lzo1x_1_compress()
Packit Service 5195f2
      * link your application with the LZO library
Packit Service 5195f2
   B) decompression
Packit Service 5195f2
      * include <lzo/lzo1x.h>
Packit Service 5195f2
        call lzo_init()
Packit Service 5195f2
        decompress your data with lzo1x_decompress()
Packit Service 5195f2
      * link your application with the LZO library
Packit Service 5195f2
Packit Service 5195f2
 The program examples/simple.c shows a fully working example.
Packit Service 5195f2
 See also LZO.FAQ for more information.
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2
 Building LZO
Packit Service 5195f2
 ------------
Packit Service 5195f2
 As LZO uses Autoconf+Automake+Libtool the building process under
Packit Service 5195f2
 UNIX systems should be very unproblematic. Shared libraries are
Packit Service 5195f2
 supported on many architectures as well.
Packit Service 5195f2
 For detailed instructions see the file INSTALL.
Packit Service 5195f2
Packit Service 5195f2
 Please note that due to the design of the ELF executable format
Packit Service 5195f2
 the performance of a shared library on i386 systems (e.g. Linux)
Packit Service 5195f2
 is a little bit slower, so you may want to link your applications
Packit Service 5195f2
 with the static version (liblzo2.a) anyway.
Packit Service 5195f2
Packit Service 5195f2
 For building under DOS, Win16, Win32, OS/2 and other systems
Packit Service 5195f2
 take a look at the file B/00readme.txt.
Packit Service 5195f2
Packit Service 5195f2
 In case of troubles (like decompression data errors) try recompiling
Packit Service 5195f2
 everything without optimizations - LZO may break the optimizer
Packit Service 5195f2
 of your compiler. See the file BUGS.
Packit Service 5195f2
Packit Service 5195f2
 LZO is written in ANSI C. In particular this means:
Packit Service 5195f2
   - your compiler must understand prototypes
Packit Service 5195f2
   - your compiler must understand prototypes in function pointers
Packit Service 5195f2
   - your compiler must correctly promote integrals ("value-preserving")
Packit Service 5195f2
   - your preprocessor must implement #elif, #error and stringizing
Packit Service 5195f2
   - you must have a conforming and correct <limits.h> header
Packit Service 5195f2
   - you must have <stddef.h>, <string.h> and other ANSI C headers
Packit Service 5195f2
   - you should have size_t and ptrdiff_t
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2
 Portability
Packit Service 5195f2
 -----------
Packit Service 5195f2
 I have built and tested LZO successfully on a variety of platforms
Packit Service 5195f2
 including DOS (16 + 32 bit), Windows 3.x (16-bit), Win32, Win64,
Packit Service 5195f2
 Linux, *BSD, HP-UX and many more.
Packit Service 5195f2
Packit Service 5195f2
 LZO is also reported to work under AIX, ConvexOS, IRIX, MacOS, PalmOS (Pilot),
Packit Service 5195f2
 PSX (Sony Playstation), Solaris, SunOS, TOS (Atari ST) and VxWorks.
Packit Service 5195f2
 Furthermore it is said that its performance on a Cray is superior
Packit Service 5195f2
 to all other machines...
Packit Service 5195f2
Packit Service 5195f2
 And I think it would be much fun to translate the decompressors
Packit Service 5195f2
 to Z-80 or 6502 assembly.
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2
 The future
Packit Service 5195f2
 ----------
Packit Service 5195f2
 Here is what I'm planning for the next months. No promises, though...
Packit Service 5195f2
Packit Service 5195f2
 - interfaces to .NET and Mono
Packit Service 5195f2
 - interfaces to Perl, Java, Python, Delphi, Visual Basic, ...
Packit Service 5195f2
 - improve documentation and API reference
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2
 Some comments about the source code
Packit Service 5195f2
 -----------------------------------
Packit Service 5195f2
 Be warned: the main source code in the 'src' directory is a
Packit Service 5195f2
 real pain to understand as I've experimented with hundreds of slightly
Packit Service 5195f2
 different versions. It contains many #if and some gotos, and
Packit Service 5195f2
 is *completely optimized for speed* and not for readability.
Packit Service 5195f2
 Code sharing of the different algorithms is implemented by stressing
Packit Service 5195f2
 the preprocessor - this can be really confusing. Lots of marcos and
Packit Service 5195f2
 assertions don't make things better.
Packit Service 5195f2
Packit Service 5195f2
 Nevertheless LZO compiles very quietly on a variety of
Packit Service 5195f2
 compilers with the highest warning levels turned on, even
Packit Service 5195f2
 in C++ mode.
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2
 Copyright
Packit Service 5195f2
 ---------
Packit Service 5195f2
 LZO is Copyright (C) 1996-2014 Markus Franz Xaver Oberhumer
Packit Service 5195f2
 All Rights Reserved.
Packit Service 5195f2
Packit Service 5195f2
 LZO is distributed under the terms of the GNU General Public License (GPL).
Packit Service 5195f2
 See the file COPYING.
Packit Service 5195f2
Packit Service 5195f2
 Special licenses for commercial and other applications which
Packit Service 5195f2
 are not willing to accept the GNU General Public License
Packit Service 5195f2
 are available by contacting the author.
Packit Service 5195f2
Packit Service 5195f2
Packit Service 5195f2