"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"[

 %common-ents;

]>

<book lang="en" id="userman" xreflabel="bzip2 Manual">

 <bookinfo>

  <title>bzip2 and libbzip2, version 1.0.6</title>

  <subtitle>A program and library for data compression</subtitle>

  <copyright>

   <year>&bz-lifespan;</year>

   <holder>Julian Seward</holder>

  </copyright>

  <releaseinfo>Version &bz-version; of &bz-date;</releaseinfo>

  <authorgroup>

   <author>

    <firstname>Julian</firstname>

    <surname>Seward</surname>

    <affiliation>

     <orgname>&bz-url;</orgname>

    </affiliation>

   </author>

  </authorgroup>

  <legalnotice>

  <para>This program, <computeroutput>bzip2</computeroutput>, the

  associated library <computeroutput>libbzip2</computeroutput>, and

  all documentation, are copyright © &bz-lifespan; Julian Seward.

  All rights reserved.</para>

  <para>Redistribution and use in source and binary forms, with

  or without modification, are permitted provided that the

  following conditions are met:</para>

  <itemizedlist mark='bullet'>

   <listitem><para>Redistributions of source code must retain the

   above copyright notice, this list of conditions and the

   following disclaimer.</para></listitem>

   <listitem><para>The origin of this software must not be

   misrepresented; you must not claim that you wrote the original

   software.  If you use this software in a product, an

   acknowledgment in the product documentation would be

   appreciated but is not required.</para></listitem>

   <listitem><para>Altered source versions must be plainly marked

   as such, and must not be misrepresented as being the original

   software.</para></listitem>

   <listitem><para>The name of the author may not be used to

   endorse or promote products derived from this software without

   specific prior written permission.</para></listitem>

  </itemizedlist>

  <para>THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY

  EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,

  THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A

  PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE

  AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,

  EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED

  TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,

  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND

  ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT

  LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING

  IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF

  THE POSSIBILITY OF SUCH DAMAGE.</para>

 <para>PATENTS: To the best of my knowledge,

 <computeroutput>bzip2</computeroutput> and

 <computeroutput>libbzip2</computeroutput> do not use any patented

 algorithms.  However, I do not have the resources to carry

 out a patent search.  Therefore I cannot give any guarantee of

 the above statement.

 </para>

</legalnotice>

</bookinfo>

<chapter id="intro" xreflabel="Introduction">

<title>Introduction</title>

<para><computeroutput>bzip2</computeroutput> compresses files

using the Burrows-Wheeler block-sorting text compression

algorithm, and Huffman coding.  Compression is generally

considerably better than that achieved by more conventional

LZ77/LZ78-based compressors, and approaches the performance of

the PPM family of statistical compressors.</para>

<para><computeroutput>bzip2</computeroutput> is built on top of

<computeroutput>libbzip2</computeroutput>, a flexible library for

handling compressed data in the

<computeroutput>bzip2</computeroutput> format.  This manual

describes both how to use the program and how to work with the

library interface.  Most of the manual is devoted to this

library, not the program, which is good news if your interest is

only in the program.</para>

<itemizedlist mark='bullet'>

 <listitem><para><xref linkend="using"/> describes how to use

 <computeroutput>bzip2</computeroutput>; this is the only part

 you need to read if you just want to know how to operate the

 program.</para></listitem>

 <listitem><para><xref linkend="libprog"/> describes the

 programming interfaces in detail, and</para></listitem>

 <listitem><para><xref linkend="misc"/> records some

 miscellaneous notes which I thought ought to be recorded

 somewhere.</para></listitem>

</itemizedlist>

</chapter>

<chapter id="using" xreflabel="How to use bzip2">

<title>How to use bzip2</title>

<para>This chapter contains a copy of the

<computeroutput>bzip2</computeroutput> man page, and nothing

else.</para>

<sect1 id="name" xreflabel="NAME">

<title>NAME</title>

<itemizedlist mark='bullet'>

 <listitem><para><computeroutput>bzip2</computeroutput>,

  <computeroutput>bunzip2</computeroutput> - a block-sorting file

  compressor, v1.0.6</para></listitem>

 <listitem><para><computeroutput>bzcat</computeroutput> -

   decompresses files to stdout</para></listitem>

 <listitem><para><computeroutput>bzip2recover</computeroutput> -

   recovers data from damaged bzip2 files</para></listitem>

</itemizedlist>

</sect1>

<sect1 id="synopsis" xreflabel="SYNOPSIS">

<title>SYNOPSIS</title>

<itemizedlist mark='bullet'>

 <listitem><para><computeroutput>bzip2</computeroutput> [

  -cdfkqstvzVL123456789 ] [ filenames ...  ]</para></listitem>

 <listitem><para><computeroutput>bunzip2</computeroutput> [

  -fkvsVL ] [ filenames ...  ]</para></listitem>

 <listitem><para><computeroutput>bzcat</computeroutput> [ -s ] [

  filenames ...  ]</para></listitem>

 <listitem><para><computeroutput>bzip2recover</computeroutput>

  filename</para></listitem>

</itemizedlist>

</sect1>

<sect1 id="description" xreflabel="DESCRIPTION">

<title>DESCRIPTION</title>

<para><computeroutput>bzip2</computeroutput> compresses files

using the Burrows-Wheeler block sorting text compression

algorithm, and Huffman coding.  Compression is generally

considerably better than that achieved by more conventional

LZ77/LZ78-based compressors, and approaches the performance of

the PPM family of statistical compressors.</para>

<para>The command-line options are deliberately very similar to

those of GNU <computeroutput>gzip</computeroutput>, but they are

not identical.</para>

<para><computeroutput>bzip2</computeroutput> expects a list of

file names to accompany the command-line flags.  Each file is

replaced by a compressed version of itself, with the name

<computeroutput>original_name.bz2</computeroutput>.  Each

compressed file has the same modification date, permissions, and,

when possible, ownership as the corresponding original, so that

these properties can be correctly restored at decompression time.

File name handling is naive in the sense that there is no

mechanism for preserving original file names, permissions,

ownerships or dates in filesystems which lack these concepts, or

have serious file name length restrictions, such as

MS-DOS.</para>

<para><computeroutput>bzip2</computeroutput> and

<computeroutput>bunzip2</computeroutput> will by default not

overwrite existing files.  If you want this to happen, specify

the <computeroutput>-f</computeroutput> flag.</para>

<para>If no file names are specified,

<computeroutput>bzip2</computeroutput> compresses from standard

input to standard output.  In this case,

<computeroutput>bzip2</computeroutput> will decline to write

compressed output to a terminal, as this would be entirely

incomprehensible and therefore pointless.</para>

<para><computeroutput>bunzip2</computeroutput> (or

<computeroutput>bzip2 -d</computeroutput>) decompresses all

specified files.  Files which were not created by

<computeroutput>bzip2</computeroutput> will be detected and

ignored, and a warning issued.

<computeroutput>bzip2</computeroutput> attempts to guess the

filename for the decompressed file from that of the compressed

file as follows:</para>

<itemizedlist mark='bullet'>

 <listitem><para><computeroutput>filename.bz2 </computeroutput>

  becomes

  <computeroutput>filename</computeroutput></para></listitem>

 <listitem><para><computeroutput>filename.bz </computeroutput>

  becomes

  <computeroutput>filename</computeroutput></para></listitem>

 <listitem><para><computeroutput>filename.tbz2</computeroutput>

  becomes

  <computeroutput>filename.tar</computeroutput></para></listitem>

 <listitem><para><computeroutput>filename.tbz </computeroutput>

  becomes

  <computeroutput>filename.tar</computeroutput></para></listitem>

 <listitem><para><computeroutput>anyothername </computeroutput>

  becomes

  <computeroutput>anyothername.out</computeroutput></para></listitem>

</itemizedlist>

<para>If the file does not end in one of the recognised endings,

<computeroutput>.bz2</computeroutput>,

<computeroutput>.bz</computeroutput>,

<computeroutput>.tbz2</computeroutput> or

<computeroutput>.tbz</computeroutput>,

<computeroutput>bzip2</computeroutput> complains that it cannot

guess the name of the original file, and uses the original name

with <computeroutput>.out</computeroutput> appended.</para>

<para>As with compression, supplying no filenames causes

decompression from standard input to standard output.</para>

<para><computeroutput>bunzip2</computeroutput> will correctly

decompress a file which is the concatenation of two or more

compressed files.  The result is the concatenation of the

corresponding uncompressed files.  Integrity testing

(<computeroutput>-t</computeroutput>) of concatenated compressed

files is also supported.</para>

<para>You can also compress or decompress files to the standard

output by giving the <computeroutput>-c</computeroutput> flag.

Multiple files may be compressed and decompressed like this.  The

resulting outputs are fed sequentially to stdout.  Compression of

multiple files in this manner generates a stream containing

multiple compressed file representations.  Such a stream can be

decompressed correctly only by

<computeroutput>bzip2</computeroutput> version 0.9.0 or later.

Earlier versions of <computeroutput>bzip2</computeroutput> will

stop after decompressing the first file in the stream.</para>

<para><computeroutput>bzcat</computeroutput> (or

<computeroutput>bzip2 -dc</computeroutput>) decompresses all

specified files to the standard output.</para>

<para><computeroutput>bzip2</computeroutput> will read arguments

from the environment variables

<computeroutput>BZIP2</computeroutput> and

<computeroutput>BZIP</computeroutput>, in that order, and will

process them before any arguments read from the command line.

This gives a convenient way to supply default arguments.</para>

<para>Compression is always performed, even if the compressed

file is slightly larger than the original.  Files of less than

about one hundred bytes tend to get larger, since the compression

mechanism has a constant overhead in the region of 50 bytes.

Random data (including the output of most file compressors) is

coded at about 8.05 bits per byte, giving an expansion of around

0.5%.</para>

<para>As a self-check for your protection,

<computeroutput>bzip2</computeroutput> uses 32-bit CRCs to make

sure that the decompressed version of a file is identical to the

original.  This guards against corruption of the compressed data,

and against undetected bugs in

<computeroutput>bzip2</computeroutput> (hopefully very unlikely).

The chances of data corruption going undetected is microscopic,

about one chance in four billion for each file processed.  Be

aware, though, that the check occurs upon decompression, so it

can only tell you that something is wrong.  It can't help you

recover the original uncompressed data.  You can use

<computeroutput>bzip2recover</computeroutput> to try to recover

data from damaged files.</para>

<para>Return values: 0 for a normal exit, 1 for environmental

problems (file not found, invalid flags, I/O errors, etc.), 2

to indicate a corrupt compressed file, 3 for an internal

consistency error (eg, bug) which caused

<computeroutput>bzip2</computeroutput> to panic.</para>

</sect1>

<sect1 id="options" xreflabel="OPTIONS">

<title>OPTIONS</title>

<variablelist>

 <varlistentry>

 <term><computeroutput>-c --stdout</computeroutput></term>

 <listitem><para>Compress or decompress to standard

  output.</para></listitem>

 </varlistentry>

 <varlistentry>

 <term><computeroutput>-d --decompress</computeroutput></term>

 <listitem><para>Force decompression.

  <computeroutput>bzip2</computeroutput>,

  <computeroutput>bunzip2</computeroutput> and

  <computeroutput>bzcat</computeroutput> are really the same

  program, and the decision about what actions to take is done on

  the basis of which name is used.  This flag overrides that

  mechanism, and forces bzip2 to decompress.</para></listitem>

 </varlistentry>

 <varlistentry>

 <term><computeroutput>-z --compress</computeroutput></term>

 <listitem><para>The complement to

  <computeroutput>-d</computeroutput>: forces compression,

  regardless of the invokation name.</para></listitem>

 </varlistentry>

 <varlistentry>

 <term><computeroutput>-t --test</computeroutput></term>

 <listitem><para>Check integrity of the specified file(s), but

  don't decompress them.  This really performs a trial

  decompression and throws away the result.</para></listitem>

 </varlistentry>

 <varlistentry>

 <term><computeroutput>-f --force</computeroutput></term>

 <listitem><para>Force overwrite of output files.  Normally,

  <computeroutput>bzip2</computeroutput> will not overwrite

  existing output files.  Also forces

  <computeroutput>bzip2</computeroutput> to break hard links to

  files, which it otherwise wouldn't do.</para>

  <para><computeroutput>bzip2</computeroutput> normally declines

  to decompress files which don't have the correct magic header

  bytes. If forced (<computeroutput>-f</computeroutput>),

  however, it will pass such files through unmodified. This is

  how GNU <computeroutput>gzip</computeroutput> behaves.</para>

 </listitem>

 </varlistentry>

 <varlistentry>

 <term><computeroutput>-k --keep</computeroutput></term>

 <listitem><para>Keep (don't delete) input files during

  compression or decompression.</para></listitem>

 </varlistentry>

 <varlistentry>

 <term><computeroutput>-s --small</computeroutput></term>

 <listitem><para>Reduce memory usage, for compression,

  decompression and testing.  Files are decompressed and tested

  using a modified algorithm which only requires 2.5 bytes per

  block byte.  This means any file can be decompressed in 2300k

  of memory, albeit at about half the normal speed.</para>

  <para>During compression, <computeroutput>-s</computeroutput>

  selects a block size of 200k, which limits memory use to around

  the same figure, at the expense of your compression ratio.  In

  short, if your machine is low on memory (8 megabytes or less),

  use <computeroutput>-s</computeroutput> for everything.  See

  <xref linkend="memory-management"/> below.</para></listitem>

 </varlistentry>

 <varlistentry>

 <term><computeroutput>-q --quiet</computeroutput></term>

 <listitem><para>Suppress non-essential warning messages.

  Messages pertaining to I/O errors and other critical events

  will not be suppressed.</para></listitem>

 </varlistentry>

 <varlistentry>

 <term><computeroutput>-v --verbose</computeroutput></term>

 <listitem><para>Verbose mode -- show the compression ratio for

  each file processed.  Further

  <computeroutput>-v</computeroutput>'s increase the verbosity

  level, spewing out lots of information which is primarily of

  interest for diagnostic purposes.</para></listitem>

 </varlistentry>

 <varlistentry>

 <term><computeroutput>-L --license -V --version</computeroutput></term>

 <listitem><para>Display the software version, license terms and

  conditions.</para></listitem>

 </varlistentry>

 <varlistentry>

 <term><computeroutput>-1</computeroutput> (or

 <computeroutput>--fast</computeroutput>) to

 <computeroutput>-9</computeroutput> (or

 <computeroutput>-best</computeroutput>)</term>

 <listitem><para>Set the block size to 100 k, 200 k ...  900 k

  when compressing.  Has no effect when decompressing.  See 

  linkend="memory-management" /> below.  The

  <computeroutput>--fast</computeroutput> and

  <computeroutput>--best</computeroutput> aliases are primarily

  for GNU <computeroutput>gzip</computeroutput> compatibility.

  In particular, <computeroutput>--fast</computeroutput> doesn't

  make things significantly faster.  And

  <computeroutput>--best</computeroutput> merely selects the

  default behaviour.</para></listitem>

 </varlistentry>

 <varlistentry>

 <term><computeroutput>--</computeroutput></term>

 <listitem><para>Treats all subsequent arguments as file names,

  even if they start with a dash.  This is so you can handle

  files with names beginning with a dash, for example:

  <computeroutput>bzip2 --

  -myfilename</computeroutput>.</para></listitem>

 </varlistentry>

 <varlistentry>

 <term><computeroutput>--repetitive-fast</computeroutput></term>

 <term><computeroutput>--repetitive-best</computeroutput></term>

 <listitem><para>These flags are redundant in versions 0.9.5 and

  above.  They provided some coarse control over the behaviour of

  the sorting algorithm in earlier versions, which was sometimes

  useful.  0.9.5 and above have an improved algorithm which

  renders these flags irrelevant.</para></listitem>

 </varlistentry>

</variablelist>

</sect1>

<sect1 id="memory-management" xreflabel="MEMORY MANAGEMENT">

<title>MEMORY MANAGEMENT</title>

<para><computeroutput>bzip2</computeroutput> compresses large

files in blocks.  The block size affects both the compression

ratio achieved, and the amount of memory needed for compression

and decompression.  The flags <computeroutput>-1</computeroutput>

through <computeroutput>-9</computeroutput> specify the block

size to be 100,000 bytes through 900,000 bytes (the default)

respectively.  At decompression time, the block size used for

compression is read from the header of the compressed file, and

<computeroutput>bunzip2</computeroutput> then allocates itself

just enough memory to decompress the file.  Since block sizes are

stored in compressed files, it follows that the flags

<computeroutput>-1</computeroutput> to

<computeroutput>-9</computeroutput> are irrelevant to and so

ignored during decompression.</para>

<para>Compression and decompression requirements, in bytes, can be

estimated as:</para>

<programlisting>

Compression:   400k + ( 8 x block size )

Decompression: 100k + ( 4 x block size ), or

               100k + ( 2.5 x block size )

</programlisting>

<para>Larger block sizes give rapidly diminishing marginal

returns.  Most of the compression comes from the first two or

three hundred k of block size, a fact worth bearing in mind when

using <computeroutput>bzip2</computeroutput> on small machines.

It is also important to appreciate that the decompression memory

requirement is set at compression time by the choice of block

size.</para>

<para>For files compressed with the default 900k block size,

<computeroutput>bunzip2</computeroutput> will require about 3700

kbytes to decompress.  To support decompression of any file on a

4 megabyte machine, <computeroutput>bunzip2</computeroutput> has

an option to decompress using approximately half this amount of

memory, about 2300 kbytes.  Decompression speed is also halved,

so you should use this option only where necessary.  The relevant

flag is <computeroutput>-s</computeroutput>.</para>

<para>In general, try and use the largest block size memory

constraints allow, since that maximises the compression achieved.

Compression and decompression speed are virtually unaffected by

block size.</para>

<para>Another significant point applies to files which fit in a

single block -- that means most files you'd encounter using a

large block size.  The amount of real memory touched is

proportional to the size of the file, since the file is smaller

than a block.  For example, compressing a file 20,000 bytes long

with the flag <computeroutput>-9</computeroutput> will cause the

compressor to allocate around 7600k of memory, but only touch

400k + 20000 * 8 = 560 kbytes of it.  Similarly, the decompressor

will allocate 3700k but only touch 100k + 20000 * 4 = 180

kbytes.</para>

<para>Here is a table which summarises the maximum memory usage

for different block sizes.  Also recorded is the total compressed

size for 14 files of the Calgary Text Compression Corpus

totalling 3,141,622 bytes.  This column gives some feel for how

compression varies with block size.  These figures tend to

understate the advantage of larger block sizes for larger files,

since the Corpus is dominated by smaller files.</para>

<programlisting>

        Compress   Decompress   Decompress   Corpus

Flag     usage      usage       -s usage     Size

 -1      1200k       500k         350k      914704

 -2      2000k       900k         600k      877703

 -3      2800k      1300k         850k      860338

 -4      3600k      1700k        1100k      846899

 -5      4400k      2100k        1350k      845160

 -6      5200k      2500k        1600k      838626

 -7      6100k      2900k        1850k      834096

 -8      6800k      3300k        2100k      828642

 -9      7600k      3700k        2350k      828642

</programlisting>

</sect1>

<sect1 id="recovering" xreflabel="RECOVERING DATA FROM DAMAGED FILES">

<title>RECOVERING DATA FROM DAMAGED FILES</title>

<para><computeroutput>bzip2</computeroutput> compresses files in

blocks, usually 900kbytes long.  Each block is handled

independently.  If a media or transmission error causes a

multi-block <computeroutput>.bz2</computeroutput> file to become

damaged, it may be possible to recover data from the undamaged

blocks in the file.</para>

<para>The compressed representation of each block is delimited by

a 48-bit pattern, which makes it possible to find the block

boundaries with reasonable certainty.  Each block also carries

its own 32-bit CRC, so damaged blocks can be distinguished from

undamaged ones.</para>

<para><computeroutput>bzip2recover</computeroutput> is a simple

program whose purpose is to search for blocks in

<computeroutput>.bz2</computeroutput> files, and write each block

out into its own <computeroutput>.bz2</computeroutput> file.  You

can then use <computeroutput>bzip2 -t</computeroutput> to test

the integrity of the resulting files, and decompress those which

are undamaged.</para>

<para><computeroutput>bzip2recover</computeroutput> takes a

single argument, the name of the damaged file, and writes a

number of files <computeroutput>rec0001file.bz2</computeroutput>,

<computeroutput>rec0002file.bz2</computeroutput>, etc, containing

the extracted blocks.  The output filenames are designed so that

the use of wildcards in subsequent processing -- for example,

<computeroutput>bzip2 -dc rec*file.bz2 >

recovered_data</computeroutput> -- lists the files in the correct

order.</para>

<para><computeroutput>bzip2recover</computeroutput> should be of

most use dealing with large <computeroutput>.bz2</computeroutput>

files, as these will contain many blocks.  It is clearly futile

to use it on damaged single-block files, since a damaged block

cannot be recovered.  If you wish to minimise any potential data

loss through media or transmission errors, you might consider

compressing with a smaller block size.</para>

</sect1>

<sect1 id="performance" xreflabel="PERFORMANCE NOTES">

<title>PERFORMANCE NOTES</title>

<para>The sorting phase of compression gathers together similar

strings in the file.  Because of this, files containing very long

runs of repeated symbols, like "aabaabaabaab ..."  (repeated

several hundred times) may compress more slowly than normal.

Versions 0.9.5 and above fare much better than previous versions

in this respect.  The ratio between worst-case and average-case

compression time is in the region of 10:1.  For previous

versions, this figure was more like 100:1.  You can use the

<computeroutput>-vvvv</computeroutput> option to monitor progress

in great detail, if you want.</para>

<para>Decompression speed is unaffected by these

phenomena.</para>

<para><computeroutput>bzip2</computeroutput> usually allocates

several megabytes of memory to operate in, and then charges all

over it in a fairly random fashion.  This means that performance,

both for compressing and decompressing, is largely determined by

the speed at which your machine can service cache misses.

Because of this, small changes to the code to reduce the miss

rate have been observed to give disproportionately large

performance improvements.  I imagine

<computeroutput>bzip2</computeroutput> will perform best on

machines with very large caches.</para>

</sect1>

<sect1 id="caveats" xreflabel="CAVEATS">

<title>CAVEATS</title>

<para>I/O error messages are not as helpful as they could be.

<computeroutput>bzip2</computeroutput> tries hard to detect I/O

errors and exit cleanly, but the details of what the problem is

sometimes seem rather misleading.</para>

<para>This manual page pertains to version &bz-version; of

<computeroutput>bzip2</computeroutput>.  Compressed data created by

this version is entirely forwards and backwards compatible with the

previous public releases, versions 0.1pl2, 0.9.0 and 0.9.5, 1.0.0,

1.0.1, 1.0.2 and 1.0.3, but with the following exception: 0.9.0 and

above can correctly decompress multiple concatenated compressed files.

0.1pl2 cannot do this; it will stop after decompressing just the first

file in the stream.</para>

<para><computeroutput>bzip2recover</computeroutput> versions

prior to 1.0.2 used 32-bit integers to represent bit positions in

compressed files, so it could not handle compressed files more

than 512 megabytes long.  Versions 1.0.2 and above use 64-bit ints

on some platforms which support them (GNU supported targets, and

Windows). To establish whether or not

<computeroutput>bzip2recover</computeroutput> was built with such

a limitation, run it without arguments. In any event you can

build yourself an unlimited version if you can recompile it with

<computeroutput>MaybeUInt64</computeroutput> set to be an

unsigned 64-bit integer.</para>

</sect1>

<sect1 id="author" xreflabel="AUTHOR">

<title>AUTHOR</title>

<para>Julian Seward,

<computeroutput>&bz-email;</computeroutput></para>

<para>The ideas embodied in

<computeroutput>bzip2</computeroutput> are due to (at least) the

following people: Michael Burrows and David Wheeler (for the

block sorting transformation), David Wheeler (again, for the

Huffman coder), Peter Fenwick (for the structured coding model in

the original <computeroutput>bzip</computeroutput>, and many

refinements), and Alistair Moffat, Radford Neal and Ian Witten

(for the arithmetic coder in the original

<computeroutput>bzip</computeroutput>).  I am much indebted for

their help, support and advice.  See the manual in the source

distribution for pointers to sources of documentation.  Christian

von Roques encouraged me to look for faster sorting algorithms,

so as to speed up compression.  Bela Lubkin encouraged me to

improve the worst-case compression performance.  

Donna Robinson XMLised the documentation.

Many people sent

patches, helped with portability problems, lent machines, gave

advice and were generally helpful.</para>

</sect1>

</chapter>

<chapter id="libprog" xreflabel="Programming with libbzip2">

<title>

Programming with <computeroutput>libbzip2</computeroutput>

</title>

<para>This chapter describes the programming interface to

<computeroutput>libbzip2</computeroutput>.</para>

<para>For general background information, particularly about

memory use and performance aspects, you'd be well advised to read

<xref linkend="using"/> as well.</para>

<sect1 id="top-level" xreflabel="Top-level structure">

<title>Top-level structure</title>

<para><computeroutput>libbzip2</computeroutput> is a flexible

library for compressing and decompressing data in the

<computeroutput>bzip2</computeroutput> data format.  Although

packaged as a single entity, it helps to regard the library as

three separate parts: the low level interface, and the high level

interface, and some utility functions.</para>

<para>The structure of

<computeroutput>libbzip2</computeroutput>'s interfaces is similar

to that of Jean-loup Gailly's and Mark Adler's excellent

<computeroutput>zlib</computeroutput> library.</para>

<para>All externally visible symbols have names beginning

<computeroutput>BZ2_</computeroutput>.  This is new in version

1.0.  The intention is to minimise pollution of the namespaces of

library clients.</para>

<para>To use any part of the library, you need to

<computeroutput>#include <bzlib.h></computeroutput>

into your sources.</para>

<sect2 id="ll-summary" xreflabel="Low-level summary">

<title>Low-level summary</title>

<para>This interface provides services for compressing and

decompressing data in memory.  There's no provision for dealing

with files, streams or any other I/O mechanisms, just straight

memory-to-memory work.  In fact, this part of the library can be

compiled without inclusion of

<computeroutput>stdio.h</computeroutput>, which may be helpful

for embedded applications.</para>

<para>The low-level part of the library has no global variables

and is therefore thread-safe.</para>

<para>Six routines make up the low level interface:

<computeroutput>BZ2_bzCompressInit</computeroutput>,

<computeroutput>BZ2_bzCompress</computeroutput>, and

<computeroutput>BZ2_bzCompressEnd</computeroutput> for

compression, and a corresponding trio

<computeroutput>BZ2_bzDecompressInit</computeroutput>,

<computeroutput>BZ2_bzDecompress</computeroutput> and

<computeroutput>BZ2_bzDecompressEnd</computeroutput> for

decompression.  The <computeroutput>*Init</computeroutput>

functions allocate memory for compression/decompression and do

other initialisations, whilst the

<computeroutput>*End</computeroutput> functions close down

operations and release memory.</para>

<para>The real work is done by

<computeroutput>BZ2_bzCompress</computeroutput> and

<computeroutput>BZ2_bzDecompress</computeroutput>.  These

compress and decompress data from a user-supplied input buffer to

a user-supplied output buffer.  These buffers can be any size;

arbitrary quantities of data are handled by making repeated calls

to these functions.  This is a flexible mechanism allowing a

consumer-pull style of activity, or producer-push, or a mixture

of both.</para>

</sect2>

<sect2 id="hl-summary" xreflabel="High-level summary">

<title>High-level summary</title>

<para>This interface provides some handy wrappers around the

low-level interface to facilitate reading and writing

<computeroutput>bzip2</computeroutput> format files

(<computeroutput>.bz2</computeroutput> files).  The routines

provide hooks to facilitate reading files in which the

<computeroutput>bzip2</computeroutput> data stream is embedded

within some larger-scale file structure, or where there are

multiple <computeroutput>bzip2</computeroutput> data streams

concatenated end-to-end.</para>

<para>For reading files,

<computeroutput>BZ2_bzReadOpen</computeroutput>,

<computeroutput>BZ2_bzRead</computeroutput>,

<computeroutput>BZ2_bzReadClose</computeroutput> and 

<computeroutput>BZ2_bzReadGetUnused</computeroutput> are

supplied.  For writing files,

<computeroutput>BZ2_bzWriteOpen</computeroutput>,

<computeroutput>BZ2_bzWrite</computeroutput> and

<computeroutput>BZ2_bzWriteFinish</computeroutput> are

available.</para>

<para>As with the low-level library, no global variables are used

so the library is per se thread-safe.  However, if I/O errors

occur whilst reading or writing the underlying compressed files,

you may have to consult <computeroutput>errno</computeroutput> to

determine the cause of the error.  In that case, you'd need a C

library which correctly supports

<computeroutput>errno</computeroutput> in a multithreaded

environment.</para>

<para>To make the library a little simpler and more portable,

<computeroutput>BZ2_bzReadOpen</computeroutput> and

<computeroutput>BZ2_bzWriteOpen</computeroutput> require you to

pass them file handles (<computeroutput>FILE*</computeroutput>s)

which have previously been opened for reading or writing

respectively.  That avoids portability problems associated with

file operations and file attributes, whilst not being much of an

imposition on the programmer.</para>

</sect2>

<sect2 id="util-fns-summary" xreflabel="Utility functions summary">

<title>Utility functions summary</title>

<para>For very simple needs,

<computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and

<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> are

provided.  These compress data in memory from one buffer to

another buffer in a single function call.  You should assess

whether these functions fulfill your memory-to-memory

compression/decompression requirements before investing effort in

understanding the more general but more complex low-level

interface.</para>

<para>Yoshioka Tsuneo

(<computeroutput>tsuneo@rr.iij4u.or.jp</computeroutput>) has

contributed some functions to give better

<computeroutput>zlib</computeroutput> compatibility.  These

functions are <computeroutput>BZ2_bzopen</computeroutput>,

<computeroutput>BZ2_bzread</computeroutput>,

<computeroutput>BZ2_bzwrite</computeroutput>,

<computeroutput>BZ2_bzflush</computeroutput>,

<computeroutput>BZ2_bzclose</computeroutput>,

<computeroutput>BZ2_bzerror</computeroutput> and

<computeroutput>BZ2_bzlibVersion</computeroutput>.  You may find

these functions more convenient for simple file reading and

writing, than those in the high-level interface.  These functions

are not (yet) officially part of the library, and are minimally

documented here.  If they break, you get to keep all the pieces.

I hope to document them properly when time permits.</para>

<para>Yoshioka also contributed modifications to allow the

library to be built as a Windows DLL.</para>

</sect2>

</sect1>

<sect1 id="err-handling" xreflabel="Error handling">

<title>Error handling</title>

<para>The library is designed to recover cleanly in all

situations, including the worst-case situation of decompressing

random data.  I'm not 100% sure that it can always do this, so

you might want to add a signal handler to catch segmentation

violations during decompression if you are feeling especially

paranoid.  I would be interested in hearing more about the

robustness of the library to corrupted compressed data.</para>

<para>Version 1.0.3 more robust in this respect than any

previous version.  Investigations with Valgrind (a tool for detecting

problems with memory management) indicate

that, at least for the few files I tested, all single-bit errors

in the decompressed data are caught properly, with no

segmentation faults, no uses of uninitialised data, no out of

range reads or writes, and no infinite looping in the decompressor.

So it's certainly pretty robust, although

I wouldn't claim it to be totally bombproof.</para>

<para>The file <computeroutput>bzlib.h</computeroutput> contains

all definitions needed to use the library.  In particular, you

should definitely not include

<computeroutput>bzlib_private.h</computeroutput>.</para>

<para>In <computeroutput>bzlib.h</computeroutput>, the various

return values are defined.  The following list is not intended as

an exhaustive description of the circumstances in which a given

value may be returned -- those descriptions are given later.

Rather, it is intended to convey the rough meaning of each return

value.  The first five actions are normal and not intended to

denote an error situation.</para>

<variablelist>