Information about dos2unix' implementation choices.

1. Smart conversion

===================

  There are some dos2unix implementations that automatically convert all type of

line breaks. For instance converting both DOS and Mac linebreaks to Unix line

breaks at once. Or automatically detect the line break type and convert to the

other side.

  Smart conversions could lead to unexpected behaviour. For instance when a

dos2unix is run on a file with only Unix line breaks and the line breaks are

flipped to the other side. This dos2unix implementation does exactly what you

tell it to do. When you run 'dos2unix' only DOS line breaks are converted to

Unix line breaks. Unix line breaks stay in the file. Seen from a DOS or Unix

perspective, a Mac line break is not a line break, so also Mac line breaks stay

untouched.  The same applies for mac2unix. Mac2unix leaves Unix and DOS line

breaks untouched.

2. Unix filter

==============

  When a standard Unix filter, e.g. sed or tr, reads input from a file it sends

its output by default to standard out. This implementation of dos2unix does by

default in-place conversion (overwriting the input file), which seems not in line.

  Dos2unix is not part of the Unix standard. Most Unixes have their

own implementation of dos2unix. There is a lot of variation in command names,

options, and behavior. The SunOS version of dos2unix, after which this version was

modeled, does by default paired conversion.

  This implementation of dos2unix has too much legacy to change the current behaviour.

Changing it would have more disadvantages than advantages. Most people expect

dos2unix to do in-place conversion. The majority of other open source implementations

also convert by default in-place. In-place conversion has the advantage that it is

very easy to convert multiple files by using wild cards.

  This implementation of dos2unix does send the output to standard-out when the

input comes from standard-in. So you can use it as filter. Note that dos2unix/

unix2dos is also used a lot on non-Unix operating systems where the filter idea

is less known.

3. Recursive conversion of files

================================

  There are implementations that have builtin functionality to do recursive

conversion of all files in a directory tree.

  This functionality is not needed in dos2unix. By using an external program,

like Unix 'find', you can do recursive conversion of directory trees. There is

no need to duplicate this.

4. Encoding conversion

======================

  Dos2unix can do several encoding conversions. First there are the conversions

of several DOS code pages to and from ISO-8859-1. These conversions are also

part of the SunOS dos2unix implementation after which this implementation has

been modeled. Although these conversions are not much used these days they have

been added for the sake completeness. Conversion of CP1252 was added, because

it is used a lot in the Western world. It's almost identical to ISO-8859-1. There

is no intention to add other conversions to and from ISO-8859-1.

  Conversion from UTF-16 was added, because the world is moving towards

Unicode.  Microsoft Windows uses by default UTF-16 format for Unicode. UTF-16

is part of Windows' core design for historical reasons. Microsoft standardized

on UCS-2, a predecessor of UTF-16, in a time when UTF-8 did not exist yet.

However a lot of Windows software is able to read UTF-8 files. In Windows

"Unicode" means usually UTF-16. For instance saving a file with Notepad in

"Unicode" encoding means in UTF-16 encoding. When you work in PowerShell and

echo some text to a file you get an UTF-16 encoded text file. UTF-16 is there

to stay, although many people would like to see otherwise and are dreaming of

an UTF-8 only world. The Unix/Linux world is moving towards UTF-8 encoding,

because it's backwards compatible with ASCII. Unix programs typically do not

support UTF-16.

  One end of the encoding spectrum is an ASCII only world, where the only

differences between DOS and Unix text files are line breaks. In English

speaking regions this is a good working environment, because ASCII is in

practice sufficient for English language. Diacritics are hardly used and can be

omitted. The other end of the spectrum is an Unicode only world. All languages

of the world are supported. Dos2unix aims to support these two ends of the

spectrum: ASCII and Unicode. The Chinese GB18030 encoding is also seen as an

Unicode transformation format. UTF-32 is not supported, because this is

practically only used as an internal format.  Other encoding transformations

are left to specialized programs like iconv and recode. The few conversion

modes to and from ISO-8859-1 are only there for legacy reasons.

  In the ASCII days DOS to Unix text file conversion, and vice versa, was only

converting line breaks.  In the Unicode era it is not only line break

conversion, but also Unicode transformation format conversion (e.g. UTF-16 to

UTF-8), and Byte Order Mark (BOM) removal or addition.

  Conversion towards UTF-16 is not supported and there is no intention to support

it in the future. UTF-8 encoded files are well supported on Windows, so

conversion to UTF-16 is not needed. And we keep on dreaming of an UTF-8 only

world...