README-file for the distribution of the Norwegian dictionaries for ISPELL.

DESCRIPTION

This distribution contains a big collection of Norwegian words (both

bokmål and nynorsk) and support files to make useful things from it.

The main file norsk.source contains 747500 words from the Norwegian

language.  Each word has a commonness indicator, and it is hyphenated

at compound points.

There is also a Makefile to assist in building dictionaries for Ispell

and other word processors, using a sensible subset of the available

words.  There is also a Makefile in the patterns directory which makes

hyphenation patterns for TeX based on the dictionary and a simple set

of hyphenation patterns that works on non-compound words.

The latest version is available at

http://spell-norwegian.alioth.debian.org/

Comments, suggestions and bug-reports to i18n-no@lister.ping.uio.no.

There is also a slashdot project with a similar goal.  We should try to

join forces with them.  <URL:http://sourceforge.net/projects/spell-no>

BUILDING A NORWEGIAN ISPELL DICTIONARY

* Get the ispell sources and unpack it.

  cd /source

  tar -zxvf ispell-3.1.20.tar.gz

  You can also unpack the sources for the Norwegian dictionary now:

  cd ispell-3.1/languages

  tar -zxvf ispell-norsk-2.0.tar.gz

* Patch Ispell

  I have made a patch for ispell based mainly on other patches found

  on the net.  If you think you have found a bug in ispell, please

  make sure that it has nothing to do with this patch before

  reporting it to the ispell manager!

  The following things are done:

  1. An attempt is made to fix the backslash bug.  The patch for this

     was found at Ken Stevens ispell.el site.

  2. Ispell can now parse html files thanks to a patch by Gerry

     Tierney.  Basically this means that a patched copy of ispell will

     ignore any mark-up tags or html entities in a html document when

     spell checking that document.  Any text inside an 'alt' attribute

     will however be checked.

     Examples: ispell index.html     # html tags will be ignored

               ispell -h README      # html tags will be ignored

               ispell -n index.html  # html tags will be spell-checked

     I have not been able to make the html mode work well when using

     ispell from emacs.  That doesn't matter too much, since ispell.el

     has its own skipping mechanism.

  3. Buildhash now accepts all characters between A and z as flags,

     not only the alphanumeric ones when MASKBITS=64.  This is needed

     by the Norwegian affix file.

  4. The AMS and breqn math environments are now skipped by ispell.

  5. Ispell gets the ability to suggest "- as a separation character

     in addition to - and space.  This only happens if such support is

     compiled in, e.g. the COMPOUNDBABEL flag must be defined, and it

     only happens in TeX mode and if the language is norsk.  It is

     useful to mark compound points in words to ensure good

     hyphenation when using LaTeX with Babel.  The Norwegian

     hyphenation patterns distributed in this package hyphenate almost

     every word in the Ispell dictionary correctly, but no guaranty is

     offered for other compound words.

  6. Added an -r switch, which is almost like the -a switch, but the

     suggestions are printed even if the word is found in the

     dictionary.  This is useful for hyphenating words and for

     eliminating rare words close to very common words.  There has to

     be some german out there wanting to make TeX hyphenate only

     compound words.

  7. Added a patch from the Redhat rpm to avoid compilation error in

     ijoin.c.

  So if you are feeling a little brave;

  cd ispell-3.1

  patch < languages/norsk/ispell-3.1.20.no.patch

  Additional patches might be needed on various systems.  The Redhat

  source RPM is a good place to look if something fails.

* CONFIGURE ISPELL The file Config.X in the ispell-3.1 distribution

  contains configuration information for ispell (no ./configure yet).

  The definitions are overridden by those in the file local.h, for

  which there is a local.h.samp.  The following local.h works for me

  on my Redhat-6.0 system.  You have to adopt the file to those

  languages you have dictionaries for.

-----------------------------------------------------------------------

#define MINIMENU        /* Display a mini-menu at the bottom of the screen */

#define USG             /* Define this on System V */

#define BINDIR  "/usr/bin"

#define LIBDIR  "/usr/lib"

#define MAN1DIR "/usr/man/man1"

#define MAN4DIR "/usr/man/man4"

#define LANGUAGES "{american,MASTERDICTS=american.med+,HASHFILES=americanmed+.ha

sh,EXTRADICT=/usr/dict/words} {norsk}"

#define MASKBITS 64

#define LOOK     "look"

#define CFLAGS   "-O3"  /* Mostly to speed up my batch operations */

#define LDFLAGS  "-s"

#define COMPOUNDBABEL

-----------------------------------------------------------------------

  It might be wise to try to build ispell only for English, to test that

  everything works, and add new languages afterwards.

  cd ispell-3.1

  make all

  This takes some time, but almost nothing compared to building the

  Norwegian dictionary.

* ADD LANGUAGES

  Get dictionaries for the languages you want to install from the

  ispell home page.  Unpack them in the appropriate directories.

  Update the LANGUAGES variable in local.h and remake.

  Make sure that there is enough free space to build the dictionary.

  If it isn't the build process will loose miserabely. About 120 MB is

  needed!

  The Norwegian dictionary can be configured.  You can choose which

  categories of words to include, and how common a word has to be to

  be included.  This is documented in the Makefile in languages/norsk.

  This flexibility has its price; it takes a very long time and a lot

  of disk space to build the dictionary, up to 120Mb.

  You can also customize the affix file to remove or add some forms of

  words.  For example you could choose to allow or disallow the

  spelling `komitéen'.  To do this you can make the file norsk.aff,

  edit it according to your needs, and make norsk.hash afterwards.

  Look for the word `valgfritt' in the file.  Bear in mind that

  norsk.aff will is dependent on norsk.aff.in, so if you touch that

  file your version will be overwritten.  It will not work as expected

  to change norsk.aff.in.

* INSTALL

  Before you install, you might want to test if ispell works.

  cd languages/norsk

  echo vurderingskriterier | ../../ispell -a -d norsk.hash

  should find vurderingskriterium.  Then

  make install

USING THE DICTIONARY

CHARACTER SETS

By default ispell assumes you use latin-1 encoding in your Norwegian

files.  To spell-check such a file you just say

ispell -d norsk mythesis.tex

In TeX you can use `{\aa}', `{\oe}', `{\o}', `\'e', `\'o' and `\^o' to

represent the special Norwegian characters.  If you do this, you have

to say

ispell -T plaintex -d norsk mythesis.tex

to spell-check a file.  The characters æøåéòô will not be recognized

then, so unfortunately you have to choose one standard.  If you use

`\aa{}' etc. instead, you should change the affix file or add a

similar entry in the affix file.

In a plain ASCII file `æ ø å' are sometimes represented `ae oe aa'.

Use

ispell -T ascii -d norsk mythesis.tex

to spell-check such a file.

The iso246 encoding puts æøå after z in the collating sequence.

If you use this encoding, say

ispell -T iso246 -d norsk mythesis.tex

Does anybody use this??

COMPOUND WORDS

The use of compound words is what makes it both fun and difficult to

produce a good and secure ispell dictionary and to make hyphenation

patterns for TeX.

Ispell has two very important switches, -B and -C, controlling whether

ispell accepts words formed by a root and another word as correct.  If

the -C flag is given, ispell will accept words as

`avdelingsbestyrerstilling', which is right, but also words as

`premierene' (premie-rene), which is wrong.  It is *not recommended*

to use the -C option with the Norwegian dictionary, since far to many

incorrect spellings will be accepted.

If you don't give the -B or -C flag, ispell will accept compound words

formed by a small subset of the words in the dictionary. The subset

depends on the configuration variables in the Makefile. This is called

controlled compoundwords mode.  It is even more safe to give the -B

option, such that only words in the dictionary are regarded as

correct.  I would do that if I had written something important.

The hyphenation patterns for TeX are only tested on words in the

dictionary, so these patterns might fail on compound words accepted in

controlled compoundwords mode.  If you want to be absolutely certain

that there will be no bad hyphens in your document, you have to use

the -B switch.  See `The hyphenation problem' below.

FIGHTING `ORD DELINGS SYNDROMET'

Most spell checkers, including ispell, suggest to split compound words

it doesn't find in its dictionary.  If people follow these suggestions

blindly, the result is disaster; they get spelling errors in the

actual document and even worse; they think they have learned the

correct spelling! (arkitekt tegnet hus i Holmenkoll åsen...)

I have done two things to fight this.  Ispell suggests `"-' in

addition to `-' and ` ' for compound words, which tells TeX that here

is a compound point and makes the spell-check skip the word next time.

The second thing is more important.  The script inorsk-maybecompound

searches a document (or standard input) for two and three words

following each other that can be written in one word, hyphenates them

using TeX and prints the compound words to standard output.  By

hyphenating one avoids words like sommer (som mer), forlenge (for

lenge) etc.  Use it!

EMACS

The version of `ispell.el' distributed with emacs-19.34 does not

support Norwegian.  I suggest you get the latest ispell.el from

ftp://kdstevens.com/pub/stevens/ispell.el.gz.  Good versions are also

found in emacs-20.[4567].

So make sure that your version of ispell.el uses the variable

ispell-local-dictionary-alist, and put a suitable subset of the

following in your .emacs file:

(setq

 ispell-local-dictionary-alist

 '(("norsk"                             ; 8 bit Norwegian mode

    "[A-Za-z\305\306\307\310\311\322\323\324\330\345\346\347\350\351\362\363\364\370]"

    "[^A-Za-z\305\306\307\310\311\322\323\324\330\345\346\347\350\351\362\363\364\370]"

    "[\".,;:]" t ("-B" "-S" "-d" "norsk") "~list" iso-8859-1)

   ("norsk7-tex"                        ; 7 bit Norwegian TeX mode

    "[A-Za-z{}\\'^`@]" "[^A-Za-z{}\\'^`@]"

    "[\".,;:]" t ("-B" "-S" "-d" "norsk" "-T" "plaintex") "~plaintex" nil)

   ("norsk7-html"                       ; 7 bit Norwegian html mode

    "[A-Za-z\&;]" "[^A-Za-z\&;]"        ; Don't use ispell's html-parser

    "[.,:]" t ("-B" "-S" "-n" "-d" "norsk") "~html" iso-8859-1)

   ("norsk7-ascii"                      ; 7 bit Norwegian (aa, ae, oe)

    "[A-Za-z]" "[^A-Za-z]"

    "[\".,;:]" t ("-B" "-S" "-d" "norsk") "~ascii" iso-8859-1)

    ("norsk7-iso246" "[][A-Za-z{}|\\]" "[^][A-Za-z{}|\\]"

     "[\".,;:]" nil ("-B" "-S" "-d" "norsk")  "~iso246" iso-8859-1)

   ("norsk-comp"                        ; 8 bit Norwegian mode

    "[A-Za-z\305\306\307\310\311\322\323\324\330\345\346\347\350\351\362\363\364\370]"

    "[^A-Za-z\305\306\307\310\311\322\323\324\330\345\346\347\350\351\362\363\364\370]"

    "[\".,;:]" t ("-S" "-d" "norsk") "~list" iso-8859-1)

   ("norsk7-tex-comp"                   ; 7 bit Norwegian TeX mode

    "[A-Za-z{}\\'^`@]" "[^A-Za-z{}\\'^`@]"

    "[\".,;:]" t ("-S" "-d" "norsk" "-T" "plaintex") "~plaintex" nil)

   ("norsk7-html-comp"                  ; 7 bit Norwegian html mode

    "[A-Za-z\&;]" "[^A-Za-z\&;]"        ; Don't use ispell's html-parser

    "[.,:]" t ("-S" "-n" "-d" "norsk") "~html" iso-8859-1)

   ("norsk7-ascii-comp"                 ; 7 bit Norwegian (aa, ae, oe)

    "[A-Za-z]" "[^A-Za-z]"

    "[\".,;:]" t ("-S" "-d" "norsk") "~ascii" iso-8859-1)

    ("norsk7-iso246" "[][A-Za-z{}|\\]" "[^][A-Za-z{}|\\]"

     "[\".,;:]" nil ("-B" "-S" "-d" "norsk")  "~iso246" iso-8859-1)

("nynorsk"                             ; 8 bit Norwegian mode

    "[A-Za-z\305\306\307\310\311\322\323\324\330\345\346\347\350\351\362\363\364\370]"

    "[^A-Za-z\305\306\307\310\311\322\323\324\330\345\346\347\350\351\362\363\364\370]"

    "[\".,;:]" t ("-B" "-S" "-d" "nynorsk") "~list" iso-8859-1)

   ("nynorsk7-tex"                        ; 7 bit Norwegian TeX mode

    "[A-Za-z{}\\'^`@]" "[^A-Za-z{}\\'^`@]"

    "[\".,;:]" t ("-B" "-S" "-d" "nynorsk" "-T" "plaintex") "~plaintex" nil)

   ("nynorsk7-html"                       ; 7 bit Norwegian html mode

    "[A-Za-z\&;]" "[^A-Za-z\&;]"        ; Don't use ispell's html-parser

    "[.,:]" t ("-B" "-S" "-n" "-d" "nynorsk") "~html" iso-8859-1)

   ("nynorsk7-ascii"                      ; 7 bit Norwegian (aa, ae, oe)

    "[A-Za-z]" "[^A-Za-z]"

    "[\".,;:]" t ("-B" "-S" "-d" "nynorsk") "~ascii" iso-8859-1)

    ("nynorsk7-iso246" "[][A-Za-z{}|\\]" "[^][A-Za-z{}|\\]"

     "[\".,;:]" nil ("-B" "-S" "-d" "nynorsk")  "~iso246" iso-8859-1)

   ("nynorsk-comp"                        ; 8 bit Norwegian mode

    "[A-Za-z\305\306\307\310\311\322\323\324\330\345\346\347\350\351\362\363\364\370]"

    "[^A-Za-z\305\306\307\310\311\322\323\324\330\345\346\347\350\351\362\363\364\370]"

    "[\".,;:]" t ("-S" "-d" "nynorsk") "~list" iso-8859-1)

   ("nynorsk7-tex-comp"                   ; 7 bit Norwegian TeX mode

    "[A-Za-z{}\\'^`@]" "[^A-Za-z{}\\'^`@]"

    "[\".,;:]" t ("-S" "-d" "nynorsk" "-T" "plaintex") "~plaintex" nil)

   ("nynorsk7-html-comp"                  ; 7 bit Norwegian html mode

    "[A-Za-z\&;]" "[^A-Za-z\&;]"        ; Don't use ispell's html-parser

    "[.,:]" t ("-S" "-n" "-d" "nynorsk") "~html" iso-8859-1)

   ("nynorsk7-ascii-comp"                 ; 7 bit Norwegian (aa, ae, oe)

    "[A-Za-z]" "[^A-Za-z]"

    "[\".,;:]" t ("-S" "-d" "nynorsk") "~ascii" iso-8859-1)

    ("nynorsk7-iso246" "[][A-Za-z{}|\\]" "[^][A-Za-z{}|\\]"

     "[\".,;:]" nil ("-B" "-S" "-d" "nynorsk")  "~iso246" iso-8859-1)

   ))

(load-library "ispell")

The above is very unpretty indeed.  It is basically four copies of the

same list.  If you come up with something better, please let me know.

I am a terrible lisp programmer!

As you see there are a lot of entries.  The -comp entries puts ispell

in controlled compoundwords mode.  Nice to do for a quick spell-check.

I recommend to delete the entries you you don't plan to use.  I like

to use the -S switch, e.g. not sort the suggestions made by ispell.

Then it is more likely that the correct suggestion will be early in

the list.

In the future I hope that ispell will be able to sort the suggestions

it makes by commonness, at least for the most common words.  That

should not be too difficult to implement.  Just load the most common

words and their frequency indicator into memory, and do the nessesary

lookups.  Or use the external look program.  Suggestions and

implementations are most welcome!

There is also a file flyspell.el around.  This also offers

spell-checking on the fly, and the interface is more like m$-word.

Flyspell-mode highlights incorrect words, and you can even click on

them to get suggestions for correct spelling.  Being able to sort on

commonness would make flyspell's auto-correction mode much more

useful!

USING ISPELL IN BATCH MODE

I find ispell's batch mode very useful.  The command

cat myfile.tex | ispell -l -d norsk | sort | uniq -c | sort -n -r -s

prints all words in myfile.tex that is not in the Norwegian

dictionary, where the most common words comes first.  Nice to spot

errors, or as a starting point for a local dictionary.

HYPHENATION IN TEX

Two sets of hyphenation patterns for the Norwegian language are

provided.  The file norskb.tex hyphenates almost as TeX used to, and

the file nohyphbc.tex only splits compound words.

It is fairly easy to install the nohyphb.tex file.  Just put it where

TeX can find it, edit the file language.dat to point to the correct

file, and remake the formats.  If you use teTeX you just say texconfig

init.

If you want to install both sets of patterns, you have a TeX capacity

problem.  The variable ssup_tree_size needs to be bigger than 65535

and trie_op_size bigger than 1501.  I use 262142 and 3501.  So you

need to change tex.ch (and omega.ch) and recompile TeX.  If you are

using teTeX that should be quite easy.  Here is a patch:

*** tex.ch~     Fri Jan 21 23:13:24 2000

--- tex.ch      Mon Jul 10 18:46:15 2000

***************

*** 196 ****

! @d ssup_trie_size == 65535

--- 196 ----

! @d ssup_trie_size == 262143

***************

*** 215 ****

! @!trie_op_size=1501; {space for ``opcodes'' in the hyphenation patterns;

--- 215 ----

! @!trie_op_size=3501; {space for ``opcodes'' in the hyphenation patterns;

***************

*** 217 ****

! @!neg_trie_op_size=-1501; {for lower |trie_op_hash| array bound;

--- 217 ----

! @!neg_trie_op_size=-3501; {for lower |trie_op_hash| array bound;

*** omega.ch~  Thu Jul 13 11:37:08 2000

--- omega.ch   Sun Jul 23 20:38:03 2000

***************

*** 125,127 ****

  @d ssup_trie_opcode == 65535

! @d ssup_trie_size == 100000

--- 125,127 ----

  @d ssup_trie_opcode == 65535

! @d ssup_trie_size == 262143

***************

*** 139,143 ****

    {Use |hash_offset=0| for compilers which cannot decrement pointers.}

! @!trie_op_size=1501; {space for ``opcodes'' in the hyphenation patterns;

    best if relatively prime to 313, 361, and 1009.}

! @!neg_trie_op_size=-1501; {for lower |trie_op_hash| array bound;

    must be equal to |-trie_op_size|.}

--- 139,143 ----

    {Use |hash_offset=0| for compilers which cannot decrement pointers.}

! @!trie_op_size=3501; {space for ``opcodes'' in the hyphenation patterns;

    best if relatively prime to 313, 361, and 1009.}

! @!neg_trie_op_size=-3501; {for lower |trie_op_hash| array bound;

    must be equal to |-trie_op_size|.}

The easiest way to use the norskbc patterns is to define the macros

\def\goodhyphens{\lefthyphenmin2\righthyphenmin2\language=\l@norskc}

\def\allhyphens{\lefthyphenmin1\righthyphenmin2\language=\l@norsk}

and change whenever you want to. A better solution might be to define

norskc as another language in the Babel system anf use the Babel

language switching system.

MAKING IT PERFECT

So you have installed these great new patterns.  But TeX still might

fail on Norwegian words not in the dictionary, so if you don't feel

particularly lucky you will have to do something about that too.

There are two strategies.  I tend to prefer the second one.

1. Mark the compound point in the compound word with "-, e.g.

   administrasjons"-sjef"-stillings-"søker.  If you have patched

   ispell, you can do this during spell-checking most of the time.

2. Use the script inorsk-hyphenmaybe to print every word in your

   document not in the dictionary (nynorsk and bokmål) hyphenated by

   TeX.  Then you can easily browse through this list and put the

   badly hyphenated words in a \hyphenation command.  The next time

   you run the script it should produce correct hyphenation.

   For example if inorsk-hyphenmaybe outputs `kon-flik-t-akse' and

   `kon-flik-t-ak-sen' you have to say \hyphenation{kon-flikt-akse

   `kon-flikt-ak-sen'} in your TeX document.

But we are not done with hyphenation yet.  Have you ever considered

the problem of hyphenating the word `villede' in TeX.  Of course you

have.  The hyphenation should be `vill-lede', thus an extra `l' should

be added.

Most languages which have such hyphenation (in particular German, with

ss) support this in Babel.  The convention is that you code villede as

vi"llede.  Of course the Norwegian dictionary supports this. Babel-3.7

will also support this for Norwegian.  Till then you can use the file

norsk.cfg to get this functionality (and some special hyphen points in

addition).  The file itself offers more information.

THE FUTURE OF HYPHENATION IN TEX

In standard TeX today it is not possible to say that one hyphen point

is better than another, e.g. I like barnehage-assistent better than

barne-hageassistent.  In the future TeX will be able to handle

multiple classes of hyphens and different penalties can be assigned to

each class.  Mathias Clasen has implemented this as a change file,

but it has not made it into the standard distributions yet.  The stuff

at the end of the patterns/Makefile is about generating hyphenation

patterns for such a TeX.

LETS MAKE THE DICTIONARY EVEN BETTER!

In the future I would like to add more word categories to the

dictionary.  If you have a lot of text from within one field of

knowledge, and would like to help, you can start by saying

cat allmytextfiles | inorsk-hyphenmaybe -e -p norskbc > mywords

You should install the hyphenation patterns norskbc for Norwegian to

get hyphenation only at compound points, and of course the full

dictionary with no words filtered out.

You will probably spot some new words, some of your own spelling

errors and some hyphenation errors.  Fix that file, add flags defined

in the affix file etc.

Next you have to learn to use the munchlist program.  Suppose you have

the words in the file mywords

gjennom-strømnings-mekanisme

gjennom-strømnings-mekanismen

gjennom-strømnings-mekanismens

gjennom-strømnings-mekanismer

gjennom-strømnings-mekanismene

cat mywords \

  | tr '-' 'Î' \

  | munchlist -v -l norsk.aff.munch \

  | tr 'Î' '-'

the output should be

gjennom-strømnings-mekanisme/AEG

which represents these five words.  (Of course this only work if

ispell and munchlist is correctly installed.)

Here is some elisp stuff I have used (provided as is, probably very badly coded):

(defun ispell-expand-affixes () (interactive)

  (shell-command-on-region (mark) (point) "sed -e \"s/[-0-9 	:]//g\" | ispell -e -d norsk"))

(defun ispell-collect-affixes () (interactive)

  (shell-command (concat

		  "echo \"" (buffer-substring-no-properties (mark) (point))

		  "\" | sed -e \"s/-/î/g\" -e \"s/[0-9 	:]//g\" | "

		  "munchlist -l norsk.aff.munch | sed -e \"s/î/-/g\" &")))

(defun ispell-expand-line () (interactive)

  (save-excursion

  (beginning-of-line)

  (let ((beg (point)))

  (end-of-line)

  (let ((end (point))))

  (shell-command-on-region beg (point) "sed -e \"s/[-0-9 	:]//g\" | ispell -d norsk -e"))))

; We have to quote the `' characters to protect them from shell

; expansion.

(defun current-line ()

  (save-excursion

    (beginning-of-line)

    (let ((beg (point)))

      (end-of-line)

      (let ((end (point)))

	(setq myvar (buffer-substring-no-properties beg end))

	(while (string-match " .*" myvar)

		 (setq myvar (replace-match "" nil nil myvar)))

	(while (string-match "\\([^\\]\\)\\([`'\"]\\|\\\\$\\)" myvar)

		 (setq myvar (replace-match "\\1\\\\\\2" nil nil myvar)))

	(while (string-match "[0-9 \t:.*]" myvar)

		 (setq myvar (replace-match "" nil nil myvar)))

	myvar))))

(defun current-region ()

  (setq myvar (buffer-substring-no-properties (mark) (point)))

  (while (string-match "\\([^\\]\\)\\([`'\"]\\|\\\\$\\)" myvar)

    (setq myvar (replace-match "\\1\\\\\\2" nil nil myvar)))

  (while (string-match "[0-9 \t]" myvar)

    (setq myvar (replace-match "" nil nil myvar)))

  myvar)