<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<book id="alex">
<bookinfo>
<date>2003-8-11</date>
<title>Alex User Guide</title>
<author>
<firstname>Chris</firstname>
<surname>Dornan</surname>
</author>
<author>
<firstname>Isaac</firstname>
<surname>Jones</surname>
</author>
<author>
<firstname>Simon</firstname>
<surname>Marlow</surname>
</author>
<address><email>ijones@syntaxpolice.org</email></address>
<!-- <copyright> -->
<!-- <year>1997-2001</year> -->
<!-- <holder>Simon Marlow</holder> -->
<!-- </copyright> -->
<abstract>
<para>Alex is a tool for generating lexical analysers in
Haskell, given a description of the tokens to be recognised in
the form of regular expressions. It is similar to the tool
<quote>lex</quote> or <quote>flex</quote> for C/C++.</para>
</abstract>
</bookinfo>
<!-- Table of contents -->
<toc></toc>
<chapter id="about">
<title>About Alex</title>
<para>Alex can always be obtained from its <ulink
url="http://www.haskell.org/alex">home page</ulink>. The latest
source code lives in the <ulink url="https://github.com/simonmar/alex">git
repository</ulink> on <literal>GitHub</literal>.
</para>
<section id="relnotes-30">
<title>Release Notes for version 3.0</title>
<itemizedlist>
<listitem>
<para>Unicode support (contributed mostly by Jean-Philippe
Bernardy, with help from Alan Zimmerman).</para>
<itemizedlist>
<listitem>
<para>
An Alex lexer now takes a UTF-8 encoded byte sequence
as input (see <xref linkend="encoding" />. If you are
using the "basic" wrapper or one of the other wrappers
that takes a Haskell <literal>String</literal> as
input, the string is automatically encoded into UTF-8
by Alex. If your input is
a <literal>ByteString</literal>, you are responsible
for ensuring that the input is UTF-8 encoded. The old
8-bit behaviour is still available via
the <option>--latin1</option> option.
</para>
</listitem>
<listitem>
<para>
Alex source files are assumed to be in UTF-8, like
Haskell source files. The lexer specification can use
Unicode characters and ranges.
</para>
</listitem>
<listitem>
<para>
<literal>alexGetChar</literal> is renamed to
<literal>alexGetByte</literal> in the generated code.
</para>
</listitem>
<listitem>
<para>
There is a new option, <option>--latin1</option>, that
restores the old behaviour.
</para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para>
Alex now does DFA minimization, which helps to reduce the
size of the generated tables, especially for lexers that
use Unicode.
</para>
</listitem>
</itemizedlist>
</section>
<section id="relnotes-22">
<title>Release Notes for version 2.2</title>
<itemizedlist>
<listitem>
<para>Cabal 1.2 is now required.</para>
</listitem>
<listitem>
<para>ByteString wrappers: use Alex to lex ByteStrings
directly.</para>
</listitem>
</itemizedlist>
</section>
<section id="relnotes-210">
<title>Release Notes for version 2.1.0</title>
<itemizedlist>
<listitem>
<para>Switch to a Cabal build system: you need a recent
version of Cabal (1.1.6 or later). If you have GHC 6.4.2,
then you need to upgrade Cabal before building Alex. GHC
6.6 is fine.</para>
</listitem>
<listitem>
<para>Slight change in the error semantics: the input
returned on error is before the erroneous character was
read, not after. This helps to give better error
messages.</para>
</listitem>
</itemizedlist>
</section>
<section id="relnotes-20">
<title>Release Notes for version 2.0</title>
<para>Alex has changed a <emphasis>lot</emphasis> between
versions 1.x and 2.0. The following is supposed to be an
exhaustive list of the changes:</para>
<section id="changes-syntax">
<title>Syntax changes</title>
<itemizedlist>
<listitem>
<para>Code blocks are now surrounded by
<literal>{...}</literal> rather than
<literal>%{...%}</literal>.</para>
</listitem>
<listitem>
<para>Character-set macros now begin with
‘<literal>$</literal>’ instead of
‘<literal>^</literal>’ and have
multi-character names.</para>
</listitem>
<listitem>
<para>Regular expression macros now begin with
‘<literal>@</literal>’ instead of
‘<literal>%</literal>’ and have
multi-character names.</para>
</listitem>
<listitem>
<para>Macro definitions are no longer surrounded by
<literal>{ ... }</literal>.</para>
</listitem>
<listitem>
<para>Rules are now of the form
<programlisting><c1,c2,...> regex { code }</programlisting>
where <literal>c1</literal>, <literal>c2</literal> are
startcodes, and <literal>code</literal> is an arbitrary
Haskell expression.</para>
</listitem>
<listitem>
<para>Regular expression syntax changes:</para>
<itemizedlist>
<listitem>
<para><literal>()</literal> is the empty regular
expression (used to be
‘<literal>$</literal>’)</para>
</listitem>
<listitem>
<para>set complement can now be expressed as
<literal>[^sets]</literal> (for similarity with lex
regular expressions).</para>
</listitem>
<listitem>
<para>The <literal>'abc'</literal> form is no longer
available, use <literal>[abc]</literal>
instead.</para>
</listitem>
<listitem>
<para>‘<literal>^</literal>’ and
‘<literal>$</literal>’ have the usual
meanings: ‘<literal>^</literal>’ matches
just after a ‘<literal>\n</literal>’, and
‘<literal>$</literal>’ matches just before
a ‘<literal>\n</literal>’.</para>
</listitem>
<listitem>
<para>‘<literal>\n</literal>’ is now the
escape character, not
‘<literal>^</literal>’.</para>
</listitem>
<listitem>
<para>The form <literal>"..."</literal> means the same
as the sequence of characters inside the quotes, the
difference being that special characters do not need
to be escaped inside <literal>"..."</literal>.</para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para>Rules can have arbitrary predicates attached to
them. This subsumes the previous left-context and
right-context facilities (although these are still allowed
as syntactic sugar).</para>
</listitem>
</itemizedlist>
</section>
<section id="changes-files">
<title>Changes in the form of an Alex file</title>
<itemizedlist>
<listitem>
<para>Each file can now only define a single grammar.
This change was made to simplify code generation.
Multiple grammars can be simulated using startcodes, or
split into separate modules.</para>
</listitem>
<listitem>
<para>The programmer experience has been simplified, and
at the same time made more flexible. See the <xref
linkend="api"/> for details.</para>
</listitem>
<listitem>
<para>You no longer need to import the
<literal>Alex</literal> module.</para>
</listitem>
</itemizedlist>
</section>
<section id="changes-usage">
<title>Usage changes</title>
<para>The command-line syntax is quite different. See <xref
linkend="invoking"/>.</para>
</section>
<section id="changes-implementation">
<title>Implementation changes</title>
<itemizedlist>
<listitem>
<para>A more efficient table representation, coupled with
standard table-compression techniques, are used to keep
the size of the generated code down.</para>
</listitem>
<listitem>
<para>When compiling a grammar with GHC, the -g switch
causes an even faster and smaller grammar to be
generated.</para>
</listitem>
<listitem>
<para>Startcodes are implemented in a different way: each
state corresponds to a different initial state in the DFA,
so the scanner doesn't have to check the startcode when it
gets to an accept state. This results in a larger, but
quicker, scanner.</para>
</listitem>
</itemizedlist>
</section>
</section>
<section id="bug-reports">
<title>Reporting bugs in Alex</title>
<para>Please report bugs in Alex to
<email>simonmar@microsoft.com</email>. There are no specific
mailing lists for the discussion of Alex-related matters, but
such topics should be fine on the <ulink
url="http://www.haskell.org/mailman/listinfo/haskell-cafe">Haskell
Cafe</ulink> mailing list.</para>
</section>
<section id="license">
<title>License</title>
<para>Copyright (c) 1995-2011, Chris Dornan and Simon Marlow.
All rights reserved.</para>
<para>Redistribution and use in source and binary forms, with or
without modification, are permitted provided that the following
conditions are met:</para>
<itemizedlist>
<listitem>
<para>Redistributions of source code must retain the above
copyright notice, this list of conditions and the following
disclaimer.</para>
</listitem>
<listitem>
<para>Redistributions in binary form must reproduce the
above copyright notice, this list of conditions and the
following disclaimer in the documentation and/or other
materials provided with the distribution.</para>
</listitem>
<listitem>
<para>Neither the name of the copyright holders, nor the
names of the contributors may be used to endorse or promote
products derived from this software without specific prior
written permission.</para>
</listitem>
</itemizedlist>
<para>THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
THE POSSIBILITY OF SUCH DAMAGE.</para>
</section>
</chapter>
<chapter id="introduction">
<title>Introduction</title>
<para>Alex is a tool for generating lexical analysers in Haskell,
given a description of the tokens to be recognised in the form of
regular expressions. It is similar to the tools
lex and flex for C/C++.</para>
<para>Alex takes a description of tokens based on regular
expressions and generates a Haskell module containing code for
scanning text efficiently. Alex is designed to be familiar to
existing lex users, although it does depart from lex in a number
of ways.</para>
<figure id="fig-tokens" float="1"><title>A simple Alex specification.</title>
<programlisting>{
module Main (main) where
}
%wrapper "basic"
$digit = 0-9 -- digits
$alpha = [a-zA-Z] -- alphabetic characters
tokens :-
$white+ ;
"--".* ;
let { \s -> Let }
in { \s -> In }
$digit+ { \s -> Int (read s) }
[\=\+\-\*\/\(\)] { \s -> Sym (head s) }
$alpha [$alpha $digit \_ \']* { \s -> Var s }
{
-- Each action has type :: String -> Token
-- The token type:
data Token =
Let |
In |
Sym Char |
Var String |
Int Int
deriving (Eq,Show)
main = do
s <- getContents
print (alexScanTokens s)
}</programlisting></figure>
<para>A sample specification is given in <xref
linkend="fig-tokens"/>. The first few lines between the
<literal>{</literal> and <literal>}</literal> provide a code scrap
(some inlined Haskell code) to be placed directly in the output,
the scrap at the top of the module is normally used to declare the
module name for the generated Haskell module, in this case
<literal>Main</literal>.</para>
<para>The next line, <literal>%wrapper "basic"</literal> controls
what kind of support code Alex should produce along with the basic
scanner. The <literal>basic</literal> wrapper selects a scanner
that tokenises a <literal>String</literal> and returns a list of
tokens. Wrappers are described fully in <xref
linkend="api"/>.</para>
<para>The next two lines define the <literal>$digit</literal> and
<literal>$alpha</literal> macros for use in the token
definitions.</para>
<para>The ‘<literal>tokens :-</literal>’ line ends the
macro definitions and starts the definition of the scanner.</para>
<para>The scanner is specified as a series of token definitions
where each token specification takes the form of</para>
<programlisting><replaceable>regexp</replaceable> { <replaceable>code</replaceable> }</programlisting>
<para>The meaning of this rule is "if the input matches
<replaceable>regexp</replaceable>, then return
<replaceable>code</replaceable>". The code part along with the
braces can be replaced by simply
‘<literal>;</literal>’, meaning that this token should
be ignored in the input stream. As you can see, we've used this
to ignore whitespace in our example.</para>
<para>Our scanner is set up so that the actions are all functions
with type <literal>String->Token</literal>. When the token is
matched, the portion of the input stream that it matched is passed
to the appropriate action function as a
<literal>String</literal>.</para>
<para>At the bottom of the file we have another code fragment,
surrounded by braces <literal>{ ... }</literal>. In this
fragment, we declare the type of the tokens, and give a
<literal>main</literal> function that we can use for testing it;
the <literal>main</literal> function just tokenises the input and
prints the results to standard output.</para>
<para>Alex has kindly provided the following function which we can
use to invoke the scanner:</para>
<programlisting>alexScanTokens :: String -> [Token]</programlisting>
<para>Alex arranges for the input stream to be tokenised, each of
the action functions to be passed the appropriate
<literal>String</literal>, and a list of <literal>Token</literal>s
returned as the result. If the input stream is lazy, the output
stream will also be produced lazily<footnote><para>that is, unless you
have any patterns that require a long lookahead.</para>
</footnote>.</para>
<para>We have demonstrated the simplest form of scanner here,
which was selected by the <literal>%wrapper "basic"</literal> line
near the top of the file. In general, actions do not have to have
type <literal>String->Token</literal>, and there's no requirement
for the scanner to return a list of tokens.</para>
<para>With this specification in the file
<literal>Tokens.x</literal>, Alex can be used to generate
<literal>Tokens.hs</literal>:</para>
<screen>$ alex Tokens.x</screen>
<para>If the module needed to be placed in a different file,
<literal>Main.hs</literal> for example, then the output filename
can be specified using the <option>-o</option> option:</para>
<screen>$ alex Tokens.x -o Main.hs</screen>
<para>The resulting module is Haskell 98 compatible. It can also
be readily used with a <ulink
url="http://www.haskell.org/happy/">Happy</ulink> parser.</para>
</chapter>
<chapter id="syntax">
<title>Alex Files</title>
<para>In this section we describe the layout of an Alex lexical
specification.</para>
<para>
</para>
<para>
We begin with the lexical syntax; elements of the lexical syntax
are referred to throughout the rest of this documentation, so
you may need to refer back to the following section several
times.
</para>
<section id="lexical">
<title>Lexical syntax</title>
<para>Alex's lexical syntax is given below. It is written as a
set of macro definitions using Alex's own syntax. These macros
are used in the BNF specification of the syntax later on.</para>
<programlisting>$digit = [0-9]
$octdig = [0-7]
$hexdig = [0-9A-Fa-f]
$special = [\.\;\,\$\|\*\+\?\#\~\-\{\}\(\)\[\]\^\/]
$graphic = $printable # $white
@string = \" ($graphic # \")* \"
@id = [A-Za-z][A-Za-z'_]*
@smac = '$' id
@rmac = '@' id
@char = ($graphic # $special) | @escape
@escape = '\\' ($printable | 'x' $hexdig+ | 'o' $octdig+ | $digit+)
@code = -- curly braces surrounding a Haskell code fragment</programlisting>
</section>
<section id="alex-files">
<title>Syntax of Alex files</title>
<para>In the following description of the Alex syntax, we use an
extended form of BNF, where optional phrases are enclosed in
square brackets (<literal>[ ... ]</literal>), and phrases which
may be repeated zero or more times are enclosed in braces
(<literal>{ ... }</literal>). Literal text is enclosed in
single quotes.</para>
<para>An Alex lexical specification is normally placed in a file
with a <literal>.x</literal> extension. Alex source files are
encoded in UTF-8, just like Haskell source
files<footnote><para>Strictly speaking, GHC source
files.</para></footnote>.
</para>
<para>The overall layout of an Alex file is:</para>
<programlisting>alex := [ @code ] [ wrapper ] [ encoding ] { macrodef } @id ':-' { rule } [ @code ]</programlisting>
<para>The file begins and ends with optional code fragments.
These code fragments are copied verbatim into the generated
source file.</para>
<para>At the top of the file, the code fragment is normally used
to declare the module name and some imports, and that is all it
should do: don't declare any functions or types in the top code
fragment, because Alex may need to inject some imports of its
own into the generated lexer code, and it does this by adding
them directly after this code fragment in the output
file.</para>
<para>Next comes an optional directives section</para>
<para>The first kind of directive is a specification:</para>
<programlisting>wrapper := '%wrapper' @string</programlisting>
<para>wrappers are described in <xref linkend="wrappers"/>. This
can be followed by an optional encoding declaration:</para>
<programlisting>encoding := '%encoding' @string</programlisting>
<para>encodings are described in <xref linkend="encoding"/>.</para>
<para>Additionally, you can specify a token type, a typeclass,
or an action type (depending on what wrapper you use):</para>
<programlisting>action type := '%action' @string</programlisting>
<programlisting>token type := '%token' @string</programlisting>
<programlisting>typeclass(es) := '%typeclass' @string</programlisting>
<para>these are described in <xref
linkend="types"/>.</para>
<section id="macrodefs">
<title>Macro definitions</title>
<para>Next, the lexer specification can contain a series of
macro definitions. There are two kinds of macros,
<firstterm>character set macros</firstterm>, which begin with
a <literal>$</literal>, and <firstterm>regular expression
macros</firstterm>, which begin with a <literal>@</literal>.
A character set macro can be used wherever a character set is
valid (see <xref linkend="charsets"/>), and a regular
expression macro can be used wherever a regular expression is
valid (see <xref linkend="regexps"/>).</para>
<programlisting>macrodef := @smac '=' set
| @rmac '=' regexp</programlisting>
</section>
<section id="rules">
<title>Rules</title>
<para>The rules are heralded by the sequence
‘<literal><replaceable>id</replaceable> :-</literal>’
in the file. It doesn't matter what you use for the
identifier, it is just there for documentation purposes. In
fact, it can be omitted, but the <literal>:-</literal> must be
left in.</para>
<para>The syntax of rules is as follows:</para>
<programlisting>rule := [ startcodes ] token
| startcodes '{' { token } '}'
token := [ left_ctx ] regexp [ right_ctx ] rhs
rhs := @code | ';'</programlisting>
<para>Each rule defines one token in the lexical
specification. When the input stream matches the regular
expression in a rule, the Alex lexer will return the value of
the expression on the right hand side, which we call the
<firstterm>action</firstterm>. The action can be any Haskell
expression. Alex only places one restriction on actions: all
the actions must have the same type. They can be values in a
token type, for example, or possibly operations in a monad.
More about how this all works is in <xref
linkend="api"/>.</para>
<para>The action may be missing, indicated by replacing it
with ‘<literal>;</literal>’, in which case the
token will be skipped in the input stream.</para>
<para>Alex will always find the longest match. For example,
if we have a rule that matches whitespace:</para>
<programlisting>$white+ ;</programlisting>
<para>Then this rule will match as much whitespace at the
beginning of the input stream as it can. Be careful: if we
had instead written this rule as</para>
<programlisting>$white* ;</programlisting>
<para>then it would also match the empty string, which would
mean that Alex could never fail to match a rule!</para>
<para>When the input stream matches more than one rule, the
rule which matches the longest prefix of the input stream
wins. If there are still several rules which match an equal
number of characters, then the rule which appears earliest in
the file wins.</para>
<section id="contexts">
<title>Contexts</title>
<para>Alex allows a left and right context to be placed on
any rule:</para>
<programlisting>
left_ctx := '^'
| set '^'
right_ctx := '$'
| '/' regexp
| '/' @code
</programlisting>
<para>The left context matches the character which
immediately precedes the token in the input stream. The
character immediately preceding the beginning of the stream
is assumed to be ‘<literal>\n</literal>’. The
special left-context ‘<literal>^</literal>’ is
shorthand for ‘<literal>\n^</literal>’.</para>
<para>Right context is rather more general. There are three
forms:</para>
<variablelist>
<varlistentry>
<term>
<literal>/ <replaceable>regexp</replaceable></literal>
</term>
<listitem>
<para>This right-context causes the rule to match if
and only if it is followed in the input stream by text
which matches
<replaceable>regexp</replaceable>.</para>
<para>NOTE: this should be used sparingly, because it
can have a serious impact on performance. Any time
this rule <emphasis>could</emphasis> match, its
right-context will be checked against the current
input stream.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>$</literal></term>
<listitem>
<para>Equivalent to
‘<literal>/\n</literal>’.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>/ { ... }</literal></term>
<listitem>
<para>This form is called a
<emphasis>predicate</emphasis> on the rule. The
Haskell expression inside the curly braces should have
type:
<programlisting>{ ... } :: user -- predicate state
-> AlexInput -- input stream before the token
-> Int -- length of the token
-> AlexInput -- input stream after the token
-> Bool -- True <=> accept the token</programlisting>
Alex will only accept the token as matching if
the predicate returns <literal>True</literal>.</para>
<para>See <xref linkend="api"/> for the meaning of the
<literal>AlexInput</literal> type. The
<literal>user</literal> argument is available for
passing into the lexer a special state which is used
by predicates; to give this argument a value, the
<literal>alexScanUser</literal> entry point to the
lexer must be used (see <xref
linkend="basic-api"/>).</para>
</listitem>
</varlistentry>
</variablelist>
</section>
<section id="startcodes">
<title>Start codes</title>
<para>Start codes are a way of adding state to a lexical
specification, such that only certain rules will match for a
given state.</para>
<para>A startcode is simply an identifier, or the special
start code ‘<literal>0</literal>’. Each rule
may be given a list of startcodes under which it
applies:</para>
<programlisting>startcode := @id | '0'
startcodes := '<' startcode { ',' startcode } '>'</programlisting>
<para>When the lexer is invoked to scan the next token from
the input stream, the start code to use is also specified
(see <xref linkend="api"/>). Only rules that mention this
start code are then enabled. Rules which do not have a list
of startcodes are available all the time.</para>
<para>Each distinct start code mentioned in the lexical
specification causes a definition of the same name to be
inserted in the generated source file, whose value is of
type <literal>Int</literal>. For example, if we mentioned
startcodes <literal>foo</literal> and <literal>bar</literal>
in the lexical spec, then Alex will create definitions such
as:
<programlisting>foo = 1
bar = 2</programlisting>
in the output file.</para>
<para>Another way to think of start codes is as a way to
define several different (but possibly overlapping) lexical
specifications in a single file, since each start code
corresponds to a different set of rules. In concrete terms,
each start code corresponds to a distinct initial state in
the state machine that Alex derives from the lexical
specification.</para>
<para>Here is an example of using startcodes as states, for
collecting the characters inside a string:</para>
<programlisting><0> ([^\"] | \n)* ;
<0> \" { begin string }
<string> [^\"] { stringchar }
<string> \" { begin 0 }</programlisting>
<para>When it sees a quotation mark, the lexer switches into
the <literal>string</literal> state and each character
thereafter causes a <literal>stringchar</literal> action,
until the next quotation mark is found, when we switch back
into the <literal>0</literal> state again.</para>
<para>From the lexer's point of view, the startcode is just
an integer passed in, which tells it which state to start
in. In order to actually use it as a state, you must have
some way for the token actions to specify new start codes -
<xref linkend="api"/> describes some ways this can be done.
In some applications, it might be necessary to keep a
<emphasis>stack</emphasis> of start codes, where at the end
of a state we pop the stack and resume parsing in the
previous state. If you want this functionality, you have to
program it yourself.</para>
</section>
</section> <!-- rules -->
</section> <!-- syntax of alex files -->
</chapter> <!-- alex files -->
<chapter id="regexps">
<title>Regular Expression</title>
<para>Regular expressions are the patterns that Alex uses to match
tokens in the input stream.</para>
<section id="regexp-syntax">
<title>Syntax of regular expressions</title>
<programlisting>regexp := rexp2 { '|' rexp2 }
rexp2 := rexp1 { rexp1 }
rexp1 := rexp0 [ '*' | '+' | '?' | repeat ]
rexp0 := set
| @rmac
| @string
| '(' [ regexp ] ')'
repeat := '{' $digit '}'
| '{' $digit ',' '}'
| '{' $digit ',' $digit '}'</programlisting>
<para>The syntax of regular expressions is fairly standard, the
only difference from normal lex-style regular expressions being
that we allow the sequence <literal>()</literal> to denote the
regular expression that matches the empty string.</para>
<para>Spaces are ignored in a regular expression, so feel free
to space out your regular expression as much as you like, even
split it over multiple lines and include comments. Literal
whitespace can be included by surrounding it with quotes
<literal>" "</literal>, or by escaping each whitespace character
with <literal>\</literal>.</para>
<variablelist>
<varlistentry>
<term><literal><replaceable>set</replaceable></literal></term>
<listitem>
<para>Matches any of the characters in
<replaceable>set</replaceable>. See <xref
linkend="charsets"/> for the syntax of sets.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>@foo</literal></term>
<listitem>
<para>Expands to the definition of the appropriate
regular expression macro.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>"..."</literal></term>
<listitem>
<para>Matches the sequence of characters in the string, in
that order.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal><replaceable>r</replaceable>*</literal></term>
<listitem>
<para>Matches zero or more occurrences of
<replaceable>r</replaceable>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal><replaceable>r</replaceable>+</literal></term>
<listitem>
<para>Matches one or more occurrences of
<replaceable>r</replaceable>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal><replaceable>r</replaceable>?</literal></term>
<listitem>
<para>Matches zero or one occurrences of
<replaceable>r</replaceable>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal><replaceable>r</replaceable>{<replaceable>n</replaceable>}</literal></term>
<listitem>
<para>Matches <replaceable>n</replaceable> occurrences of
<replaceable>r</replaceable>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal><replaceable>r</replaceable>{<replaceable>n</replaceable>,}</literal></term>
<listitem>
<para>Matches <replaceable>n</replaceable> or more occurrences of
<replaceable>r</replaceable>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal><replaceable>r</replaceable>{<replaceable>n</replaceable>,<replaceable>m</replaceable>}</literal></term>
<listitem>
<para>Matches between <replaceable>n</replaceable> and
<replaceable>m</replaceable> (inclusive) occurrences of
<replaceable>r</replaceable>.</para>
</listitem>
</varlistentry>
</variablelist>
</section>
<section id="charsets">
<title>Syntax of character sets</title>
<para>Character sets are the fundamental elements in a regular
expression. A character set is a pattern that matches a single
character. The syntax of character sets is as follows:</para>
<programlisting>set := set '#' set0
| set0
set0 := @char [ '-' @char ]
| '.'
| @smac
| '[' [^] { set } ']'
| '~' set0</programlisting>
<para>The various character set constructions are:</para>
<variablelist>
<varlistentry>
<term><literal><replaceable>char</replaceable></literal></term>
<listitem>
<para>The simplest character set is a single Unicode character.
Note that special characters such as <literal>[</literal>
and <literal>.</literal> must be escaped by prefixing them
with <literal>\</literal> (see the lexical syntax, <xref
linkend="lexical"/>, for the list of special
characters).</para>
<para>Certain non-printable characters have special escape
sequences. These are: <literal>\a</literal>,
<literal>\b</literal>, <literal>\f</literal>,
<literal>\n</literal>, <literal>\r</literal>,
<literal>\t</literal>, and <literal>\v</literal>. Other
characters can be represented by using their numerical
character values (although this may be non-portable):
<literal>\x0A</literal> is equivalent to
<literal>\n</literal>, for example.</para>
<para>Whitespace characters are ignored; to represent a
literal space, escape it with <literal>\</literal>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal><replaceable>char</replaceable>-<replaceable>char</replaceable></literal></term>
<listitem>
<para>A range of characters can be expressed by separating
the characters with a ‘<literal>-</literal>’,
all the characters with codes in the given range are
included in the set. Character ranges can also be
non-portable.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>.</literal></term>
<listitem>
<para>The built-in set ‘<literal>.</literal>’
matches all characters except newline
(<literal>\n</literal>).</para>
<para>Equivalent to the set
<literal>[\x00-\x10ffff] # \n</literal>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal><replaceable>set0</replaceable> # <replaceable>set1</replaceable></literal></term>
<listitem>
<para>Matches all the characters in
<replaceable>set0</replaceable> that are not in
<replaceable>set1</replaceable>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>[<replaceable>sets</replaceable>]</literal></term>
<listitem>
<para>The union of <replaceable>sets</replaceable>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>[^<replaceable>sets</replaceable>]</literal></term>
<listitem>
<para>The complement of the union of the
<replaceable>sets</replaceable>. Equivalent to
‘<literal>. # [<replaceable>sets</replaceable>]</literal>’.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>~<replaceable>set</replaceable></literal></term>
<listitem>
<para>The complement of <replaceable>set</replaceable>.
Equivalent to ‘<literal>. # <replaceable>set</replaceable></literal>’</para>
</listitem>
</varlistentry>
</variablelist>
<para>A set macro is written as <literal>$</literal> followed by
an identifier. There are some builtin character set
macros:</para>
<variablelist>
<varlistentry>
<term><literal>$white</literal></term>
<listitem>
<para>Matches all whitespace characters, including
newline.</para>
<para>Equivalent to the set
<literal>[\ \t\n\f\v\r]</literal>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>$printable</literal></term>
<listitem>
<para>Matches all "printable characters". Currently this
corresponds to Unicode code points 32 to 0x10ffff,
although strictly speaking there are many non-printable
code points in this region. In the future Alex may use a
more precise definition of <literal>$printable</literal>.</para>
</listitem>
</varlistentry>
</variablelist>
<para>Character set macros can be defined at the top of the file
at the same time as regular expression macros (see <xref
linkend="regexps"/>). Here are some example character set
macros:</para>
<programlisting>$lls = a-z -- little letters
$not_lls = ~a-z -- anything but little letters
$ls_ds = [a-zA-Z0-9] -- letters and digits
$sym = [ \! \@ \# \$ ] -- the symbols !, @, #, and $
$sym_q_nl = [ \' \! \@ \# \$ \n ] -- the above symbols with ' and newline
$quotable = $printable # \' -- any graphic character except '
$del = \127 -- ASCII DEL</programlisting>
</section>
</chapter>
<chapter id="api">
<title>The Interface to an Alex-generated lexer</title>
<para>This section answers the question: "How do I include an
Alex lexer in my program?"</para>
<para>Alex provides for a great deal of flexibility in how the
lexer is exposed to the rest of the program. For instance,
there's no need to parse a <literal>String</literal> directly if
you have some special character-buffer operations that avoid the
overheads of ordinary Haskell <literal>String</literal>s. You
might want Alex to keep track of the line and column number in the
input text, or you might wish to do it yourself (perhaps you use a
different tab width from the standard 8-columns, for
example).</para>
<para>The general story is this: Alex provides a basic interface
to the generated lexer (described in the next section), which you
can use to parse tokens given an abstract input type with
operations over it. You also have the option of including a
<firstterm>wrapper</firstterm>, which provides a higher-level
abstraction over the basic interface; Alex comes with several
wrappers.</para>
<section id="encoding">
<title>Unicode and UTF-8</title>
<para>Lexer specifications are written in terms of Unicode
characters, but Alex works internally on a UTF-8 encoded byte
sequence.</para>
<para>
Depending on how you use Alex, the fact that Alex uses UTF-8
encoding internally may or may not affect you. If you use one
of the wrappers (below) that takes input from a
Haskell <literal>String</literal>, then the UTF-8 encoding is
handled automatically. However, if you take input from
a <literal>ByteString</literal>, then it is your
responsibility to ensure that the input is properly UTF-8
encoded.
</para>
<para>
None of this applies if you used the <option>--latin1</option>
option to Alex or specify a Latin-1 encoding via a
<literal>%encoding</literal> declaration. In that case, the input is
just a sequence of 8-bit bytes, interpreted as characters in the Latin-1
character set.
</para>
<para>
The following (case-insenstive) encoding strings are currently
supported:
</para>
<variablelist>
<varlistentry>
<term><literal>%encoding "latin-1"</literal></term>
<term><literal>%encoding "iso-8859-1"</literal></term>
<listitem><para>Declare Latin-1 encoding as described above.</para></listitem>
</varlistentry>
<varlistentry>
<term><literal>%encoding "utf-8"</literal></term>
<term><literal>%encoding "utf8"</literal></term>
<listitem><para>Declare UTF-8 encoding. This is
the default encoding but it may be useful to explicitly declare
this to make protect against Alex being called with the
<option>--latin1</option> flag.</para></listitem>
</varlistentry>
</variablelist>
</section>
<section id="basic-api">
<title>Basic interface</title>
<para>If you compile your Alex file without a
<literal>%wrapper</literal> declaration, then you get access to
the lowest-level API to the lexer. You must provide definitions
for the following, either in the same module or imported from
another module:</para>
<programlisting>type AlexInput
alexGetByte :: AlexInput -> Maybe (Word8,AlexInput)
alexInputPrevChar :: AlexInput -> Char</programlisting>
<para>The generated lexer is independent of the input type,
which is why you have to provide a definition for the input type
yourself. Note that the input type needs to keep track of the
<emphasis>previous</emphasis> character in the input stream;
this is used for implementing patterns with a left-context
(those that begin with <literal>^</literal> or
<literal><replaceable>set</replaceable>^</literal>). If you
don't ever use patterns with a left-context in your lexical
specification, then you can safely forget about the previous
character in the input stream, and have
<literal>alexInputPrevChar</literal> return
<literal>undefined</literal>.</para>
<para>Alex will provide the following function:</para>
<programlisting>alexScan :: AlexInput -- The current input
-> Int -- The "start code"
-> AlexReturn action -- The return value
data AlexReturn action
= AlexEOF
| AlexError
!AlexInput -- Remaining input
| AlexSkip
!AlexInput -- Remaining input
!Int -- Token length
| AlexToken
!AlexInput -- Remaining input
!Int -- Token length
action -- action value</programlisting>
<para>Calling <literal>alexScan</literal> will scan a single
token from the input stream, and return a value of type
<literal>AlexReturn</literal>. The value returned is either:</para>
<variablelist>
<varlistentry>
<term><literal>AlexEOF</literal></term>
<listitem>
<para>The end-of-file was reached.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>AlexError</literal></term>
<listitem>
<para>A valid token could not be recognised.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>AlexSkip</literal></term>
<listitem>
<para>The matched token did not have an action associated
with it.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>AlexToken</literal></term>
<listitem>
<para>A token was matched, and the action associated with
it is returned.</para>
</listitem>
</varlistentry>
</variablelist>
<para>The <literal>action</literal> is simply the value of the
expression inside <literal>{...}</literal> on the
right-hand-side of the appropriate rule in the Alex file.
Alex doesn't specify what type these expressions should have, it
simply requires that they all have the same type, or else you'll
get a type error when you try to compile the generated
lexer.</para>
<para>Once you have the <literal>action</literal>, it is up to
you what to do with it. The type of <literal>action</literal>
could be a function which takes the <literal>String</literal>
representation of the token and returns a value in some token
type, or it could be a continuation that takes the new input and
calls <literal>alexScan</literal> again, building a list of
tokens as it goes.</para>
<para>This is pretty low-level stuff; you have complete
flexibility about how you use the lexer, but there might be a
fair amount of support code to write before you can actually use
it. For this reason, we also provide a selection of wrappers
that add some common functionality to this basic scheme.
Wrappers are described in the next section.</para>
<para>There is another entry point, which is useful if your
grammar contains any predicates (see <xref
linkend="contexts"/>):</para>
<programlisting>alexScanUser
:: user -- predicate state
-> AlexInput -- The current input
-> Int -- The "start code"
-> AlexReturn action</programlisting>
<para>The extra argument, of some type <literal>user</literal>,
is passed to each predicate.</para>
</section>
<section id="wrappers">
<title>Wrappers</title>
<para>To use one of the provided wrappers, include the following
declaration in your file:</para>
<programlisting>%wrapper "<replaceable>name</replaceable>"</programlisting>
<para>where <replaceable>name</replaceable> is the name of the
wrapper, eg. <literal>basic</literal>. The following sections
describe each of the wrappers that come with Alex.</para>
<section>
<title>The "basic" wrapper</title>
<para>The basic wrapper is a good way to obtain a function of
type <literal>String -> [token]</literal> from a lexer
specification, with little fuss.</para>
<para>It provides definitions for
<literal>AlexInput</literal>, <literal>alexGetByte</literal>
and <literal>alexInputPrevChar</literal> that are suitable for
lexing a <literal>String</literal> input. It also provides a
function <literal>alexScanTokens</literal> which takes a
<literal>String</literal> input and returns a list of the
tokens it contains.</para>
<para>The <literal>basic</literal> wrapper provides no support
for using startcodes; the initial startcode is always set to
zero.</para>
<para>Here is the actual code included in the lexer when the
basic wrapper is selected:</para>
<programlisting>
type AlexInput = (Char, -- previous char
[Byte], -- rest of the bytes for the current char
String) -- rest of the input string
alexGetByte :: AlexInput -> Maybe (Byte,AlexInput)
alexGetByte (c,(b:bs),s) = Just (b,(c,bs,s))
alexGetByte (c,[],[]) = Nothing
alexGetByte (_,[],(c:s)) = case utf8Encode c of
(b:bs) -> Just (b, (c, bs, s))
alexInputPrevChar :: AlexInput -> Char
alexInputPrevChar (c,_,_) = c
-- alexScanTokens :: String -> [token]
alexScanTokens str = go ('\n',[],str)
where go inp@(_,_bs,str) =
case alexScan inp 0 of
AlexEOF -> []
AlexError _ -> error "lexical error"
AlexSkip inp' len -> go inp'
AlexToken inp' len act -> act (take len str) : go inp'
</programlisting>
<para>The type signature for <literal>alexScanTokens</literal>
is commented out, because the <literal>token</literal> type is
unknown. All of the actions in your lexical specification
should have type:</para>
<programlisting>{ ... } :: String -> token</programlisting>
<para>for some type <literal>token</literal>.</para>
<para>For an example of the use of the basic wrapper, see the
file <literal>examples/Tokens.x</literal> in the Alex
distribution.</para>
</section>
<section>
<title>The "posn" wrapper</title>
<para>The posn wrapper provides slightly more functionality
than the basic wrapper: it keeps track of line and column
numbers of tokens in the input text.</para>
<para>The posn wrapper provides the following, in addition to
the straightforward definitions of
<literal>alexGetByte</literal> and
<literal>alexInputPrevChar</literal>:</para>
<programlisting>
data AlexPosn = AlexPn !Int -- absolute character offset
!Int -- line number
!Int -- column number
type AlexInput = (AlexPosn, -- current position,
Char, -- previous char
[Byte], -- rest of the bytes for the current char
String) -- current input string
--alexScanTokens :: String -> [token]
alexScanTokens str = go (alexStartPos,'\n',[],str)
where go inp@(pos,_,_,str) =
case alexScan inp 0 of
AlexEOF -> []
AlexError ((AlexPn _ line column),_,_,_) -> error $ "lexical error at " ++ (show line) ++ " line, " ++ (show column) ++ " column"
AlexSkip inp' len -> go inp'
AlexToken inp' len act -> act pos (take len str) : go inp'
</programlisting>
<para>The types of the token actions should be:</para>
<programlisting>{ ... } :: AlexPosn -> String -> token</programlisting>
<para>For an example using the <literal>posn</literal>
wrapper, see the file
<literal>examples/Tokens_posn.x</literal> in the Alex
distribution.</para>
</section>
<section>
<title>The "monad" wrapper</title>
<para>The <literal>monad</literal> wrapper is the most
flexible of the wrappers provided with Alex. It includes a
state monad which keeps track of the current input and text
position, and the startcode. It is intended to be a template
for building your own monads - feel free to copy the code and
modify it to build a monad with the facilities you
need.</para>
<programlisting>data AlexState = AlexState {
alex_pos :: !AlexPosn, -- position at current input location
alex_inp :: String, -- the current input
alex_chr :: !Char, -- the character before the input
alex_bytes :: [Byte], -- rest of the bytes for the current char
alex_scd :: !Int -- the current startcode
}
newtype Alex a = Alex { unAlex :: AlexState
-> Either String (AlexState, a) }
instance Functor Alex where ...
instance Applicative Alex where ...
instance Monad Alex where ...
runAlex :: String -> Alex a -> Either String a
type AlexInput = (AlexPosn, -- current position,
Char, -- previous char
[Byte], -- rest of the bytes for the current char
String) -- current input string
alexGetInput :: Alex AlexInput
alexSetInput :: AlexInput -> Alex ()
alexError :: String -> Alex a
alexGetStartCode :: Alex Int
alexSetStartCode :: Int -> Alex ()</programlisting>
<para>To invoke a scanner under the <literal>monad</literal>
wrapper, use <literal>alexMonadScan</literal>:</para>
<programlisting>alexMonadScan :: Alex result</programlisting>
<para>The token actions should have the following type:</para>
<programlisting>type AlexAction result = AlexInput -> Int -> Alex result
{ ... } :: AlexAction result</programlisting>
<para>The Alex file must also define a function
<literal>alexEOF</literal>, which will be executed on when the
end-of-file is scanned:</para>
<programlisting>alexEOF :: Alex result</programlisting>
<para>The <literal>monad</literal> wrapper also provides some
useful combinators for constructing token actions:</para>
<programlisting>-- skip :: AlexAction result
skip input len = alexMonadScan
-- andBegin :: AlexAction result -> Int -> AlexAction result
(act `andBegin` code) input len = do alexSetStartCode code; act input len
-- begin :: Int -> AlexAction result
begin code = skip `andBegin` code
-- token :: (AlexInput -> Int -> token) -> AlexAction token
token t input len = return (t input len)</programlisting>
</section>
<section>
<title>The "monadUserState" wrapper</title>
<para>The <literal>monadUserState</literal> wrapper is built
upon the <literal>monad</literal> wrapper. It includes a reference
to a type which must be defined in the user's program,
<literal>AlexUserState</literal>, and a call to an initialization
function which must also be defined in the user's program,
<literal>alexInitUserState</literal>. It gives great flexibility
because it is now possible to add any needed information and carry
it during the whole lexing phase.</para>
<para>The generated code is the same as in the <literal>monad</literal>
wrapper, except in 3 places:</para>
<para>1) The definition of the general state, which now refers to a
type <literal>AlexUserState</literal> that must be defined in the Alex file.</para>
<programlisting>
data AlexState = AlexState {
alex_pos :: !AlexPosn, -- position at current input location
alex_inp :: String, -- the current input
alex_chr :: !Char, -- the character before the input
alex_bytes :: [Byte], -- rest of the bytes for the current char
alex_scd :: !Int, -- the current startcode
alex_ust :: AlexUserState -- AlexUserState will be defined in the user program
}
</programlisting>
<para>2) The initialization code, where a user-specified routine (<literal>alexInitUserState</literal>) will be
called.</para>
<programlisting>
runAlex :: String -> Alex a -> Either String a
runAlex input (Alex f)
= case f (AlexState {alex_pos = alexStartPos,
alex_inp = input,
alex_chr = '\n',
alex_bytes = [],
alex_ust = alexInitUserState,
alex_scd = 0}) of Left msg -> Left msg
Right ( _, a ) -> Right a
</programlisting>
<para>3) Two helper functions (<literal>alexGetUserState</literal>
and <literal>alexSetUserState</literal>) are defined.</para>
<programlisting>
alexGetUserState :: Alex AlexUserState
alexSetUserState :: AlexUserState -> Alex ()
</programlisting>
<para>Here is an example of code in the user's Alex file defining
the type and function:</para>
<programlisting>data AlexUserState = AlexUserState
{
lexerCommentDepth :: Int
, lexerStringValue :: String
}
alexInitUserState :: AlexUserState
alexInitUserState = AlexUserState
{
lexerCommentDepth = 0
, lexerStringValue = ""
}
getLexerCommentDepth :: Alex Int
getLexerCommentDepth = do ust <- alexGetUserState; return (lexerCommentDepth ust)
setLexerCommentDepth :: Int -> Alex ()
setLexerCommentDepth ss = do ust <- alexGetUserState; alexSetUserState ust{lexerCommentDepth=ss}
getLexerStringValue :: Alex String
getLexerStringValue = do ust <- alexGetUserState; return (lexerStringValue ust)
setLexerStringValue :: String -> Alex ()
setLexerStringValue ss = do ust <- alexGetUserState; alexSetUserState ust{lexerStringValue=ss}
addCharToLexerStringValue :: Char -> Alex ()
addCharToLexerStringValue c = do ust <- alexGetUserState; alexSetUserState ust{lexerStringValue=c:(lexerStringValue ust)}
</programlisting>
</section>
<section>
<title>The "gscan" wrapper</title>
<para>The <literal>gscan</literal> wrapper is provided mainly
for historical reasons: it exposes an interface which is very
similar to that provided by Alex version 1.x. The interface
is intended to be very general, allowing actions to modify the
startcode, and pass around an arbitrary state value.</para>
<programlisting>alexGScan :: StopAction state result -> state -> String -> result
type StopAction state result
= AlexPosn -> Char -> String -> (Int,state) -> result</programlisting>
<para>The token actions should all have this type:</para>
<programlisting>{ ... } :: AlexPosn -- token position
-> Char -- previous character
-> String -- input string at token
-> Int -- length of token
-> ((Int,state) -> result) -- continuation
-> (Int,state) -- current (startcode,state)
-> result</programlisting>
</section>
<section>
<title>The bytestring wrappers</title>
<para>The <literal>basic-bytestring</literal>,
<literal>posn-bytestring</literal> and
<literal>monad-bytestring</literal> wrappers are variations on the
<literal>basic</literal>, <literal>posn</literal> and
<literal>monad</literal> wrappers that use lazy
<literal>ByteString</literal>s as the input and token types instead of
an ordinary <literal>String</literal>.</para>
<para>The point of using these wrappers is that
<literal>ByteString</literal>s provide a more memory efficient
representation of an input stream. They can also be somewhat faster to
process. Note that using these wrappers adds a dependency
on the <literal>ByteString</literal> modules, which live in the
<literal>bytestring</literal> package (or in the
<literal>base</literal> package in <literal>ghc-6.6</literal>)</para>
<para>
As mentioned earlier (<xref linkend="encoding" />), Alex
lexers internally process a UTF-8 encoded string of bytes.
This means that the <literal>ByteString</literal> supplied
as input when using one of the ByteString wrappers should be
UTF-8 encoded (or use either the <option>--latin1</option>
option or the <literal>%encoding</literal> declaration).
</para>
<para>Do note that <literal>token</literal> provides a
<emphasis>lazy</emphasis> <literal>ByteString</literal> which is not
the most compact representation for short strings. You may want to
convert to a strict <literal>ByteString</literal> or perhaps something
more compact still. Note also that by default tokens share space with
the input <literal>ByteString</literal> which has the advantage that it
does not need to make a copy but it also prevents the input from being
garbage collected. It may make sense in some applications to use
<literal>ByteString</literal>'s <literal>copy</literal> function to
unshare tokens that will be kept for a long time, to allow the original
input to be collected.</para>
<section>
<title>The "basic-bytestring" wrapper</title>
<para>The <literal>basic-bytestring</literal> wrapper is the same as
the <literal>basic</literal> wrapper but with lazy
<literal>ByteString</literal> instead of <literal>String</literal>:</para>
<programlisting>
import qualified Data.ByteString.Lazy as ByteString
data AlexInput = AlexInput { alexChar :: {-# UNPACK #-} !Char, -- previous char
alexStr :: !ByteString.ByteString, -- current input string
alexBytePos :: {-# UNPACK #-} !Int64} -- bytes consumed so far
alexGetByte :: AlexInput -> Maybe (Char,AlexInput)
alexInputPrevChar :: AlexInput -> Char
-- alexScanTokens :: ByteString.ByteString -> [token]
</programlisting>
<para>All of the actions in your lexical specification
should have type:</para>
<programlisting>{ ... } :: ByteString.ByteString -> token</programlisting>
<para>for some type <literal>token</literal>.</para>
</section>
<section>
<title>The "posn-bytestring" wrapper</title>
<para>The <literal>posn-bytestring</literal> wrapper is the same as
the <literal>posn</literal> wrapper but with lazy
<literal>ByteString</literal> instead of <literal>String</literal>:</para>
<programlisting>
import qualified Data.ByteString.Lazy as ByteString
type AlexInput = (AlexPosn, -- current position,
Char, -- previous char
ByteString.ByteString, -- current input string
Int64) -- bytes consumed so far
-- alexScanTokens :: ByteString.ByteString -> [token]
</programlisting>
<para>All of the actions in your lexical specification
should have type:</para>
<programlisting>{ ... } :: AlexPosn -> ByteString.ByteString -> token</programlisting>
<para>for some type <literal>token</literal>.</para>
</section>
<section>
<title>The "monad-bytestring" wrapper</title>
<para>The <literal>monad-bytestring</literal> wrapper is the same as
the <literal>monad</literal> wrapper but with lazy
<literal>ByteString</literal> instead of <literal>String</literal>:</para>
<programlisting>
import qualified Data.ByteString.Lazy as ByteString
data AlexState = AlexState {
alex_pos :: !AlexPosn, -- position at current input location
alex_bpos:: !Int64, -- bytes consumed so far
alex_inp :: ByteString.ByteString, -- the current input
alex_chr :: !Char, -- the character before the input
alex_scd :: !Int -- the current startcode
}
newtype Alex a = Alex { unAlex :: AlexState
-> Either String (AlexState, a) }
runAlex :: ByteString.ByteString -> Alex a -> Either String a
type AlexInput = (AlexPosn, -- current position,
Char, -- previous char
ByteString.ByteString, -- current input string
Int64) -- bytes consumed so far
-- token :: (AlexInput -> Int -> token) -> AlexAction token
</programlisting>
<para>All of the actions in your lexical specification
have the same type as in the <literal>monad</literal> wrapper. It is
only the types of the function to run the monad and the type of the
<literal>token</literal> function that change.</para>
</section>
<section>
<title>The "monadUserState-bytestring" wrapper</title>
<para>The <literal>monadUserState-bytestring</literal> wrapper is the same as
the <literal>monadUserState</literal> wrapper but with lazy
<literal>ByteString</literal> instead of <literal>String</literal>:</para>
<programlisting>
import qualified Data.ByteString.Lazy as ByteString
ata AlexState = AlexState {
alex_pos :: !AlexPosn, -- position at current input location
alex_bpos:: !Int64, -- bytes consumed so far
alex_inp :: ByteString.ByteString, -- the current input
alex_chr :: !Char, -- the character before the input
alex_scd :: !Int -- the current startcode
, alex_ust :: AlexUserState -- AlexUserState will be defined in the user program
}
newtype Alex a = Alex { unAlex :: AlexState
-> Either String (AlexState, a) }
runAlex :: ByteString.ByteString -> Alex a -> Either String a
-- token :: (AlexInput -> Int -> token) -> AlexAction token
</programlisting>
<para>All of the actions in your lexical specification
have the same type as in the <literal>monadUserState</literal> wrapper. It is
only the types of the function to run the monad and the type of the
<literal>token</literal> function that change.</para>
</section>
</section>
</section>
<section id="types">
<title>Type Signatures and Typeclasses</title>
<para>The <literal>%token</literal>, <literal>%typeclass</literal>,
and <literal>%action</literal> directives can be used to cause
Alex to emit additional type signatures in generated
code. This allows the use of typeclasses in generated lexers.</para>
<section>
<title>Generating Type Signatures with Wrappers</title>
<para>The <literal>%token</literal> directive can be used to
specify the token type when any kind of
<literal>%wrapper</literal> directive has been given.
Whenever <literal>%token</literal> is used, the
<literal>%typeclass</literal> directive can also be used to
specify one or more typeclass constraints. The following
shows a simple lexer that makes use of this to interpret the
meaning of tokens using the <literal>Read</literal>
typeclass:</para>
<programlisting>
%wrapper "basic"
%token "Token s"
%typeclass "Read s"
tokens :-
[a-zA-Z0-9]+ { mkToken }
[ \t\r\n]+ ;
{
data Token s = Tok s
mkToken :: Read s => String -> Token s
mkToken = Tok . read
lex :: Read s => String -> [Token s]
lex = alexScanTokens
}
</programlisting>
<para>Multiple typeclasses can be given by separating them with
commas, for example:</para>
<programlisting>
%typeclass "Read s, Eq s"
</programlisting>
</section>
<section>
<title>Generating Type Signatures without Wrappers</title>
<para>Type signatures can also be generated for lexers that do
not use any wrapper. Instead of the <literal>%token</literal>
directive, the <literal>%action</literal> directive is used to
specify the type of a lexer action. The
<literal>%typeclass</literal> directive can be used to specify
the typeclass in the same way as with a wrapper. The
following example shows the use of typeclasses with a
"homegrown" monadic lexer:</para>
<programlisting>
{
{-# LANGUAGE FlexibleContexts #-}
module Lexer where
import Control.Monad.State
import qualified Data.Bits
import Data.Word
}
%action "AlexInput -> Int -> m (Token s)"
%typeclass "Read s, MonadState AlexState m"
tokens :-
[a-zA-Z0-9]+ { mkToken }
[ \t\n\r]+ ;
{
alexEOF :: MonadState AlexState m => m (Token s)
alexEOF = return EOF
mkToken :: (Read s, MonadState AlexState m) =>
AlexInput -> Int -> m (Token s)
mkToken (_, _, _, s) len = return (Tok (read (take len s)))
data Token s = Tok s | EOF
lex :: (MonadState AlexState m, Read s) => String -> m (Token s)
lex input = alexMonadScan
-- "Boilerplate" code from monad wrapper has been omitted
}
</programlisting>
<para>The <literal>%token</literal> directive may only be used
with wrapper, and the <literal>%action</literal> can only be
used when no wrapper is used.</para>
<para>The <literal>%typeclass</literal> directive cannot be
given without the <literal>%token</literal> or
<literal>%action</literal> directive.</para>
</section>
</section>
</chapter>
<chapter id="invoking">
<title>Invoking Alex</title>
<para>The command line syntax for Alex is entirely
standard:</para>
<screen>$ alex { <replaceable>option</replaceable> } <replaceable>file</replaceable>.x { <replaceable>option</replaceable> }</screen>
<para>Alex expects a single
<literal><replaceable>file</replaceable>.x</literal> to be named
on the command line. By default, Alex will create
<literal><replaceable>file</replaceable>.hs</literal> containing
the Haskell source for the lexer.</para>
<para>The options that Alex accepts are listed below:</para>
<variablelist>
<varlistentry>
<term><option>-o</option> <replaceable>file</replaceable></term>
<term><option>--outfile</option>=<replaceable>file</replaceable></term>
<listitem>
<para>Specifies the filename in which the output is to be
placed. By default, this is the name of the input file with
the <literal>.x</literal> suffix replaced by
<literal>.hs</literal>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-i</option> <optional><replaceable>file</replaceable></optional></term>
<term><option>--info</option> <optional><replaceable>=file</replaceable></optional></term>
<listitem>
<para>Produces a human-readable rendition of the state
machine (DFA) that Alex derives from the lexer, in
<replaceable>file</replaceable> (default:
<literal><replaceable>file</replaceable>.info</literal>
where the input file is
<literal><replaceable>file</replaceable>.x</literal>).</para>
<para>The format of the info file is currently a bit basic,
and not particularly informative.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-t</option> <optional><replaceable>dir</replaceable></optional></term>
<term><option>--template</option>=<replaceable>dir</replaceable></term>
<listitem>
<para>Look in <replaceable>dir</replaceable> for template files.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-g</option></term>
<term><option>--ghc</option></term>
<listitem>
<para>Causes Alex to produce a lexer which is optimised for
compiling with GHC. The lexer will be significantly more
efficient, both in terms of the size of the compiled
lexer and its runtime.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-d</option></term>
<term><option>--debug</option></term>
<listitem>
<para>Causes Alex to produce a lexer which will output
debugging messages as it runs.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-l</option></term>
<term><option>--latin1</option></term>
<listitem>
<para>Disables the use of UTF-8 encoding in the generated
lexer. This has two consequences:
<itemizedlist>
<listitem>
<para>
The Alex source file is still assumed to be UTF-8
encoded, but any Unicode characters outside the range
0-255 are mapped to Latin-1 characters by taking the
code point modulo 256.
</para>
</listitem>
<listitem>
<para>
The built-in macros <literal>$printable</literal>
and '<literal>.</literal>' range over the Latin-1
character set, not the Unicode character set.
</para>
</listitem>
</itemizedlist>
Note that this currently does not disable the UTF-8 encoding
that happens in the "basic" wrappers,
so <option>--latin1</option> does not make sense in
conjunction with these wrappers (not that you would want to
do that, anyway).
Alternatively, a <literal>%encoding "latin1"</literal> declaration can be
used inside the Alex source file to request a Latin-1 mapping. See also
<xref linkend="encoding" /> for more information about the
<literal>%encoding</literal> declaration.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-?</option></term>
<term><option>--help</option></term>
<listitem>
<para>Display help and exit.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-V</option></term>
<term><option>--version</option></term>
<listitem>
<para>Output version information and exit. Note that for legacy
reasons <option>-v</option> is supported, too, but the use of it
is deprecated. <option>-v</option> will be used for verbose mode
when it is actually implemented.</para>
</listitem>
</varlistentry>
</variablelist>
</chapter>
</book>