This is fplex, a lexer generator for FP. Each source file is documented
separately and the makefile should be self-explanatory.

Input format for fplex: each line has two blank-separated parts, a
regular expression followed by the name for the construct. Regular
expressions use the special symbols + to indicate 1 or more, * to
indicate 0 or more, (a|b) for selection of a or b but not both, [a-z]
for all characters a-z, [^ab] for all characters BUT ab, and \c to
cancel any special meaning "c" may have had (i.e. \\ means backslash
and \] means ]); in addition, \n means newline, \t tab, \b blank, \s
backslash. ( and ) are used for grouping.

The output of fplex is an fp routine which calls "lex" on the pair
formed from the input and a constant encoding of the finite state
machine.  The name of the scanner which can be called by the user is
the same as the name of the input file (without the ".lex" extension)
or, if the input is not from a file, simply "lexer".

The scanner takes as input a string of characters and returns a pair
<errors, tokens>. errors will be <> in case of a successful scan; if it
is not <>, it is a vector of <errtype, text, line, col>. errtype is one
of _nochar (character not defined by syntax) or _noscan (unable to
match pattern to any available rule); text is the text that gave rise
to the error; line and col are the line and column at which the
beginning of error text is located. There is a switch in fplstd.fp
called parallel. If set to _F, a sequential finite state machine is
used. If set to _T, a parallel (but more complex and slower) finite
state machine is used.

Compiling with -n, -O, and -lnfp gives fplex that is less than 60% the
size of compiling with -lfp and none of the other switches (on Vax,
using Gcc). Stripping the binary removes another 4%. The time for the
generator to execute (for fpl.lex) seems to be about 80%, and the time
for the scanner to run is about 80% of the time before optimizing. See
log.full and log.opt for actual figures.

If the format should change in selflex.lex, or if fplstd.fp requires a
different set of functions or, finally, to regenerate fpllex.fp, type
"make lexer_boot". "lexer_boot" is purposely obscure so it won't be
invoked accidentally. Note that fpllex.fp generated in this way should
be identical to lexer.gen, as can be checked by running "make
lexer.res" (after making "lexer_boot"). Note that fpllex.sv should be
a valid version of fpllex.fp.

Optional improvements:

Now each deterministic state is the vector containing all the
non-deterministic states it could be in. It should be replaced
by an atomic value instead. Nonfinal states should be numbers,
including 0 for the start state, positive values for non-new
states, and the corresponding negative values for new states.
States that are valid terminal states should be symbols, to be
exploded when they are new state and imploded to make them back
into non-new states. The symbol representing the state should
be the name of the terminal symbol followed by 4 digits of
identification, since there may be more than one state which
returns the same terminal symbol. Eg, state "float" would be
float0000 the first time it occurs, float0001 the second time,
and so on.

It should be possible to specify inclusion of fplerrcheck in
the generated scanner. Either make it the default, or allow
specification of options in the lexer file (e.g., %error) and
modify self.lex, parsel.boot.fp, parsel.fp so that two things
are returned, instead of just a grammar: a grammar and a set
of flags.

Add symbolic definitions to grammar, so you can e.g. define
digit	0|1|2|3|4|5|6|7|8|9
and then use {digit} in a rule, as in lex. Note that the code
to parse '{expr}' is already in place, it simply leads to an
error in parsel.new.fp.

Consider adding ^expr and expr$ to the grammar, but only add them
if plan to add trailing context (unlikely)
