Token

Any set of patterns defining an input stream must eventually result in the correct sequence of parse tokens. These represent when a field starts and ends, when a record starts and ends, and the contents of fields. This allows the parse engine to build the database file correctly. The sequence of these tokens must be returned as zero or more start of record followed by zero or more start of field, field contents, end of field tokens, followed by end of record tokens. The only valid parse tokens that can be used are given in the table below.

Token

Description

<nul>

Do nothing with the input

<sor>

Flag start of a record

<sof>

Flag start of a field

<fld>

Contents of a field (can be returned multiple times)

<eof>

Flag end of a field

<eor>

Flag end of a record

<err>

Generate an error

Go to the first, previous, next, last section, table of contents.