=head1	NAME

dktok - Tokenizer

=head1	SYNOPSIS

  #include <dktok.h>

  int fct(void *d, void *tpv, char *s, int *ec);

  dk_tokenizer_t *
  dktok_new(
    size_t b, char **q, char *s, char *w, char *n, char c,
    dk_fct_tokenizer *f, void *d
  );

  void dktok_start(dk_tokenizer_t *tp);

  int dktok_add(dk_tokenizer_t *tp, char c);

  int dktok_stop(dk_tokenizer_t *tp);

  int dktok_get_error_code(dk_tokenizier_t *tp);

  unsigned long dktok_get_lineno(dk_tokenizer_t *tp);

  void dktok_reset_error_code(dk_tokenizer_t *tp);

=head1	DESCRIPTION

The dktok module splits incoming data from a stream into tokens
and allows to process the tokens by using a processing function.

The fct() function is an example for a processing function.
The I<d> parameter is a pointer to additional data used for processing,
I<tpv> is a pointer to a tokenizer, I<s> is a pointer to the buffer
containing the token, I<ec> is a pointer to a variable which can
be set if an error occures. The function must return 1 on success,
0 on errors. If the function returns 0 input processing is aborted.

The dktok_new() function creates a new tokenizer. The parameter I<b>
specifies the length of the largest token. The I<q> parameter points
to an array containing strings. Each string contains a bracket pair
(opening and closing bracket). The array must be finished by a NULL
pointer.
The I<s> parameter is a string containing all one-character-tokens
(i.e. mathematical operators).
The I<w> parameter is a string containing all white-space
characters.
The processing function is specified as parameter I<f>, additional
data as specified in I<d> is passed as parameter I<d> to the
processing callback function.
The function returns a valid pointer on success, NULL on error.
On success the tokenizer structure must be released after
usage by the dktok_delete() function.

The dktok_start() function must be invoked before feeding data
into the tokenizer.

The dktok_add() function feeds data into the tokenizer. If the return
value is positive (>0) we can continue feeding the tokenizer.

The dktok_stop() function must be run after providing the last
input to process the final token and to flush all buffers.

The dktok_get_error_code() retrieves the error code established by
the processing callback function.

The dktok_get_lineno() function returns the current line number. This
can be used to generate error messages.

The dktok_reset_error_code() function resets the internal error code.

=head1	AUTHOR

Dirk Krause

=head1	COPYRIGHT AND LICENSE

Copyright (c) 2001-2008, Dirk Krause
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

 * Redistributions of source code must retain the above
   copyright notice, this list of conditions and the
   following disclaimer.

 * Redistributions in binary form must reproduce the above
   copyright notice, this list of conditions and the following
   disclaimer in the documentation and/or other materials
   provided with the distribution.

 * Neither the name of the Dirk Krause nor the names of
   contributors may be used to endorse or promote products
   derived from this software without specific prior written
   permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
