perldoc > Plucene::Analysis::CharTokenizer(3pm)

NAME
    Plucene::Analysis::CharTokenizer - base class for character tokenisers

SYNOPSIS
            # isa Plucene::Analysis::Tokenizer

            my $next = $chartokenizer->next;

DESCRIPTION
    This is an abstract base class for simple, character-oriented tokenizers.

METHODS
  token_re
    This should be defined in subclasses.

  normalize
    This will normalise the character before it is added to the token.

  next
            my $next = $chartokenizer->next;

    This will return the next token in the string, or undef at the end of the string.