# perldoc > Plucene::Analysis::CharTokenizer

## NAME
    [Plucene::Analysis::CharTokenizer](https://www.chedong.com/phpMan.php/perldoc/Plucene%3A%3AAnalysis%3A%3ACharTokenizer/markdown) - base class for character tokenisers

## SYNOPSIS
            # isa [Plucene::Analysis::Tokenizer](https://www.chedong.com/phpMan.php/perldoc/Plucene%3A%3AAnalysis%3A%3ATokenizer/markdown)

            my $next = $chartokenizer->next;

## DESCRIPTION
    This is an abstract base class for simple, character-oriented tokenizers.

## METHODS
  token_re
    This should be defined in subclasses.

  normalize
    This will normalise the character before it is added to the token.

  next
            my $next = $chartokenizer->next;

    This will return the next token in the string, or undef at the end of the string.