# Plucene::Analysis::Tokenizer - base class for tokenizers - perldoc - [phpMan]

## NAME
    [Plucene::Analysis::Tokenizer](https://www.chedong.com/phpMan.php/perldoc/Plucene%3A%3AAnalysis%3A%3ATokenizer/markdown) - base class for tokenizers

## SYNOPSIS
            my $tokenizer = [Plucene::Analysis::Tokenizer::Subclass](https://www.chedong.com/phpMan.php/perldoc/Plucene%3A%3AAnalysis%3A%3ATokenizer%3A%3ASubclass/markdown)
                    ->new({ reader => $reader });

## DESCRIPTION
    This is an abstract base class for tokenizers.

    A Tokenizer is a TokenStream whose input is a Reader.

## METHODS
  new
            my $tokenizer = [Plucene::Analysis::Tokenizer::Subclass](https://www.chedong.com/phpMan.php/perldoc/Plucene%3A%3AAnalysis%3A%3ATokenizer%3A%3ASubclass/markdown)
                    ->new({ reader => $reader });

    This will create a new tokenizer.

  next
    This must be defined in a subclass

  close
    Close the input reader.